Home

Flexible Multilevel Multidimensional Item Analysis and Test Scoring

1. Free G v4 Interaction 1 2 1x2 Ne Q Ne 131 Free G v5 MainFffect 1 1 Free G v5 MainEffect 3 3 Free G v5 Interaction 1 3 1x3 Free G v6 MainEffect 2 2 Free G v6 MainEffect 3 3 Free G v6 Interaction 2 3 2x3 Free G v7 MainEffect 1 1 Free G v7 MainEffect 2 1 Free G v7 MainEffect 3 1 Free G v7 Interaction 1 2 1x2 Free G v7 Interaction 1 3 1x3 Free G v7 Interaction 2 3 2x3 Free G v7 Interaction 1 2 3 1x2x3 With respect to the output presented next while there are some slight changes in the estimates the reported point estimate values for the item pa rameters remain largely unchanged from the calibration that employed the higher order latent variable Differences in the estimated profile probabilities are not seen until the 4th or 5th decimal place If one looked at the sco file there are also only slight changes to the reported values the most noticeable difference between the sco file from the model employing the higher order variable and the current example is that final three values discussed in the previous sco file the higher order latent variable mean SD and variance estimates are no longer present 132 THTO99TZ O T 8L98S 60 0 T 48v T60 0 0 S81v 980 0 0 499611600 T TLVLZ660 0 1 TTIvSSE20 0 o 811608vc 0 o qoid 1 T H H o o o o 1 44ed dnoip 10 ser3rlIqeqo4d uorqeS5TJrsseT5 sso rO p
2. TL felol fa 055 95 where N is the number of level 1 units in level 2 unit J fz 05 is the dis tribution of the level 2 latent variables and Yjg iyijg YN jg is the collection of item response patterns for all ny items from the N level 1 units in level 2 unit j of group g Once the item responses are treated as fixed upon observation the marginal likelihood of Y jg is defined as LIY jj Sel Y jg Taking logs and summing over the assumed independent level 2 units as well as groups the marginal log likelihood for the structural parameters is G Jg log L Y vis Y ES 25 27 L Y jg g 1j 1 where G is the number of groups J is the number of level 2 units in group g and Y Yig Y 1 y is the collection of item response patterns for all level 1 and level 2 units in group g 8 2 Item Response Models W ijg where ap is the set of p slopes on the between latent variables and aly is the set of q slopes on the within latent variables For the kth row in g g the linear predictor is equal to nijkg ap 07 aj 0 8 2 1 Model with Pseudo Guessing Parameter This is the multilevel and multidimensional extension of the 3PL model Let the correct endorse 1 response probability be 1 guess 1 exp cCk Mijkg l where guess is the item specific pseudo guessing probability and cx is the item specific intercept Consequently Pz Yijkg O nijkg 1 0 Pz Yijkg 1 nijkg Pe
3. 1 00 0 00 0 00 0 00 1 00 0 00 1 00 0 00 0 00 1 00 The simulation control output shows that the input syntax and the param eter values are specified correctly The item parameters are invariant across groups so only parameters from Group 1 are shown The within item equality of the slopes implies that the 5 level 1 dimensions a2 a6 follow a testlet response theory model restricted bifactor with 4 testlets of 3 items each Furthermore the equality of slopes on the Within general dimension a2 and the Between general dimension a1 implies that a random intercept model is set up that provides a decomposition of the Between and Within variances for the general trait The mean of Group 1 and 0 20 lower than Group 2 but the Group 1 Between variance is 0 50 and the Group 2 between variance is 0 25 79 4 9 Internal Simulation Function Here we provide an overview for using the internal simulation function ISF which allows users to automate numerous replications of a given simulation and subsequent calibration This function uses existing simulation and calibration syntax files and merely provides a convenient way of obtaining replications it is not an alternative method for constructing simulation syntax files 4 9 1 Accessing the Internal Simulation Function Currently the ISF is available through the flex MIRTO GUI only specifically the ISF is not available if using the program through the command prompt or when
4. S860T 0 89S99 0 0 168860 0 I 909610 0 TS9060 0 69SS91 0 I 89980 0 t0S9I 0 v90666 0 I 98 070 0 LLV1ET O 687977 0 mdmoO OOS MAMO WAY O 3 9 mdmo v14600 0 988600 0 FS VSI O TvI9TT0 0 168v90 0 89Tv69 0 STLZ00 0 198700 0 9Sp8T 0 S818 60 0 vETVOO O OTS60 0 ZEZZOO 0 916600 0 1804800 864Z00 0 ZL 000 0 T88EE7 0 T94T10 0 S70L00 0 Tv6v904 0 08v60 0 420910 0 c8v661 0 046600 0 9v0 v0 0O Z9ESL 0 80TEZO O PPEEEO O 91087 0 vOCCVO 9 6800 LLTTET O Cv008T OT6670 POEO9 0 6 ccvo 801800 vOTET O 08S600 92000 LTLEZO O 6v0S 0 641100 v191 0 9800 0 992000 0 9c 810 0 9E8TZO 0 ST00 c008 0 0 Sv4SC0 CvATTO vT800T 0 Sv8660 C8vSEO vosS0c 0 S 0SS0 96 vc0 9SSv91 0 Tv10TEE O 0 O0cvS40 0 0 92 600 0 S9v600 0 1018870 o S9T9L0 0 o T4982T1 0 99ELZZ 0 860T 0 o 9vv340 0 o 9v 600 0 S v600 0 8 9TC 0 0 6 1200 0 0 91v9 0 0 99 80 0 o v9SvEE O o S 9200 0 69L00 0 TI9cEE O 66800 0 6TL00 0 Sv6100 0 81867c 0 o TO 00 0 o v6z8TT O 9807 T 0 T8vOtE O 0 S86Sv0 0 o 1769900 OETZLO 0 OTELLZ 0 0 S 0vTO 0O o TSc8Sl O 660261 0 8ZLZTE 0 o T6vvvO O o T9660T 0 c81SCT O O6 S29 0 Z8020 0 99L9 S 0 TOL9ZT O TIvS4S O 0TS0c0 0 TET498 0 0SSv00 0 SC8600 0
5. Summary of the Data and Control Parameters Missing data code Number of Items Number of Cases Number of Dimensions Item Categories Model 1 3 Graded 2 3 Graded 3 3 Graded 4 3 Graded Miscellaneous Control Values Random number seed 7474 Graded Items for Group 1 G Item a c 1 c2 ab 0 70 2 00 2 1 00 2 00 3 1 20 0 00 4 1 50 0 00 Graded Items for Group 1 Item a b 1 1 0 70 2 86 2 1 00 2 00 3 1 20 0 00 4 1 50 0 00 Group Parameter Estimates Group Label mu 1 G 0 00 The first part of the simulation control output echoes the user supplied specifications e g number of items number of simulees the number of cate gories IRT model types and the random number seed The generating item parameters are then printed One can see that the syntax statements are cor rectly interpreted When everything is set up correctly a simulated data set is generated and saved The first and last three rows of this data set are shown below The first 4 columns are the item responses in 3 categories coded as 0 63 1 and 2 Column 5 is a case ID column Column 6 contains the generating theta value for that simulee Output 4 2 Simulated Data Set Graded Model Syntax Only 0 0 246011 1 1 70243 2 0 71107 2 997 0 0161087 998 0 702542 999 1 09703 As noted previously simulations may either be conducted by using Value statements to provide generating parameter values or through the use of a parameter file To
6. The current example simulates group differences in both population means and variances as well as item parameters Each group has 10 items Group 1 has mean 0 variance 1 Group 2 has the same variance but a mean at 0 2 Group 3 has a mean of 0 2 and variance of 1 5 The items are for the most part 3PL items with the exception of Item 1 in Group 3 The lower asymptote parameter is set to 0 for that item making it a 2PL item Also the intercept parameter for Item 9 in Group 2 is chosen such that its threshold value is 0 5 standard deviation higher than the corresponding items in Groups 1 and 3 74 Example 4 6 Simulation DIF and Group Mean Differences Project Title Simulate Data Description 3 Groups 10 Items N 1000 mostly 3PLs Group 1 N 0 0 1 0 Group 2 N 0 2 1 0 Group 3 N 0 2 1 5 Item 9 in Group 2 has 0 5 higher threshold than the corresponding item in Groups 1 and 3 Item 1 in Group 3 is actually a 2PL item lt Options gt Mode Simulation ReadPRMFile genparams txt RndSeed 7474 lt Groups gt Group1 File group1 dat Varnames v1 v10 1000 Group27 File group2 dat Varnames v1 v10 1000 Group37 File group3 dat Varnames v1 v10 1000 lt Constraints gt The Options section remains substantially unchanged from previous ex amples Within the lt Groups gt section we now have labels and specifications for the three distinct groups for wh
7. There are some difference between flex MIRT and MPLUS For example MPLUS fixes the latent variable mean of the last latent class attribute profile 8 in this case to 0 while in flexMIRT we have constrained the first latent class to 0 This results in differences in the latent attribute parameter estimates but as we show in the below table these differences are simply a matter of choice of identification constraints and do not affect the primary estimates of interests i e estimated profile proportions Table 6 5 Generating latent class mean values and flex MIRT and MPLUS estimates for Rupp Templin amp Henson s Example 9 2 cl c2 c3 c4 c5 c6 c7 c8 sum generating H 0 1 00 1 00 1 00 1 00 1 00 1 00 0 exp u 1 0 367879 0 367879 0 367879 0 367879 0 367879 0 367879 1 4 207277 v 0 2481 0 0913 0 0913 0 0913 0 093 0 0913 0 0913 0 2481 flexMIRT 0 1 21583 0 91595 1 00082 1 05552 0 99914 0 97495 0 13578 exp 1 0 296464 0 400136 0 367578 0 348011 0 368196 0 377211 0 87303468 4 030631 0 2481 0 0736 0 0993 0 0912 0 0863 0 0914 0 0936 0 2166 MPLUS 0 136 1 08 0 78 0 865 0 92 0 863 0 839 0 exp 1 145682 0 339596 0 458406 0 421052 0 398519 0 421894 0 432142 1 4 617291 0 2481 0 0736 0 0993 0 0912 0 0863 0 0914 0 0936 0 2166 We will next cover the contents of the requested file that contains the individual scores 128 000000 000000 000000 0 v v99690 0 8288 0 0 v046c0 0 4918000 L6Z8L T
8. o O O Q QO N PRPRPRPRP RB O OQ GOO N PF O 000000000000000000mNn O O F F E RFP X X O BFF RP F P BP F ooooo 00 00 00 00 00 00 o000O0O0O00000000000000_ 0 ooooo O O O O 00 00 0 0 00 mW Q P Q PP F Q F F F N SE N N PF PF PF P N O OO O O O O G 000 00 00 00 0 m 00 As before the marginal X values and the local dependence LD statistic values of Chen and Thissen 1997 are reported due to the GOF Extended 22 statement but an additional feature is present in the table because of the ordi nalnature of the data TheMarginal fit Chi square and Standardized LD X2 Statistics table is presented in the following Output example The standardized LD X value for each item pair is followed by either the letter p or the letter n When the item responses are composed of ordered cate gories phi correlations are computed for both the model implied and observed bivariate tables If the model implied correlation is lower than the observed correlation for a given item pair a p is printed after the calculated X meaning positive LD If the model implied correlation is higher an n is printed indicating negative LD Either a p or an n is printed after ev ery value in the table The absolute magnitude of the individual X values should still evaluated against a critical value such as 3 0 to determine if a non ignorable level of LD is present Output 2 11 Graded Model
9. 2 5 7 8 134 2 5 7 7 8 2 4 5 7 2 7 1 7 2 5 6 7 gt 1 2 3 5 7 2 3 5 7 Dimensions 22 8 main effects 14 generated higher ord ints 4D DM G Varnames al a8 lt Constraints gt Fix G vi v20 MainEffect Free Free Free Free Free Free Free Free Free Free Free Free Free Free Free Free Free Free Free Free G vi Interaction 3 6 7 3rd order int of attr 3 6 7 G v2 Interaction 4 7 2nd order int of attr 4 7 G v3 Interaction 4 7 2nd order int of attr 4 7 G v4 Interaction 2 3 5 7 4th order int of attr 2 3 5 7 G v5 Interaction 2 4 7 8 4th order int of attr 2 4 7 8 G v6 MainEffect 7 main effect of attr 7 G v7 Interaction 1 2 7 3rd order int of attr 1 2 7 G v8 MainEffect 7 main effect of attr 7 G v9 MainEffect 2 main effect of attr 2 G v10 Interaction 2 5 7 8 4th ord int of attr 2 5 7 8 G v11 Interaction 2 5 7 3rd order int of attr 2 5 7 G v12 Interaction 7 8 2nd order int of attr 7 8 G v13 Interaction 2 4 5 7 4th ord int of attr 2 4 5 7 G v14 Interaction 2 7 2nd order int of attr 2 7 G v15 Interaction 1 7 2nd order int of attr 1 7 G v16 Interaction 2 7 2nd order int of attr 2 7 G v17 Interaction 2 5 7 3rd order int of attr 2 5 7 G v18 Interaction 2 5 6 7 4th ord int of attr 2 5 6 7 G v19 Interaction 1 2 3 5 7 5th ord int at
10. If a normal prior is chosen the subsequent columns will supply the latent factor mean s and then co variance s The latent factor mean values are entered in order with the mean of factor 1 listed first followed by mean 2 mean 3 and so on For specifying co variance values the unique elements of the latent factor variance covariance matrix are entered in row major order To demonstrate this point suppose the following 4 x 4 factor covariance matrix is to be submitted to flex MIRTO 66 fl f2 f3 f4 11 fm 2 4 7 2g 5 8 B 4 5 6 9 14 The progression of variance covariance values entered in the parameter file should be 123456789 10 If an empirical histogram prior is chosen the user must supply additional information Column 6 specifies the total number of quadrature points on which the histogram is defined and Column 7 requires a maximum value for the quadrature points which determines the range over which the points are to be distributed Similar to specifying Quadrature or FisherInf in the Options section the quadrature points are spread symmetrically around zero with the minimum and maximum value determined by the entry in Column 7 After the EH specification the mean s and unique element s in the covariance matrix are listed in the subsequent columns following the same progression of entries just covered The histogram itself is supplied next For instance if the entry in Column 6 is 101 and Col
11. Nominal 8 Tc a Identity Constraints Fix G vi v7 MainEffect Free Free Free Free Free Free Free Free Free Free Free Free Free Free G vi MainEffect 1 G v2 MainEffect 2 G v3 MainEffect 3 G v4 MainEffect 1 G v4 MainEffect 2 G v4 Interaction 1 2 1x2 G v5 MainEffect 1 1 G v5 MainEffect 3 3 G v5 Interaction 1 3 1x3 G v6 MainEffect 2 2 G v6 MainEffect 3 3 G v6 Interaction 2 3 2x3 G v7 MainEffect 1 1 G v7 MainEffect 2 1 N OQ N 124 Free G v7 MainEffect 3 1 Free G v7 Interaction 1 2 1x2 Free G v7 Interaction 1 3 1x3 Free G v7 Interaction 2 3 2x3 Free G v7 Interaction 1 2 3 1x2x3 Fix D a Slope Fix D a ScoringFn Value D a ScoringFn 1 0 0 As with the previous CDM examples the maximum number of E step and M steps have been increased from the default the E tolerance and M tolerance values have been decreased and the SE calculation method has been set to REM in the lt Options gt section The items were simulated as correct incorrect items so Ncat has been set to 2 and we will fit the 2PLM to all items As noted in the introduction there are 3 attributes measured by the items so Attributes 3 From the Q matrix we can see that Items 4 6 all measure two of the attributes and Item 7 measures all 3 to accommodate this aspect o
12. flexMIRT Vector Psychometric Group flexMIRT Flexible Multilevel Multidimensional Item Analysis and Test Scoring User s Manual Version 2 0 Authored by Carrie R Houts PhD Li Cai PhD The correct citation for the user s manual is Houts C R amp Cai L 2013 flex MIRT O user s manual version 2 Flexible multilevel multidimensional item analysis and test scoring Chapel Hill NC Vector Psychometric Group The correct citation for the software is Cai L 2013 flex MIRT version 2 Flexible multilevel multidimensional item analysis and test scoring Computer software Chapel Hill NC Vector Psychometric Group Copyright 2013 Vector Psychometric Group LLC October 2013 Contents 1 Overview 1 2 Basic Calibration and Scoring 3 Deal INCA D dto E cac y ae dae y do er phe te 3 2 2 Single Group 2PL Model Calibration 5 2 3 Single Group 3PL Model Calibration 12 2 4 Single Group 3PL EAP Scoring 15 2 5 Single Group 3PL ML Scoring 19 2 6 Graded Model Calibration and Scoring 20 2 7 1PL Model Fitted to Response Pattern Data 23 2 8 The Nominal Categories Model 29 2 84L Nominal Model as AAA PRG S 29 2 8 2 Generalized Partial Credit Model 36 2 8 3 Rating Scale Model lt s So iu eG oS ovx 37 3 Advanced Modeling 40 3 1 Multiple Group Calibration ln 40 3 2 Characterizing
13. 1 5 0 667 1 182 0 262 0 284 0 068758 0 001159 0 080632 Table D 4 Labeled sco file EAP scores 2 Factors with User supplied ID Variable Grp flexMIRT User ID 6 0 SE 1 SE 2 611 621 622 Obs 1 1 subl 0 141 0 466 0 258 0 355 0 066705 0 001763 0 126136 1 2 sub2 0 192 1 407 0 281 0 295 0 079119 0 001470 0 087174 1 3 sub3 2 023 1 684 0 527 0 392 0 277999 0 015970 0 153786 1 4 sub4 1 311 0 840 0 334 0 290 0 111558 0 003131 0 083973 1 5 sub5 0667 1 182 0 262 0 284 0 068758 0 001159 0 080632 Table D 5 Labeled sco file MAP scores 2 Factors Grp flexMIRT of Iterations 0i 0 SE 1 SE 05 611 821 822 Obs to MAPs 1 1 3 0 146 0 405 0 245 0 338 0 059838 0 001456 0 114251 1 2 4 0 200 1 400 0 263 0 287 0 068942 0 001183 0 082246 1 3 5 1 838 1 643 0 494 0368 0 244128 0 011428 0 135356 1 4 4 1 261 0 831 0 296 0 275 0 087689 0 002205 0 075827 1 5 4 0 677 1 190 0 247 0 270 0 061086 0 000914 0 072911 185 00 0 00 T 00 0 00 T 00 0 00 0 0 9 I Idnor 0 TED qdo Io TTo sr In IO soeg Jo 7 419 Teqe T add TI 000 200000 0 000 200000 0 000 F900000 0 0004 900000 0 000 9TOTS0 000 988292 0004968299 g C 9 I ga I 000 200000 0 0004 900000 0 000 r900000 0 0004 900000 0 000 e86668 T O00 TF PEVH98 E 000t9rPOTO g G 9 I pA I 000 200000 0 0004 900000 0 000 F900000 0 0004 900000 0 000 r98I6ITP I 000 9z8ZT9T 000 9S GT69T g G 9 I I 000 200000 0 0004 900000 0 000 F900000 0 0004 900000 0
14. Guessing 12 13 25 41 46 49 158 159 Interaction 101 102 112 116 122 125 126 132 135 Intercept 25 39 41 49 62 158 159 MainEffect 101 102 105 106 112 116 122 125 126 132 135 Mean 41 42 49 50 94 158 Prior 157 161 Beta 12 41 159 161 logNormal 12 161 Normal 12 13 46 161 ScoringFn 25 30 33 38 39 57 125 126 158 159 Slope 24 25 33 38 39 41 49 94 57 60 62 90 94 97 158 159 Value 61 62 125 126 157 159 160 Convergence Criteria 4 8 30 43 144 CTX2Tbl see lt Options gt CTX2Tb1 Data Format 3 21 153 Multiple Groups 41 Response Pattern Data 24 154 Description see Project Description Diagnostic Classification Models see Cognitive Diagnostic Models DIF see Options DIF DIFcontrasts see lt Options gt DIFcontrasts DIFitems see Groups DIFitems Dimensions see lt Groups gt Dimensions DM see Groups DM DMtable see lt Options gt DMtable EmpHist see Groups EmpHist Empirical Histograms 42 71 72 74 155 Epsilon see lt Options gt Epsilon Equal see lt Constraints gt Equal Etol see lt Options gt Etol Factor Loadings 25 26 55 92 FactorLoadings see lt Options gt FactorLoadings File see lt Groups gt File FisherInf see lt Options gt FisherInf FitNullModel see lt Options gt FitNullModel Fix see Constraints Fix Fixed Effects Calibration 97 98 Fixed Theta Calibra
15. In this chapter we will provide examples for using flex MIRT when multiple groups multiple factors and or a hierarchical data structure need to be modeled 3 1 Multiple Group Calibration The data set used to illustrate 2PL and 3PL examples in the previous chapter is actually made up of responses from two groups of examinees 3rd and 4th graders Using this already familiar data set we will demonstrate multiple group analysis Example 3 1 2 Group 3PL Using Normal Priors with EAP Scoring Project Title G341 19 Description 12 Items 3PL 2 Groups Calibration with Normal Prior Saving Parameter Estimates Followed by Combined Scoring Run for EAP Scores lt Options gt Mode Calibration Progress Yes SaveSCO Yes SavePRM Yes SaveInf Yes Score EAP FisherInf 81 4 0 SE Xpd 40 GOF Extended M2 Full FitNullModel Yes lt Groups gt ZGrade4 File g341 19 grpl dat Varnames vl v12 N 1314 Ncats vl v12 2 Model vi vi2 ThreePL BetaPriors vi vi2 1 5 Grade37 File g341 19 grp2 dat Varnames v1 v12 N 1530 Ncats vl v12 2 Model vi vi2 ThreePL BetaPriors vi vi2 1 5 Constraints Free Grade3 Mean 1 Free Grade3 Cov 1 1 Equal Grade3 vi v12 Guessing Grade4 vi v12 Guessing Equal Grade3 vi v12 Intercept Grade4 vi v12 Intercept Equal Grade3 vi v12 Slope Grade4 vi v12 Slope Pri
16. Item Parameters 34 Nominal Model 2 Output GOF values 35 DIF Analyses Output Table 47 Two Level Mode Means and Variances 52 Correlated Four Factor Model Output Loadings Means and Variances MEM PC wal ee ee a te ee S R R Q s S A 55 Nominal Model Scoring Function Estimates Excerpt 58 Simulation Control Output Graded Model Syntax Only 63 Simulated Data Set Graded Model Syntax Only 64 Simulated Data Set Graded Model Using PRM File 65 Simulation Control Output for Multilevel Bifactor Model 78 vii 4 5 4 6 4 7 5 1 5 2 5 3 5 4 5 9 6 1 6 2 6 3 6 4 6 5 6 6 6 7 6 8 Beginning Columns of Output from 5 Replications 82 Item Parameters with Parameter Number Labels 82 ISF output Switch from Model 1 to Model 2 Values 83 Correlated Four Factor Model Output Processing Pane Output 91 Correlated Four Factor Model Output Loadings Means and Variances e a g ce va e dee tes da do tetuer uc le dea iare d 92 Two Level MIRT Model Processing Pane Output 95 Multilevel MIRT Model Item and Group Parameters 96 Fixed Effects Calibration Output Item Parameters 98 C RUM Output Item Parameters and Attribute Profile Pos terior Probabilities tae amp dl eR ROG YR ENG UR 107 C RUM Output SCO Output aom vo ox x 109 DINA output Item Parameter Estimates 113 DINA S
17. N 586 Dimensions 4 53 Ncats v2 v21 5 Model v2 v21 Graded 7 Constraints Fix v2 v21 Slope Free v2 v5 Slope 1 Free v6 v9 Slope 2 Free v10 v15 Slope 3 Free v16 v21 Slope 4 Free Cov 2 1 Free Cov 3 1 Free Cov 3 2 Free Cov 4 1 Free Cov 4 2 Free Cov 4 3 Points of note in the syntax relevant to the fitted multidimensional model are the Dimensions statement in the lt Groups gt section and the various constraints encountered in the lt Constraints gt section As described earlier we are fitting a model with four factors hence the Dimensions 4 statement Within the lt Constraints gt section we are assigning the items to the appropriate factors The first step in assigning items to factors to fix all slope values to 0 via Fix v2 v21 Slope which allows us in the next step to free the individual items to load on only the appropriate factors As can be seen in the four Free statements we are specifying that Items 2 through 5 load on the first factor Items 6 through 9 load on the second factor Items 10 through 15 load are assigned to the third factor and so on This corresponds to the description provided earlier in which Items 2 to 5 belong to the Family factor only and Items 6 to 9 belong to only the Money factor etc The final constraints such as Free Cov 2 1 are used to tell flexMIRT to freely estimate the covariance correlation between the facto
18. SEM lt Groups gt Group1 File QOL DAT Varnames vi v35 N 586 Ncats vi v35 7 Model vi v35 Nominal 7 Tc vi v35 38 gt 0 0 0 0 0 O 1 1 0 0 O O 2 1 1 0 0 O 3 l l 1 0 0 0 5 1 1 1 1 1 6 0 0 0 O O lt Constraints gt Fix vi v35 ScoringFn Equal vl v35 Slope Equal vi v35 Intercept 2 Equal vi v35 Intercept 3 Equal vi v35 Intercept 4 Equal vi v35 Intercept 5 Equal vi v35 Intercept 6 Thus far we have covered how to use flexMIRT to calibrate several of the most commonly used single group unidimensional IRT models and to obtain IRT scale scores While these types of models are useful the greatest strength of flex MIRT is its ability to readily handle complex models e g multidi mensional bifactor testlet etc and more complex data structures such as multilevel data In the next chapter we will provide instruction and examples building intuitively from the basic syntax already presented to demonstrate how flex MIRT syntax files are constructed to handle more complex models and data structures 39 CHAPTER 3 Advanced Modeling The previous chapter focused on basic unidimensional single group models to introduce the flex MIRT syntax output and available IRT models While these examples are useful in their own right it is expected that a user work ing with real world data will need more advanced modeling options
19. SaveSco Yes No SaveDbg Yes No DMtable Yes No NormalMetric3PL Yes No SlopeThreshold Yes No Specific to CDMs the command DMtable controls whether the diagnostic classification probability table is printed in the irt file The default is to print the table but this may be modified by changing the command to DMtable No Modifying this setting may be helpful if the probability table will be large The final two statements in this section are provided primarily to facili tate the use of parameter estimates obtained from other programs but users may find them convenient for other purposes Both NormalMetric3PL and SlopeThreshold are used to print save out or read in parameters that are 139 in more traditional IRT metrics than the MIRT compatible logistic metric flex MIRT uses these keywords are only available for use in conjunction with unidimensional models When NormalMetric3PL Yes the program will print normal metric parameter estimates in the output and save those values into the prm file if requested Specifically lex MIRTO will 1 print save a 1 702 rather than the typical logistic metric slope values 2 instead of the intercept bx a flexMIRT will print output the difficulty b parameter value and 3 rather than the logit guessing parameter the program will report save the customary g value Note that even if the normal metric keyword is used during calibration any priors that may be speci
20. provides two general methods for DIF testing Langer 2008 1 a DIF sweep for all items and 2 a more focused examination where candidate DIF items are tested with designated anchor items Both methods utilize the Wald test We will first cover the more general DIF sweep again using the end of grade testing data As the syntax is quite similar to what was presented earlier only the relevant changes will be noted in the excerpt The full syntax file may be found in the corresponding folder on the flex MIRTO support website Example 3 5 DIF Test All Items Excerpts lt Options gt DIF TestAll DIFcontrasts 1 0 1 0 lt Constraints gt 45 Prior Grade3 vi v12 Guessing Normal 1 1 0 5 Prior Grade4 vi v12 Guessing Normal 1 1 0 5 First in conjunction with the use of the 3PL model a logit normal prior is applied to the guessing parameters which was a previously unseen option With respect to the DIF analysis the relevant new statements are contained in the Options section To enable the general DIF sweep testing all items DIF TestAll is specified Also required when testing for DIF are values that supply flexMIRT with the entries for a DIF contrast matrix specifying how contrasts among groups should be constructed In this example we are conducting a straight forward comparison of Grade3 against Grade4 so the statement appears as DIFcontrasts 1 0 1 0 with spaces between the s
21. v3 v12 Intercept Grade4 v3 v12 Intercept Equal Grade3 v3 v12 Slope Grade4 v3 v12 Slope The keyword used with the DIF statement is now TestCandidates as opposed to TestA11 in the previous example The DIFcontrasts statement remains unchanged To specify candidate DIF items within the Groups sec tion we simply list the candidate item names using the DIFitems statement found as the last entry within each group subsection of the excerpted code Here we are testing Items 1 and 2 for possible DIF As can be seen in the Constraints section the item parameters for the anchor items v3 v12 are set equal across groups The DIF table printed in the output is structured as in the previous example and will not be covered here 3 5 Multiple Group Multilevel Model Our final multiple group example will cover the flex MIRT syntax options for handling nested data Using the student cognitive outcomes data from the 2000 Program for International Student Assessment see Adams amp Wu 2002 31 mathematics items with mixed formats multiple choice and constructed response from students in both the United States and Ireland will be analyzed The US data set is comprised of 2115 students representing 152 different schools and the Irish data set has 2125 student observations coming from 139 different schools From the general description of the data it is clear that the complete set of data for this examples has two groups U
22. 0001 Category 0 Category 1 Score Observed Expected bserved Expected 0 1 0 4 1 4 5 1 1 4 1 0 4 8 0 6 1 By specifying Score SSC in the command file we requested a summed score to EAP scale score conversion table be printed which is shown below With a total of 5 items there are 6 possible summed scores which are listed in the first column Going across a row after the sum score the EAP score and the posterior standard deviation associated with that particular summed score value are presented Additionally the table also includes model implied and observed probabilities for the summed scores appearing in columns labeled P and 0 respectively Immediately after the table the marginal reliability of the scale scores is printed as well as the value of a chi square statistic that provides a test of the underlying normality of the latent variable distribution Output 2 14 1PL Grouped Data Output Summed Score to Scale Score Conversion Summed Score to Scale Score Conversion Table Summed Score EAP SD P 0 0 00 910 797 0 0023654 0 0030000 00 429 800 0 0208898 0 0200000 00 941 809 0 0885974 0 0850000 0 2370000 0 3570000 1 2 3 00 439 823 0 2274460 4 00 084 841 0 3648997 5 00 632 0 864 0 2958018 0 2980000 Summed score based latent distribution fit S D2 0 9 p 0 8166 Marginal reliability of the scaled scores for summed scores 0 29427 Scrolling further down in the full output file we encounter se
23. 1 A Generalized Formulation Consider a linear predictor for a level 1 unit i e g individual in level 2 unit j e g school in a group g e g country Wis Ag Big Ag Or where y j is an ng x 1 vector of linear predictors of the n items in this group AP is an Ng X p matrix of item slopes on the p level 2 between latent dimen sions or latent attributes and interaction terms and A is an ng x q matrix of item slopes on the q level 1 within latent dimensions or latent attributes and interaction terms with 07 and er being the between and within latent dimensions or attributes and Biibule interactions Note that both AP and AW are implicitly assumed to be functions of d unknown structural param ster in 6 subject to appropriate identification conditions or user specified constraints At the item level the IRT model is a conditional density fz Yijkglnijkg fe YijrglO 07 where k 1 ng and Yijkg is the item response to item k from level 1 unit in level 2 unit 7 in group g Conditionally on the latent variable levels the item responses are independent such that B f yijg 05 055 ijg ll fe yijxg 07 015 Let the distribution of 87 be fz 0 wo the level 1 latent variables can be 162 integrated out as fs yi485 S Yil 0 055 fe 01 4015 B jg the marginal Assuming further conditional independence of level 1 units on 0 distribution of all item responses in a level 2 unit is Nj fX
24. 5 8 2 Multi level MIRT Model 5 4 Fixed Effects Calibration ee Cognitive Diagnostic Models 6 1 General CDM Modeling Framework 6 2 CDM Specific Syntax Options i233 9 ey ws 6 3 CDM Modeling Examples sv Si xU ERRpaRAR Pa ies 6 3 1 Introductory CDM Examples 6 3 2 Replicating Published Examples Details of the Syntax 7 1 The Project Section xl poesie Er exe ae bs ee ho 7 2 The Options Section vus ho Ae RE SLE 7 3 The Groups Section eacus pex ud e adum due San d tlg d 7 4 The Constraints Section Soi o5 x Vac ES estu Y eret You Models 8 1 A Generalized Formulation lll 8 2 Item Response Models cll 8 2 1 Model with Pseudo Guessing Parameter 8 2 2 Graded Response Model ii 85 85 8T 89 89 93 97 99 100 101 102 103 123 138 138 138 151 157 162 162 163 163 164 8 2 8 Nominal Categories Model Appendices A Quick Start Installation and User Guide Al Installing flexMIRT 4d ua UG RC REESE A 2 Using flexMIRT 452 5 4 4 23m SERENA UM B flexMIRT via the Command Line Interface B 1 flexMIRT via the CLI single flerun B 2 flexMIRT via the CLI batch mode C Plotting C 1 Plotting ICCs TCCs and Information Functions D Labeled Output Files E Licensing Information E 1 Commercial use of flexMIRT E 2 Classroom Discounts for flexMIRT Referen
25. 50 94 97 151 125 135 156 152 EmpHist 43 154 155 File 5 6 12 16 19 21 24 30 Ta 58 154 156 Identity 33 125 132 33 37 39 41 43 44 49 50 54 57 60 62 64 69 71 73 75 77 90 94 97 105 112 Tc 39 58 154 156 Identity 33 Varnames 5 6 12 16 19 21 24 116 122 125 132 135 151 FixedTheta 97 Generate 101 102 116 135 136 156 InteractionEffects 101 112 114 121 122 125 132 ItemWeight 154 155 Key 151 153 Missing 50 151 152 25 30 33 37 39 41 43 45 49 50 54 57 60 62 64 69 71 73 75 77 90 94 97 105 112 116 122 125 132 135 151 152 HabermanResTbl see lt Dptions gt HabermanResTbl 196 Hierarchical Data see Multidimensional IRT Multi level Model Imputations see lt Options gt Imputations Information Function 10 11 42 143 144 InitGain see lt Options gt InitGain InteractionEffects see lt Groups gt InteractionEffects ItemWeights see lt Groups gt ItemWeights JSI see lt Options gt JSI Key see lt Groups gt Key Local Dependence LD 15 23 28 146 Jackknife Slope Index JSI 25 28 147 Standardized Chi square 15 22 23 28 145 logDetInf see lt Options gt logDetInf M2 see lt Options gt M2 Marginal Reliability 10 11 27 MaxE see lt Options gt MaxE MaxM see lt Options gt MaxM MaxMLscore see lt Options gt MaxMLscore MCsize see lt Options gt MCsiz
26. 8 Probability 0 04 0 2 0 0 Theta Similarly for obtaining information function plots the function call of flexmirt inf is used and the name and location of an inf file is provided in the parentheses So the example syntax line flexmirt inf C Users laptop Documents flexMIRT flexMIRT plotting fit2 inf txt 182 is telling R to use the information plotting function flexmirt inf to plot the information values found in the file fit2 inf txt As noted in the comments in the R code file the information plotting function pulls information from the calibration file that created the inf txt file The program assumes that the inf file and the original calibration syntax file flexmirt will be in same folder If for some reason the flexmirt file is not in the same folder as the information file the plotting function will return an error and is unable to run The provided plot gives a representative of the graphics produced by the IIF TIF plotting function Figure C 2 IIF produced by provided R code IIF for Group 1 Item 1 1 0 0 8 Information 0 4 3 2 1 0 1 2 3 The R programs provided are free software you can redistribute and or modify them under the terms of the GNU General Public License http www gnu org copyleft gpl html as published by the Free Software Foundation either version 2 of the License or at your option any later version The programs are distribut
27. Calibration Extended GOF Table Marginal fit Chi square and Standardized LD X2 Statistics for Group 1 Groupi Marginal Item Chi2 o oc 5gogmwn e o o g O O B B OC U CO CO B 0 1 0 0 0 1 0 1 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 1 0 0 0 0 0 0 0 0 a IPP Y p ITSO o 2 7 1PL Model Fitted to Response Pattern Data Up to this point analyses have been conducted on raw individual data in which each line in the data set represents responses from an individual flex MIRT is also able to handle data that are grouped by response patterns 23 Example 2 6 1PL Calibration and Scoring with Response Pattern Data lt Project gt Title LSAT 6 Description 5 Items 1PL N 1000 Grouped Data lt Options gt Mode Calibration GOF Complete JSI Yes HabermanResTbl SX2Tbl Yes FactorLoadings SaveSco No Score SSC lt Groups gt Group1 File lsat6grp dat Varnames vi v5 w Select vi v5 only the first 5 variables CaseWeight w N 30 Ncats vi vb 2 Model vi v5 Graded 2 lt Constraints gt Equal vi v5 Slope Entire Block of Comments The syntax file above uses the well known LSAT6 data set in which the individuals have been grouped by response patterns There are 1000 examinees and 5 dichotomous items The first use of constraints in the lt Constraints gt section is also presented demonstrating how to fit a 1PL model in flexMIRT An
28. Example 2 4 3PL ML Scoring example Project Title 3PL ML Scoring Example Description 12 items 1 Factor 1 Group Scoring lt Options gt Mode scoring ReadPRMFile 3PL_example prm txt Score ML MaxMLscore 3 5 MinMLscore 5 5 saveSCO YES lt Groups gt Group1 File g341 19 dat Varnames v1 v12 N 2844 lt Constraints gt This syntax is the same as the previous scoring example save for the change from Score EAP to Score ML and the additional commands MaxMLscore and MinMLscore used to reign in the extreme ML estimates We present the first 10 cases of the associated sco txt file where the scores were saved 19 Output 2 8 3PL ML Scoring Output 198234 427265 0 113716 427265 675141 0 637464 382694 205723 500000 859248 573388 184667 680658 184667 625639 606990 792826 628273 990000 658306 O ON O O 50 N eO O O O O O Or B E RP PPP cO cO 2 3 2 3 3 3 3 3 2 3 o o Like the previous sco file the first column indicates group membership and the second column provides the flex MIRT 9 assigned observation number The third column reports the number of iterations to reach the reported score The fourth and fifth columns are the ML theta estimate and the associated SE respectively As can be seen for observation 9 our specified maximum of 3 5 was applied Because the reported value was assi
29. If the return code is 0 the replication completed successfully if it takes any value other than 0 a problem was encountered Column 3 reports the log likelihood Column 4 gives the number of estimation cycles that were completed to fit the model the 5th column reports the time the calibration run took and the 6th column give the inverse of the condition number The inverse condition number is equal to the ratio of the smallest to largest eigen value of the asymptotic covariance matrix of the item parameters if it is very small it shows instability of the obtained solution Output 4 5 Beginning Columns of Output from 5 Replications O 4 42522e 003 46 1 41000e 001 8 61952e 003 2 87409e 000 1 99261e 000 7 62795e 001 0 4 44206e 003 40 1 25000e 001 1 52923e 002 3 00412e 000 1 88355e 000 7 38170e 001 O 4 28512e 003 41 1 25000e 001 8 18986e 003 2 78522e 000 1 87415e 000 7 44810e 001 O 4 33888e 003 31 1 09000e 001 1 63897e 002 2 74887e 000 1 86261e 000 6 77367e 001 O 4 37110e 003 29 1 09000e 001 1 82420e 002 2 58602e 000 1 84299e 000 8 12819e 001 At the seventh column reporting of the item and latent variables param eters begins The parameters are reported in the same order as they are numbered in a corresponding one off calibration run output It is strongly rec ommended that a single run of the calibration syntax file be conducted prior to employing the ISF to both ensure the syntax works properly and to obtain the ordering of parame
30. Non Normal Population Distributions 42 3 3 Multiple Group Scoring Only 2s wx be Rogue 43 ob DIE Analyse cu iR xe LER BS dk eS eS 45 3 5 Multiple Group Multilevel Model 49 3 6 Multidimensional Model 4 Correlated Factors 53 3 7 The Bifactor Model coor ase ia es 56 3 7 1 Bifactor Model Using the Reparameterized Nominal Cat egories Model _ uis Roman pg s 56 3 1 2 Bifactor Model Using the 2PL 429v 59 4 Simulation 61 4 1 A Syntax Only Example Using Value Statement 61 4 2 The First Example Redone Using Parameter File 64 4 3 The Parameter File Layout 2 Ryo amp ape ms 4 3 1 Column Progression for Groups 4 3 2 Column Progression for Items 4 4 A 3PL Model Simulation Example 4 5 Simulating Data with Non Normal Theta Distributions 4 6 Simulating Bifactor Model Data 4 7 Simulating Group Differences 4 8 Multilevel Bifactor Model Simulation 4 9 Internal Simulation Function 256i gk 3j 4 9 1 Accessing the Internal Simulation Function 4 9 2 The Simulation Output File Metropolis Hastings Robbins Monro Estimation for High Dimensional Models 5 1 Overview of MH RM Estimation 5 2 MH RM Specific Syntax Options 5 3 Examples Utilizing MH RM Estimation 5 3 1 MIRT Model with Correlated Factors
31. SZL9L9 0 TII6IC O 8vT000 0 S SZ00 0 8168290 696600 0 S18v48 0 60Z8Z0 0 099cS 0 691v60 0 Tccess O T 9 S0 0 T66ZZT 0 978100 0 8vSS61 0 OZEZS0 O c 0 cT O 418T00 0 8vTEVC O T69100 0 SC6810 0 v66000 0 Sccvss 0 8S6 00 0 86TTCT 0 88TTTO 0 vC89 V O 68T9 0 0 SS4 0c 0 Tv1LECO O T9 CT T oT t 8T9 0v 0 6 T TS8EZT T 8 T T092v0 T T 9 T 8t SST I 6ZLESL O 8TSZ00 0 s T TTITv497 0 v T 0S6880 1 T SSL89b 0 e T 6 9 98 0 T OOTTEL 80v0c0 90ESLZ 8v90T1T1 8veTel 6800c0 889 99 668v00 681109 118600 608822 8T600 960062 Tv4c0 8v161c TOESS0 984109 SS 0S0 T 0 0 T o o T o o T o 0 o 0 T 4 o 0 T 0 o 4 o 0 T o o 109 The first two columns of the sco file give the group and observation num ber similar to a standard flexMIRT IRT score output file Recall that for this example there are 4 attributes and 16 possible attribute profiles The es timated probability of the respondent having mastered each of the attributes individually is given starting after the observation number For example for observation 1 the probability they have mastery of attribute 1 is 0 054 for attribute 2 it is 0 128 and for attribute 3 it is 0 165 and for attribute 4 it is 0 375 The following four columns after the separator are the estimated sta
32. Scoring lt Options gt Mode Scoring ReadPRMFile 3PL_example prm txt Score EAP saveSCO YES lt Groups gt Group1 File g341 19 dat Varnames v1 v12 N 2844 lt Constraints gt For scoring runs the lt Project gt section stays the same The first change encountered is in the Options section in that Mode is now set to Scoring Using the command ReadPRMFile the name of the file that contains the item parameter estimates is provided In this case the file was created by the previous calibration run The type of IRT scale scores is selected using the Score command In this case Expected A Posteriori EAP scores will be saved We may also specify ML for maximum likelihood scoring MAP for Maximum A Posteriori scoring or SSC for summed score to EAP conversion tables We have also specified that the scores should be saved into a sepa rate file which will be labeled with the extension sco txt The command requesting flex MIRT to output the estimated scores SaveSco Yes is redundant for most scoring methods but it is needed if individual scores are desired for the SSC method Without the SaveSco Yes command SSC will produce only the summed score to scale score conversion table Notice that this may be a desirable feature in some situations when only the conversion table is needed with actual scoring occurring at a much later stage The output of scoring runs differs from calibration
33. WinFlexMIRT executable is found This is done using standard change directory com mands For example if flexMIRTO was located in the folder C flexmirt you would type cd C flexmirt where cd stands for change directory and the remaining text is the path to the folder Command Prompt Microsoft Windows Version 6 1 76611 Copyright lt c gt 2009 Microsoft Corporation All rights reserved IC Users UPG gt cd c f lexmirt ic Nf lexmirt gt If a correct location has been specified the working directory will be changed which can be seen in the above image otherwise an error message of The system cannot find the path specified will be reported Once the CLI has been pointed to the folder that contains the Win FlexMIRT executable all that remains is to supply the command file name with the appropriate WinFlexMIRT call If the command file is found in the same folder as the executable the call is of the form WinflexMIRT exe r mysyntax flexmirt where the r tells WinFlexMIRT to run in the CLI and mysyntax flexmirt is the command file you wish to run 175 If the command file is found in a different folder or has spaces in the file name then the full path to the command tile must be given in quotation marks For example if we wished to submit the command file FL 20item flexmirt for analysis and it is located in the folder C Users VPG Documents FL analyses Final sca
34. YT OT O 1v O ST cr O L 0 T 3eup d 98 s c 3eUuL 4d 9 sS T 3eu Hd T NO1H 103 XT1IP N eouerieAoj eouerIeA STQeTIeA 149987 00 0 1D T I nu dd Teqe1 dnoi15 Se3eurasq Teyowereg dnoi5 yoee UTYAIM sQuepuodsei QZ sun ZT OOT swear 9 eqeq peaernurg o3 TSPON LUIN T9A9T OML IFA QA GA v gA TA TA Teqey UeiT 19 T dno15 103 sue31 Tdz yore upy squepuodsaz Oz sarun zT OOT swear 9 eqeq peaernurg o3 epo LUIN T9A9T OML IFA SIojourereq dno15 pue woy pow IYIN PAM p mdmo 96 5 4 Fixed Effects Calibration New to flexMIRT is the ability to perform fixed effects aka fixed theta cal ibration in which the user supplies individual theta values which are fixed to be used in calibrating items This mode of calibration is potentially useful in field testing experimental items Fixed effects calibration is only available in conjunction with MH RM estimation The keyword to induce flex MIRT to perform fixed effects calibration is FixedTheta FixedTheta is a group specific command that is used to set the variable s that contain s the known indi vidual theta values for that group in the calibration By implementing fixed effects calibration in flexMIRT as group specific it is possible to construct a calibration in which one group is based on fixed theta values and the other group is based on random estimated thetas Example 5 4 Fixed Effects Calibration lt Project gt Title LSAT 6 Fixed Eff
35. Yijkg l mijkg guessy 7 163 8 2 2 Graded Response Model Suppose item k has K graded categories Let the cumulative response proba bilities be PelYijkg gt 0 mgkg 10 1 1 exp Ckg 1 Hike Pz Yijkg gt Tong 1 Px Yjing gt K 1 m e Vijkg Inijkg 1 exp ckg K 1 T jkg Pr yijkg K nijkg 9 0 where the c s are item intercepts and the boundary cases are defined for con sistency Then the category response probabilities are PelYijkg l mijkg PelYijrg Unigeg Pe vijkg gt l 1 nijng 8 2 3 Nominal Categories Model The nominal model fit by flexMIRT is the multilevel extension of the repa rameterized nominal model due to Thissen et al 2010 Suppose item k has K nominal categories Category l s response probability is defined as exp Skg Mijkg Ckg 1 K 1 y ml exp Skg mflijkg T Ckg m Pz Yijkg l nijkg where skg is the scoring function value for category l and cpg is the intercept value for category l Dropping the kg subscript for a minute Thissen et al 2010 pointed out that for identification the following restrictions should be in place s 0 sk 1 K 1 co 0 This can be accomplished by reparameterizaton Let 0 1 CQ s 4 anda EE SK 1 Ck 1 The vector is a K 2 x 1 vector of scoring function contrasts that defines the ordering of categories and y is a K 1 x 1 vector of intercept contrasts 164 Th
36. as the previ ous although we have changed the parameter file name The added feature here is that a non normal distribution will be specified in the generating pa rameter file EHgenparams txt This change only effects that values entered into the Group row of the parameter file An examination of the generating parameter file will find that the 5th column of the Group row been changed from the value of 0 seen in Table 4 3 to a value of 1 As noted in Table 4 2 this indicates that an empirical histogram prior will be used The 6th column then provides the number of points to be used in constructing the histogram 121 and the next value indicates the range over which the points are spread from 6 to 6 here as indicated by the value of 6 in the 7th column of the Group row Following the specification of the empirical histogram parameters the latent variable mean s and covariances are listed After all means and covariances are listed 71 the heights of the empirical histogram are provided For our current example we need to provide 121 values per the entry in Column 6 Figure 4 1 provides a graphical display of the empirical histogram provided in the parameter file to demonstrate the extent of non normality Figure 4 1 Empirical Histogram From EHgenparams txt Normalized Density Ordinates 0 02 0 03 0 04 i I I 0 01 l 0 00 1 4 6 Simulating Bifactor Model Data We will be using item parameters obtained fr
37. assessment at hand Basic DINO Fit to Simulated Data The final basic example using this simulated dataset will fit the disjunctive deterministic noisy or gate DINO model Unlike the previous models the DINO allows for both non zero main effect and interaction terms However and again using the LDCM framework constraints are placed such that the main effect slopes are set equal and the interaction term is 1 times the main effect constrained to equality slope This is done because an item may be endorsed answered correctly if any one of the attributes is mastered and there is assumed to be no added benefit to having mastery of several all attributes over mastering only one or a subset of the attributes the 1 applied to the interaction term serves to remove such a benefit from the model Example 6 4 DINO Fit to Simulated Data Project Title Calibrate and Score Simulated DM Data Description DINO Model Options Mode Calibration MaxE 20000 Etol 1e 5 SE REM SavePrm Yes SaveCov Yes SaveSco Yes Score EAP Groups 4G File DMsim dat Varnames v1 v15 115 N 3000 Ncats vl v15 Model v1 v15 2 Graded 2 Attributes 4 Generate 3 4 Dimensions 5 4 ME and 1 2nd order int AD Varnames ai a4 DM G lt Constraints gt Fix G v1 v15 MainEffect Free G v1 v3 MainEffect 1 Free G v4 v6 MainEffect 2 Free G v7 v9 Main
38. batch examples ex6 2D flexmirt in 886 lt ms gt Batch completed with errors in 00 01 22 336 Average process time 259 86ms log saved here C flexmirt batch examples batch test_log txt C f lexmirt gt Once flexMIRT has processed all of the syntax files named in the batch file it will report that it has finished the batch and print the total and average processing time per file It will also give the name and location for a summary log of the batch run the log file will be located in the same directory as the batch file and will be named with the batch file name and a _log txt suffix 178 The batch log from the example batch is presented here and as in the command prompt screen shows that broken flexmirt was not completed due to errors and the run time for each successfully completed syntax file and the batch as a whole File Edit Format View Help 11 28 2012 11 28 2012 11 28 2012 11 28 2012 11 28 2012 11 28 2012 YE 11 28 2012 11 28 2012 NNNNNNNNN Found 7 pu to run in c flexmirt batch examples batch test csv Finished c flexmirt batch examples Mexi 2PL flexmirt in 124 ms Finished c flexmirt batch examples ex1 3PL flexmirt in 156 ms Finished c flexmirt batch examples ex2 flexmirt in 15 ms Finished c flexmirt batch examples ex3 calib flexmirt in 605 ms Errors in C Nf lexmirt batch EE Finished c flexmirt batch examples Xex5 flexmirt in 18 ms Finished c flexmirt b
39. because these cycles do not contribute much in the fixed effects case and eliminating them improves the time efficiency of the estimation When FixedTheta is specified and the program can successfully run a problem flex MIRT will print in the main output a notification that fixed effects calibration was performed prior to reporting the item parameter esti mates for the group Such a notification is presented in the example program output Also with regard to the output please note that for a single group fixed effects analysis or a fixed effects analysis that applies to all the groups in a multiple group run the SE of the log likelihood approximation will be reported as 0 Output 5 5 Fixed Effects Calibration Output Item Parameters Fixed effects calibration in Group 1 Groupi LSAT 6 5 Items 1PL N 1000 2PL Items for Group 1 Groupi Item Label P 1 vi 2 v2 3 v3 4 v4 5 v5 98 CHAPTER 6 Cognitive Diagnostic Models Over the last several decades a growing field of interest and research has been the development and estimation of latent variable models that locate respon dents in multidimensional space for the purpose of diagnosis in this context diagnosis includes the usual meaning such as diagnosing a psychological dis order but also encompasses activities such as identifying a student s mastery of content areas in an educational setting Such models have been variously termed cognitive diagnostic mode
40. been requested The overall fit statistics are followed by three members of the power divergence family of fit statistics e g Cressie amp Read 1984 for assessing latent distribution fit specifically the summed score likelihood based G D and X These indices have been found to accurately detect departures from normality within the latent distribution while appropriately ignoring other forms of model misspecification such as multidimensionality Li amp Cai 2012 Output 4 7 ISF output Switch from Model 1 to Model 2 Values 1 29309e 314 1 47651e 317 4 52136E 03 31 9 70097e 315 1 47651e 317 4 40664E 03 24 2 32371e 315 1 47651e 317 4 44256E 03 24 1 04469e 314 1 47651e 317 4 45391E 03 21 1 02812e 314 1 47651e 317 4 58027E 03 22 If the ISF was instructed to fit multiple models to the data then the values for next model will follow on the same line after the latent distribution fit statistics of the current model The new model reporting begins with the replication number and follows the same general order of reporting detailed previously If we fit two models to a given set of simulated data the output columns at the point where the reporting for the first model ends and the second model begins will look something like the values presented in Output 83 4 7 although we have added the break line for emphasis Note however that the numbering of parameters may change from model t
41. blank space will not be interpreted correctly by the program The number of valid response categories observed in the data file is specified with the NCats statements Within the parentheses of the NCats statement the names of the variables affected by the command are listed For example if Item 24 28 and 32 40 all had four response options this could be entered as NCats v24 v28 v32 v40 4 in flex MIRTO syntax The internal representation of item response data in flexMIRT is zero based i e response options must start at zero and go up That being said the program is capable of recoding the data when old and new values are specified with the Code command If the first 10 items are read from a data file that had four response options 1 2 3 4 it would be necessary to recode these to zero based prior to fitting the IRT model The recode statement could be Code vi v10 1 2 3 4 0 1 2 3 The original values are listed first and the values to which they should be recoded appear within the second set of parentheses If it is necessary to also collapse the responses so that the original codes 1 and 2 become a single response option 0 and the original 3 and 4 become a second option 1 that Code command could be Code v1 v10 1 2 3 4 0 0 1 1 Note that recoding does not change the values of the number of response options in the data file as defined by the NCats 152 statements The Key statement also provides a specifi
42. combination of Mode Calibration Algorithm MHRM and a random number seed via RndSeed is required All other listed commands are optional with the default values shown above Imputations controls the number of imputations from the MH step per RM cycle A larger value will lead to smoother behavior from the algorithm In most cases a value of 1 should be sufficient for obtaining accurate point estimations however this number may be increased for better standard errors Thinning refers to the MCMC practice of retaining only every kth draw from a chain Thinning is used in part to reduce the possible autocorrelation that may exist between adjacent draws The Thinning statement sets the interval for the MH sampler meaning if Thinning 10 every 10th draw by the sampler will be retained The Burnin statement controls the number of draws that are discarded from the start of the MH sampler These draws are discarded to avoid using 87 values that were sampled before the chain had fully converged to the target distribution Burnin 10 tells flexMIRT to discard the first 10 values obtained by the MH sampler Stage1 determines the number of Stage I constant gain cycles The Stage I iterations are used to improve default or user supplied starting values for the estimation that occurs in Stage II Stage2 specifies the number of Stage II Stochastic EM constant gain cycles The Stage II iterations are used to further improve startin
43. employed Multiple Ta and Tc statments may be supplied allowing for a large degree of flexibility in the parameterization of the nominal 156 model The final command DIFitems in the lt Groups gt section supplies candidate DIF items This command is required when DIF TestCandidates has been invoked in the Options section The names of candidate items are listed across separated by commas e g DIFitems vi v4 v8 7 4 The Constraints Section Several of the examples have shown that there is no required statement in the Constraints section and that it may be left blank after the required section header However as models become more complex especially if they include multiple groups constraints will more than likely become necessary Listed in the left most column of Table 7 1 are the types of constraints available Value indicates the starting or fixed value of a parameter Free indicates a parameter which is freely estimated Equal indicates parameters which will be constrained equal Fix fixes a parameter at a specific value Coeff applies a provided coefficient to a given parameter enabling proportionality restrictions AddConst applies a additive constant to specified parameters and Prior provides a prior distribution for that parameter Table 7 1 Constraint Commands and Keywords Constraint Type Parameter Prior Distribution Free Intercept Normal y 7 Equal Slope logNormal u c Fix Guessing B
44. examination of the data file Isat6grp dat shows that that the data for these five items have been grouped by response patterns with the last variable in the file providing the number of observations per pattern After naming the group and file name in the Groups section as usual we use the Select statement to select only the first 5 variables that will be subject to 24 IRT analysis i e we exclude the weighting variable which was named w in the Varnames statement Note that this is also the first time that we have encountered comments in the syntax file A comment begins with 2 slashes following the standard C C convention They may appear anywhere in the syntax but not within a command statement In addition one may also use a pair of to create entire blocks of comments that span more than one lines as shown at the end of the syntax file Returning to the example syntax we use the CaseWeight command to identify the variable that contains response pattern frequencies i e the num ber of cases per response pattern The rest of the Groups statements those defining number of categories etc are as before Note that the graded model with 2 categories is specified in the Model statement which is equivalent to a 2PL model To obtain a 1PL model the slope discrimination parameters must be constrained equal across items To accomplish this an equality constraint is specified in the Constraints sec tion usin
45. example to fit the graded model with four cate gories to the hypothetical items previously mentioned the Model statement is Model v24 v28 v32 v40 Graded 4 The question mark in parentheses following the Graded Nominal and GPC keywords in Example Box 7 7 are place holders for the number of categories the model should accommodate When items are of mixed formats multiple choice free response Likert type scale multiple NCats and Model statements may be needed to correctly de scribe the data and the chosen IRT models We have covered the required commands within the Groups section as well as several optional commands that are useful for selecting or modifying data In addition to the syntax statements previously covered there are other additional more specialized commands which are also available 153 Example 7 8 Groups Specialized Data Model Descriptors Groups gt CaseWeight var Dimensions 1 Primary 0 Between 0 Nlevel2 Cluster var CaseID var EmpHist Yes No ItemWeight vars BetaPriors vars Attributes InteractionEffects Generate DM group Ta vars Trend Identity Tc vars Trend Identity DIFitems vars The CaseWeight command is needed when response pattern by frequency aka grouped data is entered It provides the program with the variable name for the column that contains the number of
46. extension being listed in the output files The layout of the item parameter output table presented in Output 2 5 is essentially the same as before with the slopes thresholds and difficulty values and their standard errors reported first In the last four columns of the table the estimates for the lower asymptote both in its logit form and in terms of the pseudo guessing probability are reported as well as standard errors 13 Le ve Go TE ez ez 69 GZ 9 oS TZ 61 6 9T 80 T 61 OT 61 L 00 v 60 T S 4180T d tc CE er 8T er 8T ST CG er 9c CEN L Ge v6 Go 96 6 ge Te og 9 0 4G 6 vC 84 TZ A 81 9A ST SA Gl yA 6 aa 13 9 ZA oe 3 TA e s e Ha T qeT wedy Tdno1n I dnozy 103 swear TAE O O O O O O O O O OO O Qm O O O O HATA OTA AO O m O O O O O O O O OV O O O m O O O O O O O O OP O O OI O O O O O O O O O OOOI O vd d o od o od o 4 od 4 o4 o Y 0 0 0 0 0 me 0 0 0 0 0 0 s seyeutyse Teqyewezed SurAes uotTyerqt Te dnozy I 109984 T sweat ZI e duexe Tag 9x3 wid 6T Tpes 9TTJ eaeura3se Teyewered 3x8 4X4 42T 6T Tpes szeqyewezed T o13uo9 pue sqTns rx 2x8 saTt4 andang s1ojourereq WY 3nd3n uorjeuqie Td dnory 8ut g z 3nd3nQ 14 In addition to the added parameters and output files the GOF statement was set to Ex
47. factor null model 2loglikelihood 2885 67 Akaike Information Criterion AIC 2895 67 Bayesian Information Criterion BIC 2917 58 Full information fit statistics of the fitted model Degrees G2 of freedom Probability FOhat RMSEA 8 45 6 0 2064 0 0143 0 03 Degrees X2 of freedom Probability FOhat RMSEA 8 99 6 0 1739 0 0152 0 03 Full information fit statistics of the zero factor null model Degrees G2 of freedom Probability FOhat RMSEA 127 43 10 0 0001 0 2153 0 14 Degrees X2 of freedom Probability FOhat RMSEA 124 58 10 0 0001 0 2104 0 14 With the zero factor null model Tucker Lewis non normed fit index based on G2 is 0 97 35 With the zero factor null model Tucker Lewis non normed fit index based on X2 is 0 96 Limited information fit statistics of the fitted model Degrees M2 of freedom Probability FOhat RMSEA 0 17 6 0 9999 0 0003 0 00 Note M2 is based on full marginal tables Note Model based weight matrix is used Limited information fit statistics of the zero factor null model Degrees M2 of freedom Probability FOhat RMSEA 117 87 10 0 0001 0 1991 0 13 Note M2 is based on full marginal tables Note Model based weight matrix is used With the zero factor null model Tucker Lewis non normed fit index based on M2 is 1 09 2 8 2 Generalized Partial Credit Model We return the QOL data again to illustrate Muraki s 1992 Generalized Par tial Credit GPC model The GPC model is a constrained speci
48. flex MIRT O has some data manipulation capabilities recoding scoring using a provided answer key etc it is expected that the user has completed necessary pre analyses such as dimensionality tests en suring that all response options have observations etc prior to submitting data to the program Missing data are indicated by 9 by default although the missing data code may be modified to any numeric value between 127 and 127 All syntax and data files used in this manual are available in the Support section on the flex MIRT website http www flexMIRT com Support 2 1 Overview Generally speaking flex MIRT syntax files are divided into four required sec tions each of which provides instructions for distinct parts of the analysis The four sections which will be discussed in more detail are 1 Project where you provide a title and a general description of the analysis for record keeping 2 Options where the type of analysis and options of the analysis are specified 3 Groups where the input data and the IRT models are specified 4 Constraints where parameter constraints such as equality or fixed parameter values or univariate priors may be specified All four section headers must be present in every command file e g even if no constraints are applied to the parameters the Constraints section header must still be declared in the syntax The other requirement is that all sta
49. flexMIRT via the CLI batch mode A newer feature in flex MIRT is the ability to submit multiple syntax files via a single batch file The feature is only available in software releases after December 1 2012 If you are using a version of flex MIRT 1 0 released prior to that date you will need to download the latest version of flex MIRT from the website and reauthenticate your license to have access to batch mode processing The batch file consists of a single column of comma separate values in which the user provides the locations and file names of the command files to be run The flexMIRT syntax files in the batch file must ALWAYS be referenced with the full path Below is an example batch file saved as a CSV in Excel Home Insert Page Layout Formulas Data Review V P amp cut Calibri 1 7 A A A3 Copy 7 Paste Hex E GV Format Painter Br 2 4 Clipboard F Font F Aligi III II l M c flexmirt batch examplesVex1 2PL flexmirt C flexmirt batch examples ex1 3PL flexmirt c flexmirt batch examples ex2 flexmirt c flexmirt batch examples ex3 calib flexmirt C flexmirt batch examples broken flexmirt c flexmirt batch examples ex5 flexmirt c flexmirt batch examples ex6 2D flexmirt AN DU P WN P 177 After directing the CLI to the folder that contains the WinFlexMIRT ex ecutable see above the batch mode is called using WinflexMIRT exe b batch csv where the b tells WinFl
50. generate the 2nd 3rd and 4th order interaction terms of the attributes as dimensions of the model If only the second order interaction effects are desired those would be created by specifying InteractionEffects 2 Similar to the InteractionEffects keyword Generate sets up higher or der interactions effects However rather than creating all possible interaction effects Generate creates only those effects specified in the statement For ex ample Generate 3 6 7 4 7 is used to generate the interaction effect of attributes 3 6 and 7 and separately the interaction effect of attributes 4 and 7 There is no limit to the number of interaction effects that may be specified in a Generate statement and the order in which interaction effects are specified does not matter The DM command stands for diagnostic model and is used to set the group containing observed data this is the group to which the higher order DM will be applied The DM command is optional one could in principle fit a model without the DM higher order latent variables The commands Ta and Tc are used in conjunction with the nominal cat egories model supplying contrasts for the scoring function and intercepts respectively The default for both Ta and Tc is a trend contrast matrix key word is Trend but an identity matrix keyword is Identity which allows for equality constraints or a user supplied matrix denoted by in the syntax box may also be
51. is equal to 0 11 1 0 11 0 10 which implies that the school achievement levels are much more homogeneous 52 3 6 Multidimensional Model 4 Correlated Factors Although the previous example was multidimensional estimating two factors it was also dealing with nested hierarchical data Because the fitting of a multidimensional model was mixed in with the hierarchical modeling syntax we provide a more straight forward multidimensional example by fitting a model with four correlated latent factors to a single group The QOL dataset was introduced in Chapter 2 but a previously unmentioned aspect of the scale is it was designed to assess both the overall quality of life as well as the quality of life in 7 more specific subdomains family money health leisure living safety and social with each subdomain containing 4 to 6 items Item 1 is a global quality of life item We will be fitting a four factor model to a subset of the items of the previously introduced QOL scale items from the family money health and leisure factors and allowing the latent factors to correlate Example 3 8 Four Factor GRM Model with Correlated Factors Project Project Title QOL Data Description Items 2 21 with Correlated Factors lt Options gt Mode Calibration Quadrature 21 5 0 SavePRM Yes Processor 4 FactorLoadings Yes lt Groups gt Group1 File QOL DAT Varnames vl v35 Select v2 v21
52. it is being called by another program such as R The ISF is found under the flex MIRT menu bar option of the flex MIRTO GUI Figure 4 2 Accessing the ISF through the GUI flexMIRT TM for Windows l File Edit FlexMIRT Help D a a Run F5 Save and Run Ctrl F5 Progress Ctrl K Once Simulation is selected a pop up will appear where the user enters the information for the simulation study to be conducted As can be seen in the next image the user specifies the desired number of replications the name and location of the existing simulation syntax file which will be responsible for the data creation and the location s and name s of the calibration syntax for the model s that the users wishes to fit to the simulated data Currently there is no way to save the simulated data but in an upcoming update flexMIRT will output the simulated data sets to external files by default The simulated datasets will be output to the file name specified in the simulation syntax file but will have the replication number added to the output file name For instance if we are outputting the data to the file SIM dat via the File SIM dat command in the simulation syntax the 80 individual replications will be saved to files SIM 0 dat SIM 1 dat etc However in the calibration syntax to direct flex MIRT to the datafile we will still use the statement File SIM dat without any mention of the replicatio
53. observations or weights for each response pattern The Dimensions statement is used to specify the total number of dimen sions in the model The default is a single level unidimensional model i e Dimensions 1 Changing the number of dimensions is the first step in setting up multidimensional and multilevel models The Primary statement is used to set the number of primary or general dimensions factors that will be used in a two tier multidimensional model In the bifactor or standard testlet model Primary 1 Several of the syntax statements in the currently discussed box are closely associated with multilevel models To initiate the basic structure for a 2 level model the Between command is used which specifies that number of latent variables at level 2 aka Between When multilevel data are simulated it is necessary to tell flexMIRT how many level 2 units should be generated This is specified by the Nlevel2 command This command is only necessary when 154 simulating multilevel data as the program is able to determine the number of unique level 2 units from the data file provided that the Cluster command is employed Cluster names the variable that contains unique identifying values for the level 2 units e g schools in which students are nested CaseID is used to specify the variable that provides identifiers for the level 1 units The EmpHist statement is used to enable the empirical histogram EH characterization of
54. q lu SD q lu 000 0 0 3 00 2 37 0 41 1 91 0 80 o 00 0 1 6 00 5 47 0 23 1 43 0 80 0 0 0 1 O 2 00 2 47 0 30 1 43 0 80 0 0 0 1 1 11 00 8 25 0 96 0 94 0 81 0 O 1 0 0 1 00 0 85 0 16 1 43 0 80 0 0 10 1 1 00 2 84 1 09 0 94 0 81 11001 56 00 55 17 0 12 0 44 0 82 1 101 21 00 24 96 0 80 0 44 0 82 0 1 1 173 00 177 91 0 41 0 08 0 84 28 1 1 1 O O 11 00 8 59 0 83 0 44 0 82 1 1 1 O 1 61 00 61 23 0 03 0 08 0 84 1 1 1 1 9 28 00 27 71 0 06 0 08 0 84 1 1 1 1 1 298 00 295 80 0 15 0 63 0 86 2 8 The Nominal Categories Model Our last set of basic examples will give demonstrations of using flex MIRT to fit the reparameterized nominal categories model Thissen Cai amp Bock 2010 As discussed in both Thissen and Steinberg 1988 and Thissen et al 2010 models equivalent to the Partial Credit Model Masters 1982 and Muraki s 1992 Generalized Partial Credit Model among others may be obtained as restricted versions of the full rank nominal model by the choice of contrast matrices and constraints 2 8 1 Nominal Model The first data set contains responses to items in a quality of life scale First reported in Lehman 1988 the data set consists of responses from 586 patients to 35 items from the classic Quality of Life QOL Scale for the Chronically Mentally Ill All items have 7 response categories terrible unhappy mostly dissatisfied mixed about equally satisfied and dissatisfied mostly satisfied pleased and de
55. range for higher dimensional more complex models Generally speaking increasing the Proposa18td value will result in lowered acceptance rates while decreasing the value will result in higher acceptance rates Users are directed to Roberts and Rosenthal 2001 for optimal scaling choice of dispersion constants and long term acceptance rates of Metropolis samplers MCSize is the Monte Carlo size for final log likelihood AIC and BIC ap proximations 7 3 The Groups Section The Groups section includes commands for specifying group names number of items models to calibrate as well as more advanced features such as com mands for setting up the structure of hierarchical or multidimensional models as well as empirical histogram priors Example 7 7 Groups General Data Model Descriptors Groups gt 96 GroupName File dat N Varnames vars Select vars Missing 9 Neats vars Code vars Key vars Model vars ThreePL Graded Nominal GPC Prior to any other group commands a label must be assigned to a group and it will become a token that all subsequent command statements may refer to Even if only one group is present it must be given a label The name provided to the group is arbitrary and at the discretion of the user however spaces should not be used when naming groups and the chosen name must be enclosed by percent signs The File statement
56. specifying InteractionEffects 2 Similar to the InteractionEffects keyword Generate sets up higher or der interactions effects However rather than creating all possible interaction effects Generate creates only those effects specified in the statement For ex ample Generate 3 6 7 4 7 is used to generate the interaction effect of attributes 3 6 and 7 and separately the interaction effect of attributes 4 and 7 There is no limit to the number of interaction effects that may be specified in a Generate statement and the order in which interaction effects are specified does not matter The DM command stands for diagnostic model and is used to set the group containing observed data this is the group to which the higher order DM will be applied The DM command is optional one could in principle fit a model without the DM higher order latent variables In the Constraints section the Coeff command allows user s to im pose proportionality restrictions on the item parameters i e MainEffects and Interactions For instance a DINO model may be parameterized as a logis tic model with interactions but the interaction coefficients are 1 0 times the main effect slopes With the Coeff keyword we can change the sign of those specific interaction terms effectively enabling flex MIRT to fit more CDMs than it would be capable of fitting if restricted to only the MainEffect and Interaction keywords The Fix and Free comman
57. the IRT slope estimate 7 indexes the item impacted and k indexes 146 the removed item and se a k is the SE of the item removed slope parameter The resulting n item by n item matrix with empty diagonals is inspected by the user and item pairs with JSI values substantially larger than the other values should be noted as possibly exhibiting LD The FactorLoadings statement may be invoked when in addition to the IRT parameters normal metric factor loadings should be printed as part of the general output file The factor loading values are converted from the IRT parameters as described in Wirth and Edwards 2007 among others The final two statements of this group are tied to the Null independence model needed for incremental fit indices The FitNullModel command when invoked with the keyword Yes tells flexMIRT to fit the Null model and print the resulting basic GOF measures in the general output file When the fit ting of the Null model is requested it becomes possible for the Tucker Lewis Index to be calculated Additionally when the zero factor Null model is re quested the resulting parameter values from fitting this model may be used as starting values for the IRT model of interest by supplying the command StartValFromNull Yes Obviously it is necessary that the Null model is requested via the FitNullModel command for the StartValFromNull com mand to be functional 147 Example 7 5 Options DIF Analysis Comm
58. the latent variable distribution simultaneously estimated with the item parameters The program will calibrate item parameters against the estimated EH prior which is based on the normalized accumulated poste rior densities for all the response patterns at each of the quadrature nodes and flex MIRT will save the EH prior in the prm file Additionally for a concur rent scoring run or any future scoring runs that use the saved parameter file containing the EH values the program will recognize that the group contains EH weights and will use them to compute EAPs or SSCs Empirical his tograms may only be used with single level unidimensional or bifactor testlet models When used with a bifactor type model only the general dimension will be characterized using the EH While category weights are typically taken to be the same that the cat egory value e g a response of 3 is assigned a weight of 3 this does not have to be the case If alternate category weights for items are desired the ItemWeights statement may be used to assign such alternative weights to items For dichotomously scored items a statement of ItemWeights vi v3 0 0 0 5 could be used to assign for variables v1 to v3 a weight of 0 to all 0 responses and a weight of 0 5 to all 1 responses For polytomous items the same format is holds Suppose variables v4 to v8 had 5 possible responses categories weights alternate to the default values of 0 1 2 3 4 could be assigned wi
59. the population distribution e g factor inter correlations means and variances is ignored when scoring individual dimen sions of MIRT models Additionally ML scoring can lead to score information matrices that are not positive definite making the SEs for some estimates undefined flex MIRT offers a multiple dispersed starting value option for use with ML and MAP scoring runs For unidimensional models the default theta start ing value in MAP ML scoring is 0 When Mstarts Yes flex MIRTO will do 12 additional restarts using a range of starting values from 2 75 to 2 75 in step sizes of 0 5 The best solution as determined by the highest log likelihood will be picked from the 13 values Note that using Mstarts Yes will in crease the time needed to complete scoring but provides protection against local modes in 3PL or nominal model scoring When a Simulation run is specified by the Mode command the random number generator must be seeded An integer value must be explicitly pro vided using Rndseed command There is no default value The ReadPRMFile command is used to read parameters values from a text file to lex MIRTG which may be used as starting values for a calibration run as fixed item parameters for a scoring run and as true generating parameter values for a simulation run The proper layout for a parameter file is covered in detail in Chapter 4 Simulation flex MIRT has several options available for computing the s
60. the pro gram On the first start up flexMIRT will open a pane where you must register the software if you opt to cancel at this point you will receive a warning message and flex MIRT 8 will completely shut down You may either supply the flexMIRT account username and password you set up earlier or may copy the license code from your My Account webpage into the appropriate box Fie Edt FlexMIRT Help D gy Ga Q run Progress Usemame Password 171 A 2 Using flexMIRT With the software registered you are now able to conduct analyses The flex MIRT Support page https flexmirt vpgcentral com Support has the User s Manual and numerous example syntax command files with accompany ing datasets to help you become acquainted with the types of analyses that may be conducted and how command files should be structured By select ing New under File or using the New icon a command file containing necessary statements for a basic analysis opens Once you have modified this command file it must be saved prior to being able to submit the analysis to flexMIRT the Run button is disabled until the file has been saved to prevent the example code from being over written To provide a brief tutorial we will focus on the first example found in the folder Example 2 1 Existing files are opened using the Open command under the File option of the engine viewing pane which is the pane opene
61. to have an acceptance rate lower than is desirable As noted earlier lowering the ProposalStd value will generally lead to higher acceptance rates Outside of the Options section no syntax was changed from the example presented in Chapter 3 the same model was fit by both the 90 MH RM and EM estimation routines Output 5 1 Correlated Four Factor Model Output Processing Pane Output MH RM Stage I AR 0 45 0 46 0 39 0 35 0 30 0 27 0 27 0 25 0 24 0 23 0 24 0 23 0 24 0 o O O Q QO N erer Ne O oooo ooo 000000 m w AR MH RM Stage II 1 AR 0 19 0 2 AR 0 19 0 3 AR 0 18 0 4 AR 0 19 0 5 AR 0 20 0 MH RM Stage III 1 AR 0 19 0 00 31 Gam 0 50 W 0 Pt 12 Chg 0 2178 2 AR 0 19 0 00 81 Gam 0 33 W 0 P 0 Chg 0 0777 3 AR 0 20 0 00 63 Gam 0 31 W 0 P 12 Chg 0 0943 As noted previously acceptance rates are printed in the processing pane for each iteration at all three stages of the MH RM estimation Given the complex structure of the model and the IRT model applied to the items the desired goal for our fitted model is to have the reported acceptance rate have an average value between 0 2 and 0 3 Reading across the output flex MIRT is reporting the iteration number the letters AR for acceptance rate followed by the level 1 acceptance and then the level 2 acceptan
62. v18 1 81 3 66 0 14 0 86 0 13 0 85 v19 3 82 4 87 0 02 0 74 0 02 0 76 v20 4 14 5 68 0 02 0 82 0 01 0 84 Note g is found from flexMIRT parameters as exp c 1 ezp c and 1 s is found as ezp c 4 a 1 ezp c a 137 CHAPTER 7 Details of the Syntax This section provides detailed coverage of all the syntax statements and as sociated options that are available in flex MIRT Required statements are listed in bold optional statements are in normal font When keywords are required with statements the possible options are listed after the statement and the default setting is underlined If numeric values are required with a statement such as for a convergence criterion the default value is presented If there is no default value a question mark will denote where values need be entered 7 1 The Project Section Example 7 1 Project section commands Title Description Both of these statements must be present in the lt Project gt section al though the actual content within the quotes may be left blank 7 2 The Options Section The lt 0ptions gt section is where the type of analysis to be conducted is spec ified and where technical details of the analysis may be modified including convergence criteria scoring methods and the level of detail desired in the output As indicated by the bold the Mode statement is the only required command and determines the type of analysis that
63. value for MinExp is set at 1 0 and when expected counts lower than the value set by MinExp are encountered the summed score groups are collapsed towards the center The M2 statement requests that the limited information fit statistics Mo first introduced by Maydeu Olivares and Joe 2005 Full and expanded in Maydeu Olivares Cai and Hernandez 2011 OrdinalSW and Cai and Hansen 2012 Ordinal be printed The None option suppresses M It should be noted that regardless of M2 request the M values will not be printed unless GOF is set to something other than Basic The command HabermanResTbl is invoked when standardized residuals for each response pattern sometimes referred to as Haberman Residuals see Haberman 1979 are desired In addition to the standardized residuals the observed and expected frequencies as well as the associated EAPs and SD for each response pattern are also reported in the requested table There is an additional LD index that flexMIRT can compute with the command JSI The statement JSI Yes requests that the Jackknife Slope Index JSI Edwards amp Cai 2011 a recently introduced LD detection tech nique be calculated Based on the observation that locally dependent items often exhibit inflated slopes the JSI procedure finds for each item pair the change in the slope parameter of item 7 when item k is removed from the scale Specifically a single JSI values is obtained by IS 7 1 where a is
64. values Coarse will be printed when the original threading model is used 144 Another subset of commands available in the Options section deals with the calculation and reporting of various GOF and local dependence LD in dices Example 7 4 Options GOF and LD Indices Commands Options gt GOF Basic Extended Complete CTX2Tbl Yes No SX2Tbl Yes No MinExp 1 0 M2 None Full Ordinal OrdinalSW HabermanResTbl Yes No JSI Yes No FactorLoadings Yes No FitNullModel Yes No Start ValFromNull Yes No The GOF statement controls the extent of GOF indices that are reported Basic Extended and Complete are the three possible levels of GOF report ing Basic the default value prints the 2x log likelihood the AIC BIC values and when appropriate the likelihood ratio G2 and Pearson X full information fit statistics The Extended option results in all the indices in cluded in the Basic reporting as well as the marginal fit X for each item and standardized LD X for each item pair e g Chen amp Thissen 1997 The last keyword Complete prints all indices included in the Extended reporting and also includes per group the item fit index S X Orlando amp Thissen 2000 Additionally under GOF Complete mode an additional statistic S D is printed below the SSC to EAP conversion table This statistic is distributed approximately as a central X variable with degrees of fr
65. variable are zero and all SD and variance estimates are 1 00 because the nominal model was fit which has zero slope and is a reparameterization of the profile probabilities Unstructured Higher Order Latent Space It is also possible to the fit CDMs in flexMIRT without using the higher order latent variable at all This is accomplished by removing all references to the D group in setting up the model We present this syntax below The only changes are the removal of the 4D group from the Groups section as well as omitting the parameter constraints that were applied to that group 130 Example 6 7 Rupp Templin amp Henson Example 9 2 without D group lt Project gt Title Rupp Templin Henson Example 9 2 redone Description LCDM model 7 items 3 attributes lt Options gt Mode Calibration MaxE 20000 Etol le 6 MaxM 50 Mtol le 9 GOF Extended SE REM SavePrm Yes SaveCov Yes SaveSco Yes Score EAP Groups AG File ch9data dat N 10000 Varnames vi v7 truec Select vi v7 Ncats vi v7 2 Model vi v7 Graded 2 Attributes 3 InteractionEffects 2 3 generate 2nd and 3rd order ints Dimensions 7 3 main 3 2nd order 1 3rd order lt Constraints gt Fix G vi v7 MainEffect Free G vi MainEffect 1 Free G v2 MainEffect 2 Free G v3 MainEffect 3 Free G v4 MainEffect 1 Free G v4 MainEffect 2
66. vl v12 2 Model vi vi2 ThreePL BetaPriors vi vi2 1 5 EmpHist Yes The primary point of interest here is the EmpHist Yes statement found as the last command provided for each group With this option invoked flex MIRT will estimate the item parameters and the population theta dis tribution simultaneously The latent variable distribution is estimated as an empirical histogram e g Mislevy 1984 Woods 2007 By empirical his togram we mean the normalized accumulated posterior densities for all re sponse patterns at each quadrature node that is an empirical characterization of the shape of the examinee ability distribution in the population The use of an empirical histogram is in contrast to a typical calibration run in which the latent trait is assumed to be normally distributed This estimation method is currently only available for single level unidimensional and single level bifactor or testlet response models with one general dimension In the Options section the convergence criterion for the E step of the EM algorithm has been increased from the default of 1e 4 to 5e 4 to accommodate the added uncertainty due to the estimation of the empirical histogram prior 3 3 Multiple Group Scoring Only Scoring runs with multiple groups proceed in much the same way as the single group scoring analyses The current scoring example makes uses of the item parameters obtained from the first multiple group exam
67. we read in our data from a named file provide flex MIRT with the variable names and specify the sample size The items were simulated as correct incorrect items so Ncat has been set to 2 and we will fit the 2PLM to all items For the first statement specific to the CDM we tell flex MIRTO that 4 latent attributes will be used Because the C RUM model uses only main effect terms the total number of dimensions is equal to the number of attributes In the D group we inform flex MIRT that the higher order portion of the model we are setting up will be applied to the group with observed data group G using the DM G statement We then name the four attributes via Varnames al a4 If no other information is given regarding categories or a model to be fit lexMIRT by default will assume there are two categories for each of the attributes and will fit the attributes with the 2PLM In the Constraints section we assign items onto attributes according to the Q matrix First we fix all items to 0 and then free items onto the appropriate main effect and interaction terms Items 1 12 only have main effects for a 105 single attribute specifled in the lt Constraints gt section From the Q matrix we know that items 13 15 will load on both attributes 3 and 4 These specifications are incorporated in the statements Free G v7 v9 v13 v15 MainEffect 3 and Free G v10 v15 MainEffect 4 in which items 13 15 appear in the lis
68. 0 00 0 00 0 00 1 v10 1 8 2 2 3 09 1 91 0 00 0 00 1 74 0 00 0 00 0 00 0 00 1 vll 1 8 2 2 2 57 1 42 0 00 0 00 0 71 0 00 0 00 0 00 0 00 1 v12 1 8 2 2 1 88 1 40 0 00 0 00 0 72 0 00 0 00 0 00 0 00 1 v13 1 8 2 2 2 80 1 57 0 00 0 00 0 13 0 00 0 00 0 00 0 00 1 v14 1 8 2 2 5 16 4 02 0 00 0 00 3 80 0 00 0 00 0 00 0 00 1 v15 1 8 2 2 2 09 249 0 00 0 00 066 0 00 0 00 0 00 0 00 1 v34 1 8 2 2 2 95 1 67 0 00 0 00 0 00 0 00 0 00 0 00 1 30 1 v35 1 8 2 2 3 22 1 40 0 00 0 00 0 00 0 00 0 00 0 00 0 44 Type Label Grp Fac Prior pa ee Ls 011 021 022 031 032 033 0 Groupl 1 8 0 0 00 0 00 1 00 0 00 1 00 0 00 0 00 1 00 Following the parameter file format the first column is for entering whether the values to follow on the row refer to an item or a group with 1 denoting items and 0 denoting groups We will now follow the first line across discussing the columns in turn The next column supplies a variable name for the line and the following column specifies to which group the item belongs In this case there is only one group so the entire column consists of 1s In Column 4 we specify that there are 8 factors in the model and Columns 5 and 6 respectively tell flex MIRT that the IRT model is the graded model type 73 2 with two possible response categories The combination of the these three values informs flex MIRT to expect 9 item parameters one intercept and 8 slopes Scanning down the columns of slope parameters it becomes clear that the a column
69. 0 96814 1 29641 1 409536 1 v9 3 1 1 2 2 79306 0 730972 1 172468 1 v10 3 1 1 2 1 65629 0 14263 1 102673 Type Label Grp of Factors Prior pa 011 0 Groupl 1 1 0 0 1 0 Group2 2 1 0 0 2 1 0 Group3 3 1 0 0 2 1 5 Note that for item v1 in group 3 a different model the 2PL rather than the 3PL is used This is why the item parameter columns logit g or c and c or a are labeled as they are the columns have different meanings depending on the model fit to an item 187 APPENDIX E Licensing Information Purchasing a license for lexMIRTO is easy If you intend to use flex MIRT for academic purposes simply register at flexmirt vpgcentral com Account LogOn and once logged in follow the instructions on screen to download a trial version At any point in your trial you can log in to your account and purchase a flex MIRT license Your yearly subscription will begin the moment you purchase the program Newer versions of flexMIRT will be updated automatically on any computer with an internet connection and on which a valid copy of the program has been installed Each single user license is good for three installs of lex MIRT so you can easily work with flexMIRT on your academic home and laptop computers at no ad ditional cost If additional installations are required pricing information may be obtained by contacting sales VP Gcentral com Your flex MIRT license does not automatically renew You will be ask
70. 0 c8 0 9 YON ToT T A 00 0 00 T Aqtquepl 19 0 08 0 c grauep l T eudpe q a s Z eudte 4d a s I eudre ta saqseaaqoo e s e did TeqeT u 4I Idnozo9 I dnozy 103 su 4I TOF saserquoo uorqoun4 SUTIODS pue s doTS T DOW TeutTuon SIojourereq Wy mdmo c I9polN IeurttoN 8T 7 Mamo 34 Although several new command statements were introduced in this ex ample only two result in additions to the output file Both the M2 and FitNullModel statements provide additional GOF information that is printed near the bottom of the output file with the other overall GOF index values The fitted model s 2xlog likelihood AIC and BIC are printed as before but additional fit statistics for the zero factor null model are also included under the corresponding fitted model section The estimation of the inde pendence model permits the calculation of the non normed fit index NNFI aka Tucker Lewis Index The usual full information fit statistics are printed but limited information statistics based on Ma for the fitted and zero factor model are also given Taken together the fit indice values indicate that the nominal model provides excellent fit to this data Output 2 19 Nominal Model 2 Output GOF values Statistics based on the loglikelihood of the fitted model 2loglikelihood 2766 69 Akaike Information Criterion AIC 2184 69 Bayesian Information Criterion BIC 2824 14 Statistics based on the loglikelihood of the zero
71. 000 ercOI c 000 9 T608 000t97022 g G 9 I GA I 000 900000 0 000 00000 0 000 00000 0 000 00000 0 000 00000 0 000 c99II9c 6 000 9z3863T 3_ C6 C 9 I I I 9D SD vo p en To 2 syeg jo epo N sIiopeg jo Z ZdiS eqeTq adAL dnory T s1032 9 YIM TZ og Wd popoqey 2 C LL 00 T 000 0 I I timo 0 Tlo TI IO S1032e4 Jo 419 PqeT adAL 000 9EZEDH T 000 9T8ESE Z TOO0 9L88 L p 000 9 6 TZ T 000cT9PLLYE Y 9S 4 I I GA I 000 8S6Z2 T 000 992vI9 C I00 9vP29v 000 cer8ST9 I 000 T908TITT S 4 I I yA I 000 90P28 T 000 90I86 C I00 98 6 48 C 000 c 961609 I O00 9pPTOPOD lt 4 I I I 000 9Grp0GS I 000 c9cSVI6 C I00 946c9 v 000 TVE6L T 000 F9ESOSETD q 4 I I GA I 00009728689 I _000 606EE E TOO 92380T p I00 96IIST 6 000T9S06vC S G I I IA I D vo 5 c2 To sep Jo Z PPO S1032e4 Jo Z diy joqey addy dnorD T 109984 T UM WAD 919 wd poroqeT 9 G ALL 186 Table D 8 Labeled prm File 3PL with 1 Factor 3 groups Type Label Grp of Factors Model of Cats logit g orc cora a 1 vl 1 1 1 2 1 33972 0 68782 1 301362 1 v2 1 1 1 2 0 96814 1 29641 1 409536 1 v9 1 1 1 2 2 79306 0 730972 1 172468 1 v10 1 1 1 2 1 65629 0 14263 1 102673 1 v1 2 1 1 2 1 33972 0 68782 1 301362 1 v2 2 1 1 2 0 96814 1 29641 1 409536 1 v9 2 1 1 2 2 79306 0 144738 1 172468 1 v10 2 1 1 2 1 65629 0 14263 1 102673 1 v1 3 1 2 2 0 68782 1 301362 1 v2 3 1 1 2
72. 000 individuals these 2000 individuals are nested within 100 higher order units 20 per unit This type of structure could be seen if 2000 students from 100 different schools took a brief assessment in which items were graded as right wrong Example 5 3 MHRM Two level MIRT Model Project Title Fit Two level MIRT Model to Simulated Data Description 6 Items 100 L2 units 20 respondents within each lt Options gt Mode Calibration Rndseed 10 Algorithm MHRM ProposalStd 1 0 ProposalStd2 1 0 Processors 2 MCsize 10000 Groups Lar File simL2 dat Varnames vi v6 12id Select vi v6 Cluster 12id Dimensions 4 Between 2 N 2000 Ncats vi v6 2 Model vi v6 Graded 2 Constraints Fix vi v6 Slope fix all slopes to begin with 93 Free vi v3 Slope 1 level 2 factor Free v4 v6 Slope 2 level 2 factor Free vi v3 Slope 3 level 1 factor Free v4 v6 Slope 4 level 1 factor Ne N Equal Gr vi v3 Slope 1 Gr vi v3 Slope 3 cross level equality Equal Gr v4 v6 Slope 2 Gr v4 v6 Slope 4 Free Cov 1 1 Free Cov 2 2 Free Cov 1 2 Free Cov 3 4 As before we have told flexMIRT to use the MH RM algorithm and provided a random number seed In this example we are using both the ProposalStd and ProposalStd2 keywords because we now have two levels to our model The adjustments to the ProposalStd will aff
73. 04 Maximum number of M step iterations 100 Convergence criterion for iterative M steps 1 00e 007 Number of rectangular quadrature points 49 Minimum Maximum quadrature points 6 00 6 00 Standard error computation algorithm Cross product approximation Miscellaneous Control Values Z tolerance max abs logit value 50 00 Number of free parameters 24 Number of cycles completed 27 Number of processor cores used 1 Maximum parameter change P 0 000083709 13 Processing times in seconds E step computations 0 23 M step computations 0 02 Standard error computations 0 22 Goodness of fit statistics 0 00 Total 0 47 Output Files Text results and control parameters 2PLM example irt txt Convergence and Numerical Stability flexMIRT R engine status Normal termination First order test Convergence criteria satisfied Condition number of information matrix 46 0221 Second order test Solution is a possible local maximum Following the flex MIRT version and copyright information the output begins by printing the title and description provided in the Project section of the syntax file A broad summary of the data and model is provided on the next 4 lines listing the missing value code number of items sample size and total number of dimensions In the next section the number of categories and the IRT models are listed for each item The various control values are listed e g convergence criteria the m
74. 257 270 Langer M M 2008 A reexamination of Lord s Wald test for differen tial item functioning using item response theory and modern error esti mation Unpublished doctoral dissertation Department of Psychology University of North Carolina at Chapel Hill Lehman A F 1988 A quality of life interview for the chronically mentally ill Evaluation and Program Planning 11 51 62 Li Z amp Cai L 2012 July Summed score likelihood based indices for testing latent variable distribution fit in item response theory Paper presented at the annual International Meeting of the Psychometric Society Lincoln NE Retrieved from http www cse ucla edu downloads files SD2 final 4 pdf Louis T A 1982 Finding the observed information matrix when using the EM algoritm Journal of the Royal Statistical Society Series B 44 226 233 Masters G N 1982 A Rasch model for partial credit scoring Psychome trika 47 149 174 Maydeu Olivares A Cai L amp Hernandez A 2011 Comparing the fit of IRT and factor analysis models Structural Equation Modeling 18 333 356 Maydeu Olivares A amp Joe H 2005 Limited and full information estima tion and testing in 2 contingency tables A unified framework Journal 191 of the American Statistical Association 100 1009 1020 Metropolis N Rosenbluth A W Rosenbluth M N Teller A H amp Teller E 1953 Equations of state space calculations by fa
75. 32 135 138 Title 4 5 8 12 16 19 21 24 30 33 41 44 45 54 60 62 64 69 71 73 75 77 90 94 97 105 112 116 122 125 132 135 138 ProposalStd see lt Options gt ProposalStd ProposalStd2 see lt Options gt ProposalStd2 199 Quadrature see lt Options gt Quadrature Quadrature Points 8 ReadPRMFile see lt Options gt ReadPRMFile Response Pattern Standardized Resid uals 28 29 Restricted Latent Class Models see Cognitive Diagnostic Models Rndseed see lt Options gt Rndseed SaveCov see lt Options gt SaveCov SaveDbg see lt Options gt SaveDbg SaveInf see Options SaveInf SavePrm see Options SavePrm SaveSco see lt Options gt SaveSco Score see lt Options gt Score Scoring 142 Expected a posteriori EAP 16 109 129 130 142 Maximum a posteriori MAP 20 21 43 44 142 Maximum Likelihood ML 142 Sum Score Conversion to EAP SSC 16 27 44 45 112 114 115 142 152 Supplying Item Parameters 16 18 20 43 45 SE see lt Options gt SE Select see Groups Select SEMtol see Options SEMtol Simulation 61 2PL 72 79 3PL 68 72 74 76 Graded Response Model 61 65 Item Parameters 63 64 69 75 76 79 Parameter File 64 76 Value statements 61 62 64 Multidimensional IRT Bifactor Model 72 74 77 79 Multi level Model 77 79 Multiple Groups 62 74 79 SlopeThreshold see lt Options gt SlopeThreshold SmartSEM see lt O
76. 4 36 6 00 15 v15 0 00 1 32 1 09 2 25 3 22 4 16 6 00 16 v16 0 00 1 52 1 71 2 65 3 96 4 99 6 00 An examination of the output excerpt presented shows that a number of scoring function values exhibit reversal of categories from the presumed order such as in Items 13 and 15 or category boundaries that are very close together such as categories 2 and 3 in Items 16 As Preston Reise Cai and Hays 2011 explained when the estimated scoring function values deviates from the presumed order of 0 1 2 6 the nominal model provides additional category specific information that other ordinal polytomous IRT models e g partial credit or graded do not provide As noted earlier the items were scored on a 7 point scale indicating the extent to which the respondent is satisfied with the statement of the item where O terrible 1 unhappy 2 mostly dis satisfied 3 mixed about equally satisfied and dissatisfied 4 mostly satisfied 5 pleased and 6 delighted There may be some uncertainty as to how people interpret the order of the middle categories 3 7 2 Bifactor Model Using the 2PL Due to the undesirable properties when analyzing the QOL data with all 7 categories we will again demonstrate how flexMIRT may be used to recode data We will fit the same bifactor factor pattern to the QOL data but rather than utilizing all 7 categories we will collapse the data into dichotomous responses and fit the 2PLM Example 3 10 Bifactor Structure 2P
77. 73 75 77 87 90 94 141 142 149 SaveCov 105 112 116 122 125 132 135 139 SaveDbg 139 SaveInf 41 42 139 143 SavePrm 12 33 41 54 60 90 105 112 116 122 125 132 135 139 SaveSco 16 19 21 24 26 41 105 112 116 122 125 132 135 139 Score 16 141 142 EAP 16 41 105 112 116 122 125 132 135 142 MAP 16 21 43 44 142 ML 16 19 20 142 SSC 16 24 26 45 112 114 115 142 SE 141 143 FDM 143 Fisher 143 Mstep 143 REM 105 112 116 122 125 132 135 136 143 Sandwich 143 SEM 33 143 Xpd 41 42 143 SEMtol 141 144 SlopeThreshold 22 139 140 SmartSEM 143 SStol 141 144 Stage1 87 88 97 149 150 Stage2 87 88 97 149 150 Stage3 87 88 149 150 Stage4 87 88 149 150 StartValFromNull 145 147 SX2Tb1 24 26 33 145 146 TechOut 139 Thinning 87 149 WindowSize 87 88 149 150 OrthDIFcontrasts see lt Dptions gt OrthDIFcontrasts Perturbation see lt Dptions gt Perturbation PISA data 49 Primary see lt Groups gt Primary Prior see lt Constraints gt Prior Prior Distribution 5 12 13 41 42 46 160 PriorInf see lt Options gt PriorInf Processors see lt Dptions gt Processors Progress see lt Options gt Progress Project 3 5 8 12 16 19 21 24 30 33 41 44 45 138 Description 4 5 8 12 16 19 21 24 30 33 41 44 45 54 60 62 64 69 71 73 75 77 90 94 97 105 112 116 122 125 1
78. 8 0 097 0 0000093 0 001025 0 000006 0 001256 0 000147 0 000085 0 028217 0 000035 0 000017 0 000168 0 009475 0 143 0 0001508 0 001332 0 000013 0 002420 0 000214 0 000173 0 034470 0 000084 0 000041 0 000363 0 020508 0 129 0 0805989 0 182430 0 027666 0 163028 0 002884 0 002254 0 009474 0 005347 0 003948 0 001326 0 016527 0 087 0 0356330 0 091595 0 013076 0 099016 0 000808 0 000623 0 002789 0 001972 0 001464 0 000291 0 007550 0 060 0 0078999 0 032539 0 004564 0 050540 0 000175 0 000147 0 000739 0 000572 0 000493 0 000057 0 003578 For this example each summed score is listed with four EAP values cor responding to the 4 attributes specified in the model 4 SD values the ex 114 pected proportion of respondents who will obtain the listed summed score listed in the column labeled p and the error variance covariance matrix associated with that summed score For example given a summed score of 0 the probability that the individual has mastered attribute 1 is 0 001 the probability that they have mastered attribute 3 is 0 026 and so on For in dividuals with a summed score of 15 those probabilities increase to 0 966 and 0 999 respectively Rather than simply providing the most likely at tribute profile associated with a summed score flex MIRT prints the individ ual attribute probabilities so researchers have sufficient information to apply mastery indifference non mastery cut off values appropriate for the specific
79. 9980 L1 v8V O 822900 0 99v 00 00S0ST O OT06ZO v618 0 v61T00 0 89v000 vILETZ O 602200 6229b 0 v 1000 0 O v000 878 90 0 T8T9T0 800c C 0 vSST00 0 6 9000 T8164T 0 166v00 L6cecv O O 6E00 0 661000 amp LLL07 0 367200 Oc8SSv O 82900 0 89v 00 6896ST 0 LE06Z0 TI966 0 0680600 090100 9 9S81 0 vezseo 1 TO60tv O 699600 0 969700 e1e vec o 998TS0 11 L71870 MU 26LT00 0 MU OTTv00 0 v6S O NU 80280 0 MU 660v90 0 cCv86v O NU 86 T00 0 NU 880700 0 8PPLEE O MU Tv 9v00 0 MU 66860 0 191681 0 NU c6v100 0 NU 118100 0 02Sv8v O 0 29 000 0 0 TS9Z00 0 SCTIVET O 0 c9S010 0 MU 181860 0 TOZT6T O me 69T010 0 NU 929220 0 80T0 0 MU 806vv0 0 MU vCELEO O0 S690v 0 0 6 9970 0 NU 9Cv6v0 0 61102 0 6Z8960 0 TET6Tv O 929960 0 SEV9LZT O 6LEL80 0 608780 0 8 6 v O 048SC 0 808L6E 0 TvO9TEE O 92000 0 97411070 T84800 0 CS9 v0 0O 92000 0 ZISTTO O z68000 0 1v8600 0 v6 00 0 OETTOO O 1600000 9SE T00 0 T8 000 0 9 v600 0 66 v00 0 L0 LT0 0 819100 0 v6tTZO O0 671300 0 1v06c0 0 Sv1600 0 046 11 0 SS8CvVl O I 6848 0 0 vOVevo O 1 96 0 9v1600 0 T48 1T 0 9ELTHT O I Tv618v0 0 SST CO 0 00 290 0 O 000 0 808pez 0 L0 S0 0 Z8E00 0 O066LT0 0 810S0 0
80. AVED ON YOUR COMPUTER source C Users laptop Documents flexMIRT flexMIRT plotting flexmirt R For plotting ICCs and TCCs use the function call flexmirt icc parameter file name PROVIDE THE FULL PATH TO THE prm FILE TO BE PLOTTED 1PL example flexmirt icc C Users laptop Documents flexMIRT flexMIRT plotting SM3 prm txt 2PL example flexmirt icc C Users laptop Documents flexMIRT flexMIRT plotting Preschool 2PL prm txt 3PL example flexmirt icc C Users laptop Documents flexMIRT flexMIRT plotting Preschool 3PL prm txt Nominal Model Example flexmirt icc C Users laptop Documents flexMIRT flexMIRT plotting PreschoolNum Nom prm txt Graded Example flexmirt icc C Users laptop Documents flexMIRT flexMIRT plotting LiberalConservative prm txt GPC Example flexmirt icc C Users laptop Documents flexMIRT flexMIRT plotting LiberalConservative GPC prm txt For plotting IIFs AND TIFs use the function call flexmirt inf inf file name NOTE THE INF PLOTTING FUNCTION PULLS INFORMATION FROM THE flexmirt CALIBRATION FILE THAT CREATED THE inf FILE IT IS ASSUMED THAT THE inf FILE and the flexmirt FILE ARE IN THE SAME LOCATION information function example flexmirt inf C Users laptop Documents flexMIRT flexMIRT plotting LiberalConservative inf txt information function example multiple grp flexmirt inf C Users laptop Documents flexMIRT flexMIRT plotting fit2 inf txt As may be seen
81. Effect 3 Free G v10 v12 MainEffect 4 items 13 15 fit with DINO model that depend on int of attribs 3 4 Free G v13 vi5 MainEffect 3 Free G v13 v1i5 MainEffect 4 Free G v13 v15 Interaction 3 4 Equal G v13 MainEffect 3 G v13 MainEffect 4 G v13 Interaction 3 4 Coeff G v13 Interaction 3 4 1 Equal G v14 MainEffect 3 G v14 MainEffect 4 G v14 Interaction 3 4 Coeff G v14 Interaction 3 4 1 Equal G v15 MainEffect 3 G v15 MainEffect 4 G v15 Interaction 3 4 Coeff G v15 Interaction 3 4 1 Again the maximum number of E step and M steps have been increased from the default the E tolerance and M tolerance values have been decreased and the SE calculation method has been set to REM in the Options section As noted in the output from the previous example there were several latent dimensions that had no items loading onto them Here rather than using the general InteractionEffects keyword we have used the more targeted Generate to construct only those interaction terms which will be used in the model As seen in the Q matrix and discussed earlier only the interaction of attributes 3 and 4 is needed and is specified by the Generate 3 4 statement If additional interactions were needed they may be added by placing a comma after the last interaction term and listing the other desired interactions For example if the 3 way interaction of attri
82. Effect 3 Free G v10 v12 MainEffect 4 item 13 15 fit with DINA model that depends on int of attribs 3 4 Free G vi3 v15 Interaction 3 4 All syntax in the lt Options gt section remains unchanged from the previous example with the exception of requesting sum score to EAP conversion tables via Score SSC In the G group we have declared the data file variable names sample size number of categories and IRT model to be fit to each item as in the previous example We again specify that there are 4 attributes with main effects but because we are using the DINA model we will also need to construct dimensions that will represent the interaction effects This is accomplished via InteractionEffects 2 which is requesting that all second order interaction terms i e attribute 1 with attribute 2 attribute 1 with attribute 3 etc be created for the model Because we have requested all second order interactions of which there are 6 possible with the 4 attributes the total number of dimensions for the final model is 10 4 main effect terms and 6 second order interactions Syntax in the D group remains unchanged from the previous example In the lt Constraints gt section as with the previous example we initially fix all items to zero and then free items onto the appropriate attributes Unlike the previous example which used only main effects we are also using the parameter keyword Interaction to free items 13 15 o
83. Epsilon 87 88 149 150 Etol 30 33 37 38 43 57 58 105 112 116 122 125 132 135 136 141 144 FactorLoadings 24 25 54 55 90 92 145 147 FisherInf 41 42 141 143 144 FitNullModel 33 35 41 145 147 GOF 145 Basic 11 145 Complete 24 26 145 Extended 12 15 21 23 33 41 125 132 145 HabermanResTbl 24 25 28 29 33 145 146 Imputations 87 149 InitGain 87 88 149 150 J81 25 28 140 198 JSI 145 LDX2 24 logDetInf 141 144 M2 145 146 Full 33 35 41 MaxE 105 112 116 122 125 132 135 136 141 144 MaxM 105 112 116 122 125 132 135 136 141 144 MaxMLscore 142 MCsize 87 89 94 149 151 MinExp 145 146 MinMLscore 142 Mode 4 5 138 139 141 Calibration 4 5 12 21 24 30 33 37 39 41 54 57 60 87 90 94 97 105 112 116 122 125 132 135 149 Scoring 4 16 19 43 45 138 Simulation 4 62 64 69 71 73 75 77 138 142 Mstarts 142 Mtol 30 33 37 38 57 58 105 112 116 122 125 132 135 136 141 144 NewThreadModel 135 136 141 144 NormalMetric3PL 139 140 NumDec 139 OrthDIFcontrasts 148 Perturbation 143 PriorInf 141 144 Processors 37 38 54 57 60 90 94 135 141 144 Progress 41 42 69 138 ProposalStd 87 89 91 94 149 151 ProposalStd2 87 89 94 149 151 Quadrature 38 54 55 57 58 60 90 94 122 141 144 ReadPRMFile 16 19 43 45 64 69 71 73 75 77 141 142 Rndseed 62 64 69 71
84. J Long Eds Testing structural equation models pp 136 162 Newbury Park CA Sage Cacioppo J T Petty R E amp Kao C F 1984 The efficient assessment of need for cognition Journal of Personality Assessment 48 306 307 Cai L 2008 SEM of another flavour Two new applications of the supple mented EM algorithm British Journal of Mathematical and Statistical Psychology 61 309 329 Cai L 2010a A two tier full information item factor analysis model with applications Psychometrika 75 581 612 Cai L 2010b High dimensional exploratory item factor analysis by a Metropolis Hastings Robbins Monro algorithm Psychometrika 75 33 57 Cai L 2010c Metropolis Hastings Robbins Monro algorithm for confirma tory item factor analysis Journal of Educational and Behavioral Statis tics 85 307 335 Cai L amp Hansen M 2012 Limited information goodness of fit testing of hierarchical item factor models British Journal of Mathematical and Statistical Psychology 66 245 276 189 Cai L Yang J S amp Hansen M 2011 Generalized full information item bifactor analysis Psychological Methods 16 221 248 Celeux G Chauveau D amp Diebolt J 1995 On stochastic versions of the EM algorithm Tech Rep No 2514 The French National Institute for Research in Computer Science and Control Celeux G amp Diebolt J 1991 A stochastic approximation type EM al gorithm for the m
85. L Project Title QOL Data Description 35 Items Bifactor Showing Recoding lt Options gt Mode Calibration Quadrature 21 5 0 SavePRM Yes Processor 4 59 lt Groups gt Group1 File QOL DAT Varnames vi v35 N 586 Dimensions 8 Primary 1 Ncats vi v35 7 Code vl v35 0 1 2 8 4 556 0 0 0 1 1 1 1 Model vi v35 Graded 2 lt Constraints gt Fix vi v35 Slope Free vi v35 Slope 1 Free v2 v5 Slope 2 Free v6 v9 Slope 3 Free v10 v15 Slope 4 Free v16 v21 Slope 5 Free v22 v26 Slope 6 Free v27 v31 Slope 7 Free v32 v35 Slope 8 The Code command is used to collapse the 7 original categories into 2 A point of note here is that the Ncats value provided refers to the number of categories in the raw data file not the number of categories that will be used for modeling which is specified with the Model statement An examination of the Code statement shows that original responses of 0 1 and 2 will be recoded into Os and raw responses of 3 6 will become 1s after the recoding by the program The bifactor structure remains unchanged from the previous example as may be seen in the in the lt Constraints gt section 60 CHAPTER 4 Simulation Simulation studies are common in the IRT literature Compared to other commercially available IRT software programs e g Bilog IRTPro Multilog and Parscale flex MIRT is unique
86. LT 9T 0 TO 90 TO vo cr or PT 03 8T Go 0 80 TO LO PT er OT VG 61 80 90 TIT co TT OT 8T LT ec GT 6 Cv TS Oc ge LT LE GT LT 93 0c ET ZE 19 c 8 cv 9p 96 ET VG 66 9T OT ev 18 VG 29 S PT VG TT T 0 9 8T O T 9 T 0 S cr 80 ev S8 aA S8 T cr Oc 60 S O u N ou O O O O O ooocococococo 00 m OOo O O O O O On On One O O O O O O O O OO P O O O O O O O O O O OO OOO OOo O O O O POE PS O On On One O O O O O O O O O O OI O O O O O O O O OPO OP OO O cc c c c c 00 00 0 0 9 9 0 0 GS OS S 6 6 O O O O O O O O OPOP OO O O O O O O O O On On On One O O O O O O O O OPS O OI O O O O O O ooococococoo O O O O O O O O O OPOP NO O 8 Z 9 S Y 1 o o I SP o I 00 o I N 4 I O o CN e I N I g oO p IM uoraedqr e dZ dnozo T 109984 T suejt ZI e duexe WIdZ suom unq uorjeuriogju yndyng uoneiqipe Tdz e c Mamo 10 The item and test information functions values for which are presented in Output 2 3 are the focus of the next section of the output file The title and description are printed once again followed by the label for the information table As may be seen the values of theta used in the table range from 2 8 to 2 8 increasing in steps of 0 4 The information function va
87. OSEDO 879870 SSvI00 NN ZTTT80 0 8 8000 0 9LZTEO 0 S 090 0 9999Z0 0 8E8ZZT O eo8v8c O vPvv9vo O 8r0S 8 0 696016 0 9v0S 6 0 cOvETTI O T 000000 T 000000 T 000000 0 9 19900 0 O 9Z0 0 DETLEV O 9cecec O 08 4T10 0 O 0ZEO O 896LZT 680120 NN 9880 Z 0 606820 0 8TC T0 0 882v210 0 v81660 0 91498T 0 S0s08v O 69v LZ 0 LOTZEV O SSC8 9 0 vIvT80 0 9STS O T 000000 T 000000 T 000000 0 790000 0 S0 890 0 097000 0 6Z0Z80 0 808Z00 0 6ZE809 0 L7 600 899877 NN 609ZT0 0 171900 0 L6 T00 0 T8LLTZ 0 vVlvEO 0 608ZT 0 v8TTT O 0219997 0 0064S 0 699Z10 0 967619 0 8v80ST 0 T 000000 T 000000 T 000000 0 8 Svv608 0 v9EIST O cvT1610 0 9TTTOO O GZGLTO O TOETOO O 001000 200000 NN Z TOET O 0T0Z00 0 09100 0 0S6610 0 642000 0 S488T10 0 Sv109 0 EZTIT O 68c9 T O cTC9v8 O 9 9676 0 490186 0 T 000000 T 000000 T 000000 0 8 TeT6E8 0 O0v6020 0 vv8610 0 vST000 0 T808TT 0 OLTTOO O 29000 900000 NN ICO O 00 000 0 S6v100 0 0ScOCO 0 T08T00 0 1vSS0T O0 T9S vl O O cYT O 088vCE O 0 2226 0 cC 626 0 020088 0 I 000000 T 000000 T 000000 0 966800 0 9 4200 0 v8TOTV O 6LZE90 0 T69990 0O LEVETO O LO6STY 0L96Z0 NN 242 680 0 8 200 0 898800 0 891780 0 6 ST 0 0 S8 6vc O 096863 0 G6928Z 0 ve2667 0 8410060 090160 0 S6TS v O T 000000 T 000000 T 000000 0 T 0000000 v6000 0 9866000 0 c 8S0v0 0 4Tc000 0 9102v0 0 vo8seo 0T6828 NN S 6T 0 0 TvOSTO0 0 09
88. S and Irish students and within each country students level 1 units are nested within schools level 2 units Because the full command file PISA00mathL2 flexmirt is quite long only commands relevant to multilevel aspect of the analysis will be highlighted 49 Example 3 7 Multiple Group Multilevel Model Excerpts lt Groups gt AUSA File PISAOOMath US dat Missing 9 Varnames SchID StdID ViewRoomQ1 BricksQ1 CarpenterQ01 PipelinesQ1 Select ViewRoomQ1 PipelinesQ1 CaseID StdID Cluster SchID Dimensions 2 Between 1 N 2115 Ireland File PISAO0Math IE dat CaseID StdID Cluster SchID Dimensions 2 Between 1 N 2125 lt Constraints gt Free USA Cov 1 1 Free Ireland Mean 1 Free Ireland Cov 1 1 The data files are specified as usual within each group We have also set a miss ing data code using Missing 9 so all 9s in the data set will be interpreted as missing values rather than the default missing value of 9 The variables are named in the typical fashion the first two variables are of note because 50 they indicate student ID numbers and school ID numbers respectively and will not be selected for IRT analysis In general a multilevel analysis has two kinds of latent dimensions Between and Within The Between dimensions will always precede the Within dimensions when flex MIRT is assigning di mensions For this example there are onl
89. Specialized Data Model Descriptors Constraints Across Groups Specifying a Prior Distribution and Freely Estimating Theta 4 2 2222 R plotting code ee Linde da ow kee S EX EES vi List of Output 2 1 2 2 2 3 2 4 2 5 2 6 2 7 2 8 2 9 2 10 2 11 2 12 2 15 2 14 2 15 2 16 2 17 2 18 2 19 3 1 3 2 3 3 3 4 4 1 4 2 4 3 4 4 2PL Calibration Output Summary and Controls 6 2PL Calibration Output Item and Group Parameters 9 2PL Calibration Output Information Functions 10 2PLM Calibration Output Goodness of Fit Indices 11 Single Group 3PL Calibration Output Item Parameters 14 Single Group 3PL Calibration Output Extended GOF Table 15 3PL EAP Scoring Output 2 53 xonesE Rovere hs ok E Wes 17 3PL ML Scoring Output 2 2 3 d deed unb 20 Graded Model Calibration Slopes and Intercepts 22 Graded Model Calibration Slopes and Thresholds 22 Graded Model Calibration Extended GOF Table 23 1PL Grouped Data Output Item Parameters and Factor Load ino PEE PLE TTE 26 IPL Grouped Data Output Item Fit Diagnostics 2 1PL Grouped Data Output Summed Score to Scale Score Con VETSION 2 03 e URL to ogi git ve co G2 Ie fey a IRI ERE STE s 27 IPL Grouped Data Output JSI matrix 28 1PL Grouped Data Output Haberman Residual Table Excerpt 28 Nominal Model Output Item Parameters 3l Nominal Model 2 Output
90. TARITITSSPTI SSOID pue sejnqriaay LUI orasouSetq 970 ets 8 zzo vz 46 A 4940 gts T6 e 00 0 9 Sa 00 0 0 6L T TC SA O0c O 98 T 61 TO so z 8T v gt 000 000 Tc O voz 9r 00 0 TA ue 00 0 90 0 18 T TA ers ze ers 1o leae1 weat 5 1 dnoz5 103 sue31 dc sej3nqri33e g sweat Tepou WAIT Z 6 rdurexa uosueg urldue ddny sIojourereq WY 3nd3n uosusy ur duro ddny 9 9 mdmo 127 The output section labeled 2PL Items for Group 1 G presents the item parameters for the 7 items with estimated versus fixed at zero loadings mir roring the Q matrix specifications with the factors labeled al a3 representing main effects for attributes 1 3 and a4 a7 representing the 2nd and 3rd order interactions For those wishing to compare flex MIRTO to MPLUS the program used to estimate the book example this item parameter section cor responds with the New Additional Parameters section of MPLUS output and the point estimates agree with those given in Figure 9 15 pg 218 of Rupp et al 2010 The next section of interest is labeled Diagnostic IRT Attributes and Cross classification Probabilities for Group 1 G which reports the estimated proportions of respondents in each attribute profile This section in flexMIRT corresponds to the MPLUS output section labeled FINAL CLASS COUNTS AND PROPORTIONS FOR THE LATENT CLASSES and the reported values match those given in Figure 9 13 of Rupp et al 2010
91. Varnames v1 v12 Dimensions 6 Between 1 1000 Nlevel2 100 Constraints Skipping down to the Groups section and focusing on the first group s subsection we have provided the data file name variable labels and the total number of dimensions of the generating model with the Dimensions state TT ment The command Between 1 indicates that the model has one level 2 between latent variable and 5 level 1 within latent variables The Nlevel2 command is required when simulating multilevel data and determines the num ber of level 2 units e g schools in which students are nested In this case the number of level 2 units is equal to 100 In conjunction with the total number of level 1 cases specified in the N 1000 line one can deduce that the number of level 1 units within each level 2 unit is equal to 10 lex MIRT will distribute the total sample size as evenly as possible across the level 2 units If the sample size value is not a multiple of the number of level 2 units the remainder observations will be added to the final level 2 unit Note that while the specifications for the second group are the same as those in the first group in this example it is by no means a requirement As mentioned earlier the number of items type of IRT model and the latent dimensionality can be different across groups Output 4 4 Simulation Control Output for Multilevel Bifactor Model Two level Bifactor Model Two Gr
92. a The items which are affected by the constraint are then specified in parentheses followed by another comma and then the keyword for the parameter that is affected e g Slope Guessing etc For a unidimensional model in Groupt the slope parameters for v1 through v5 can be set equal using the following syntax Equal Groupi vi v5 Slope If there is only one group the group name can be omitted Equal vi v5 Slope The following syntax fixes the second slope of v1 v2 to 0 the default value unless modified by Value statement Fix v1 v2 Slope 2 The following syntax fixes the second factor mean of Group2 to 0 the default value unless modified by Value statement Fix Group2 Mean 2 158 The following syntax fixes the second factor variance of Group2 to 1 the default value unless modified by Value statement Fix Group2 Cov 2 2 The following syntax fixes the covariance between factors 2 and 3 in Group2 to 0 the default value unless modified by Value statement Fix Group2 Cov 3 2 The following syntax fixes the first scoring function contrast for v1 to 1 the default value unless modified by Value statement Fix v1 ScoringFn 1 The following syntax fixes the second scoring function contrast for v1 to 0 the default value unless modified by Value statement Fix v1 ScoringFn 2 The following syntax frees the third slope for items v1 v5 in Group2 Free Group2 vi v5 Slope 3 Cro
93. a model was given in Figure 6 1 where ys are the observed items zs are the discrete attributes and s are the testlet effects For the 24 item dataset we will be modeling items 1 6 are members of testlet 1 items 7 12 belong to testlet 2 items 13 18 to testlet 3 and items 119 19 24 are members of the final testlet The Q matrix for mapping items onto attributes supplied in Table 6 3 Table 6 3 Q matrix for Testlet DINA Example Item Attribute 1 Attribute 2 Attribute 3 Attribute 4 Item 1 1 1 0 0 Item 2 0 0 1 1 Item 3 1 0 1 0 Item 4 0 1 0 1 Item 5 1 0 0 1 Item 6 0 1 1 0 Item 7 1 1 0 0 Item 8 0 0 1 1 Item 9 1 0 1 0 Item 10 0 1 0 1 Item 11 1 0 0 1 Item 12 0 1 1 0 Item 13 1 1 0 0 Item 14 0 0 1 1 Item 15 1 0 1 0 Item 16 0 1 0 1 Item 17 1 0 0 1 Item 18 0 1 1 0 Item 19 1 1 0 0 Item 20 0 0 1 1 Item 21 1 0 1 0 Item 22 0 1 0 1 Item 23 1 0 0 1 Item 24 0 1 1 0 As can be seen in syntax the lt Options gt section is largely unchanged from the previous examples although we have reduced the number quadrature points from the default due to the high dimensional nature of this problem In the first group G the data file variable names etc have been given As with previous CDM examples we specify the number of attributes Attributes 4 and also request that flexMIRT create dimensions for the interaction ef fects From the Q matrix every possible second order interaction is needed so 120 we use Interacti
94. ain effects gives us a total of 22 latent dimensions in the model In the Constraints section all main effect terms are initially fixed to 0 and then Free statements are added to replicate the Q matrix for the problem which may be found in Table 8 on pg 347 of de la Torre amp Douglas 2004 Because a DINA is fit to all items only the highest order interaction of at tributes for each item is specified and main effect terms are specified for only those items that depend on a single attribute i e items 6 8 and 9 The last statement in the Constraints section sets the latent attribute slope parameters to be equal across attributes producing a restricted higher order model As with all examples the full output file is available on the program s Support site The item parameter table for this example is extremely wide 22 latent dimension columns an intercept column and all associated SEs and is not presented here due to size However we have summarized the flex MIRTO parameter estimates in Table 6 6 Please note that the slopes are presented in a single column in this table for efficiency of space the es timated slope values are not all on the same latent dimension as can be seen in the full output file In addition to the flex MIRT estimates we also present the original slippage and guessing parameter values reported by de la Torre and Douglas 2004 in their Table 9 and the flexMIRT estimates con verted to this paramete
95. al case of the nominal model To set up a GPC model using the reparameterized nominal model one fixes all scoring function contrasts This results in the first scoring function contrast being fixed to 1 0 and the other contrasts fixed to 0 0 With such constraints in place the category scoring function values are equal to 0 1 k which is the necessary structure for fitting the GPC Because the GPC is such a popular special case of the nominal model in flexMIRT 2 0 we have added GPC as an option to the ModelQ statement This op tion will automatically fix the scoring functions as needed freeing the user from making such technical specifications The following syntax demonstrates the use of this new model keyword As with the Graded model specification when one uses the GPC keyword in the Model statement the model name is immediately followed by the number of categories in parentheses Example 2 9 Generalized Partial Credit Model Project Title QOL Data 36 Description 35 Items GPC lt Options gt Mode Calibration Etol le 4 Mtol le 5 Processors 2 lt Groups gt Group1 File QOL DAT Varnames vi v35 N 586 Ncats v1 v35 Model v1 v35 lt Constraints gt 7 GPC 7 2 8 3 Rating Scale Model Andrich s 1978 Rating Scale RS model may also be obtained as a spe cial case of the reparameterized nominal model To obtain the RS model in flex MIRT it is necessary to fix
96. ality Example 6 5 Testlet Structure with DINA model Project Title Fit HO DINA Model with Testlets Description HO DINA Model 4 attributes lt Options gt Mode Calibration MaxE 10000 Etol 1e 4 Mtol 1e 7 Quadrature 25 6 0 Processors 2 GOF Extended SE REM SaveDbg Yes SaveSco Yes 121 Score EAP lt Groups gt AG File fitDINAt dat Varnames vi v24 Ncats vi v24 2 Model vi v24 Graded 2 Attributes 4 InteractionEffects 2 6 possible 2nd order ints Primary 10 number of dimensions to CDM Dimensions 14 total dims including testlet effects AD Varnames al a4 DM G lt Constraints gt Fix G vi v24 MainEffect Free G v1 v7 v13 v19 Interaction 1 2 Free G v3 v9 v15 v21 Interaction 1 3 Free G v5 v11 v17 v23 Interaction 1 4 Free G v6 v12 v18 v24 Interaction 2 3 Free G v4 v10 v16 v22 Interaction 2 4 Free G v2 v8 v14 v20 Interaction 3 4 Free G v1 v6 Slope 11 Free G v7 v12 S1ope 12 Free G v13 v18 S1ope 13 Free G v19 v24 S1ope 14 Equal G vi v6 Slope 11 Equal G v7 v12 Slope 12 Equal G v13 v18 Slope 13 Equal G v19 v24 Slope 14 Equal D a1 a4 Slope 1 122 6 3 2 Replicating Published Examples Because users are most likely acquainted with fitting CDMs in other programs we also provide syntax and output for well known examples
97. ance it is expected that about 1596 of the population have mastered none of the measured attributes attribute profile 0 0 0 0 around 4 have mastered at tributes 2 and 4 profile 0 1 0 1 and 18 4 of the population have mastered all 4 attributes profile 1 1 1 1 We will next cover the structure of the individual scoring file sco that was requested in the Options section In the sco output excerpt presented we have given only the first 10 cases and have included line breaks and a separator not in the output file for ease of presentation 108 16v100 0 0S9Z20 0 vLvTOO 0 29000 0 S81000 0 Sv0000 0 L0 T00 O c v800 0 8 v1 0 0 c 8170 0 490000 0 692000 0 990000 0 Z10000 0 vv0000 0 T00000 0 00000 0 62 000 0 v1000 0 Cvv000 0 81v000 0 6v3L00 0 TTv000 0 v6 100 0 0T0000 0 TIT000 0 84Te00 0 618 00 0 68910 0 8 4400 0 T90000 0 686000 0 0900000 T60000 0 200000 0 600000 0 6 0000 0 990000 106000 0 408000 0 608200 0 66ZEE0 0 89200 0 96000 0 96000 0 2 Z000 0 009000 0 92v00 0 TOZALIO O 0c0010 0 96ST00 0 u c19100 0 TLST00 0 Z0000 0 Z SOT00 0 f 860000 0 1 010000 0 6000 0 n 86Z000 0 1 976000 0 amp 8S00 0 ZZSE00 Oc90ST1 O 606c0 86088 0 S 600 0 9ZETOO 8SvEzZ O 02
98. ands Options gt DIF None Test All TestCandidates DIFcontrasts OrthDIFcontrasts Yes No Within the Options section there are three commands related to the implementation of DIF analyses The first command is a request for a DIF analysis DIF By default no DIF analysis is conducted but this may be modified by the user to request either a comprehensive DIF sweep keyword is TestA11 testing all items or a focused DIF analysis which looks for DIF only in the candidate items supplied by the user keyword is TestCandidates If the TestCandidates option is selected the DIFitems command found in the Groups section is required When a DIF analysis is requested by invoking one of the DIF options DIFcontrasts becomes a required command This command is used to supply flex MIRTO with the DIF contrast matrix which is used in constructing the contrasts among groups The total number of contrasts is one fewer than the total number of groups being analyzed For a 2 group DIF analyses that means there is one allowable contrast which will typically be of the form DIFcontrasts 1 1 For analyses with more than 2 groups any ANOVA contrasts can be specified The Helmert contrast which is orthogonal is a reasonable choice e g for 3 groups DIFcontrasts 2 0 1 0 1 0 0 0 1 0 1 0 When group size is highly unbalanced OrthDIFcontrasts offers additional re weighting of the contrast coefficients such th
99. are the slope labeled a and intercept labeled c In addition for unidimensional models the difficulty value labeled b is also reported this value may be found from the other two parameters as b c a Below the item parameter table the group parameters are printed In this example we had only one group so the mean and variance were fixed at 0 and 1 respectively for model identification That these parameters were not estimated is indicated by both the lack of an assigned parameter number and the dashes in the standard error columns 9 0 sez00g uz 4qed esuodsey 103 TITAETTO Y Teur3rIeH 88 0 78 0 08 0 S O TL 0 99 0 29 I 0 IS9 0 8p 0 S O Sv O 8p 0 eS O e s peaoedxg O T CVT 88 T LL T TO C 62 S TO i 48 Tv v 8 v 98 vV 8 v 98 UOTIRUIOJUT 389 Go 20 00 00 TO 00 TO vo c0 90 60 90 8e vo Oc 68 ee 60 9T 20 90 vo 6 6e LT 02 VG 20 cr Go ge LO GTA TTA OTA 6A QA JA QA GA pA A ZA TA TeqeT eq ur tdnozy I dnozy 103 08 Z o3 08 Z OI e404 JO sen eA ST 3 SANTA uorqoun4 uor3euzogju U 141I N H 10 60 TO 00 c0 00 TO 90 0 80 00 0 00 0 TO 0 T0 0 c0O O 0 O SO FO 0 90 0 80 0 ZT O 9T O OZ O vc 88 VS oo 9T cT 80 v O 60 TT TO 00 0 00 co 20 Go or er PT c0 TO vo 00 0 60 LO cr
100. at they become orthogonal with respect to the group sizes Langer 2008 This option is turned off by default 148 Finally there are several commands that are specific to or used primarily by the MH RM algorithm Example 7 6 Options MH RM Specific Options Options gt Algorithm BAEM MHRM Imputations 1 Thinning 10 Burnin 10 Stagel 200 Stage2 100 Stage3 2000 Stage4 0 InitGain 1 0 Alpha 1 0 Epsilon 1 0 WindowSize 3 ProposalStd 1 0 ProposalStd2 1 0 MCsize 25000 To call the MH RM algorithm the combination of Mode Calibration Algorithm MHRM and arandom number seed via RndSeed is required All other listed commands are optional with the default values shown above Imputations controls the number of imputations from the MH step per RM cycle A larger value will lead to smoother behavior from the algorithm In most cases a value of 1 should be sufficient for obtaining accurate point estimations however this number may be increased for better standard errors Thinning refers to the MCMC practice of retaining only every kth draw from a chain Thinning is used in part to reduce the possible autocorrelation that may exist between adjacent draws The Thinning statement sets the interval for the MH sampler meaning if Thinning 10 every 10th draw by the sampler will be retained The Burnin statement controls the number of draws that are discar
101. atch examples ex6 2D flexmirt in 886 ms Batch completed with errors in 00 01 22 336 Average process time 259 86 ms The typical output files for the individual runs submitted in batch mode will be in the same folder as the corresponding flexmirt syntax command file not with the batch file 179 APPENDIX C Plotting In unidimensional IRT graphs e g tracelines and information curves form an integral component of item analysis in practice flexMIRT is built to handle multilevel and multidimensional IRT models In this context there currently exists no universally accepted consensus with regards to graphical procedures We do realize however that there is a need for publication quality graphics and we chose to provide a set of R programs We implemented the provided graphing functions in R because it is incredibly powerful allows maximal customizability and is a free program While flex MIRT does not have an internal plotting function supplemen tal R code to read in flexMIRT parameter files and plot item characteris tic curves ICCs test characteristic curves TCCs item information func tions IIFs and test information functions TIFs is available on the Support webpage http www flexMIRT com Support in a folder labeled flexMIRT plotting examples Please note that the provided code is usable with single factor models only C 1 Plotting ICCs TCOs and Information Functions The provided R code rea
102. ation and as required when Mode Simulation provided a seed value for the random number generator which will be used to create the data Because we are not using Value statements we have also provided a file name where item and group parameters may be found In the lt Groups gt section we label our group and use the File statement to provide a name for file where the data will be stored once generated We want to simulate data for 10 variables so we provide labels for 10 variable names with the Varnames statement and finally using N specify the number of cases we which to have generated The variable and group labels provided here must match those provided in the second column of the parameter file As may be seen here and as mentioned previously the syntax command file is only used to set up the very basic structure of the analysis and the specifics are left to be input through the parameter file to which we will now turn our attention As covered in Section 4 1 the actual parameter file to be submitted to flexMIRT Exlgenparms txt in this case may not contain row headers but they are included in Table 4 3 for illustrative purposes only In this parameter file we have arranged all the item specifications first and the group parameters are in the bottom row This is completely arbitrary and the rows of the file 69 may be arranged in any way the user wishes However the columns must follow the arrangement previously detailed Readin
103. aximum number of iterations number of quadrature points used number of free parameters in the model if any control values were changed from defaults the information printed here serves as verification Next the program processing time is listed both in total and broken down into various stages Any output files that were generated by the analysis are named in the next section The final part of this subsection of the output reports if flex MIRT terminated normally and if the specified convergence criteria were met The next section of the output provides the estimated item and group parameters Output 2 2 2PL Calibration Output Item and Group Parameters 2PLM example 12 items 1 Factor 1 Group 2PL Calibration 2PL Items for Group 1 Item Label P a vi 2 05 v2 4 23 v3 6 84 v4 8 04 v 10 85 v6 12 34 v7 14 90 v8 16 97 v9 18 89 viO 20 32 vii 22 87 vi2 24 01 1 2 3 4 5 6 7 8 9 10 11 PB O PF P O F P O F O F O O O O O O O O O O O O O O O O O O O O O O O a O O O O O O O O O O O O m pa N 2PLM example 12 items 1 Factor 1 Group 2PL Calibration Group Parameter Estimates Group Label PH mu 1 Groupi Following a reprinting of the title and description each item is listed in its own row with the provided label and assigned parameter numbers estimated item parameters and standard error values going across The IRT model fit in this example was the 2PLM so the reported item parameters
104. bbons R D Bock R D Hedeker D Weiss D J Segawa E Bhaumik D K Grochocinski V J 2007 Full information item bifactor analysis of graded response data Applied Psychological Measurement 31 4 19 Gibbons R D amp Hedeker D 1992 Full information item bifactor analysis Psychometrika 57 423 436 Glas C A W Wainer H amp Bradlow E T 2000 Maximum marginal likelihood and expected a posteriori estimation in testlet based adaptive 190 testing In W J van der Linden amp C A W Glas Eds Computer ized adaptive testing Theory and practice pp 271 288 Boston MA Kluwer Academic Publishers Gu M G amp Kong F H 1998 A stochastic approximation algorithm with Markov chain Monte Carlo method for incomplete data estimation problems The Proceedings of the National Academy of Sciences 95 7270 7274 Haberman S J 1979 The analysis of qualitative data New York Academic Press Hartz S M 2002 A Bayesian framework for the unified model for assess ing cognitive abilities Blending theory with practicality Unpublished doctoral dissertation Department of Statistics University of Illinois at Urbana Champaign Hastings W K 1970 Monte Carlo simulation methods using Markov chains and their applications Biometrika 57 97 109 Jamshidian M amp Jennrich R I 2000 Standard errors for EM estimation Journal of the Royal Statistical Society Series B 62
105. butes 2 3 and 4 was desired in addition to the 3 4 interaction we could specify that using a single generate statement of Generate 3 4 2 3 4 116 As before we free the first 12 items onto the appropriate single attributes Following the comment in the syntax we free items 13 15 on both main effect terms for attributes 3 and 4 as well as the interaction of the two attributes To induce the equality specified in the model we set the main effects and interac tion terms equal to each of the items via a statement such as Equal G v13 MainEffect 3 G v13 MainEffect 4 G v13 Interaction 3 4 and then apply the 1 to the interaction with a coefficient constraint of the form Coeff G v13 Interaction 3 4 1 117 6A 8 LA 9A SA yA gA TA e qe us21 slo joure Ie d wo j ro I mdmo ON IG Q 9 3nd jn O 118 As with the DINA model the estimated item values reported are in the LCDM parameterization but DINO parameters of guessing and slippage may be obtained by conversion Again guessing is found from the reported param eters as ezp c 1 ezp c and one minus the slippage term 1 s is found as ezp c a 1 ezp c a The a term used in the conversion should be the value reported in one of the main effect columns not the negative value associated with the interaction term One aspect of the output that we haven t discussed as of yet is the model fit We present the 2 log l
106. c form of recoding that may be useful when the observed responses from multiple choice items have been read in Rather than writing numerous Code statements to convert the observed responses into correct incorrect 0 1 data the Key statement allows the user to provide a scoring key for the items and instruct flex MIRT to score the items internally For instance if the correct responses to Items 1 through 4 were 2 0 1 and 3 respectively the Key statement to score these items would be Key vi v4 2 0 1 3 Multiple Code and Key statements are permit ted to easily accommodate situations in which items have variable numbers of response options or when scoring a large number of items multiple scoring keys would result in greater user readability of the command file by limiting command line length It should be noted that both the Code and the Key com mands are completely internal to flex MIRT and the original data file will not be modified Although flexMIRT can do some basic data management it is not built to be a data management tool It is generally advisable to per form all necessary data cleaning and data management in a general statistical software package such as SAS or SPSS before exporting a space comma or tab delimited file for flex MIRT analysis The Model command follows the same general format as Ncats but instead of placing a number after the equals sign one of the four keywords for the IRT model must be chosen For
107. c100 0 CcCCSvVO O 904100 0 49 6 0 0 0T698T 0 DS9ITTZ O TTv86T O 6vc9 0 0 9 v1v0 O cS0Tv0 O T 000000 T 000000 T 000000 0 T OIT000 0 469810 0 26100 0 c987221 0 8 2900 0 86899T 0 TTOLDT Z88187 NN 968v T 0 TS20Z0 0 8SvC0 0 CETVPST O CCT6T0 0 ZZE6ST O 686198 0 96Sz6 0 TSTEGE O TELO9T O vLe06T O 3188610 T 000000 T 000000 T 000000 0 T 000000 0 v6000 0 9866000 0 c 8S0v0 0 4Tc000 0 9102v0 0 vO8seo0 0T76828 NN SE 6T E0 O TvOSTIO0 0 09c100 0 ZEGVO O 904100 0 419 6 0 0 0T698T 0 DS9TTZ O TTIv861 0 6vc9 0 0 9 v1v0 O cS0Tv0 0 T 000000 T 000000 T 000000 0 Z 990000 0 667000 0 S6v820 0 9Cve80 0 S T 00 0 SI9T0 0 96ZLTS 0 600 NN 002062 0 069800 0 9948T0 0 6Sv6T10 0 T99c00 0 v809 T O c0TO6vV O vP6vetr o 96889 0 66868 0 S8610 0 98vc9T 0 T o r4 oos yndyng uosuog ur duroT ddny 4 9 mdmg 129 As in the previous sco output we have given the first 10 cases and have added a line break and separators The first two columns of the sco file give the group and observation number similar to a standard flexMIRT IRT score output file Recall that for this example there are 3 attributes and 8 possible attribute profiles The probability of the respondent having mastered each of the attributes individually is given starting after the observation number For example for observation 1 the probability they have mastery of attribute 1 is 0 16 for attribute 2 it is 0 02 and for attr
108. ce rate The current model had only 1 level so all level 2 values will be 0 00 The final value labeled Fn is an indicator that the program is or is not working towards a converged solution While not the log likelihood it may be interpreted in a similar fashion meaning that less negative values indicate a better fit Examining the first several rows of the provided iteration history the acceptance rate starts somewhat high but quickly drops down to the near desired level by the 10th iteration of Stage I during this time the Fn value also rapidly drops initially 91 and then stabilizes For all of Stages II and III both the acceptance rate and the function value fluctuate with the acceptance rate staying near the desired value of 0 20 While the output should be closely examined for issues this pattern of values in the processing pane is an early indicator that the MH RM algorithm is working towards a good solution We present the same section of the output was that given for Example 3 8 Output 5 2 Correlated Four Factor Model Output Loadings Means and Variances Factor Loadings for Group 1 Groupi Item Label lambda 1 s e lambda 2 s e lambda 3 s e lambda 4 s e v2 0 03 0 Mayaca v3 0 03 v4 0 01 v5 0 01 v6 em v7 v8 v9 v10 v11 v12 v13 vi4 vid v16 v17 0 02 0 04 0 02 0 02 o oc 5ogmwn e 90909999999909092929 9999999992920929992999 0909999999909092929 90929999992920299292999 QOL Data It
109. ces Index 111 166 167 167 172 174 174 177 180 180 184 188 188 188 189 194 List of Tables 4 1 4 2 4 3 4 4 4 5 6 1 6 2 6 3 6 4 6 5 6 6 7 1 D 1 D 2 D 3 D 4 D 5 D 6 D 7 D 8 Simulation Graded Model PRM File Content 65 Key Codes for Columns With Set Options 66 Simulation 3PL Parameter File with Labeled Columns 70 Labeled Parameter File for Bifactor Model Simulation 73 Simulating DIF and Group Mean Differences Parameter Values 76 Q matrix for Basic CDM Demonstrations 103 Model Fit for Basic CMDs Fit to Simulated Data 119 Q matrix for Testlet DINA Example 120 Q matrix for Rupp Templin amp Henson s Example 9 2 123 Generating latent class mean values and flex MIRT and MPLUS estimates for Rupp Templin amp Henson s Example 9 2 128 Guessing and Slippage Parameters from flexMIRT and MCMC 137 Constraint Commands and Keywords 157 Labeled sco file EAP scores 1 Factor 184 Labeled sco file MAP scores 1 Factor 184 Labeled sco file EAP scores 2 Factors 185 Labeled sco file EAP scores 2 Factors with User supplied ID Variable O A Y 185 Labeled sco file MAP scores 2 Factors 185 Labeled prm file GRM with 1 Factor 1 Group 186 Labeled prm file 2PL with 6 Factors 1 Group 186 Labeled prm File 3PL wi
110. ctions related to LD detection We have previously discussed the Marginal fit and Standard ized LD X Statistics table but the matrix immediately below that in the 2L output is new The JSI Yes statement in the command file produces a matrix labeled Edwards Houts Cai Jackknife Slope Diagnostic Index which provides values for a newly introduced LD detection method Edwards amp Cai 2011 the calculation of which is described in detail later in the manual Output 2 15 1PL Grouped Data Output JSI matrix Edwards Houts Cai Jackknife Slope Diagnostic Index for Group 1 Groupl Item 1 2 3 4 5 1 0 3 0 3 0 3 0 3 0 7 0 7 0 7 2 5 2 5 2 5 2 5 1 4 1 4 1 4 1 4 4 1 zc For now it is sufficient to know that a value noticeably larger than others in the matrix indicates an item pair that should be examined for possible LD The last new table is the result of the statement HabermanResTbl Yes A portion of this table is reproduced below In this table observed response patterns are listed by row For a given response pattern the observed and model implied frequencies the standardized residual EAP score and standard deviation associated with the EAP score are printed Output 2 16 1PL Grouped Data Output Haberman Residual Table Ex cerpt Response Pattern bserved and Expected Frequencies Standardized Residuals EAPs and SDs Group 1 Groupi Item Frequencies Standard 1 2 3 4 5 Observed Expected Residual EAP
111. d when the program is first initiated Once the desired flex MIRT command file is located and selected it will be opened in a separate pane labeled Syntax Editor with the file name listed after File Edt lexMIRT Help D a run Progress License valid expires 03 07 2032 ND Syntax Editor 2PLM_example Fie Edt FlexMIRT Help D EZ Pl X a A ji run lt Project gt Title 2PLM example Description 12 items 1 Factor 1 Group 2PLM Calibration lt Options gt Mode Calibration File g341 19 dat Varnames vl v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 N 2844 Ncats vi vi2 2 Model vi vi2 Graded 2 Constraints 172 The command file is submitted for analyses by clicking the Run button on the far right of the icon toolbar flexMIRTO will print its progress e g EM iteration number etc in the engine pane and when complete will open the Output Viewer pane which will contain all results Additionally the results are written to an external text file which may be opened at a later time with flex MIRTO or a text editor such as Notepad wed Output Viewer 2PLM example 125759 eel rs flexMIRT R 2 00 64 bit Flexible Multilevel Multidimensional Item Analysis and Test Scoring c 2012 2013 Vector Psychometric Group LLC Chapel Hill NC USA 2PLM example 12 items 1 Factor 1 Group 2PLM Calibration Summary of the Data and Dimensions Missing data cod
112. ded from the start of the MH sampler These draws are discarded to avoid using values that were sampled before the chain had fully converged to the target 149 distribution Burnin 10 tells flexMIRTO to discard the first 10 values obtained by the MH sampler Stage1 determines the number of Stage I constant gain cycles The Stage I iterations are used to improve default or user supplied starting values for the estimation that occurs in Stage II Stage2 specifies the number of Stage II Stochastic EM constant gain cycles The Stage II iterations are used to further improve starting values for the MH RM estimation that occurs in Stage III Stage3 sets the maximum number of allowed MH RM cycles to be performed Stage4 determines the method by which SEs are found If Stage4 0 then SEs will be approximated recursively see Cai 2010b If a non zero value is given then the Louis formula Louis 1982 is used directly If the Louis formula is to be used the supplied value determines the number of iterations of the SE estimation routine this number will typically need to be large e g 1000 or more InitGain is the gain constant for Stage I and Stage II If the algorithm is initially taking steps that are too large this value may be reduced from the default to force smaller steps Alpha and Epsilon are both Stage III decay speed tuning constants Alpha is the first tuning constant and is analogous to the InitGain used in Stages I and II E
113. demonstrate this point we will now generate data with the parameters values just used but supply them to flex MIRT via a parameter file 4 2 The First Example Redone Using Parameter File Example 4 2 Simulation Graded Model Using PRM File Project Title Simulate Data Description 4 Items Graded Use PRM File lt Options gt Mode Simulation Rndseed 7474 ReadPrmFile simib prm txt lt Groups gt 4G File simlb dat Varnames vl v4 1000 lt Constraints gt The ReadPrmFile statement is invoked to read the parameter file The group specification only contains the file name the variable names and the number of simulees Note that the Constraints section is empty because the values of the generating parameters are set in the parameter file The content of the 64 Table 4 1 Simulation Graded Model PRM File Content 1 v2 1 1 2 3 2 1 10 1 v3 1 1 2 3 0 1 12 1 vl 1 1 2 3 2 1 07 1 v4 1 1 2 3 0 1 15 particular parameter file is shown in Table 4 1 The parameter file has 4 rows corresponding to the 4 items Take row 1 as an example The entry in the first column of 1 specifies that this is a row containing item parameters This is a predefined option The other option is 0 which corresponds to group specifications The second column specifies the variable name Note that the items are not ordered as v1 to v4 but there is no problem since the items are checked against the list of variabl
114. ds item parameters from a flex MIRT parameter output file with the extension prm txt For information functions the code reads the needed values from a flex MIRT information output file which has the extension inf txt flex MIRTG does not generate these files by default so it is necessary to explicitly request that the parameters and information values be saved from the calibration run using the SavePRM Yes and the SaveInf Yes commands in the Options section respectively Additionally by default the Information Function output file will only contain the Information value for a single point at the theta value of 0 To change the number of points and the range over which they are spaced one uses the FisherInf 21 3 0 command in the Options section This specific command requests 180 that information values at 21 points spaced evenly between 3 0 and 3 0 be written to the generated inf txt file the choice of 21 and 3 0 are used as placeholders and both values can be modified by the user The entirety of the example plotting code found in the file DrawICCs Infs R in the available flexMIRT plotting examples folder is presented below Be cause we don t wish to assume that users are familiar with R we will briefly discuss the main points of the R syntax Example C 1 R plotting code calls the pre defined plotting functions REQUIRED AS FIRST STEP ALTER THE PATH SO IT DIRECTS TO WHERE flexmirt R IS S
115. ds noted in the Constraints section aren t limited to those particular commands or to CDMs but are provided to show that rather than using the typical IRT parameter keywords Slope etc users may employ the parameter keywords MainEffect and Interaction to allow syntax specification that is in keeping with the CDM conceptualiza tion terminology for the models 6 3 CDM Modeling Examples In this section we will provide several examples of CDMs fit in flexMIRT We will discuss basic examples that demonstrate how to fit several of the more well known CDMs that appear in the literature and then using datasets that have been fit elsewhere demonstrate the comparability of lex MIRT estimates to those from other programs and estimation methods 102 6 3 1 Introductory CDM Examples Here we provide straight forward demonstrations of several widely discussed models We hope that in addition to providing simple concrete examples for the chosen models from these examples users will begin to see the flexibility flex MIRT has in fitting CDMs gain insight into the details of our syntax for CDMs and be able to extrapolate this to fit the wide variety of alternative CDMs that are not specifically covered but may be fit using flexMIRT For the examples in this section we will use simulated data for 15 dichotomous items that were generated to measure four underlying attributes The Q matrix which assigns items to attributes is present
116. e 9 Number of Items 12 Number of Cases Latent Dimensions 1 Item Categories Mode1 1 Graded Graded Graded Graded Graded Graded Graded Graded Graded Graded Graded Graded lO CO Oy n bp Q N p o P P NNNNNNNNNNN IN m N Bock Aitkin EM Algorithm Control Values Maximum number of cycles 500 Convergence criterion 1 00e 004 Maximum number of M step iterations 100 173 APPENDIX B flexMIRT via the Command Line Interface This appendix provides a brief overview of using flex MIRTO via the Command Prompt which may be useful to those wishing to batch submit a large number of files or to control flexMIRT via another program e g R B 1 flexMIRT via the CLI single file run Begin by accessing the Command Prompt via the Windows Start button All Programs Accessories Command Prompt n Accessories fa Calculator Pictures EX Command Prompt Connect to a Network Projector Ii Connect to a Projector cj Getting Started A Math Input Panel a Notepad Paint Y Remote Desktop Connection 13j Run Q Snipping Tool t Sound Recorder Default Programs Sticky Notes Sync Center Help and Support Windows Explorer B WordPad Ji Ease of Access System Tools Le Tablet PC Windows PowerShell Music Games Computer Control Panel Devices and Printers 4 Back Search programs and files 174 You will now need to direct the CLI to the folder where the
117. e Metropolis Hastings Robbins Monro Algorithm 85 86 92 93 MinExp see lt Options gt MinExp MinMLscore see lt Options gt MinMLscore Missing see lt Groups gt Missing Missing Data 3 152 Mode see lt Options gt Mode Model see lt Groups gt Model Models see also Calibration 3PL 163 Generalized Formulation 162 Graded Response Model 164 Nominal Categories 164 Monte Carlo Study see Simulation Mstarts see lt Options gt Mstarts Mtol see lt Options gt Mtol Multidimensional IRT Bifactor Model 37 39 56 60 119 122 Correlated Factors 53 54 56 90 94 Multi level Model 49 52 93 95 Quadrature Points 55 58 Multiple Groups Calibration 40 Differential Item Functioning DIF 45 Anchor Items 49 Candidate DIF Items 48 DIF Sweep 45 Measurement Invariance 42 79 160 Scoring 43 N see lt Groups gt N Ncats see lt Groups gt Ncats Nested Data see Multidimensional IRT Multi level Model 197 NewThreadModel see lt Options gt NewThreadModel Nlevel2 see lt Groups gt Nlevel2 Non Normal Theta Estimation see Emprical Histograms NormalMetric3PL see lt Options gt NormalMetric3PL NumDec see lt Options gt NumDec Options 4 5 12 16 19 21 24 30 33 41 44 45 138 Algorithm 87 90 91 94 97 149 Alpha 87 88 149 150 Burnin 87 88 149 150 CTX2Tb1 145 DIF 148 TestA11 46 148 TestCandidates 49 148 DIFcontrasts 46 49 148 DMtable 101 139
118. e 4 Mtol 1e 9 SEMtol 1e 3 SStol 1e 4 Processors 1 NewThreadModel Yes No As detailed previously when Mode is set to something other than Calibration additional statements are required It was noted earlier that specifying SaveSCO Yes during a calibration run triggers a combined scoring run If a scoring run is specified either by explicitly setting Mode to Scoring or implicitly by requesting a combined scoring run an IRT scoring method is required As may be deduced from the lack of an underlined scoring option there is no default The options available for scoring are Expected A Posteriori keyword is EAP Maximum A Posteriori keyword is MAP Maximum Likelihood keyword is ML and Summed Score Conversions to EAP scores keyword is SSC which in addition to individual scores provides a conversion table in the output that translates summed scores to EAP scale scores When using ML scoring there are two additional keywords that must be employed Users are required specify the desired maximum and minimum 141 scores to be assigned by flexMIRT using the MaxMLscore and MinMLscore keywords respectively There are no default values for the ML minimum and maximum scores Efforts have been made to make ML scoring as robust as possible but it is not recommended for use with multi dimensional models Because ML scoring does not use information from the population distribution essential statistical information about
119. e and Score DM Data Description C RUM Model lt Options gt Mode Calibration MaxE 20000 MaxM 5 Mtol 0 Etol le 5 SE REM SavePrm Yes SaveCov Yes SaveSco Yes Score EAP lt Groups gt AG File DMsim dat Varnames vi vi5 N 3000 Neats vi vi5 2 Model vi vi5 Graded 2 Attributes Dimensions H ds oa 104 AD Varnames al a4 DM G lt Constraints gt Fix G vi v15 MainEffect Free G v1 v3 MainEffect 1 Free G v4 v6 MainEffect 2 Free G v7 v9 vi3 vi15 MainEffect 3 Free G v10 v15 MainEffect 4 The first points of note in the syntax occur in the lt Options gt section The maximum number of E steps has been increased from the default the E tol erance and M tolerance values have been decreased and the SE calculation method has been set to REM Richardson extrapolation method as experi ence has found the REM SEs to be preferable to estimates from other methods These changes in the Options section are made to improve the resulting SE estimates and are recommended for any CDMs fit using flexMIRT The specifications for the CDM begin in the Groups section Although the naming of the groups is arbitrary all examples will use the custom of defining group G as the group containing observed data and group D as the group containing the higher order latent dimensions attributes As with a typical IRT model in group G
120. e available for use with single group analyses as well For instance the Progress statement via Yes No keywords determines whether flexMIRT will print progress updates in the console window Using SaveInf Yes we have requested that the Fisher information function for the items and the total scale be saved to an external file and have specified via FisherInf 81 4 0 that we would like infor mation values calculated at 81 points equally spaced from 4 0 to 4 0 The method of standard error calculation has been specified as empirical cross products approximation using SE Xpd all possible SE calculation methods and corresponding keywords are covered in Chapter 5 Details of the Syntax 3 2 Characterizing Non Normal Population Distributions This example is substantially the same as the analysis just discussed with an additional option included in the calibration portion As the command file does not change drastically only the modified sections are presented here The full syntax file g341 19 calib EH flexmirt is available in the examples provided on the flex MIRT website Example 3 2 2 Group 3PL with Empirical Histogram Excerpts lt Options gt Etol 5e 4 lt Groups gt ZGrade4 File g341 19 grpl dat Varnames vl v12 N 1314 Ncats vl v12 2 42 Model vi vi2 ThreePL BetaPriors vi vi2 1 5 EmpHist Yes Grade37 File g341 19 grp2 dat Varnames v1 v12 N 1530 Ncats
121. e estimated popula tion discrepancy function value labeled FOhat and the RMSEA Root Mean Square Error of Approximation e g Browne amp Cudeck 1993 In this exam ple flexMIRT has reported that the underlying contingency table was too sparse to compute the full information fit values because the total number of cross classifications is 212 4096 which is considerably larger than the sample ll size of 2844 2 3 Single Group 3PL Model Calibration The data in the previous section may also be suitable for the 3PL model which explicitly models examinees guessing Example 2 2 Single Group 3PL Calibration Syntax Project Title 3PL example Description 12 items 1 Factor 1 Group Calibration saving parameter estimates lt Options gt Mode Calibration GOF Extended savePRM Yes lt Groups gt Group1 File g341 19 dat Varnames v1 v12 N 2844 Ncats vi vi2 2 Model vi vi2 ThreePL lt Constraints gt Prior vi vi12 Guessing Beta 1 0 4 0 The syntax for this run differs from the previous one in several ways In the Options section the parameter estimates are to be saved to a sep arate output file savePRM Yes and additional GOF statistics were re quested by specifying GOF Extended as an option In the Groups sec tion the fitted model is now the 3PL model Model keyword is ThreePL In the Constraints section a prior distribution is impo
122. e matrices T and Te are fixed K x K 1 matrices of contrast coefh cients By an appropriate choice of T the identification restrictions will be automatically satisfied Thissen et al 2010 propose the use of the following linear Fourier contrast matrix 1 das e da est T 2 t32 t3 K 1 K 1 0 gt 0 where a typical element t m for 1 2 K and m 1 2 K 1 takes its value from a Fourier sine series SC K 1 For instance for K 4 the contrast matrix is 0 0 0 F 1 866 866 2 866 866 3 0 0 and for K 5 the contrast matrix is 0 0 0 0 1 707 1 7O7 F 2 1 o 1 3 707 1 707 4 0 0 0 For the nominal model one can verify that identification conditions are all satisfied For the GPC model all the w contrasts are set to zero 165 Appendices 166 APPENDIX A Quick Start Installation and User Guide This guide is designed to provide an overview for installing flex MIRT from the webpage and initializing the program for use A 1 Installing flexMIRT To obtain the flexMIRT installer you must first register for an account on the flexMIRT website The registration page may be found by selecting the My Account link in the flexMIRT website banner or visited directly at https flexmirt vpgcentral com Account LogOn From your account page you will be able change your password view your current flexMIRT license and expiration date information manage
123. e model In M Nering amp R Ostini Eds Handbook of polyto mous item response theory models Developments and applications pp 43 75 New York NY Taylor amp Francis Thissen D amp Steinberg L 1988 Data analysis using item response theory Psychological Bulletin 104 385 395 Tian W Cai L Thissen D amp Xin T 2012 Numerical differen tiation methods for computing error covariance matrices in item re sponse theory modeling An evaluation and a new proposal Edu cational and Psychological Measurement Advance online publication doi 10 1177 0013164412465875 Tittering D M 1984 Recursive parameter estimation using incomplete 192 data Journal of the Royal Statistical Society Series B 46 257 267 Wirth R J amp Edwards M C 2007 Item factor analysis Current ap proaches and future directions Psychological Methods 12 58 79 Woods C 2007 Empirical histograms in item response theory with ordinal data Educational and Psychological Measurement 67 73 87 Woods C Cai L amp Wang M 2013 The Langer improved Wald test for DIF testing with multiple groups Evaluation and comparison to two group irt Educational and Psychological Measurement 79 532 547 193 Index AddConst see Constraints AddConst Algorithm see lt Options gt Algorithm Alpha see lt Options gt Alpha Attributes see Groups Attributes BetaPriors see Groups BetaPrior
124. e names in the syntax file and matched automatically Column 3 indicates that this item belongs to group 1 Column 4 indicates the number of factors in this case the model is unidimensional The next two columns taken together set the IRT model to graded type 2 on column 5 with 3 categories column 6 Then the item intercepts from the intercept corresponding to the lowest category to the highest category and the item slope are specified for each item The resulting simulated data set is exactly the same as the previous example Output 4 3 Simulated Data Set Graded Model Using PRM File 02100 0 246011 22221 1 70243 220 71107 2 2 1 2 997 0 0161087 2 2 2 1 998 0 702542 O O 1 0 999 1 09703 4 3 The Parameter File Layout Although the progression of columns of the parameter file must proceed in a prescribed fashion the general format of the file is plain ASCII in which delimiters may be spaces or tabs Column headers are not permitted which is why the order of values in the columns must follow the rigid structure 65 For arranging the values in the parameter file we have found it easiest to use a spreadsheet program e g Excel to initially set up the entries and when complete export paste the values into the flat ASCII text editor As discussed earlier there are set options from which the user must select for certain columns Table 4 2 provides a summary of these options Table 4 2 Key Codes for Columns Wi
125. ect the level 1 acceptance rate and adjustments to the ProposalStd2 value will affect the level 2 acceptance rates In the lt Groups gt section we inform flexMIRT we will be fitting a model with 4 latent dimensions Dimensions 4 two of which will be used as higher level between factors Between 2 and we then indicate a variable that will supply the level 2 group membership Cluster 12id Within the lt Constraints gt section we create a structure that has 2 between and 2 within factors via the first group of Fix and Free statements and assign items to factors as desired We also constrain the slopes across the levels to equality using Equal statements and then allow the variances of the level 2 factors and the covariance of the within factors and the covariance of the between factors to be freely estimated It is worth noting here that there is nothing outside of the lt Options gt section that is uniquely related to the requested MH RM estimation this specific model can be estimated using the default Bock Aitkin EM albeit with noticeable time loss Larger models with more factors would most likely require MH RM to complete successfully We will cover the processing pane output prior to addressing the results of the estimation In particular we will highlight the differences between the previous example and the processing pane output for a model with a second level 94 Output 5 3 Two Level MIRT Model P
126. ects Calibration Description 5 Items 1PL N 1000 lt Options gt Mode Calibration Algorithm MHRM Stagel 0 Stage2 0 lt Groups gt Group1 File lsat6 dat Varnames vi v5 id map Select vi v5 N 1000 NCats vl v5 2 Model vi v5 Graded 2 FixedTheta map Constraints Equal vi v5 S1lope 97 For any group that will be calibrated with fixed ability values theta is read in as part of the datafile for that group In this example the theta values for Group1 are in the variable map the last variable listed in the Varnames statement As in earlier examples because we have provided variables that are not to be calibrated we then use a Select statement to specify only those items which will be subjected to IRT analysis To provide lex MIRT with the theta values we use the statement FixedTheta map to tell the program the variable that holds the fixed theta values Although not seen in this example FixedTheta works under multidimensionality currently the number of variables listed in FixedTheta vars must match the number of factors specified via Dimensions for that group Of final note in the fixed effects calibration syntax the Stage1 and Stage2 keywords in the Options section which determine the number of constant gain and stochastic EM constant gain cycles of the MH RM estimation method are both set to 0 This is recommended for fixed effects calibration in lexMIRT
127. ed after your year with flex MIRT if you would like to renew your subscription If you desire a different subscription length please contact sales VPGcentral com for pricing information E 1 Commercial use of flexMIRT This version of flex MIRT is intended for non commercial use only as defined in the fexMIRT End User s Licensing Agreement If you are interested in pur chasing flex MIRT for commercial use please contact sales VPGcentral com E 2 Classroom Discounts for flexMIRT For educators who are interested in using flex MIRT in their classroom bulk ed ucational discounts are available Please contact salesQVP Gcentral com for more information 188 References Adams R amp Wu M 2002 PISA 2000 technical report Paris Organization for Economic Cooperation and Development Andrich D 1978 A rating formulation for ordered response categories Psychometrika 43 561 573 Bock R D amp Aitkin M 1981 Marginal maximum likelihood estimation of item parameters Application of an EM algorithm Psychometrika 46 443 459 Bock R D Gibbons R amp Muraki E 1988 Full information item factor analysis Applied Psychological Measurement 12 261 280 Bradlow E T Wainer H amp Wang X 1999 A Bayesian random effects model for testlets Psychometrika 64 153 168 Browne M W amp Cudeck R 1993 Alternative ways of assessing model fit In K Bollen amp
128. ed by adding the statement SlopeThreshold Yes to the lt Options gt section When the 21 SlopeThreshold keyword is invoked threshold values rather than intercept values will be saved to the prm file Output 2 9 Graded Model Calibration Slopes and Intercepts Graded Items for Group 1 Groupi Item Label Pi a s e P c 1 1 vi 5 58 0 06 1 25 v2 10 85 06 6 35 v3 15 79 06 11 04 v4 20 73 06 16 11 v5 25 44 05 21 35 v6 30 19 05 26 10 v7 35 24 05 31 27 v8 40 72 04 36 18 v9 45 08 05 41 46 50 37 05 46 72 55 62 06 51 34 60 44 05 56 41 65 12 05 61 19 70 37 05 66 71 75 57 06 71 30 80 80 04 76 13 85 17 05 81 28 90 61 04 86 83 o N o O O Q BAUN O OOpPEFE PE PF F Rp RR O Ol P GOO N PF O N 0o0o0000000000000000nu ORORRRRARARRORRARAR AR RR ooo 0 0 0 0 00 0 00 00 000 QO N E QO N G5 S GS N N Q N Aa S A BW O O O O O O O O O O O 00 00 00 08 B O Op P PF OF oO o0o00O0O00000O000000000o00o00omNn lt O O Q O O O Qy O Q O O O O on m 00 Graded Items for Group 1 Groupi Item Label P a s e bi o vi 5 v2 10 v3 15 v4 20 v5 25 v6 30 v7 35 v8 40 v9 45 50 55 60 65 70 75 80 85 90 58 85 79 73 44 19 24 72 08 37 62 44 12 37 57 80 17 61 o 06 05 06 35 06 26 06 38 05 01 05 TT 05 63 04 05 05 29 05 45 06 68 05 05 05 95 05 52 7 06 04 05 04
129. ed in Table 6 1 As can be seen the majority of items use only 1 attribute but the final 3 items are assigned to both attributes 3 and 4 These items could be fit with a variety of models and we will present examples for three different CDMs Table 6 1 Q matrix for Basic CDM Demonstrations Item Attribute 1 Attribute 2 Attribute 3 Attribute 4 Item 1 1 0 0 0 Item 2 1 0 0 0 Item 3 1 0 0 0 Item 4 0 1 0 0 Item 5 0 1 0 0 Item 6 0 1 0 0 Item 7 0 0 1 0 Item 8 0 0 1 0 Item 9 0 0 1 0 Item 10 0 0 0 1 Item 11 0 0 0 1 Item 12 0 0 0 1 Item 13 0 0 1 1 Item 14 0 0 1 1 Item 15 0 0 1 1 Basic C RUM Fit to Simulated Data For the first example the majority of items will only depend on the main effect of a single attribute and we will fit the compensatory reparameterized unified model C RUM to the final three items The C RUM model e g Hartz 103 2002 Choi Rupp amp Pan 2013 is a CDM that when items are informed by multiple attributes allows for mastery of one attribute to compensate for non mastery on other attributes The C RUM estimates for each item an intercept term and as many slope terms for an item as there are 1 entries in the Q matrix As detailed in Choi et al 2013 C RUM is a special case of the log linear cognitive diagnosis model LCDM in which all interaction terms are set to 0 meaning only main effects are estimated Example 6 2 C RUM Model Fit To Simulated Data Project Title Calibrat
130. ed in the hope that they will be useful but WITHOUT ANY WARRANTY without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE See the GNU General Public License for more details 183 APPENDIX D Labeled Output Files Due to the lack of labels in some of the output files that flex MIRT creates we provide this appendix to give labeled examples of several file examples specifically sco and prm files For each example we will provide labels at the top of each column which would not normally appear and a small selection of lines from the output file Table D 1 Labeled sco file EAP scores 1 Factor Grp flexMIRT Obs SE 6 1 1 0179 0 501 1 2 0 957 0 609 1 3 0 033 0 576 1 4 0 957 0 609 1 5 0 519 0 515 Table D 2 Labeled sco file MAP scores 1 Factor Grp flexMIRT iterations SE 0O Obs to MAP 1 1 2 0 150 0 494 1 3 0 939 0 564 1 3 2 0 077 0 567 1 4 3 0 939 0 564 1 5 2 0489 0 518 Note Scores from maximum likelihood Score iterations to score column 184 ML will have a similar Z of Table D 3 Labeled sco file EAP scores 2 Factors Grp flexMIRT 01 05 SE 01 SE 02 011 621 622 Obs 1 1 0 141 0 466 0 258 0 355 0 066705 0 001763 0 126136 1 2 0 192 1 407 0 281 0 295 0 079119 0 001470 0 087174 1 3 2 023 1 684 0 527 0 392 0 277999 0 015970 0 153786 1 4 1 311 0 840 0 334 0 290 0 111558 0 003131 0 083973
131. eedom equal to number of summed scores 1 2 and provides a test of latent variable distribution assumption In both the Extended and Complete GOF modes the standardized LD X statistics for each item pair may be suppressed by setting the CTX2Tb1 command to No As noted in the discussion of the LD statistic values of Chen and Thissen 1997 when applied to ordinal data i e Example 2 5 phi correlations are examined to determine if difference between the observed 145 and model implied item correlation is positive or negative When polytomous data are involved there may be times that the contingency table of two items needs to be collapsed from the edges to the middle so that the cells are not too sparse Even if collapsing occurs for the Chen Thissen LD calculations the phi correlations determining the p or n after each LD value will be based on the original non collapsed tables The SX2Tb1 command is invoked when the user wishes to be able to exam ine the detailed item fit tables used in the calculation of the S X statistics As described in Orlando and Thissen 2000 when expected frequencies for responses become small the accuracy of the X approximation deteriorates flex MIRTO employs an algorithm to evaluate the expected probabilities prior to final calculations and collapse the table when needed The MinExp com mand allows the user to set the value at which expected counts are considered too small the default
132. el in which each item is restricted to load on at most two factors The typical set up for a bifactor model is one factor on which all the items load termed the general factor and several more specific factors termed group specific factors onto which only subsets of the items can load A common substantive example of this would be a reading test made up of several short reading passages that each have several items probing the students comprehension for a particular passage The general factor in this case would be Reading and each passage would have its own specific factor onto which only the items related to that passage load Note that Bradlow Wainer and Wang s 1999 testlet response theory model can be understood as a further restricted bifactor model with within item proportionality constraints on the general and group specific slopes see also Glas Wainer amp Bradlow 2000 With generalized capability to impose constraints lex MIRT can estimate testlet models too Chief among the benefits of the item bifactor model is computational efficiency in which full information maximum marginal likelihood estimation of item parameters can proceed with 2 dimensional integration regardless of how many group specific factors are in the model due to a result first discovered by Gibbons and Hedeker 1992 The bifactor model in flex MIRT is in fact obtained as a two tier model Cai 2010a flex MIRT is fully capable of multip
133. ems 2 21 Correlated Factors Group Parameter Estimates Group Label PH mu 1 s e PR mu 2 1 Groupi 0 00 gt 0 00 Latent Variable Variance Covariance Matrix for Group1 Groupi P Theta 1 s e P Theta 2 s e P Theta 3 s e Pit Theta 4 s e 1 00 141 0 40 0 03 1 00 gt 142 0 54 0 03 143 0 49 0 03 1 00 rri 144 0 54 0 03 145 0 55 0 03 146 0 68 0 02 From the excerpt of the output provided it may again be observed that the output from the MH RM is formatted identically to output from EM esti mations Comparing across the two estimation routine results the parameter estimates are nearly identical indicating that the EM and MH RM estimation routines are reaching a similar converged solution In favor of the MH RM however is the fact that the total time needed to estimate the model and generate output was around 4 minutes compared to the 54 minutes used by the default Bock and Aitkin EM 92 5 3 2 Multi level MIRT Model With the four factor example we have demonstrated the time benefit that can be obtained through the use of the flexMIRT MH RM algorithm when esti mating high dimensional models The MH RM as implemented in flexMIRT is also able to fit multi level models a feature that to our knowledge is not available in any other commercially available software We will provide syntax for and output from fitting such a model The data used in this example was simulated to have six dichotomous items with responses from 2
134. eta a 1 8 1 Value ScoringFn Coeff Mean AddConst Cov Prior MainEffect Interaction Presented in the second column of Table 7 1 are the keywords for parame ters that may have constraints applied to them Intercept refers to the loca tion parameter Slope refers to the slope discrimination parameter Guessing is the lower asymptote parameter in the 3PL model ScoringFn is the scoring function contrast parameters of the nominal model Mean and Cov refer to the mean and covariance matrix elements of the latent variable distribution 157 MainEffect and Interaction are available so users may employ syntax spec ifications that are in keeping with the conceptualization terminology used in cognitive diagnostic models The question marks in the parentheses following the parameter keywords permits the referencing of specific parameters For example when there are 4 latent variables in the model Slope 1 refers to the slope on the first factor Mean 2 refers to the mean of the second factor and Cov 3 2 refers to the covariance of factors 3 and 2 When an item has 5 categories Intercept 2 in a graded model or ScoringFn 3 in a nominal model each refers to specific item parameters For the Free Equal and Fix constraints the general format for applying restrictions to the model is the same First the type of constraint to be applied and the group to which it is being applied are listed followed by a comm
135. eters it is necessary to supply the contrast coefficient values as well as the contrasts In other words the built in contrasts are automatically computed but any user defined contrasts need to be entered into the program The contrast coefficients are given after the contrast selection column e g Column 7 but before the actual contrast values The contrast coefficients of a user supplied contrast matrix are ex pected to be in row major order That is for a contrast matrix applied to nominal items with k categories the coefficients from that k x k 1 matrix would be entered in the following order T 1 1 T 1 2 T 1 3 T L k 1 T 2 1 T 2 k 1 T k k 1 4 4 A 3PL Model Simulation Example In this example we will simulate item responses to 10 3PL items for a single group of 500 simulees The latent trait distribution is standard normal The 68 syntax file is presented first Example 4 3 Simulation 3PL Model Normal Population Project Title Fit Model Description 10 Items 3PL N 500 Normal Priors lt Options gt Mode Simulation Progress No ReadPRMFile genparams txt RndSeed 1 lt Groups gt Group1 File simi dat Varnames v1 v10 N 500 lt Constraints gt As always the command file begins with the lt Project gt section where a ti tle and general description are provided In the lt 0ptions gt section we have set the engine mode to Simul
136. exMIRT to run in the CLI batch mode and batch csv is the name of the batch file If the batch file is in a different directory than the flex MIRT executable or contains spaces in the file name the full path and file name should be provided and enclosed in quotes as was shown with the single run command line run Once the batch file has been submitted progress will be printed The number of files found in the batch file is first reported and then the progress for individual runs starting and finishing will be given If flex MIRT is unable to complete a run successfully the fact that an error was detected in the run will also be reported As can be seen the file broken flexmirt did not complete and that fact is reported here quite plainly Cf lexmirt gt winflexmirt exe b C flexmirt batch examples batch test csu License valid expires 63 07 2632 Found 7 configs to run in C flexmirt batch examples batch test csu Finished C flexmirt batch examples ex1 2PL flexmirt in 124 ms gt Finished C flexmirt batch examples ex1 3PL flexmirt in 156 ms gt Finished C flexmirt batch examples ex2 flexmirt in 15 lt ms gt Starting 4 of 7 C flexmirt batch examples ex3 calib f lexmirt Starting 5 of 7 C flexmirt batch examples broken f lexmirt Errors in C flexmirt batch examples broken f lexmirt Starting 6 of 7 C flexmirt batch examples ex5 f lexmirt Starting 7 of 7 C flexmirt batch examplesrex6 2D f lexmirt Finished C flexmirt
137. f the item assignment we ask flexMIRT to generate all second and third order effects for us via the InteractionEffects command Finally we set Dimensions 7 as the total number of dimensions of the model is 3 main effects 3 second order interactions 1 third order interaction In the D group we inform flexMIRT that the diagnostic model we are setting up will be applied to group G using the DM G statement Within the D group we specify that the attribute variable will be called a and that it has 8 categories reflecting the 8 possible attribute profiles Additionally we tell flexMIRT that the attribute variable will be fit with the nominal model and we use an identity matrix for the nominal model intercepts In the lt Constraints gt section we assign items onto attributes according to the Q matrix First we fix all items to 0 and then free items onto the appro priate main effect and interaction terms For example from the Q matrix we see that Item 4 is informed by attributes 1 and 2 and the interaction of those attributes In the lt Constraints gt section we free Item 4 with the statements Free G v4 MainEffect 1 Free G v4 MainEffect 2 and Free G v4 Interaction 1 2 The final three constraints are applied to the higher level group group D The first two statements fix the slope and scoring function values and the final Value statement is used to fix the first scoring 125 function value to 0 for model ident
138. fer to the contrast matrices for the scoring function and intercept parameters respectively The options for the scoring function and intercept contrasts include a Trend matrix which uses a Fourier basis aug mented with a linear column an Identity matrix or a user supplied matrix In this example both of the contrast matrices have been set to type Identity to facilitate equality constraints Within the Constraints section we en counter a more complicated set of constraints than in the previous example Suffice it to say that these constraints are applied to the scoring functions and intercepts so that the solution matches with Thissen and Steinberg s 1988 preferred testlet solution Estimated item parameters are presented in Output 2 18 33 68 0 cv 0 cv 0 000 GUS 8 0 8 0 000 g9 0 08 0 98 0 000 TV S 46 0 0070 00 0 grauepI T Y 4 T 10399e9 leqeT1 wey Tdnoxg T dnozy 103 sueg TeuTWoN sTeqouwereg ZL6T X20g TeuUTsTI9 YEN 2 e 2 e ozo 390 8 830 cv 0O 6 830 cv 0O 6 Karauep YPN Z 06 0 g9 0 S YvY O 0870 v T O 98 0 Karauep gjrauep T a s ewweS Hg a s Z ewweS q a s T euueg q sysezqU0D eqeT u 4I Idnozo9 I dnozy 10 sue3 TOF sasgserquoo a3deoze3u epoxy TeuruoN 00 97 0 97 0 00 0 UI TEN Z 00 ToT 00 0 00 0 KgrauepI T ys es es TS Teqe7 u 4I Idnoz9 q dnozy senqpeA uorqSun4 Sur4oS5S TOPON TeuruoN 97 0 L T 0 97 0 L 00 T Aqtquepl 79
139. fied are applied to the MIRT compatible metrics prior applied to intercept not difficulty normal guessing prior still applied to logit g distribution etc If NormalMetric3PL is set to Yes for a scoring run then flexMIRT will expect the all parameter values read in from the prm file for scoring to be in the normal metric By default flexMIRT prints both the intercept and difficulty parameters in the output for undimensional models but saves only the intercept values into a requested prm file When SlopeThreshold Yes rather than saving the intercept values the program will write the difficulty b values to the prm file If SlopeThreshold is set to Yes in conjunction with a 3PL model the psuedo guessing parameters gs will be printed saved rather than the default logit guessing parameter values Users should be aware that unlike the behavior when NormalMetric3PL is used the SlopeThreshold keyword has no effect on the metric of the slope a values printed to the prm are in the logistic metric regardless of the SlopeThreshold setting 140 Example 7 3 Options Technical Specifications Options gt Score EAP MAP ML SSC MaxMLscore MinMLscore Mstarts Yes No Rndseed ReadPRMFile txt SE Xpd Mstep SEM Sandwich Fisher FDM REM SmartSEM Yes No Perturbation 0 001 FisherInf 1 0 0 PriorInf Yes No logDetInf Yes No Quadrature 49 6 0 MaxE 500 MaxM 100 Etol 1
140. g across for the first item the 1 in the first column denotes that the values to follow refer to an item We then provide a variable name v1 and assign this item to the first group which is the only group in this example The next column indicates that there is 1 latent factor in the model and the subsequent column Column 5 says that we will be providing parameters for the 3PL model indicated by the 1 The 6th column value tells flexMIRT that the item will have two response options as per the 3PL model Based on the number of factors the IRT model and the number of response categories we now supply the appropriate number of item parameters in this case 3 parameters 1 for the lower asymptote 1 for the intercept and 1 for the slope The order of parameters values must be guessing in the logit metric intercept in the c a x b metric where b is the difficulty threshold and then the slope parameter a Table 4 3 Simulation 3PL Parameter File with Labeled Columns Type Label Group Factors Model Cat logit g c a 1 v1 1 1 1 2 L34 069 1 30 1 v2 1 1 1 2 0 97 1 30 1 41 1 v3 1 1 1 2 1 64 0 42 1 05 1 v4 1 1 1 2 2 10 040 1 26 1 v5 1 1 1 2 1 59 1 35 2 43 1 v6 1 1 1 2 1 72 0 56 1 67 1 v7 1 1 1 2 1 13 0 26 3 12 1 v8 1 1 1 2 1 71 1 56 2 09 1 v9 1 1 1 2 2 79 0 73 1 17 1 v10 1 1 1 2 1 66 0 14 1 10 Type Label Group Factors Prior Type Mean Var 0 Groupl 1 1 0 0 00 1 00 The final row of the f
141. g the Equal statement on the Slope parameters of items v1 v5 The keywords for other items parameters are Intercept for the location param eter in all models Guessing for the guessing parameter lower asymptote in the 3PL model and ScoringFn for the scoring function parameters in the Nominal Categories model There are several new commands in the Options section For the good ness of fit indices the full gamut of available GOF indices detailed in Chapter 7 of the manual are requested by the keyword Complete The JSI statement requests the Jackknife Slope Index JSI an LD detection tool for all pairs of items The HabermanResTbl command requests a table that lists the ob served and expected frequencies and standardized residuals for each response pattern as well as the associated SEs EAPs and SDs When SX2Tb1 is re quested detailed item fit tables will be printed in addition to the Orlando and Thissen 2000 S X item fit chi square statistics The command FactorLoadings Yes requests the printing of the factor loading values for each item which may be converted from the respective slope values e g Takane amp de Leeuw 1987 Wirth amp Edwards 2007 Even though the scores are not saved SaveSco No the scoring method of SSC is specified As noted previously the SSC method will print the requested summed score to EAP scale score conversion table in the output although no individual scores are calculated saved The c
142. g to further speed up computations by spreading the load automatically to the multiple available cores or process ing units This user s manual is intended both as a guide to those trying to navigate the complex features implemented in flexMIRT for the first time and as a reference source It is assumed that the user is familiar with the theory and applications of IRT Please visit the website http www flexMIRT com for product updates technical support and feedback CHAPTER 2 Basic Calibration and Scoring We begin by discussing the basic structure of flex MIRT syntax We provide examples that demonstrate some standard IRT analyses such as calibration and scoring Syntax files for flex MIRT may be created in the GUI by select ing New or created using a flat text program e g Notepad If New is selected within the flex MIRT9 GUI a command file containing the skeleton code for a basic analysis opens Once you have modified this command file it must be saved prior to being able to submit the analysis to lexMIRT 9 the Run button is disabled until the file has been saved to prevent the skeleton code from being over written The file name extension for flex MIRT syntax files is flexmirt Data files must use space tab or comma delimiters between variables and may not contain a variable name header Data should be ar ranged so columns represent items and each observation has its own row in the dataset Although
143. g values for the MH RM estimation that occurs in Stage III Stage3 sets the maximum number of allowed MH RM cycles to be performed Stage4 determines the method by which SEs are found If Stage4 0 then SEs will be approximated recursively see Cai 2010b If a non zero value is given then the Louis formula Louis 1982 is used directly If the Louis formula is to be used the supplied value determines the number of iterations of the SE estimation routine this number will typically need to be large e g 1000 or more InitGain is the gain constant for Stage I and Stage II If the algorithm is initially taking steps that are too large this value may be reduced from the default to force smaller steps Alpha and Epsilon are both Stage III decay speed tuning constants Alpha is the first tuning constant and is analogous to the InitGain used in Stages I and II Epsilon controls the rate at which the sequence of gain constants converge to zero Specified Epsilon values must be in the range 0 5 1 with a value closer to 1 indicating a faster rate of convergence to zero WindowSize allows the user to set the convergence monitor window size Convergence of the MH RM algorithm is monitored by calculating a window of successive dif ferences in parameter estimates with the iterations terminating only when all differences in the window are less than 0 0001 The default value of the win dow size is set at 3 to prevent premature stoppage due to random va
144. gned as a result of our MaxMLscore statement the reported SE is 99 99 to indicate that the score was assigned rather than estimated 2 6 Graded Model Calibration and Scoring Example syntax for a combined calibration and scoring run is now presented flex MIRT code for the graded response model which may be used with two or more ordered response options e g Samejima 1969 is also introduced This example employs data from 3000 simulees using parameter values re ported in Table 1 of Edwards 2009 paper as generating values These data were constructed to mimic responses collected on the Need for Cognition Scale Cacioppo Petty amp Kao 1984 which is an 18 item scale that elicits responses on a 5 point Likert type scale asking participants to indicate the extent to which the content of the item prompt describes them Example 2 5 Graded Model Combined Calibration and Scoring Syntax with Recoding Project Title NCS example Description 18 items 1D 1 Group Calibration and Scoring 20 lt Options gt Mode Calibration SavescO YES Score MAP GOF Extended lt Groups gt Group1 File NCSsim dat Varnames vi v18 N 3000 Ncats vi vi8 5 Code vi v18 1 2 3 4 5 0 1 2 3 4 Model vi vi8 Graded 5 Constraints The only new command encountered in this example is the Code statement in the Groups section The internal representation of item re s
145. here is only one group The group name is enclosed in percent signs and should be something easy to remember e g Groupi or Grade4 or Female or 4US The group names become important in specifying constraints in multiple group analyses The name is arbitrary but spaces are not allowed when specifying group names After a group name has been given the data file name where the item responses are located is specified as well as the number of participants in the group variable names the number of possible response categories for each item and the IRT model to be estimated e g threePL Graded 5 etc 4 The final section lt Constraints gt is reserved for specifying constraints and prior distributions on the item parameters and latent variable means and co variances Although the section header is required there are no mandatory commands that must be included Constraints and prior distributions may be placed on almost any aspect of the model including means covariances and the various item parameters Details on how to specify constraints will be discussed in examples that utilize constraints 2 2 Single Group 2PL Model Calibration The first several examples will use data from two test forms that resemble those used in North Carolina end of grade testing Data are responses to 12 multiple choice items collected from 2844 test takers with responses coded as correct 1 or incorrect 0 These dichotomous responses
146. ibute 3 it is 0 60 The next three columns are the estimated SEs for those probabilities respectively The last six entries on the first line of each observation contain the entries for the estimated attribute variance covariance matrix Finally on what has been moved to the second line for each respondent but is a continuation of the same in the actual sco file the posterior probabilities for each of the attribute profiles 8 possible for this example are given For observation 1 the posterior probability of being in attribute profile 1 is 0 30 the probability of being in attribute profile 2 is 0 52 and so on The order in which the probabilities are given matches the order used in the output table Diagnostic IRT Attributes and Cross classification Probabilities Referring to the output presented for this example we find that attribute profile 1 corresponds to 0 0 0 meaning no attributes are mastered and attribute profile 2 corresponds to 0 0 1 meaning only attribute 3 has been mastered etc After the probabilities for each of the attribute profiles are presented the most likely attribute profile for the respondent is given for respondent 1 that is attribute profile 2 for respondents 6 and 7 that is attribute profile 8 The last three values given in the sco file are the higher order latent variable point estimate followed by the SD and variance for that estimate In this case all individual theta estimates for the higher order
147. ich we wish to generate data Following the other examples the details of the simulation are set up in the parameter file genparams txt The contents of the parameter file presented in part in Table 4 5 are now discussed As before the first column still indicates whether the row refers to an item or a group and the second column provides a label The first indication that this is a multiple group problem appears in Column 3 which is the column designated for assigning the row item or group to a specific group defined in the syntax In previous examples with only one group all rows displayed a 1 75 Table 4 5 Simulating DIF and Group Mean Differences Parameter Values 1 vl 1 1 1 2 1 33972 0 68782 1 301362 1 v2 1 1 1 2 0 96814 1 29641 1 409536 1 v9 1 1 1 2 2 79306 0 730972 1 172468 1 vl0 1 1 1 2 1 65629 0 14263 1 102673 1 vl 2 1 1 2 1 33972 0 68782 1 301362 1 v2 2 1 1 2 0 96814 1 29641 1 409536 1 v9 2 1 1 2 2 79306 0 144738 1 172468 1 vl0 2 1 1 2 1 65629 0 14263 1 102673 1 vl 3 1 2 2 0 68782 1 301362 1 v2 3 1 1 2 0 96814 1 29641 1 409536 1 v9 3 1 1 2 2 79306 0 730972 1 172468 1 vl0 3 1 1 2 1 65629 0 14263 1 102673 0 Groupl 1 1 0 0 1 0 Group 2 1 0 02 1 0 Group3 3 1 O0 02 1 5 in this column where now the first 10 items 4 of which are displayed in the table have 1s the next 10 rows have 2s and so on With respect to the differences across the groups as noted earlier Item 1 in group 3 is a 2PL ite
148. ient for lex MIRT to locate it If however the command file and the data are in different directories it is necessary to specify the full path in the File statement For example an alternate statement might be File C Users VPG Documents flexMIRT example 2 1 g341 19 dat When providing names for the variables in the Varnames statement the name for every variable was fully written out as v1 v2 v12 For the Ncats and Model statements a shortcut was used which is only available when variables names end with consecutive integers As exemplified in the syntax vi v2 v12 can be shortened to v1 v12 This shorthand may be employed in any statement where variable names are provided For the Model statement the Graded Response Model with two categories is specified as Graded 2 This model is formally equivalent to the 2PL model and as such flexMIRT does not implement the 2PL model separately Al though not used in this example it is useful to be aware that multiple Ncat and Model statements can be specified within one group which is needed for an analysis of mixed item types For example if items 1 through 6 in the current data were dichotmous and items 7 through 12 were 5 category polytomous items two Ncat statements i e Ncats vi v6 2 Ncats v7 v12 5 and two Model statements i e Model v1 v6 Graded 2 Model v7 v12 Graded 5 would be provided to fit appropriate models to the items There are no const
149. ification purposes With the syntax created we can now run the CDM in flexMIRT and turn to the output 126 vl O vt L ewe tq 0070 9 eudte sd vC O ers ers 16 0 T L 0 9 ewes qq o s 0070 S eudre sd ors Tvl O 16 0 00 T 90 T 00 T 76 0 ec T 00 0 00 0 00 0 00 0 000 000 000 000 000 e e T 8 L 9 3 v g e T 2080 729 TeqeT1 Weil q z dnozp5 103 suez euruoN sI q ueredq 2461 X909 TRUTBTID 00 I ZI TEO 90T TT E T O 00 7I OT EVO Z6 0 6 90 GET 8 farauepI e H G euwe2 gd o s p eure gg o s g euueS8 sq os g euue8 amp d os I euueS amp d sasei3uo TeqeT uo3I q iz dnox9 103 sue3I zor saserquoo adeoiequI Tepow reutuoN 00 S 00 00 00 00 T 00 0 e T 9s ss ys es zs Ts Teqe1 921 a z dnozy sanTe uora2unj Surroog T POW TRUTUON 000 000 puoi 000 e T y eudie sd e s eudie gd e s c eudre gd T eudie 4d sqserquop as e sd ede 991 a z dnoz5 103 sue4I 103 SISP1JUOJ UOTIDUNJ SUT109S pue sedo g Tepoj TeuTWON se3nqri33e g suo3t TSpou WAIT Z 6 oTduexg uosueg urldue ddny Br feet aoe EO ones e I SP ers P es TP d leqeT Weil a z dnoz5 103 swear 2d s anqr zqqe sueqT lepou WAIT Z 6 oTduexg uosueg urldue ddny 4600991c 0 T 098S 60 0 T TTI6v T160 0 o 0817 980 0 o 86S6TT60 0 T S6 46660 0 T 19689 10 0 0 0 e 6T10018vc 0 qozd ui T T T T o 0 o 0 Y 2384 TOF SOTITTTAeQoIg UO
150. ikelihood AIC and BIC values for each of the three models fit to our simulated data in Table 6 2 As with the output for more standard IRT models these values are reported at the bottom of the irt flexMIRT output file As can be seen in Table 6 2 the fit of the models indicates that using DINO for the final three items was the preferred model based on all three reported fit indices It is informative to know that the same data set was used in all three examples and this data was generated using the DINO model for the last three items The advantage for the DINO is some times small but that is most likely a result of the majority of items being fit by single main effects the specification of which did not vary regardless of the model that was used for the items with more complex attribute relationships Table 6 2 Model Fit for Basic CMDs Fit to Simulated Data CRUM DINA DINO 2loglikelihood 45799 53 45873 59 45798 31 Akaike Information Criterion AIC 45881 53 45949 59 45874 31 Bayesian Information Criterion BIC 46156 21 46204 17 46128 89 Basic DINA with Testlet Structure For the final example in this section we employ a new simulated dataset that contains responses to 24 dichotomous items that are to be modeled with 4 attributes In addition to being generated from a CDM this data was simulated to incorporate a testlet structure such as when multiple reading items use the same passage as a prompt A graphical depiction of such
151. ile is used to describe the group The first 0 indicates that the row is describing a group The second column provides the group a label Group1 which should correspond to the label provided in the command file We assign the Group1 label to the first group by the 1 in the 3rd column and specify that a unidimensional model will be generated by indicating 1 factor in the 4th column As noted in Table 4 2 the fifth column of a Group line indicates the type of prior distribution to be applied in this case a normal prior signified by the 0 After a normal prior is indicated a group mean of 0 00 and a variance of 1 00 are supplied 70 4 5 Simulating Data with Non Normal Theta Distribu tions In this example we will cover data simulated from models with non normal population distribution This is accomplished using the empirical histograms method previously used in Calibration Example 3 2 The empirical his togram prior may only be used for unidimensional or bifactor models testlet included as previously noted in the discussion of Example 3 2 Example 4 4 Simulation Empirical Histogram Prior Project Title Fit Model Description 10 Items 3PL N 500 Non Normal Priors lt Options gt Mode Simulation ReadPRMFile EHgenparams txt RndSeed 1 lt Groups gt Group1 File sim2 dat Varnames v1 v10 N 500 lt Constraints gt The command file for this example is substantively the same
152. imension reduction algorithm it supports the non parametric estimation of latent density shapes using empirical histograms for both unidimensional and hierarchical e g bifactor and testlet response theory item factor models in any number of groups with support for con straints on group means and variances Finally it has a full featured built in Monte Carlo simulation module that can generate data from all models imple mented in flexMIRT flexMIRT is easy to use It has an intuitive syntax It can import data natively in space comma or tab delimited formats Windows based flexMIRT with a friendly graphical user interface GUI is available in both 32 bit and 64 bit flavors flexMIRT is designed with cross platform compat ibility and large scale production use from the beginning The computational engine of flexMIRT is written using standard C which ensures that it can run on any platform where a C compiler exists A newly designed memory allocation scheme helps flexMIRT efficiently handle thousands of items and millions of respondents with no imposed upper limit on the size of the problem flexMIRT is fast For multidimensional and multilevel models that per mit dimension reduction flexMIRT automatically reduces the amount of quadrature computation to achieve a dramatic reduction in computational time As modern CPU architecture trends toward multi core design flexMIRTO implements two ways to use multi threadin
153. in that it facilitates simulations by offering the user the capability of generating data from all implemented IRT models including multidimensional bifactor and multilevel models The command syntax for generating data has the same basic structure as the model fitting syntax In general it will involve either a relatively small amount of code in the syntax file leaving the majority of the model settings and generating parameter values in the parameter file Alternatively one can fully specify the models and generating parameter values directly in the syntax without using the parameter file or using it only as a supplement Note that a parameter file saved from a calibration run is completely compatible in simulations This provides a convenient mechanism so that the user may conduct simulations with generating parameters found in empirical data analysis 4 1 A Syntax Only Example Using Value Statement In this example we will use the syntax file to specify simulation of a data set consisting of 1000 simulees responses to 4 graded items each scored with 3 cat egories The generating parameter values are specified in the Constraints section In simulation mode the Value statement can be used to specify generating parameters as is done in the following example 61 Example 4 1 Simulation Graded Model Syntax Only lt Project gt Title Simulate Data Description 4 Items Graded Syntax Dnly lt Options gt Mode Si
154. in the Grade 3 group relative to the fixed values of the Grade 4 group which has a mean of 0 and variance of 1 The format of the Value Coeff and AddConst constraints is similar to the previous types of constraints but with an additional piece of information After specifying the constraint type group items and parameter a numeric value is specified as well which is separated from the parameter keyword by a comma For example a coefficient command could be something like Coeff G vi vO Slope 1 702 where we are applying a coefficient in group G for variables v1 v10 of 1 702 to the slopes effectively incorporating the scaling constant D into the estimation The Value statement is a more general purpose constraint command In calibration mode it can be used to specify the values of fixed parameters or it can be used to supply starting values In scoring mode instead of using a parameter file which is more convenient one can in principle use the Value statement to fix item and group parameters to the calibrated estimates In simulation mode the Value statement can be used to specify generating pa rameters For a Value constraint on one or more parameters if the parameters are free the numeric values supplied become starting values If the parameters are fixed Value can be used to change the default fixing values For example for a unidimensional model in a single group the following syntax results in a set of constraints that f
155. ing the z probabilities Le P x1 0K 01 99 TH P zi fi 04 with a set of higher order latent variables 0 so that each of the attributes zs is treated as if it is an observed item 6 2 CDM Specific Syntax Options Example 6 1 Options Groups and lt Constraints gt CDM Specific Options Options gt DMtable Yes No Groups gt Attributes InteractionEffects Generate DM group Constraints gt Coeff group vars parameter Fix group vars MainEffect Free group vars Interaction In the Options section the user may control whether or not the some times large diagnostic classification probability table is printed in the output via the DMtable command Attributes is used to set the number of main effects present in the skill space that is being assessed This is the keyword that triggers flexMIRT to model discrete latent variables rather than the default continuous latent vari ables InteractionEffects instructs flex MIRT to automatically generate all possible interaction effects For instance with 4 attributes one may specify 101 InteractionEffects 2 3 4 and in addition to the 4 main effects the program with also generate the 2nd 3rd and 4th order interaction terms of the attributes as dimensions of the model If only the second order interaction effects are desired those would be created by
156. ings Means and Variances Factor Loadings for Group 1 Groupi Item Label lambda 1 s e lambda 2 s e lambda 3 s e lambda 4 s e v2 0 03 0 dotes v3 0 03 v4 0 02 v5 0 02 v6 v7 v8 v9 vid vil vi2 vi3 v14 v15 v16 v17 v18 v19 v20 0 03 0 05 0 02 0 03 V0ADOAONA Ooooooooooooooooooo 0 0 0 0 0 0 o 0 0 E 0 Mu o 0 0 o 0 0 0 0 0 S O O O O O O oO oO O O O O O O oO O O oO OoOoooooooooooooooooo Items 1 21 Correlated Factors with Correlated Factors Group Parameter Estimates Group Label PH mu 1 s e PR mu 2 1 Groupi 0 00 es 0 00 Latent Variable Variance Covariance Matrix for Groupi Groupi P Theta 1 s e P Theta 2 s e P Theta 3 s e Pit Theta 4 s e 1 00 aisi 141 0 41 0 04 1 00 Toc 142 0 55 0 03 143 0 50 0 04 1 00 144 0 55 0 04 145 0 56 0 03 146 0 69 95 The pattern of the estimated loadings indicates that the items were as signed to the factors as expected with certain loadings restricted to 0 00 and having no associated SE Because there is only a single group the latent vari able means were also restricted to zero and have no associated SE indicating they were not estimated The final section in the output excerpt reports the correlation matrix of the latent variable for example the correlation between the first two standardized factors is estimated to be 0 41 with a SE of 0 04 3 7 The Bifactor Model The item bifactor model is special multidimensional IRT mod
157. is for specifying the path and file name in which item response data is held or to be saved for Simulation runs The number of 151 cases in the data file is set using the N statement The File and N commands are listed as required here but there is one exception When the engine is placed under Scoring mode and scoring method SSC is selected the input file is optional when only the score conversion table is made without scoring individuals Varnames assigns labels to variables by which they will be referred to later in commands and in the output Variable names are separated by commas following the equals sign of the Varnames statement When variable names end with consecutive integers e g vl v2 v100 a short cut is available that can be both time and space saving Rather than listing all the variables a dash between the name of the first variable and the last variable i e v1 v100 will indicate to flexMIRT that all variables between v1 and 100 are desired The optional Select statement allows for a subset of the variables initially read from the data file to be submitted to IRT analysis The default for Select is to use all variables defined in Varnames Missing values may be represented by a specific number in data files The default missing value is 9 but that may be modified with the Missing com mand However missing values may only be numeric values between 127 and 127 Common missing value symbols such as a dot or a
158. is for the general factor with all item loading onto it and the next 7 columns represent the group specific factors onto which only a subset of items load Turning now to the group specification in the bottom row of the table the first column signifies that this line is supplying group information due the value of 0 Labeling assigning the group number and the number of factors in the model all similar to what was done in the item rows come next In Column 5 we enter information regarding the type of population distribution prior We have chosen Normal priors via the option 0 Because we have chosen Normal priors all that is left to specify are the latent variable means for each of the eight factors followed by the unique elements of the covariance matrix 4 7 Simulating Group Differences As seen in the multiple group calibration examples each group has its own self contained subsection within the lt Groups gt section While this can lead to some redundancy in the naming of variables etc it does permit greater flexibility For example flexMIRT does not require that the number of items the number of categories the kind of IRT models employed or the number of latent dimensions to remain the same across groups One may specify one group to have a normal prior distribution and use empirical histograms in the other group In particular one may even set up a multilevel model for one group and a single level model for the other
159. ixes the item slopes to 1 0 and then frees the estimation of the variance of the latent variable effectively creating a Rasch type model Fix vi v5 Slope Value vl v5 Slope 1 0 Free Cov 1 1 For applying prior distributions the format is quite similar to the other constraints but with the addition of a distribution type and its associated pair of parameter values after a colon The syntax still follows the general 160 constraint format of type group items parameter with the addition of the requested prior distribution following a colon In the across groups constraint example previously used we specified a beta prior for the guessing parameters The prior chosen was a beta distribution with o 1 1 0 and 8 1 4 0 As noted in Table 7 1 the Normal distribution and the logNormal distributions are also available when assigning priors For the Normal the prior mean and standard deviation are needed For the logNormal the prior mean and standard deviation on the logarithmic scale are needed For the Normal prior if it is used on the guessing parameter it is in fact imposed as a logit normal prior on the logit of the guessing probability 161 CHAPTER 8 Models In this chapter some more technical aspects of the IRT model implemented in flexMIRT are discussed We begin with a generalized formulation focusing on the linear predictor portion of the model More specific issues are discussed next 8
160. ixture problem Tech Rep No 1383 The French National Institute for Research in Computer Science and Control Chen W H amp Thissen D 1997 Local dependence indices for item pairs us ing item response theory Journal of Educational and Behavioral Statis tics 22 265 289 Choi H J Rupp A A amp Pan M 2013 Standardized diagnostic assess ment design and analysis Key ideas from modern measurement theory In M M C Mok Ed Self directed learning oriented assessments in the Asia Pacific pp 61 85 New York NY Springer Cressie N amp Read T R C 1984 Multinomial goodness of fit tests Journal of the Royal Statistical Society Series B 46 440 464 de la Torre J amp Douglas J A 2004 Higher order latent trait models for cognitive diagnosis Psychometrika 69 333 353 Delyon B Lavielle M amp Moulines E 1999 Convergence of a stochastic approximation version of the EM algorithm The Annals of Statistics 27 94 128 Edwards M C 2009 An introduction to item response theory using the Need for Cognition scale Social and Personality Psychology Compass 3 4 507 529 Edwards M C amp Cai L 2011 July A new procedure for detecting depar tures from local independence in item response models Paper presented at the annual meeting of American Psychological Association Wash ington D C Retrieved from http faculty psy ohio state edu edwards documents APA8 2 11 pdf Gi
161. l is fitted in each country which permits the decomposition of between school versus within school variance 51 Output 3 2 Two Level Model Means and Variances PISA 2000 Math Data Two Level Unidimensional 31 Items US Data 2115 Students 152 Schools IE Data 2125 Students 139 Schools Summary of the Data and Control Parameters Group USA Ireland Missing data code 9 9 Number of Items 31 31 Number of L 2 Units Number of L 1 Units Number of Dimensions Between Dimensions Within Dimensions Group Parameter Estimates Group Label P mu 1 e PH mu 2 1 USA 0 00 0 00 2 Ireland 79 0 36 0 00 Latent Variable Variance Covariance Matrix for Groupl P Theta 1 s e P Theta 2 s e 1 0 52 0 00 1 00 Latent Variable Variance Covariance Matrix for Group2 Ireland P Theta 1 s e Pi Theta 2 s e 80 0 11 0 00 00 We can see from the output above that the summary of data shows the mul tilevel syntax has been correctly interpreted On average the overall school level achievement in Ireland is 0 36 standard deviations higher than that of the US The more interesting result however is in the variance decomposi tions In the US with a fixed within school variance of 1 0 for identification the between school variance is estimated as 0 52 This implies an intra class correlation of 0 52 1 0 52 0 34 showing a significant amount of school level variations in the US On the other hand the intra class correlation for Ireland
162. l would take a non trivial amount of computing time flexMIRT is able to by pass individual scoring and produce only the neces sary table The syntax to do so is straight forward and presented below Example 3 4 2 Group 3PL Summed Score to EAP Conversion Table lt Project gt Title G341 19 Description 12 Items 3PL 2 Groups Summed Score to EAP from Parameter Estimates Using Estimated Empirical Histogram Prior Make Table Only lt Options gt 44 Mode Scoring ReadPRMFile g341 19 calib EH prm txt Score SSC lt Groups gt Grade4 Varnames v1 v12 Z Grade3 Varnames vi v12 Constraints Just as a typical scoring run we have specified the parameter file now us ing parameters obtained from the multiple group calibration which estimated theta via an empirical histogram We have set the scoring method to produce the desired SSC Summed Score to EAP Conversion table In the Groups section we still provide labels and variable names for both groups but the sample size and data file names are omitted because they are not needed for producing the conversion table without scoring any individuals 3 4 DIF Analyses Up to this point the multiple group examples have assumed the invariance of item parameters across groups However when multiple groups are involved it is generally prudent to check that assumption by looking for differential item functioning DIF between groups flex MIRT
163. le the WinFlexMIRT call would be WinflexMIRT exe r C Users VPG Documents FL analyses Final scale FL_20item flexmirt Once a command file has been successfully submitted the same progress information that is shown in the engine pane of the flexMIRT graphical user interface will be printed in the CLI i a 1 3764 29219 6139 1 3569 29219 6139 1 3381 29219 6138 1 3194 29219 6138 1 3013 29219 6138 1 2828 29219 6138 1 2654 29219 6138 1 2475 29219 6138 1 2298 29219 6137 1 2122 29219 6137 1 1952 29219 6137 1 1781 29219 6137 1 1618 29219 6137 1 1456 29219 6137 1 1292 29219 6136 1 1134 29219 6136 1 8975 29219 6136 1 0820 29219 6136 1 0666 29219 6136 1 0513 29219 6136 1 0365 29219 6136 1 0034 29219 6136 1 80027 29219 6135 346 0 9905 29219 6135 AINAN AIN AN AIN AIN AN AN AN AN AN AN AN AON AN AEN AN AN AN AN I AI I A 41 gt Computing item parameter covariance matrix Computing goodness of fit statistics Writing output file FL_2 item prm txt Writing output file FL 2Bitem irt txt Initializing integration methods done Initializing segment structures done Initializing item structures done Initializing data structures done Writing output file FL_2 item ssc txt E Finished in 673ms When the run has completed successfully the output files may be found in the folder in which the command file is located 176 B 2
164. le group bifactor or two tier modeling as detailed in Cai Yang and Hansen 2011 3 7 1 Bifactor Model Using the Reparameterized Nominal Cate gories Model Here we use the full 35 item Quality of Life data to illustrate a bifactor analysis specifying a general QOL factor and 7 specific factors to capture the subdomains mentioned earlier Gibbons et al 2007 reported a graded bifactor analysis of this data set using the original 7 categories and we will 56 replicate this structure but employ a full rank nominal model Example 3 9 Bifactor Structure Nominal Model lt Project gt Title QOL Data Description 35 Items Bifactor Nominal Model lt Options gt Mode Calibration Quadrature 21 5 0 Etol le 4 Mtol le 5 Processors 2 lt Groups gt Group1 File QOL DAT Varnames vl v35 N 586 Ncats vl v35 7 Model vi v35 Nominal 7 Dimensions 8 Primary 1 lt Constraints gt Fix vi v35 ScoringFn 1 Fix vi v35 Slope Free vi v35 Slope 1 Free v2 v5 Slope 2 Free v6 v9 Slope 3 Free v10 v15 Slope 4 Free v16 v21 Slope 5 Free v22 v26 Slope 6 Free v27 v31 Slope 7 Free v32 v35 Slope 8 With regards to the bifactor structure in the lt Groups gt section a multidi mensional model with 8 dimensions is first specified This is the total number of domains factors both general and specific The Primary command spec ifies out of the to
165. licenses on your various computers with a purchased license users are allowed up to 3 separate installs of the program as well as make payments for new or renewed licenses J MIRT Home Purchase Support My Account Contact ex Vector Psychometric Group Sign In Email Address New User Password Register Here Forgot your password Log On After selecting Register Here you will be taken to the registration page and asked to enter your name email address and any affiliation information you would like to supply 167 Home Purchase Support My Account Contact flexMIRT Vector Psychometric Group Create a New Account Account Information Name Jane Doe Email address You will use this to verify your account and login flexmirt vpgcentral com Institution Vector Psychometric Group LLC Department Psychometrics Once you have submitted this information via the Register button an email confirming your registration information including the initial password for your flexMIRT account will be sent to the address you provided The con firmation email will originate from info flexmirt com please add this email address to your safe list and if you have not received an email after 10 minutes check that the email has not been redirected to your spam folder With the password obtained from the email you may now log in to your flexMIRT account You may change your password using the Change Pass wo
166. lighted Example 2 7 Nominal Model Calibration lt Project gt Title QOL Data Description 35 Items Nominal lt Options gt Mode Calibration Etol 1e 4 Mtol 1e 5 Processors 2 lt Groups gt Group1 File QOL DAT Varnames vi v35 N 586 Ncats vi v35 7 20 Model vi v35 Nominal 7 lt Constraints gt Fix vl v35 ScoringFn 1 In the lt Groups gt section we have identified the data file named the variables specified the number of response options and chosen to fit the Nominal model with 7 categories to the data In the Options section the convergence cri teria for both the E step and M iterations of the EM algorithm have been adjusted from the default values by the Eto1 and Mtol commands respec tively so that a tightly converged solution can be obtained Because the nominal categories model estimates parameters that differ from the models previously discussed excerpts from the item parameter tables will be presented from the output and briefly discussed 30 6 00 97 4 00 ec E 00 04 i 00 vv x 00 ev 00 e 90 00 s6 00 TA T I 10393e9 Toqe ue21 Tdnozg 10g sue4 TRutuoyn sI q ueredq ZL6T xX2og Teur3r I0 1 0 Liv 4907 ST O v O STy pueilp SEA se OL 0 SOT S T ZTO 97 0 eoy PuerL ve vt tv O Te v9 O 0 vl O vero 61 pue e c vS 0 6 vo T 8 sto sto L pueda TA T euuel d Z euueS gq a s T e
167. ls CDMs diagnostic classification models cognitive psychometric models restricted latent class models and structured IRT models among others The goal of this chapter is not to provide a didac tic on CDMs interested readers are directed to Rupp Templin and Henson 2010 for a thorough discussion of CDMs and related models but to demon strate for those already familiar with the models how flexMIRT may be used to fit them 99 6 1 General CDM Modeling Framework Complete coverage of the models flex MIRT fits to CDMs is provided in the Models chapter but we will briefly introduce the models here Figure 6 1 provides a schematic for a generic flexMIRT CDM with a single higher order factor to facilitate the discussion of the mathematical equations Figure 6 1 Generic Higher order CDM The extended CDMs used by flexMIRT combine a linear predictor with a link function The linear predictor may be represented as K K k1 1 n a y Bk Tk y 25 Pry ka Tk Tko 6 1 ky 1 ki 1 ko 1 p higher order interaction terms gt k k k 1 where a B and are item parameters zs are 0 1 attributes and s are continuous latent variables The kernel linear predictor 7 can be turned into item response probabilities through the use of appropriate link functions such 100 as the logit presented below 1 P ui 1 x E mi 1 x 1 exp n For the higher order portion of the CDM flexMIRT is model
168. lues for each item are presented on the appropriate row and the Test Information Function TIF values which sum the item information values at each theta value plus the contribution from the prior are printed towards the bottom of the table Additionally the expected standard error associated with each theta value is printed Finally the marginal reliability which is defined as 1 minus the averaged error variance divided by the prior variance is reported underneath the table After the information and reliability section the overall goodness of fit GOF indices are presented which appear in the next output box Output 2 4 2PLM Calibration Output Goodness of Fit Indices Statistics based on the loglikelihood of the fitted model 2loglikelihood 33408 05 Akaike Information Criterion AIC 33456 05 Bayesian Information Criterion BIC 33598 92 Full information fit statistics of the fitted model The table is too sparse to compute the general multinomial goodness of fit statistics Limited information fit statistics of the fitted model The M2 statistics were not requested When no option is specified the GOF reporting will provide only basic fit indices These include the values for the 2xLog Likelihood the Akaike Information Criterion AIC Bayesian Information Criterion BIC and when appropriate the Pearson X statistic labeled X2 the likelihood ratio statis tic labeled G2 their associated degrees of freedom th
169. m so it is specified as a graded model with two response categories while item 1 in both Groups 1 and 2 are 3PL items DIF also occurs for Item 9 The same generating IRT model is used and the guessing and discrimination parameter values are identical across groups but in Group 2 the difficulty parameter value is 0 5 higher in the b metric than the value in Groups 1 and 3 Mean differences among the groups are also generated The last 3 rows refer to Groups 1 through 3 respectively After labeling specifying group membership and declaring the number of factors normal population distributions are chosen for all groups via the Os in the 5 column The sixth column provides the population means and the seventh column provides population variances 76 4 8 Multilevel Bifactor Model Simulation Our final example will illustrate how one may simulate item responses accord ing to a multilevel bifactor model in two groups We will generate data for 12 dichotomously scored graded items Example 4 7 Simulation Multilevel Model with Bifactor Structure Project Title Two level Bifactor Model Two Groups Description 12 Items 4 Group Factors 100 L2 units 10 respondents within each lt Options gt Mode Simulation Rndseed 7471 ReadPrmFile MGsimL2 prm txt Groups G17 File simL2 g1 dat Varnames v1 v12 Dimensions 6 Between 1 1000 Nlevel2 100 1 27 File simL2 g2 dat
170. may be fitted with either the 2PL model if guessing is assumed to be negligible or with the 3PL model which explicitly models guessing with an additional parameter Example syntax for fitting the 2PL model is presented first Example 2 1 2PL Calibration Syntax Project Title 2PL example Description 12 items 1 Factor 1 Group 2PL Calibration lt Options gt Mode Calibration lt Groups gt Group1 File g341 19 dat Varnames v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 N 2844 Ncats vi vi2 2 Model vi v12 Graded 2 lt Constraints gt Following the previous discussion section headers denoted by their enclosure within lt and gt precede the commands The lt Project gt header is followed by a title and description both of which will be printed in the output The lt Options gt section by the Mode command declares this to be a calibration run After the lt Groups gt section header a name for the first and only group is provided which is enclosed within s Then the name of the data file File variable names Varnames number of examinees N number of response categories in the data for each item Ncats and the IRT model to be fit to each item Model are provided In this example only the file name was provided in the File statement When the command file flexmirt and the data file are in the same folder the name of the data file is suffic
171. mulation Rndseed 7474 lt Groups gt AG File sim a dat Varnames vi v4 1000 Model vi v4 Graded 3 lt Constraints gt Value v1 Slope 0 7 Value v2 Slope 1 0 Value v3 Slope 1 2 Value v4 Slope 1 5 Value v1 v2 Intercept 1 2 0 Value v1 v2 Intercept 2 1 0 Value v3 v4 Intercept 1 0 Value v3 v4 Intercept 2 1 0 One can see in the Options section flex MIRT is set to run in Simu lation mode When this is case a random number seed must be supplied via the Rndseed statement The group specification remains the same as in cal ibration runs In this case the File statement instructs flex MIRT 9 to save the simulated item responses to simla dat The variables are named and the total number of simulees is set to 1000 The Model statement sets the gener ating IRT model to the graded response model with 3 categories Next in the Constraints section the generating parameter values are set For instance the first constraint says that the value of the slope parameter for item v1 should be set to 0 7 The 5th constraint specifies that the first intercept term for item vl and v2 should be equal to 2 0 In a single group the group name can be omitted so Value v1 Slope 0 7 and Value G v1 S10pe 0 7 would have the same effect For multiple group simulations the group name must be specified in each Value statement 62 Output 4 1 Simulation Control Output Graded Model Syntax Only
172. n Empirical Histogram Prior Simulation Bifactor Structure From Existing Parameter File Simulation DIF and Group Mean Differences Simulation Multilevel Model with Bifactor Structure 12 16 19 5 1 5 2 5 3 5 4 6 1 6 2 6 3 6 4 6 5 6 6 6 7 6 8 7 1 1 2 7 3 7 4 7 5 7 6 T 7 8 7 9 C 1 Options Engine Mode Settings to Call MH RM Estimation and MH RM Specific Options e fata e a a MHRM 4 Factor MIRT Model MHRM Two level MIRT Model Fixed Effects Calibration x domm oer ERR ERA eed lt Options gt Groups and lt Constraints gt CDM Specific Op DIOUS AP1 355 Gases Bon ids de ehh Y qub p be u a Mex RV ba a ese C RUM Model Fit To Simulated Data DINA Fit to Simulated Datos alias Sa Red DINO Fit to Simulated Datars iz bow eae xx Testlet Structure with DINA model Rupp Templin amp Henson Example 9 2 Rupp Templin amp Henson Example 9 2 without D group de la Torre Subtraction Syntax 0 6 446 452 pias a Project section commands Options Engine Mode and Output Statements Options Technical Specifications Options GOF and LD Indices Commands lt Options gt DIF Analysis Commands lt Options gt MH RM Specific Options Groups General Data Model Descriptors Groups
173. n number When all the syntax files have been identified and a name for the output file is specified in the Save Simulation Output to blank the Run button will be enabled Figure 4 3 Completed ISF pop up window File Edit FiexMIRT Help D Gg Ba run Progress FlexMirt Simulation Number of Replications 5 2 Data Generating Model E PnC SIM test sim1 flexmirt Fit Model 1 EAPnC SIM test Vit 1 flexmirt E Fit Model 2 E Fit Model 3 Save Simulation Output to Sim 1 output tx Once the simultion has been submitted flexMIRTO will begin the process of simulating data and fitting the specified model s to the data The progress of each replication is printed in the engine viewing pane as with any one off simulation or calibration When the specified number of replications have completed a single output file is created The format of the ISF output file is optimized to be read into some other software program e g SAS and as such appears quite different from the normal flex MIRTO output The output file has no labeled sections or column headers etc Due to this the layout of the ISF output file will be described in detail 8l 4 9 2 The Simulation Output File The first 6 columns in the output for every model summary will contain the same information Column 1 has the simulation replication number Column 2 prints what is called the return code and is an indicator of successful repli cations
174. ndard errors for those probabilities respectively The entries that have been moved to the second line of each observation contain the estimated attribute posterior variance covariance matrix For ob servation 1 those values would enter the matrix as al a2 a3 a4 al 0 050755 a2 0 023774 0 109961 a3 0 024396 0 033344 0 137477 a4 0 029047 0 049426 0 051856 0 234379 In what we have made the third line for each respondent are the posterior probabilities for each of the possible attribute profiles As noted earlier with 4 attributes there are 16 possible profiles and therefore 16 values on the third line for this example For observation 1 the posterior probability of being in attribute profile 1 is 0 548 the probability of possessing attribute profile 2 is 0 204 and so on The order in which the attribute profile probabilities are given matches the order of the probability profiles used in the output table Diagnostic IRT Attributes and Cross classification Probabilities Looking back at the output presented for this example we find that attribute profile 1 corresponds to 0 0 0 0 meaning no attributes are mastered and attribute profile 2 corresponds to 0 0 0 1 meaning only attribute 4 has been mastered etc Finally in what has been moved to the fourth line for each respondent the most likely attribute profile is listed followed by the individual s estimated theta score for the higher order latent variable as well a
175. nto the interaction of attributes 3 and 4 As noted when introducing the model for items that depend on more than one attribute only the highest order interaction terms is allowed to be non zero so we do not free items 13 15 onto the main effects of attributes 3 or 4 Because the majority of the output has the same structure as the previous example s regardless of specific CDM fit to items we will only briefly discuss outputted estimates For this example we highlight the output the relates to the item parameters and the sum score to EAP conversion table in the ssc file that is printed due to Score SSC 112 O CN 0 i0 n gt 6A ga A ga ga pa ga za TA Teqe o 2000000000000 c000o0o0o0oooooooo c0c02000000000000 c0c02000000000000 OQ m i O 00 oo S oo o oo o oo Q Q 0 s 10 ON 0 A a a a 00 0 0 0 0 0 0 0 o 0 00 0 10 0 0 0 0 0 0 e 10 E o E o E 10 00 o E o E o E o o o STA TA epa TTA TTA OTA 6 8 LA ga SA v gA TA TA TeqeT uWe3 5 I dnozo 103 sweat Idz o o o o o o o oo ooo0 o o o o o o o o o o oo ooo0 o o o o T G c i ON 00 o o dooooooooooooooo o Lo sojeurns 1ojeurereq Wy 3nd3no vNIG 9 mdmo 113 The output excerpt presents the estimated parameters for the items with estimated versus fixed at zero loadings mirroring the Q matrix s
176. o a better window and converged standard errors The Perturbation command controls a value used by both the forward difference and Richardson extrapolation methods of SE estimation These methods require the selection of a perturbation constant that is used in the denominator of the finite difference approximation the choice of this perturba tion constant may exert substantial influence on the accuracy of the final item parameter SE estimates Tian Cai Thissen amp Xin 2012 As noted above the flex MIRT default of this perturbation constant is set to 0 001 the value found to produce the consistently least poor estimates from the two methods when studied in both simulations and as applied to real data Tian et al 2012 While the forward difference and Richardson extrapolation methods are available to users Tian et al 2012 found them to be less accurate than the supplemented EM algorithm SEs when applied to standard IRT models with no noticeable savings with respect to time when the smart window for the supplemented EM SEs was employed The information function for the individual items and full scale is printed in the general output i e the irt txt file using theta values from 2 8 to 2 8 changing by steps of 0 4 The SaveInf option addressed earlier allows the user to output the information function values to a separate inf txt file which is useful for additional computations external to lexMIRT or when informa
177. o model As stated before to determine the reporting order of parameters a single run of any model being fit is strongly recommended before using the ISF 84 CHAPTER 5 Metropolis Hastings Robbins Monro Estimation for High Dimensional Models Although Bock and Aitkin s 1981 EM algorithm made IRT parameter estima tion practical the method does have shortcomings with the most noticeable one being its limited ability to generalize to truly high dimensional models This is due primarily to the need to evaluate high dimensional integrals in the likelihood function for the item parameters As the number of dimensions of a model increases linearly the number of quadrature points increases expo nentially making EM estimation unwieldy and computationally expensive for models with more than a handful of latent factors A Metropolis Hastings Robbins Monro MH RM Cai 2010b 2010c al gorithm has been implemented in flexMIRT to allow for the estimation of higher dimensional models We will briefly discuss the algorithm to acquaint those users less familiar with this recently introduced method cover the avail able flex MIRT options and syntax commands that are specific to this esti mation routine and then provide working examples 5 1 Overview of MH RM Estimation In brief the MH RM algorithm is a data augmented Robbins Monro type RM Robbins amp Monro 1951 stochastic approximation SA algorithm driven by the random imputa
178. odel Yes 32 SavePRM Yes lt Groups gt Group1 File PreschoolNum dat Varnames Identify Match Identify3 Identify4 Match3 Match4 Select Identify Match N 592 Ncats Identify Match 4 Code Identify Match 0 3 4 34 0 1 2 3 Model Identify Match Nominal 4 Ta Identify Match Identity Tc Identify Match Identity lt Constraints gt Fix Identify ScoringFn 2 Equal Match ScoringFn 2 Groupi Match ScoringFn 3 Equal Groupi Match Intercept 1 Group1 Match Intercept 2 Within the Options section there are several new commands The method of standard error calculation has been changed from the default of empirical cross product approximation Xpd to Supplemented EM algorithm keyword is SEM see Cai 2008 We have also requested the full version of the limited information GOF M statistic Cai amp Hansen 2012 Maydeu Olivares amp Joe 2005 and additionally specified that the zero factor Null Model should be estimated and the associated incremental GOF indices be printed In the Groups section we have identified the data file named the vari ables selected only the pseudo items to retain for analysis specified the num ber of response options recoded the response options into zero based sequen tial but nevertheless nominal codes and chosen to fit the Nominal model with 4 categories to the data The statements Ta and Tc are specific to the nominal model and re
179. ods Cai and Wang 2013 found that the Wald 2 procedure implemented via DIF TestAll in flexMIRT tends to result in inflated Type I error rates and suggested that this procedure may be best used for identifying anchors for a subsequent DIF analysis that examines only candidate items for DIF rather than as a final DIF analysis from which conclusions regarding item functioning are drawn Additionally it is also recommended that when using the TestA11 DIF sweep option that the standard error method be changed from the default Conclusions reported in Woods et al 2013 were based on the use of supplemented EM SEs SE SEM flexMIRT may also be used for more traditional DIF testing of test can didate items against designated anchor items Only relevant excerpts of the command file g341 19 difcand flexmirt are presented Again the full file is available on the support website Example 3 6 DIF Test Candidate Items Excerpts lt Options gt DIF TestCandidates DIFcontrasts 1 0 1 0 lt Groups gt ZGrade4 File g341 19 grpl dat Varnames vl v12 N 1314 Ncats vl v12 2 Model vi vi2 ThreePL DIFitems vl v2 ZGrade3 File g341 19 grp2 dat Varnames vl v12 48 N 1530 Ncats vl v12 2 Model vi vi2 ThreePL DIFitems vl v2 lt Constraints gt Free Grade3 Mean 1 Free Grade3 Cov 1 1 Equal Grade3 v3 v12 Guessing Grade4 v3 v12 Guessing Equal Grade3
180. om our bifactor analysis in Ex ample 3 10 to simulate items with a bifactor structure from the 2PL model As noted previously when not employing Value statements only the very ba sic structure of the analysis is supplied in the simulation syntax file and the details of the model are left to the parameter file Table 4 4 presents a labeled version of the parameter file QOLbifac prm txt for illustrative purposes Example 4 5 Simulation Bifactor Structure From Existing Parameter File Project Title QOL Data Description Simulation From PRM File 35 Items Bifactor T2 lt Options gt Mode Simulation ReadPRMFile QOLbifac prm txt Rndseed 7474 lt Groups gt Group1 File QOLbifacsim DAT Varnames vi v35 Dimensions 8 Primary 1 N 1000 lt Constraints gt Table 4 4 Labeled Parameter File for Bifactor Model Simulation Type Label Grp Fac Model Cat c a1 a2 a3 04 as a6 a7 ag 1 vl 1 8 2 2 2 13 2 26 0 00 0 00 0 00 0 00 0 00 0 00 0 00 1 v2 1 8 2 2 3 78 2 31 2 51 0 00 0 00 0 00 0 00 0 00 0 00 1 v3 1 8 2 2 1 60 1 62 1 42 0 00 0 00 0 00 0 00 0 00 0 00 1 v4 1 8 2 2 376 386 4 89 0 00 0 00 0 00 0 00 0 00 0 00 1 v5 1 8 2 2 2 87 2 77 3 08 0 00 0 00 0 00 0 00 0 00 0 00 1 v6 1 8 2 2 0 86 2 77 0 00 320 0 00 0 00 0 00 0 00 0 00 1 v7 1 8 2 2 1 38 1 38 0 00 1 50 0 00 0 00 0 00 0 00 0 00 1 v8 1 8 2 2 049 2 60 0 00 2 86 0 00 0 00 0 00 0 00 0 00 1 v9 1 8 2 2 0 80 1 78 0 00 1 97 0 00 0 00
181. ommands introduced in this example add several 25 new parts to the output file Output 2 12 1PL Grouped Data Output Item Parameters and Factor Loadings LSAT 6 5 Items 1PL N 1000 Grouped Data 2PL Items for Group 1 Groupi Item Label P a s e PH 1 vi 0 76 0 00 v2 0 76 0 00 v3 0 76 0 00 v4 0 76 0 00 v5 0 76 0 00 LSAT 6 5 Items 1PL N 1000 Grouped Data Factor Loadings for Group 1 Groupi Item Label lambda 1 1 vi 0 41 v2 0 41 v3 0 41 v4 0 41 v5 0 41 The item parameters are printed in their customary order but note that the 1PL equal slope constraint specified in the syntax has been interpreted correctly as evidenced by all values in the a column being equal The factor loadings are presented below the typical item parameter table with loadings appearing in the lambda1 column The next new portion of the output are the values and tables associated with the summed score based item fit diagnostic e g Orlando amp Thissen 2000 which are printed as a result of the GOF Complete and SX2Tbl Yes commands respectively The S X value and table for the first item is presented When the p value associated with the S X value is greater than the a level typically 0 05 the item should be examined for possible misfit 26 Output 2 13 1PL Grouped Data Output Item Fit Diagnostics Orlando Thissen Bjorner Summed Score Based Item Diagnostic Tables and X2s Group 1 Groupi Item 1 S X2 1 48 5 p lt 0
182. onEffects 2 to generate those dimensions automat ically To incorporate the testlet structure into our model we will be fitting a bifactor type model in which the CDM main effect and interactions are the primary general dimensions and the testlet effects are considered the specific dimensions To indicate this to flex MIRT we set Primary 10 which is the total number of CDM dimensions 4 main effects 4 6 interactions and set the total number of dimensions for the model equal to 14 10 CDM dimensions 4 testlet effect dimensions In the Constraints section the first group of statements is used to assign items to the appropriate interaction terms based on the Q matrix for the DINA model The second group of constraints beginning with Free G vi v6 Slope 11 is used to assign items to their testlet dimension As noted earlier the first 10 latent dimensions are taken up by the CDM so the first available dimension for a testlet effect is dimension 11 hence Slope 11 in the noted Free statement The specific dimensions are specified as a restricted bifactor model where the testlet effect is presumed to be a testlet specific random intercept term This requires the equality of the slopes within factors and is accomplished with the final group of constraints With the final Equal constraint we are fitting a Rasch type model to the higher order dimensions constraining the slopes of attribute 1 through attribute 4 in group D to equ
183. onal IRT models As for the item types flex MIRT can estimate any arbitrary mixtures of 3 parameter logistic 3PL model logistic graded response model which includes 2PL and 1PL as special cases and the nominal categories model including any of its restricted sub models such as generalized partial credit model partial credit model and rating scale model for both single level and multilevel data in any number of groups flex MIRT also has some of the richest psychometric and statistical fea tures New to version 2 0 we have implemented features that allows users to fit Cognitive Diagnostic Models CDMs In addition to fully implement ing many recently developed multilevel and multidimensional IRT models flex MIRT supports numerous methods for estimating item parameter stan dard errors Supplemented EM empirical cross product approximation Fisher expected information matrix Richardson extrapolation forward difference and sandwich covariance matrix A multitude of model fit statistics for dimen sionality analysis item fit testing and latent variable normality diagnosis are included in flexMIRT Its multiple group estimation features easily facilitate studies involving differential item function DIF and test linking including vertical scaling A key innovation in flex MIRT is its ability to relax the ubiquitous mul tivariate normality assumption made in virtually all multidimensional IRT models With an extended d
184. or Grade3 vi v12 Guessing Beta 1 0 4 0 Two groups Grade4 and Grade3 are specified in the Groups sec tion n important point to highlight is that the data for each group should be stored as individual files To that end g341 19 dat is split into 2 new files g341 19 grp1 dat and g341 19 grp2 dat As may be seen in the N statements for each group the original sample size of 2844 is now divided between the two groups Within each group the basic specification state ments VarNames Ncats etc remain the same There are now two sets of them one per group new option not specific to multiple group analysis is BetaPriors This statement imposes a prior on the item uniquenesses in the form of a Beta a 1 0 distribution with a user specified a parameter in this case a 1 5 see Bock Gibbons amp Muraki 1988 41 Within the lt Constraints gt section we are allowing the mean and variance of the second group Grade3 to be estimated by freeing those parameters via the first two constraint statements Additionally we have specified a model that assumes measurement invariance by constraining the item parameters to be equal across the two groups Finally we supplied a Beta distribution prior for the guessing parameters There are new commands encountered in the Options section as well It should be noted that these newly encountered commands are not specific in any way to the multi group format and ar
185. oups 12 Items 4 Group Factors 100 L2 units 10 respondents within each Summary of the Data and Control Parameters Group G1 G2 Missing data code 9 9 Number of Items 12 12 Number of L 2 Units 100 100 Number of L 1 Units 1000 1000 Number of Dimensions 6 6 Between Dimensions 1 Within Dimensions 5 5 Miscellaneous Control Values Random number seed 7471 2PL Items for Group 1 G1 Item a1 a2 a 3 a 4 a5 a6 c 1 2 00 2 00 1 50 0 00 0 00 0 00 1 50 2 1 50 1 50 1 50 0 00 0 00 0 00 0 50 3 1 00 1 00 1 20 0 00 0 00 0 00 0 50 4 2 00 2 00 0 00 1 20 0 00 0 00 1 50 5 1 50 1 50 0 00 1 50 0 00 0 00 1 50 6 1 00 1 00 0 00 1 80 0 00 0 00 0 50 7 1 50 1 50 0 00 0 00 1 20 0 00 0 50 8 1 50 1 50 0 00 0 00 1 50 0 00 1 50 9 1 00 1 00 0 00 0 00 1 50 0 00 1 50 10 2 00 2 00 0 00 0 00 0 00 1 80 0 50 11 2 00 2 00 0 00 0 00 0 00 2 00 0 50 12 1 00 1 00 0 00 0 00 0 00 1 50 1 50 Group Latent Variable Means 78 Group 1 2 Label G1 G2 mu 1 mu 2 mu 3 mu 4 mu 5 mu 6 0 00 0 00 0 00 0 00 0 00 0 00 0 20 0 00 0 00 0 00 0 00 0 00 Latent Variable Variance Covariance Matrix for Groupi G1 Theta 1 Theta 2 Theta 3 Theta 4 Theta 5 Theta 6 50 00 00 00 00 00 1 00 0 00 0 00 0 00 0 00 1 00 0 00 0 00 0 00 1 00 0 00 1 00 0 00 0 00 1 00 Latent Variable Variance Covariance Matrix for Group2 G2 Theta 1 Theta 2 Theta 3 Theta 4 Theta 5 Theta 6 0 00 00 00 00 00 0 0 0 0 0 25 1 00 0 00 0 00 0 00 0 00
186. pe of contrast matrix used for the intercepts Tc and the estimated y parameters For convenience the parameters are also reported in the original parameterization in the final table of the output presented The next example employs a set of data regarding numerical knowledge from pre school aged children most notably analyzed in Thissen and Stein berg 1988 to illustrate the concept of testlets As described by Thissen and Steinberg 1988 the data were collected from 4 tasks that involved asking 592 young children to identify the numerals representing the quantities of 3 and 4 and separately match the numeral for each number to a correct quantity of blocks From these items called Identify3 Identify4 Match3 and Match4 pseudo items were created that summarized a child s performance on both numbers For the new pseudo items Identify and Match the response options are 0 child could do the requested action for neither number 3 child could identify match 3 only 4 child could identify match 4 only and 34 child could identify match both 3 and 4 successfully We will again fit the Nominal Model but with some additional constraints imposed Example 2 8 Nominal Model Calibration 2 Project Title Pre School Numerical Knowledge Description 2 Redefined Items Nominal lt Options gt Mode Calibration SE SEM Etol 5e 5 Mtol 1e 9 GOF Extended M2 Full SX2Tb1 Yes HabermanResTbl Yes FitNullM
187. pecifications The factors labeled a1 a4 represent main effects for attributes 1 4 and a5 a10 represent the requested 2nd order interactions Items 1 12 have estimates for main effects on only the attributes specified in the Q matrix and item 13 15 have estimates on only the interaction of attributes 3 and 4 which is labeled as a10 in the parameter table One may notice that di mensions a5 a9 that have no items assigned to them these dimensions correspond to the unused second order interaction effects that were created via the InteractionEffects keyword The estimated item values reported are in the LCDM parameterization but the classic DINA parameters of guess ing and slippage may be obtained by conversion Guessing is found from the reported flex MIRTO parameters as ezp c 1 exp c and one minus the slippage term 1 s is found as ezp c a 1 ezp c a In the ssc file that was generated by the scoring request we find tables that provide conversions from sum scores to the probability of mastery for each attribute We present an excerpt of these table below Output 6 4 DINA Summed Score Conversion Tables from ssc File Summed Score to Attribute Profile Probability Conversion Table Summed Score EAP 1 EAP 2 EAP 3 EAP 4 SD 1 SD 2 SD 3 SD 4 P Error Covariance Matrix 0 00 0 001 0 001 0 026 0 005 0 030 0 026 0 158 0 070 0 0000002 0 000876 0 000003 0 000700 0 000115 0 000045 0 024902 0 000017 0 000007 0 000092 0 00494
188. ple Example 3 1 which employed normal priors The syntax is presented below and as with the single group scoring syntax in the lt Options gt section Mode is set to Scoring the type of scores requested is specified MAPs in this case and the file containing the relevant item parameters generated by our previous calibration run is given using the ReadPRMFile statement Because of the distinct groups two sets of data descriptor statements e g syntax for declaring group label data file 43 name variable names and sample size are given Because this example is so similar to the single group scoring runs no further discussion is provided For explications of any of the included syntax statements the user should refer to a previous single group example e g Example 2 3 Example 3 3 2 Group 3PL MAP Scoring from Parameter Estimates Project Title G341 19 Description 12 Items 3PL 2 Groups MAP from Parameter Estimates lt Options gt Mode Scoring ReadPRMFile g341 19 calib normal prm txt Score MAP lt Groups gt Grade4 File g341 19 grp1 dat Varnames v1 v12 1314 Grade3 File g341 19 grp1 dat Varnames v1 v12 1530 lt Constraints gt It some situations it may be useful to obtain the sum score conversion table without actually having to score data For example when only the SSC table is wanted and the sample size is large enough that calculating scores for every individua
189. ponse data in flex MIRT is zero based i e response options must start at zero and go up The raw item response data file for this example NC Ssim dat labels the categories as 1 through 5 so recoding is necessary The Code statement does this for variables v1 v18 by specifying the original val ues in the first pair of parentheses and following a comma the desired re coded values in a second set of parentheses Thus a command such as Code v1 v18 1 2 3 4 5 4 3 2 1 0 accomplishes both reverse cod ing and makes the recoded item response data zero based Another point of note in this command file is that even though this is a calibration run we have requested MAP scores be saved This combination of commands is a short cut to a combined calibration and scoring run in which the calibration is explicitly requested and the SaveSco Yes command implicitly requests a scoring run using the estimated parameter values from the calibration With respect to the output the primary change from previous examples are the additional parameter values in the item parameter table As may be seen from the two item parameter estimate output tables below there are multiple thresholds or equivalently intercepts for each item For conve nience both of these parameterizations thresholds and intercepts are pre sented in separate tables However only the intercept values will be saved to the prm file if it is requested This behavior can be modifi
190. psilon controls the rate at which the sequence of gain constants converge to zero Specified Epsilon values must be in the range 0 5 1 with a value closer to 1 indicating a faster rate of convergence to zero WindowSize allows the user to set the convergence monitor window size Convergence of the MH RM algorithm is monitored by calculating a window of successive dif ferences in parameter estimates with the iterations terminating only when all differences in the window are less than 0 0001 The default value of the win dow size is set at 3 to prevent premature stoppage due to random variation ProposalStd and ProposalStd2 control the dispersion of the Metropolis proposal densities for the first and second level of the specified model respec tively If a single level model is specified only the ProposalStd command will be used Although default values of 1 0 have been set for both of these commands these values need to be adjusted on a case by case basis The values used will depend on the complexity of the model the number of items the type of items e g dichotmous polytomous and the model fit to the items e g graded 3PLM Nominal The user should choose ProposalStd and ProposalStd2 values so that the long term average of the acceptance rates which is printed for each iteration in the progress window 150 or level 1 and level 2 if applicable are around 0 5 for lower dimensional models lt 3 factors and in the 0 2 0 3
191. ptions gt SmartSEM SStol see Options SStol Stagel see Options Stagel Stage2 see Options Stage2 Stage3 see Options Stage3 Stage4 see Options Stage4 Standard Error SE Method EM algorithm 143 Empirical Cross Product Approx imation 8 33 42 143 Fisher Information Matrix 143 Forward Difference 143 Richardson Extrapolation 105 143 Sandwich Covariance Matrix 143 Supplemented EM 33 143 StartValFromNull see lt Options gt StartValFromNull SX2Tbl see Options SX2Tbl Ta see Groups Ta Tc see Groups Tc TechOut see Options TechOut Thinning see Options Thinning Title see Project Title Value see Constraints Value Varnames see Groups Varnames Weighted Data see Data Format Re sponse Pattern Data WindowSize see Options WindowSize 200
192. raints placed on this model so after the section header is declared the command file syntax ends At the completion of estimation flex MIRTO automatically opens the results in the Output viewer The out put is also saved into a text file named with the command file name and irt txt as the extension The command file used in the first example is 2PLM_example flexmirt and the output is saved to 2PLM_ example irt txt The output is presented in a straight forward fashion so those familiar with IRT and its parameters and fit values should find it easily understandable For completeness however we will review the output and highlight relevant sections Due to the length of the output file it will be separated into parts which are discussed in turn Output 2 1 2PL Calibration Output Summary and Controls flexMIRT R 2 00 64 bit Flexible Multilevel Multidimensional Item Analysis and Test Scorin c 2012 2013 Vector Psychometric Group LLC Chapel Hill NC US 2PLM example 12 items 1 Factor 1 Group 2PLM Calibration Summary of the Data and Dimensions Missing data code 9 Number of Items 12 Number of Cases 2844 Latent Dimensions 1 Item Categories Model Graded Graded Graded Graded Graded Graded Graded Graded Graded Graded Graded Graded e O QO O OOF Q N P N N N N N N N N N N N N N Bock Aitkin EM Algorithm Control Values Maximum number of cycles 500 Convergence criterion 1 00e 0
193. rd link in the upper right hand corner Before you are able to download the installer you must accept the license agreement which can be accessed by clicking the link in the green box 168 Log O Home Purchase Support My Account Contact flexMIRT Vector Psychometric Group flexmirt amp vpgcentral com My Account Change Password Log Off Account Details flexmirt vpgcentral com Jane Doe Vector Psychometric Group LLC Psychometrics License Options Save Change license Type Trial You will be taken to the End User License Agreement Once you have reviewed the agreement there is an I Agree check box at the bottom of the page which you must endorse to continue After you have accepted the license agreement you will be able to access the flex MIRT installer via the newly available Download Software link Home Purchase Support My Account Contact flexMIR Vector Psychometric Group flexmirt vpgcentral com My Account Change Password Log Off Account Details flexmirt vpgcentral com Download Software Jane Doe Vector Psychometric Group LLC Psychometrics License Options 169 After selecting Download a pop up will appear where you will select the version of lex MIRTO appropriate for your computer The flex MIRT installer should be saved to a folder where you will be able to locate it la
194. riation ProposalStd and ProposalStd2 control the dispersion of the Metropolis proposal densities for the first and second level of the specified model respec tively If a single level model is specified only the ProposalStd command will be used Although default values of 1 0 have been set for both of these commands these values need to be adjusted on a case by case basis The values used will depend on the complexity of the model the number of items the type of items e g dichotmous polytomous and the model fit to the items e g graded 3PLM Nominal The user should choose ProposalStd and ProposalStd2 values so that the long term average of the 88 acceptance rates which is printed for each iteration in the progress window or level 1 and level 2 if applicable are around 0 5 for lower dimensional models lt 3 factors and in the 0 2 0 3 range for higher dimensional more complex models Generally speaking increasing the Proposa18td value will result in lowered acceptance rates while decreasing the value will result in higher acceptance rates Users are directed to Roberts and Rosenthal 2001 for optimal scaling choice of dispersion constants and long term acceptance rates of Metropolis samplers MCSize is the Monte Carlo size for final log likelihood AIC and BIC ap proximations 5 3 Examples Utilizing MH RM Estimation All of the calibration examples presented previously in the manual are able to be estimated u
195. rior of the individual latent traits z 0 Y B as the unique invariant distribution e Approximation In the second step based on the imputed data the complete data log likelihood and its derivatives are evaluated so that the ascent directions for the item and latent density parameters can be determined later e Robbins Monro Update In the third step RM stochastic approximation filters are applied when updating the estimates of item and latent den sity parameters First the RM filter will be applied to obtain a recursive stochastic approximation of the conditional expectation of the complete data information matrix Next we use the RM filter again when updat ing the new parameter estimates Cai 2010c showed that the sequence of parameters converges with probability 1 to a local maximum of the likelihood of the parameter estimates given the observed data For more detailed descriptions of the estimation method see Cai 2010b 2010c 86 5 2 MH RM Specific Syntax Options Example 5 1 Options Engine Mode Settings to Call MH RM Estimation and MH RM Specific Options Options gt Mode Calibration Algorithm MHRM RndSeed 7 Imputations 1 Thinning 10 Burnin 10 Stagel 200 Stage2 100 Stage3 2000 Stage4 0 InitGain 1 0 Alpha 1 0 Epsilon 1 0 WindowSize 3 ProposalStd 1 0 ProposalStd2 1 0 MCsize 25000 To call the MH RM algorithm the
196. rization Comparing the point estimates across the two programs one notices some differences but overall the values are quite simi lar Regressing the original guessing parameter estimates onto the converted flexMIRT values gives an intercept value close to zero 0 006 a regression coefficient on 0 97 and an R value of 0 99 A similar comparison of the slip page values also results in a low intercept 0 01 a regression coefficient near 1 0 99 and a high R value 0 98 Given the comparability of the parameter 136 values it is interesting to note that flexMIRT completed its calibration in 20 5 seconds highlighting the extreme efficiency that may be obtained when estimating higher order CDMs via MML Table 6 6 Guessing and Slippage Parameters from flexMIRT and MCMC de la Torre flexMIRT amp Douglas Item Intercept c Slope a g 1 s g 1 s vl 4 59 6 50 0 01 0 87 0 04 0 90 v2 3 42 6 90 0 03 0 97 0 03 0 96 v3 4 49 6 47 0 01 0 88 0 00 0 88 v4 1 25 3 21 0 22 0 88 022 0 89 v5 0 84 2 35 0 30 0 82 0 30 0 82 v6 4 99 8 09 0 01 0 96 0 03 0 96 v 3 86 5 25 0 02 0 80 0 03 0 81 v8 0 20 1 65 0 45 0 81 0 44 0 81 v9 1 63 2 75 0 16 0 75 0 18 0 75 v10 3 51 4 82 0 03 0 79 0 03 0 79 vll 2 66 5 24 0 07 0 93 0 07 0 93 v12 1 91 4 74 0 13 0 94 0 13 0 96 v13 4 30 5 02 0 01 0 67 0 02 0 67 v14 9 20 5 79 0 04 0 93 0 05 0 93 v15 3 43 5 51 0 03 0 89 0 04 0 90 v16 2 25 4 23 0 10 0 88 0 10 0 88 v17 3 00 4 83 0 05 0 86 0 04 0 86
197. rocessing Pane Output MH RM Stage I AR 0 47 0 AR 0 48 0 AR 0 47 0 AR 0 46 0 AR 0 45 0 AR 0 44 0 AR 0 44 0 AR 0 43 0 AR 0 42 0 AR 0 43 0 AR 0 43 0 dt dt dt dt dt dt dt dt HH o O O Q QO N OoOoooo0000000 MH RM Stage II 1 AR 0 41 0 2 AR 0 41 0 3 AR 0 42 0 4 AR 0 42 0 5 AR 0 41 0 MH RM Stage III 0 41 0 0 41 0 0 41 0 0 41 0 While the general structure of the reported values is the same as the pre vious processing pane rather than 0 00 being consistently repeated as the second value after the AR as in the first example there is now a value that changes with each iteration this is the acceptance rate for the second level To be concrete in Iteration 1 of Stage I the reported level 1 acceptance rate is 0 47 and the level 2 acceptance rate is 0 63 As noted before the desired range of acceptance rates is between 0 20 and 0 30 for large or complex mod els with values around 0 50 being acceptable for smaller less dimensionally complex models Output 5 4 presents the parameters for the estimated model It may again be observed that the item and group parameter output from the MH RM is formatted identically to output from EM estimations Additionally it appears that the estimation completed successfully resulting in reasonable item and group parameter values and SEs 95 p e3 ul td 00 0 00 0 00 0 00 0 VT O 88 0
198. rs Using the lower triangle of the variance covariance matrix of the latent factors to determine positions the variance of the first factor is found at Cov 1 1 the variance of the second factor at Cov 2 2 and the covariance of the two factors is located in Row 2 Column 1 i e Cov 2 1 of the matrix and so on 54 Additionally in the lt 0ptions gt section the Quadrature command is en countered for the first time This statement allows the user specify the number of quadrature points to be used 21 in this case and the range over which those points should be spread from 5 0 to 5 0 here Reducing the number of quadrature points from the default value 49 points per dimension may be necessary when fitting multidimensional models because the total number of points used is exponentially related to the number of dimensions With higher dimensional models the total number of quadrature points and therefore pro cessing time may become unwieldy if the default value is maintained Within the output the various sections are arranged as with the other ex amples In the output file the factor loadings are printed below the estimated item parameters as requested by the FactorLoadings Yes statement As seen before the group means of the latent variables are reported followed by the estimated latent variable variance covariance matrix This section of the output is presented below Output 3 3 Correlated Four Factor Model Output Load
199. runs so we have repro duced the entire scoring control output file which has the extension ssc txt in Output 2 7 below 16 Output 2 7 3PL EAP Scoring Output 3PL EAP scoring 12 items 1 Factor 1 Group Scoring Summary of the Data and Control Parameters Missing data code 9 Number of Items 12 Number of Cases 2844 Number of Dimensions 1 Item Categories Model 1 2 3PL 2 2 3PL 3 2 3PL 4 2 3PL 5 2 3PL 6 2 3PL 7 2 3PL 8 2 3PL 9 2 3PL 10 2 3PL 11 2 3PL 12 2 3PL Scoring Control Values Response pattern EAPs are computed Miscellaneous Control Values Dutput Files Text results and control parameters g341 19 ssc txt Text scale score file g341 19 sco txt 3PL EAP scoring 12 items 1 Factor 1 Group Scoring 3PL Items for Group 1 Groupi Item a c b logit g g 17 1 1 84 0 56 0 31 1 09 0 25 2 1 27 2 77 2 17 2 00 0 12 3 1 50 0 98 0 65 1 19 0 23 4 1 31 0 46 0 35 1 19 0 23 5 1 35 0 42 0 31 1 08 0 25 6 1 39 1 91 1 37 1 93 0 13 7 1 96 4 27 2 18 2 22 0 10 8 1 05 1 24 1 19 1 56 0 17 9 1 94 3 31 1 71 2 59 0 07 10 1 35 2 28 1 69 2 23 0 10 11 1 77 1 40 0 79 1 25 0 22 12 1 55 0 52 0 33 1 37 0 20 3PL EAP scoring 12 items 1 Factor 1 Group Scoring Group Parameter Estimates Group Label mu s2 sd 1 Group1 0 00 1 00 1 00 As with the calibration output the file starts with a general data summary and echoing of values that were provided to flexMIRT In the scoring mode thi
200. s Between see Groups Between Between Groups Factor see Multidimensional IRT Multi level Model Burnin see lt Options gt Burnin Calibration IPL 24 25 97 98 Parameter Estimates 26 98 2PL 5 6 9 58 60 94 96 105 112 116 122 125 Parameter Estimates 8 9 96 107 113 3PL 12 40 42 43 46 48 49 Parameter Estimates 13 14 Graded Response Model 20 49 53 54 90 94 Parameter Estimates 21 22 Nominal Categories 29 30 32 33 57 125 156 Generalized Partial Credit Model 36 37 Parameter Estimates 30 31 34 99 Rating Scale Model 37 39 CaseID see Groups CaseID CaseWeight see Groups CaseWeight Cluster see Groups Cluster Code see Groups Code Coeff see Constraints Coeff Cognitive Diagnostic Models 99 103 C RUM 104 105 DINA 111 112 122 134 135 DINO 115 116 LCDM 104 111 115 123 125 132 Q matrix 103 106 120 123 Cognitive Psychometric Models see Cognitive Diagnostic Models Constraints 4 5 12 16 19 21 24 30 33 38 39 41 42 58 157 AddConst 157 160 Coeff 101 102 116 117 157 160 Cov 41 42 49 50 54 90 94 158 160 Equal 24 25 33 38 39 41 49 94 97 116 117 121 122 135 136 157 159 Fix 30 33 38 39 54 57 60 90 94 105 106 112 116 121 122 125 126 132 135 157 194 160 Free 41 49 50 54 57 60 90 94 105 106 112 116 121 122 125 126 132 135 136 157 159 160
201. s section of the output is much shorter due to the fact that fewer control parameters are available for modification Of note in this section is the line labeled Scoring Control Values under which the type of IRT scale scores is printed Below that the output files generated are named and tables con taining the item and group parameters as read in from the parameter file are printed to allow for the user to verify the parameter values were read in as expected The individual scores were saved into a separate file with the extension _sco txt Opening the individual score file 3PLM_ score example sco txt reveals that the columns in the score file are formatted as follows the group number the observation number the scale score and the standard error as sociated with the scale score Optionally following the group number and the observation number flex MIRT will print the case ID number if the caseID variable is given in the command syntax 18 2 5 Single Group 3PL ML Scoring As noted maximum likelihood scores are also available One issue that may occur when using ML scores is that extreme response patterns on either end of the continuum e g all correct or all incorrect are associated with undefined or unreasonable theta estimates When Score ML the user is required to specify the desired maximum and minimum scores via the MaxMLscore and MinMLscore keywords to obtain practical theta estimates for such cases
202. s the the SD and variance estimates for the higher order theta estimate For respondent 1 the most likely attribute profile is profile 1 corresponding to attribute profile 0 0 0 0 mastery of no attributes this attribute can be found to have the highest posterior probability listed on the line above 110 Basic DINA Fit to Simulated Data For the second example we will fit the deterministic input noisy and gate DINA model to the final three items Unlike the C RUM this model is non compensatory in nature not allowing mastery on one attribute to make up for lack of mastery on other attributes This is accomplished in the LCDM framework by constraining main effects to zero and allowing only the most complex attribute interaction term for an item to take on a non zero value Example 6 3 DINA Fit to Simulated Data Project Title Calibrate and Score DM Data Description DINA Model lt Options gt Mode Calibration MaxE 20000 MaxM 5 Etol 1e 5 SE REM SavePrm Yes SaveCov Yes SaveSco Yes Score SSC lt Groups gt AG File DMsim dat Varnames vi v15 N 3000 Ncats vi vi15 2 Model vi vi5 Graded 2 Attributes 4 InteractionEffects 2 Dimensions 10 4 ME and 6 2nd order ints AD Varnames al a4 DM G 111 lt Constraints gt Fix G vi v15 MainEffect Free G vi v3 MainEffect 1 Free G v4 v6 MainFffect 2 Free G v7 v9 Main
203. sed on the Guessing parameters for all items In this example the Beta 1 0 4 0 distribution was used which corresponds to a prior sample size of 5 with 1 0 1 0 4 0 0 2 as its mode This prior indicates that the prior guessing probability is equal to 0 2 as these multiple choice items have 5 options The Normal and LogNormal distributions are also available when specifying priors In relation to the lower asymptote the Normal prior applies a normal distribution prior on the logit of the lower asymptote parameter A prior of Normal 1 39 0 5 is a reason 12 able prior for the lower asymptote with a mode around 0 20 in the typical g metric Other common normal priors for the logit g parameter would be N 1 09 0 5 for items with 4 possible response options N 0 69 0 5 for items with 3 response options and N 0 0 0 5 for True False type items The stan dard deviation of 0 5 is standard across all noted priors and is based primarily on experience in that it provides a prior that is diffuse enough to allow es timated guessing parameters to move away from the mean value if necessary but not so diffuse as to be uninformative The output for this analysis is quite similar to that of the previous run so only differences will be highlighted As noted in the syntax discussion flex MIRT has been instructed to save the item and group parameter values into a separate file This results in an additional file name with the prm txt
204. sing MH RM estimation However requests for additional things such as LD indices more complete GOF statistics etc will most likely be ignored by flex MIRT O because their behavior in conjunction with the MH RM algorithm is still under research 5 3 1 MIRT Model with Correlated Factors For a first example we will refit the 4 correlated factors model applied to the 35 item QOL dataset originally presented in Example 3 8 Example 5 2 MHRM 4 Factor MIRT Model Project Title QOL Data Description Items 2 21 Correlated Factors lt Options gt Mode Calibration Algorithm MHRM RndSeed 432 ProposalStd 0 5 SavePRM Yes Processor 4 FactorLoadings Yes 89 lt Groups gt Group1 File QOL DAT Varnames vl v35 Select vl v21 N 586 Dimensions 4 Ncats vl v21 7 Model vi v21 Graded 7 lt Constraints gt Fix vi v21 Slope Free v1 v2 v5 Slope 1 Free vi v6 v9 S1ope 2 Free v1 v10 v15 Slope 3 Free vi v16 v21 Slope 4 Free Cov 2 1 Free Cov 3 1 Free Cov 3 2 Free Cov 4 1 Free Cov 4 2 Free Cov 4 3 In the Options section we have specified that flexMIRT should use the MH RM algorithm via Algorithm MHRM In addition we have provided the required random number seed for the MH RM estimation and set ProposalStd 0 5 This value was adjusted from the default of 1 0 because a first run of the algorithm was observed
205. ss group constraints are also allowed The basic constraint format is still used with the addition of the constraint in the second group following a colon In one of our multiple group examples it was desired that the item parameters for 12 items be the same across the 2 groups To accomplish this the Equal constraints in the excerpted code below were employed Example 7 9 Constraints Across Groups Specifying a Prior Distribution and Freely Estimating Theta lt Constraints gt Free Grade3 Mean 1 Free Grade3 Cov 1 1 Equal Grade3 vi v12 Guessing Grade4 vi v12 Guessing Equal Grade3 vi v12 Intercept Grade4 vi v12 Intercept Equal Grade3 vi v12 Slope Grade4 vi v12 Slope Prior Grade3 vi v12 Guessing Beta 1 0 4 0 159 As can be seen for the three constraints that set the item parameters across groups the general constraint format of type group items parameter is maintained for both groups with a colon between the listing for the Grade 3 and Grade 4 groups If more than two groups are to be constrained equal additional colons followed by the additional group information may be added Also of note in this syntax excerpt by default the latent variable mean vector and covariance matrix are initially constrained to zero and identity respec tively Using constraints the first two lines in the example above specify that the mean and variance for the latent trait are to be freely estimated
206. st computing machines Journal of Chemical Physics 21 1087 1092 Mislevy R 1984 Estimating latent distributions Psychometrika 49 359 381 Muraki E 1992 A generalized partial credit model Application of an EM algorithm Applied Psychological Measurement 16 159 176 Orlando M amp Thissen D 2000 Likelihood based item fit indices for dichotomous item response theory models Applied Psychological Mea surement 24 50 64 Preston K Reise S Cai L amp Hays R 2011 Using the nominal response model to evaluate response category discrimination in the PROMIS emo tional distress item pools Educational and Psychological Measurement 71 523 550 Robbins H amp Monro S 1951 A stochastic approximation method The Annals of Mathematical Statistics 22 400 407 Roberts G O amp Rosenthal J S 2001 Optimal scaling for various Metropolis Hastings algorithms Statistical Science 16 351 367 Rupp A A Templin J amp Henson R A 2010 Diagnostic measurement Theory methods and applications New York The Guilford Press Samejima F 1969 Estimation of latent ability using a response pattern of graded scores Psychometric monograph No 17 Takane Y amp de Leeuw J 1987 On the relationship between item response theory and factor analysis of discretized variables Psychometrika 52 393 408 Thissen D Cai L amp Bock R D 2010 The nominal categories item respons
207. t 1 2 3 5 7 G v20 Interaction 2 3 5 7 4th order int att 2 3 5 7 Equal D al a8 Slope restricted higher order model As with the previous CDM example the maximum number of E steps has been increased from the default the E tolerance and M tolerance val ues have been decreased and the SE calculation method has been set to REM in the Options section As noted earlier these changes are made primarily to improve the resulting SE estimates We have also requested the newly implemented multi core processing model be used for this analy sis via NewThreadModel Yes This model is requested due to the high 135 dimensional nature of the current problem Additional information regarding the new thread model is available in the Syntax chapter In the first group section have used the more targeted Generate to con struct only those interaction terms which will be used in the model The needed interaction effects are generated in the order they will be used for the items e g item 1 requires the interaction of attributes 3 6 and 7 but this not necessary as noted earlier as many interactions as needed may be con structed from a single Generate statement and the order in which interactions are listed does not matter Counting the interactions specified in the Generate statement we find there are 14 interactions needed to correctly assign items to correspond to the Q matrix Adding the 14 interactions with the 8 m
208. t of items freed onto both attribute 3 and attribute 4 106 684TTv8T 0 S6v 6000 0 TS8846ST0 O0 469 000 0 90LL72TL0 0 T66897100 0 8SS 6S10 0 ATTSTIOO O 646800TT 0 AvST S300 0 807906 0 0 Cv 08400 0 ATTC6T6T 0 T60688 0 0 T8ZEZ99T 0 TE8ZOOST O qozd gt uo st 0104107010410 POOF HOCH HOCH HOO H Om p O O O O O O O O uu u i AOOO G gt Q C Q OHA uu unus BH H 5 T dnox1 103 s rarTrqeqorq uorqeorgzrsseT5 ssoxzo pue S9INQTAIIY LUYI oTasouSerq se 000 00 0 STA ST ve 000 000 UTA a EE 00 0 00 0 ETA T 00 0 00 0 TTA eT 00 0 00 0 TTA TT 00 0 00 0 OTA OT 00 0 00 0 6 00 0 00 0 8 00 0 00 0 A 88 T 000 94 Xa 00 0 g v c 00 0 pA 00 0 T6 T 9 EA 00 0 T6 T sz ZA 000 68 T to TA c T d TeqeT o o T4 CN OO sf LO ONO T4 CN O sf MOOR wo H o m H s mnrrmqeqoid ION sOd epgoiq OMqHJJY pue siojouereg WN MAMO WAY O T 9 Mmdmo 107 The first section of the output excerpt presents the item parameters esti mates with freely estimated versus fixed at zero parameters mirroring the Q matrix specifications The factors labeled al a4 represent main effects slope parameters for attributes 1 4 respectively and c is the intercept parameter The next section of interest is labeled Diagnostic IRT Attributes and Cross classification Probabilities for Group 1 G which reports the estimated proportion of the population in each attribute profile For inst
209. tal number of dimensions how many are primary general 57 dimensions As noted in the data description the QOL scale is made of 1 general domain and 7 subdomains Once the dimensionality of the model is set up in the lt Groups gt section the details of which items load onto which specific factors are given in the lt Constraints gt section The first constraint serves to fix all item slopes to zero facilitating the subsequent freeing of parameters The next 8 lines set up the factor structure by 1 freeing all the item slopes on the first dimension which will be the general factor and 2 assigning the items to the appropriate subdomains based on the item content Note that it is absolutely necessary to ensure the the primary dimension s precede the group specific dimensions when freeing items to load on the factors flexMIRT automatically reduces the dimensionality of integration to 2 and consequently the analysis runs ex tremely fast The Model vi v35 Nominal 7 statement in the lt Groups gt section indicates that we wish to fit the Nominal Model for 7 response categories to the items In the current code we have opted not to modify either the scoring functions contrasts which may accomplished via the Ta statement or the intercept contrast matrix which may accomplished via the Tc statement Without changes the default option of Trend matrices will be used for both sets of parameters In the lt Constraints gt sec
210. tandard er rors of estimated item group parameters The standard error method may be changed by the user through the SE command The default method is empir ical cross product approximation keyword is Xpd Standard errors may also be calculated via the supplemented EM algorithm keyword is SEM Cai 2008 from the Fisher expected information matrix keyword is Fisher via the sandwich covariance matrix keyword is Sandwich using the forward differ ence method keyword is FDM e g Jamshidian amp Jennrich 2000 or from the Richardson extrapolation method keyword is REM e g Jamshidian amp Jen nrich 2000 Mstep values from the EM algorithm can be requested although technically the M step standard errors are incorrect They tend to be too 142 small but under some circumstances e g huge calibration sample size and thousands of item parameters it may be more efficient to bypass the more elaborate and correct standard error computations and focus only on item parameter point estimates As an additional note three of the available standard error methods have additional options that may be adjusted by the user By default the supple mented EM algorithm SEs come from an optimized window of iterations If you find that the SEs are failing to converge in that optimized window the optional command SmartSEM may be useful By setting SmartSEM No it will allow flex MIRT to use the full EM iteration history which may lead t
211. technical specifications deal with convergence criteria values and maximum number of iterations MaxE determines the maximum allowed number of E steps in the EM algorithm MaxM the number of allowable iterations in each of the iterative M steps Etol and Mtol set the convergence criteria for the E and M steps respectively SEMtol sets the convergence criterion for the Supplemented EM algorithm used in calculating standard errors and finally SStol determines the criterion for declaring two summed scores as equivalent useful for scoring with weights The Processors statement lets the user determine how many processors threads an analysis can utilize When NewThreadModel Yes a new multi core processing model is employed The new thread model is an updated im plementation of multi core processing that may be more efficient for very high dimensional models with a large number of quadrature points to evaluate The level of granularity is smaller than the classical threading model implemented in flexMIRT a classical threading model trades more memory for speed and this new model does not do that nearly as much so the parallel speed up may not be as significant as the classical approach for low dimensional analyses with large N User are encouraged to experiment with the threading options When the NewThreadModel is active Parallelization granularity Fine may be more optimal for high dimensionality will be printed in output with other control
212. tements in the flex MIRT syntax file must end with a semi colon Finally the commands in the syntax are not case sensitive but file names on certain operating systems may be Within the Project section there are two required commands Title and Description The name of the title and the contents of the description must be enclosed in double quotation marks The Options section is where the more technical aspects of the analysis are specified It is in this section that analysis details such as convergence criteria scoring methods standard error estimation methods or requests for various fit indices may be specified The only required command is Mode where the user is asked to specify what type of analysis will be conducted The three options available are Calibration Scoring and Simulation Depend ing on the Mode of the analysis there may be additional required commands such as the type of IRT scoring to be conducted in a Scoring analysis These details will be covered as they emerge in the examples As noted previously Groups is the section where information regarding the groups of item response data is provided This includes the name s of the file s containing the data set s the number of items the number of groups and participants and the number of response options for each item among other things Following the Groups section header a group name must be provided for the first group even if t
213. tended so additional fit statistics namely marginal X values and the local dependence LD statistics of Chen and Thissen 1997 are included in the output A table containing these values is printed between the group parameter values and the Information Function values table This table is presented below The values in the second column of this table are the item specific marginal X which may be tested against a chi square variable with degrees of freedom equal to the number of categories for that item minus 1 In this case all items fit well at the level of univariate margins The remaining cells provide pairwise diagnostic information for possible local dependence These are standardized X values so values larger than 3 in absolute magnitude should be examined further Output 2 6 Single Group 3PL Calibration Output Extended GOF Table Marginal fit Chi square and Standardized LD X2 Statistics for Group 1 Group1 Marginal Item Chi2 1 2 3 4 5 6 7 8 0 0 Ooooooooooo 0 0 0 0 0 0 0 0 0 0 0 wwr 5000 Qo 2 4 Single Group 3PL EAP Scoring There are situations when existing item parameters will be used and only scoring for a new set of data is desired This next example will introduce a command file that does just that using the saved parameter estimates from a prior calibration run 15 Example 2 3 3PL EAP Scoring example lt Project gt Title 3PL EAP scoring Description 12 items 1 Factor 1 Group
214. ter flexmirt vpgcentral com My Account Change Password Log Off Home Purchase Support My Account Contact flexMIRT Vector Psychometric Group Account Details flexmirtevpgcentral com Download Software Jane Doe H Download FlexMIRT x Current Engine Version 2 26 Current GUI Version 2 0 5 1 Windows 32bit Windows 64bit Locate the downloaded installer flexmirtInstaller and double click to ac tivate it NOTE IT MAY BE NECESSARY FOR SOME USERS ESPE CIALLY THOSE WITHOUT ADMINISTRATIVE PRIVILEGES TO RIGHT CLICK ON THE INSTALLER AND SELECT Run as Administrator FOR THE PROGRAM TO FUNCTION PROPERLY ONCE IT HAS BEEN IN STALLED Welcome to the FlexMIRT 2 0 64 bit for Windows Setup Wizard The Setup Wizard will install FiexMIRT 2 0 64 bit for Windows on your computer Click Next to continue or Cancel 170 The second step of the installer will ask which folder you wish to install the program into If you would like the program installed somewhere other than the default location this can be accomplished by selecting the Change button and directing the installer to the desired folder location Install FexMIRT 2 0 64 bit for Windows to Follow the remaining steps as directed by the installer until the program indicates that it has finished installing Once flex MIRT have been installed a shortcut will appear on the desktop Double click this to initiate
215. ters For example below is the item parameter portion of the flex MIRT output for a single run fitting the model found in the file Fitl flexmirt Output 4 6 Item Parameters with Parameter Number Labels Graded Items for Group 1 Item 1 The order in which the parameters will be printed in the ISF output cor 82 responds to the values list in the columns labeled PZ in the output From this output excerpt we can see that in the ISF output file the two intercept parameters and then the slope parameter for Item 1 will be printed first as indicated by the 1 2 and 3 respectively in the Pf column These val ues are followed by the intercept parameters and slope for Item 2 and so on Once the item parameters have been reported the latent variable mean s and covariance s are printed After all point estimates are given the respective standard error for each parameter is printed in the same order as the point estimates The next six columns of the ISF output following all parameter point estimates and SEs are fixed flexMIRT 9 will report in order the G X and M overall fit statistics It should be noted here that the M column will always print regardless of whether the statistic was requested or not If the statistic was not requested via the GOF and M2 statements in the Options section placeholder values of 0 0000 will be printed and are not indicative of the actual value of the Ma statistic had it
216. th 1 Factor 3 groups 187 List of Examples 2 1 2 2 2 3 2 4 2 5 2 6 2 7 2 8 2 9 2 10 2 11 3 1 3 2 3 3 3 4 3 5 3 6 3 7 3 8 3 9 3 10 4 1 4 2 4 3 4 4 4 5 4 6 4 7 2PL Calibration Syntax A ee Rep ee ee ee e urs Single Group 3PL Calibration Syntax aPL EAP Scoring example e o ero de verd 3PL ML Scoring example u a woe a Graded Mode Combined Calibration and Scoring Syntax with TRC COU yes cs EE het h ha n GU ITLEURIUCHITELTIE 1PL Calibration and Scoring with Response Pattern Data Nominal Model Calibration Nominal Model Calibration2 Generalized Partial Credit Model Rating Scale Model with Trend Contrasts Rating Scale Model with Triangle Contrasts 2 Group 3PL Using Normal Priors with EAP Scoring 2 Group 3PL with Empirical Histogram Excerpts 2 Group 3PL MAP Scoring from Parameter Estimates 2 Group 3PL Summed Score to EAP Conversion Table DIF Test All Items Excerpts basal ae el we x3 DIF Test Candidate Items Excerpts Multiple Group Multilevel Model Excerpts Four Factor GRM Model with Correlated Factors Bifactor Structure Nominal Model Bifactor Structure 2PL 2 21 ala Bis Ak ie as Simulation Graded Model Syntax Only Simulation Graded Model Using PRM File Simulation 3PL Model Normal Population Simulatio
217. th Set Options Column 1 Column 5 Items Column 5 Groups Select Columns Entry Type IRT Model Prior Type Nominal Contrast Type Value Meaning Value Meaning Value Meaning Value Meaning 0 Group 1 3PL 0 Normal Prior 0 Irend 1 Item 2 Graded 1 EH Prior 1 Identity 3 Nominal 2 User specified As indicated in Table 4 2 the first column of the parameter file indicates whether the information to be supplied across the row refers to an item or a group The information that goes into the various subsequent columns differs somewhat depending on whether item or group is indicated in column 1 so the arrangement for these two parts of the parameter file will be discussed separately 4 3 1 Column Progression for Groups With a 0 in the first column of the parameter file flex MIRTO will interpret all subsequent information on the line as applicable to a group The 2nd column is used to supply a group label which should correspond to the label provided in the syntax file The third column indicates to which group this group belongs i e 1 if it belongs to the first group 2 if it belongs to the second group etc Column 4 requires the user to specify the number of latent variables in the generating model within the group just specified The 5th column is reserved for indicating the type of population distribution prior to be applied to the latent trait The acceptable values are 0 for normal and 1 for empirical histogram see Table 4 2
218. th the command ItemWeights vi v3 0 0 0 1 2 0 3 0 3 1 Due to this statement scores of 0 are assigned a category weight of 0 1s a weight of 0 1 2s a weight of 2 0 all 3s given a weight of 3 0 and responses of 4 a weight of 3 1 Multiple ItemWeights statements are permitted to handle situations when the number of responses options per item is variable 155 BetaPriors is used to assign item specific beta distribution priors on the uniquenesses the rational and use of applying priors to the item uniquenesses was discussed in Bock et al 1988 on pp 269 270 The Beta distribution applied in flex MIRT has an a parameter set to the user specified value and the 8 parameter fixed at 1 For example the statement BetaPriors vi v5 1 6 would apply to the uniquenesses of Items 1 through 5 a prior distribution of the form Beta 1 6 1 Multiple BetaPriors statements are permitted The next four commands are used exclusively for the fitting of cognitive diagnostic models Attributes is used to set the number of main effects present in the skill space that is being assessed This is the keyword that triggers flex MIRT to model discrete latent variables rather than the default continuous latent variables InteractionEffects instructs flex MIRT to automatically generate all possible interaction effects For instance with 4 attributes one may specify InteractionEffects 2 3 4 and in addition to the 4 main effects the program with also
219. the scoring functions for all items Addition ally we need to constrain the slopes for all items to be equal as well as set each intercept to be equal across the items This is all accomplished via the constraints imposed in Example 2 10 Example 2 10 Rating Scale Model with Trend Contrasts Project Title QOL Data Description 35 Items RS Using Default Trend Contrasts lt Options gt Mode Calibration SE SEM lt Groups gt Group1 File QOL DAT Varnames vi v35 37 N 586 Ncats vl v35 7 Model vi v35 Nominal 7 lt Constraints gt Fix vi v35 ScoringFn Equal vi v35 Slope Equal vi v35 Intercept 2 Equal vi v35 Intercept 3 Equal vi v35 Intercept 4 Equal vi v35 Intercept 5 Equal vi v35 Intercept 6 The previous RS fitting used the flex MIRT default trend contrasts To obtain results for the RS model similar to those that would be produced by Multilog the user may provide flex MIRT with the triangle contrasts utilized by Multilog This is done through the Tc command and rather than using one of the available keywords supplying the values that compose the desired contrast The triangle contrast is shown in the Tc statement in the following example syntax Example 2 11 Rating Scale Model with Triangle Contrasts Project Title QOL Data Description 35 Items RS Multilog style Triangle Contrasts lt Options gt Mode Calibration SE
220. there are brief explanations provided in the R code for those less familiar with the program and its language Although we have provided numerous examples these are primarily to demonstrate that the provided plotting functions work with a variety of models There are actually only two commands that are needed to plot any given set of parameters First R needs to load the pre defined plotting functions we have provided these are found in the file flexmirt R and the file is loaded into R using the source call seen on the first non comment line the first line without a sign at the beginning 181 Once the functions have been loaded into R the next step is to provide the functions with the location and file name of the values one wishes to plot For obtaining ICCs and TCCs the function call used is flexmirt icc and within the parentheses the user supplies the prm file name with location path that contains the parameters that will define the ICCs and TCCs For instance the line flexmirt icc C Users laptop Documents flexMIRT flexMIRT plotting Preschool 2PL prm txt calls the trace line plotting function and tells the function to create plots based on the parameter values found in the file Preschool 2PL prm txt The graph presented here is representative of the plot output that will be obtained from the ICC TCC plotting function Figure C 1 GRM ICCproduced by provided R code ICC for item Views 1 0 0
221. tion see Fixed Effects Calibration FixedTheta see lt Groups gt FixedTheta Free see lt Constraints gt Free Generate see lt Groups gt Generate GOF see lt Options gt GOF Goodness of Fit GOF 145 Item Level 15 Marginal Chi Square 15 22 23 195 Orlando Thissen Bjorner S X 25 Model 5 6 153 27 GPC 37 Overall 11 12 35 36 119 GPC 153 Limited Information Ma 33 35 Graded O 5 6 21 24 25 54 36 60 62 90 94 97 105 112 Group Label 4 5 Groups 4 5 12 16 19 21 24 25 30 33 37 39 41 43 45 49 54 57 60 90 94 97 Attributes 101 105 112 116 121 122 125 132 135 156 BetaPriors 41 43 154 156 Between 50 51 77 94 154 CaseID 18 50 51 154 155 CaseWeight 24 25 154 Cluster 50 51 94 154 155 Code 21 33 60 151 153 DIFitems 49 148 154 157 Dimensions 37 39 50 51 54 57 60 73 77 78 90 94 105 112 116 121 122 125 132 135 116 122 125 132 135 153 Nominal 30 33 38 39 57 125 153 threePL 12 41 43 49 N 5 6 12 16 19 21 24 30 33 37 39 41 43 44 49 50 54 57 60 62 64 69 71 73 75 77 78 90 94 97 105 112 116 122 125 132 135 151 152 Ncats 5 6 12 21 24 30 33 37 39 41 43 49 54 57 58 60 90 94 97 105 112 116 122 125 132 135 151 152 Nleve12 77 78 154 Primary 37 39 57 58 60 73 154 121 154 DM 101 102 105 112 116 122 Select 24 25 33
222. tion the first constraint imposed fixes the first scoring function contrast for all 35 items the default value is 1 0 as required for identification see Thissen et al 2010 With the default contrast matrices the fixed first scoring function contrast of 1 0 ensures that the first and last scoring function values are fixed to 0 and 6 respectively The remaining 5 scoring function values are reparameterized by the contrast matrices and become estimatable parameters as shown below In the lt Options gt section the number of quadrature points is reduced for time efficiency as was done with the other MIRT models using the Quadrature statement and the tolerance values are adjusted down to obtain tighter solu tions Output 3 4 Nominal Model Scoring Function Estimates Excerpt Nominal Model Scoring Function Values s Group 1 Groupi Item Label s 1 s 2 s3 s 4 s 5 s 6 s 7 1 vi 0 00 0 76 1 18 1 46 3 29 4 61 6 00 2 v2 0 00 0 79 1 53 2 58 4 18 4 77 6 00 3 v3 0 00 1 49 1 91 2 08 3 77 4 21 6 00 4 v4 0 00 0 77 1 65 2 48 4 05 4 76 6 00 5 v5 0 00 0 99 1 71 2 47 3 93 4 78 6 00 6 v6 0 00 0 93 2 46 3 13 4 35 5 12 6 00 58 7 v7 0 00 0 79 2 27 2 96 4 25 5 03 6 00 8 v8 0 00 1 50 2 30 2 92 4 27 4 96 6 00 9 v9 0 00 1 53 2 54 2 77 4 18 4 85 6 00 10 v10 0 00 1 41 2 59 2 86 3 93 4 53 6 00 11 vil 0 00 0 88 1 60 1 89 2 98 4 35 6 00 12 v12 0 00 0 11 1 06 1 35 2 51 4 15 6 00 13 v13 0 00 1 45 1 71 1 63 2 71 3 86 6 00 14 vi4 0 00 1 55 1 68 2 59 3 54
223. tion functions are plotted By default the inf txt file has only one theta value at 0 0 saved The FisherInf command allows the number of points and the symmetric minimum maximum values of those points to be modified For instance if information values were desired for 81 points with theta values ranging from 4 0 to 4 0 the command to obtain this output in conjunction with SaveInf Yes would be FisherInf 81 4 0 with the total number of points listed first and the maximum theta value to be used listed after comma Using the PriorInf command the user controls if 143 flexMIRT includes the contribution from the prior in the test information output the default setting of Yes includes the prior contribution Note that for multidimensional models flexMIRT will try to print by de fault the trace of the Fisher information matrix calculated at every grid point as defined by the direct product of the points specified in the FisherInf com mand For high dimensional models this is not advisable If the logDetInf Yes command is used the information will be taken as the log determinant of the Fisher information matrix rather than the trace The Quadrature command which can modify the number of quadrature points used in the E step of the EM algorithm is specified in a similar fashion as the FisherInf command The default quadrature rule has 49 rectangular points over 6 to 4 6 The next several statements listed within the
224. tions produced by a Metropolis Hastings sampler MH Hastings 1970 Metropolis Rosenbluth Rosenbluth Teller amp Teller 1953 which is a sampling method based on the principles of Markov chain Monte Carlo MCMC The MH RM algorithm is motivated by Tittering s 1984 recursive algorithm for incomplete data estimation and is a close relative of Gu and Kong s 1998 SA algorithm It can also be conceived of as an extension of the Stochastic Approximation EM algorithm SAEM Celeux amp Diebolt 1991 Celeux Chauveau amp Diebolt 1995 Delyon Lavielle amp Moulines 1999 when 85 linear sufficient statistics do not naturally exist in the complete data model Within the flexMIRT implementation the MH RM iterations use ini tial values found through an unweighted least squares factor extraction stage Stage I and further improved upon during a supplemented EM like stage Stage II both Stage I and Stage II use a fixed number of cycles Stage III is home to the MH RM algorithm and where the primary estimation of parameters occurs Cycles in Stage III terminate when the estimates stabilize Cycle j 4 1 of the MH RM algorithm for multidimensional IRT consists of three steps e Imputation In the first steps conditional on provisional item and la tent density parameter estimates p from the previous cycle random samples of the individual latent traits 09 are imputed using the MH sampler from a Markov chain having the poste
225. to demonstrate that comparable parameter estimates may be obtained from flex MIRT and benefits of either computation time savings or simplicity of syntax are ob tained The first example is a replication of Example 9 2 presented in Diagnos tic Measurement Theory Methods and Applications by Rupp et al 2010 The simulated data presented in this example uses 7 dichotomous items that measure 3 attributes The Q matrix given by the authors is duplicated here Table 6 4 Q matrix for Rupp Templin amp Henson s Example 9 2 Item Attribute 1 Attribute 2 Attribute 3 Item 1 1 0 0 Item 2 0 1 0 Item 3 0 0 1 Item 4 1 1 0 Item 5 1 0 1 Item 6 0 1 1 Item 7 1 1 1 With the Q matrix we are now able to generate our flexMIRT CDM syntax Example 6 6 Rupp Templin amp Henson Example 9 2 lt Project gt Title Rupp Templin Henson Example 9 2 Description Saturated LCDM model 7 items 3 attributes lt Options gt Mode Calibration MaxE 20000 Etol 1e 6 MaxM 50 Mtol le 9 GOF Extended SE REM SavePrm Yes 123 SaveCov Yes SaveSco Yes Score EAP lt Groups gt 4G File ch9data dat N 10000 Varnames vi v7 truec Select vl v7 Ncats vl v7 2 Model vi v7 Graded 2 Attributes 3 InteractionEffects 2 3 generate 2nd and 3rd order ints Dimensions 7 3 main 3 2nd order 1 3rd order AD DM G Varnames a Ncats a 8 Model a
226. ue SAINQTAIAY LUI 2rasougerq ecc 83 0 ets TC 29 0 iv oc L 81 61 0 sts T phas 00 0 ga ST Bez 00 0 TO 641 vt ga 8T O 98 T 8T O so z TT pa oT ooo 00 0 00 0 10 0 voz CE 00 0 TA rete 00 0 1070 48 T TA ers ers Te leqe1 uazr 5 dnoz5 103 su qI Tdz dnois q moy s1o3ourereq WJ 3nd3n uosuog ur dus r ddny g 9 3nd3n Q 133 de la Torre Subtraction Example In a demonstration of a higher order latent trait CDM to non simulated data de la Torre and Douglas 2004 fit a higher order DINA model to a 20 item fraction subtraction test given to 2144 examinees In the noted paper an 8 attribute model was estimated using MCMC with final parameter estimates based on averaging the estimates from 10 parallel chains each with 20000 iterations of which the first 10000 were discarded as burn in We provide syntax and output of a flex MIRT replication of this analysis Example 6 8 de la Torre Subtraction Syntax Project Title Tatsuoka Subtraction Data Description Restricted Higher order DINA Model lt Options gt Mode Calibration MaxE 20000 MaxM 5 Mtol 0 Etol le 5 GOF Extended SE REM SavePrm Yes SaveCov Yes SaveSco Yes Score EAP Processors 4 NewThreadModel Yes Groups AG File subtraction csv Varnames vi v20 Ncats vi v20 2 Model vi v20 Graded 2 Attributes 8 Generate 3 6 7 4 7 2 3 5 7 2 4 7 8 1 2 7
227. ummed Score Conversion Tables from ssc File 114 DINO Output Item Parameters 118 Rupp Templin Henson Output Item Parameters 127 Rupp Templin Henson Output sco File 129 Rupp Templin Henson Output Item Parameters without D PTOUP ey eder A eun Qua roS us ob datei 133 viii CHAPTER 1 Overview Vector Psychometric Group LLC a North Carolina based psychometric soft ware and consulting company is pleased to announce the immediate avail ability of flex MIRT Version 2 0 a substantial update to our multilevel and multiple group item response theory IRT software package for item analysis and test scoring flex MIRT fits a variety of unidimensional and multidimen sional IRT models to single level and multilevel data using maximum marginal likelihood or modal Bayes via Bock Aitkin EM with generalized dimension reduction or MH RM estimation algorithms The generalized dimension re duction EM algorithm coupled with arbitrary user defined parameter con straints makes flex MIRT one of the most flexible IRT software programs on the market today and the MH RM algorithm allows users to efficiently estimate high dimensional models flexMIRT produces IRT scale scores us ing maximum likelihood ML maximum a posteriori MAP and expected a posteriori EAP estimation It optionally produces summed score to IRT scale score EAP conversion tables for unidimensional and multidimensi
228. umn 7 is 5 then 101 points will be evenly spaced over the range of 5 to 5 The user must then supply as the last 101 entries of this particular row the 101 ordinates of the empirical histogram i e the heights of the histogram at the 101 quadrature points An example of simulating data with EH priors will be presented later 4 3 2 Column Progression for Items The first four columns of a row supplying item level information have the same generic content as groups First a value indicating item information will follow i e 1 is given in Column 1 then a variable name is assigned in Column 2 This name must correspond to a variable in the Varnames statement in the command file Column 3 assigns the item to a specific group and Column four requires the entry of the number of latent factors in the generating model At this point Column 5 the content of columns diverges from what was covered in the groups subsection The generating IRT model for the item is set in Column 5 with the possible models coded as indicated in Table 4 2 For instance an item to be generated using Graded model parameters would have a 2 in the 5th column The subsequent column supplies the number of possible response categories for the item 67 Once an IRT model is chosen the item parameters must be entered one value per column The progression of expected parameters varies slightly de pending on the which IRT model was selected in Column 5 If the 3PL model
229. upplied values for each contrast When k groups are present k 1 contrasts must be specified e g for 3 groups DIFcontrasts 2 0 1 0 1 0 0 0 1 0 1 0 with commas in between Because the syntax file is free form the contrasts may appear on more than one line For example DIFcontrasts 2 0 1 0 1 0 0 0 1 0 1 0 is entirely permissible From the DIF analysis an additional table is presented in the output 46 0 0 0 0 0 0 0 0 0 0 0 0 do oo oo o o 44 o o o 4 Y oOoocooooooooo v cd oco oc oco od od od o 4 4 oOooooooooooo c cd co co oc 4 od d c 4 a KD r NOCH HOONOOW oOooooooooooo oOooooooooooo cO 0 0 CO C 0 mm LO i0 Q0 Qo F CO QO 10 O oor 4ooomoddo Yd 8 T I FP ZX Te101 zdzo Idz5 UT siI qumu We YT sue3 Td 10 so5rqsrqeqs AIG 60 Y LO OO O LO LO O O0 O m tH po a0 N gt lt o qer yndyng sesATeuy JIC T yndyno 47 For each item that is present in both groups a line in the table is added that reports the various Wald tests for that item As we fit the 3PL to the items we are provided with an overall test for DIF a test of DIF with respect to the guessing parameter labeled X2g a test of the slope assuming g is fixed across groups labeled X2alg and a test of the intercept assuming a and g are fixed Each of the reported Wald values also has the associated degrees of freedom and p value printed Wo
230. uueg d sise1quoy eqeT u 4I qdnozy I dno19 toy su qI Toy sqsezquog 43d S z quI epoy ISUTUON Te y E A i Sea se ore PV ve ve 60 ST I Ta T gs vs es zs Ts leqe1 u 4I Tdnoz5 1 dnozy sanTep uorqoun4 SUTIOIS Tepoj TeuruowN O0c 0 T 6 0 CTD TTT zz co OT LTS 246 0 60v 00 T puexr vTrO vero DID ses se T O TOv S4 O 3 00v c8 0 H vv O 86 Ov I 86 0 16 00 T pueal O 6v 0 0t pes be gT 0 AY vL O 1 O 9T 040 t 900 vl TAI TO T T 0071 pueda 470 vV O 8T ZA 4 61 0 S 6 0 Oro b LEO T O c 680 T T To 00 T puexL TO 8 0 9 T T 9 eudte dd a s g eudte Hd a s or eyudte q a s Z eudie dd a s I eudie Hd sysezquop a s e Hd Teqe7 921 Tdnoz5 I dno1y 107 sw I 103 saserquoo UOTIDUNJ Sur o5s pue sadoTS Tepoy TeuruoN sIojourereq WY 3nd3n opo N euruon 4T c mdmo 3l The first 3 tables in Output 2 17 provide parameter estimates using the reparameterized version of the model see Thissen et al 2010 In the first ta ble the overall slope value labeled a the type of x contrast matrix Ta used and the estimated contrast values labeled alpha 1 alpha 3 are reported There are 7 categories so there are 6 scoring function contrasts The second table labeled Nominal Model Scoring Function Values reports the scoring function values for the categories which are calculated as Tga where are the estimated scoring contrasts The third table of item parameters reports the ty
231. was selected the guessing parameter in logit metric is given in Column 7 followed by the intercept parameter and then all discrimination slope param eter values if there is more than one factor If the Graded Model is selected in Column 5 the expected order for parameter values is the ordered intercept parameters starting at the lowest in Column 7 and going up followed by the discrimination parameter s With respect to the Nominal model the parameters are entered in the reparameterized form detailed in Thissen et al 2010 The type of scoring contrast is given in Column 7 using one of the code values given in Table 4 2 If Trend 0 or Identity 1 contrasts are specified in Column 7 the scoring function contrast values of which there are the number of categories minus one must be supplied starting from Column 8 After those m 1 values are given the slope parameter s are entered Immediately after the slope parameter the next column the column number of which will vary depending on number of categories and slopes used is used to indicate the contrast type for the intercept parameters Again if Trend or Identity contrasts are selected the columns immediately following the contrast selection contain the appropriate number of intercept parameter values again always equal to the number of categories minus one In the rare occasion that user supplied contrasts are selected for either the scoring functions or the intercept param
232. will be conducted If Mode Scoring or Mode Simulation is selected some additional technical commands covered in the next group of statements become required The setting of Progress determines if flex MIRT will print detailed progress in 138 formation in the console window e g iteration number and log likelihood value TechOut determines if controls values such as tolerance values pro cessing times names of outputted files etc will be printed in the preamble of the output NumDec determines the number of decimal places reported for item parameters in the output file the default is 2 The other commands allow for the optional saving of additional output into separate files SaveCov refers to the covariance matrix of the parameter estimates SaveInf the Fisher infor mation function values of the items and the test SavePrm the item and group parameter estimates SaveSco the individual IRT scale scores and SaveDbg additional technical information Each of the these additional output requests are saved to files with corresponding extensions that is the SaveInf com mand results in an output file with inf txt appended to the command file name SaveCov appends cov txt etc Example 7 2 Options Engine Mode and Output Statements Options gt Mode Calibration Scoring Simulation Progress Yes No TechOut Yes No NumDec 2 SaveCov Yes No Savelnf Yes No SavePrm Yes No
233. y two dimensions total so we specify the total number of dimensions to be used for the model via Dimensions 2 and also denote that at level 2 there is 1 between school dimension Between 1 This implies that there is 1 within school dimension at the student level In the estimation and output flexMIRT will use the first dimension for between school variations and the second dimension for within group vari ations When a 2 level model has been requested it is required that a variable defining the higher level units be specified via the Cluster statement We also provide a variable that identifies the first level data individuals with the CaseID statement The latter can be convenient if the individual IDs are unique so that the scale scores computed by flexMIRT can be merged into other data sets for further analysis The Constraints section contains extensive statements for setting up the necessary priors and equality constraints both within and across the two groups so that the desired hierarchical model is estimated With the between group equality constraints the item parameters for both level 1 and level 2 models are set equal across the two countries thereby creating anchoring so that the mean of Ireland schools can be freely estimated The within group equality constraints are set up so that the item slopes for the within dimension is equal to those for the between dimension This implies that a random intercept only mode

Flexible Multilevel Multidimensional Item Analysis and Test Scoring

Contents

Download Pdf Manuals

Related Search

Related Contents