Differential item functioning analysis system User`s Manual
Contents
1. txt which can be accessed by DIFAS Note that there cannot be any variable names in the first row as DIFAS can read in only numeric values and assumes all entries are data and are to be included in the analyses DIFAS 5 0 User s Manual 9 Importing the Data To import a data file use the top portion of the main window of DIFAS to select the desired file When selecting the desired file follow these steps STEP 1 SPECIFY DELIMITER USED IN DATA FILE Specify the delimiter used in the file space comma or tab as listed by the options on upper left portion of the Main Window By default DIFAS sets the delimiter to space but you may have to change this to comma or tab Not changing delimiter appropriately will cause DIFAS to not recognize the data when importing STEP 2 SPECIFY FILE TYPE Specify whether the file has the extension txt or dat using the drop down list box By default DIFAS sets the file type to txt Note however that if you are importing a file created from SPSS you may have to select the dat extension STEP 3 LOCATE DESIRED DATA FILE Using the drive list box and the directory list box in the top portion of the Main Window locate and select the desired file Clicking on the desired file will cause the file s name to appear in the Selected File box in the upper right side of the Main Window STEP 4 CLICK IMPORT BUTTON To import the data of the selected file to DI2. Mantel Haenszel chi square Mantel Haenszel common log odds ratio and estimated standard error Standardized Mantel Haenszel common log odds ratio Breslow Day test of trend in odds ratio heterogeneity ETS classification scheme The DIF procedures that DIFAS runs for polytomously scored items include Mantel s chi square Lui Agresti cumulative common log odds ratio and estimated standard error Standardized Lui Agresti cumulative common log odds ratio Cox s noncentrality parameter estimator and estimated standard error Standardized Cox s noncentrality parameter estimator The DTF procedures that DIFAS runs consists of estimates of the variance in the DIF effect across the items of the test or scale The DSF procedures that DIFAS runs consists of estimates of the step level common log odds ratios using the cumulative and adjacent categories dichotomization scheme All of the procedures described above are discussed in greater detail in Chapter 3 In addition DIFAS can compute a item level descriptive statistics b frequencies for the reference and focal groups at each level of the stratifying variable and c the mean of each item conditional on 10 intervals of the stratifying variable DIFAS 5 0 User s Manual 3 System Requirements The following hardware and software is required for running DIFAS e Microsoft Windows 95 or later or Microsoft Windows NT 3 51 or later e 24 MB RAM for Windows 95 or later 32 MB RAM for M
3. DIFAS 5 0 Differential item functioning analysis system User s Manual 2012 Randall D Penfield CONTENTS Chapter 1 Getting Started with DIFAS System requirements Running DIFAS A diagram of the DIFAS Main Window Conducting analyses using DIFAS Chapter 2 Importing Data Preparing the data for import Importing the data Navigating the imported data file Chapter 3 Conducting Analyses Descriptives Frequencies DIF analyses for dichotomous items DIF analyses for polytomous items Differential test functioning Differential step functioning Missing Data Chapter 4 Managing Output Properties of the output Editing the output Saving output files Opening output files Printing Output DIFAS 5 0 User s Manual 2 CHAPTER 1 Getting Started with DIFAS DIFAS is a Windows based program that performs a variety of functions related to assessing the presence of differential item functioning DIF in items differential test functioning DTF across all items of a test or scale and differential step functioning DSF for ordinal polytomous items DIFAS is exclusively point and click and is aimed at providing users with the capability of conducting sophisticated DIF and DTF analyses in a user friendly environment Although a multitude of parametric and nonparametric DIF detection procedures exist DIFAS performs only nonparametric DIF analyses The DIF procedures that DIFAS runs for dichotomously scored items include
4. 0 0 07 0 Var 3 0 08 0 11 0 02 0 15 0 19 0 13 0 01 0 18 0 06 0 Var 4 0 03 0 15 0 06 0 28 0 08 0 22 0 09 0 14 0 03 0 Var 5 0 0 0 12 0 12 0 08 0 01 0 1 0 01 0 09 0 Var 6 0 03 0 11 0 05 0 07 0 22 0 09 0 09 0 04 0 07 0 Var 7 0 02 0 09 0 06 0 12 0 42 0 O 1 0 03 0 04 0 Var 8 0 02 0 05 0 03 0 17 0 36 0 09 0 25 0 02 0 07 0 Var 9 0 03 0 08 0 16 0 12 0 13 0 05 0 2 0 05 0 03 0 Var 10 0 04 0 04 0 01 0 04 0 06 0 25 0 16 0 08 0 13 0 Var 11 0 03 0 02 0 05 0 05 0 07 0 04 0 03 0 08 0 07 0 Each of the DIF statistics conducted by DIFAS for dichotomous items will now be briefly described DIFAS 5 0 User s Manual 15 Mantel Haenszel Chi Square MH CHI The Mantel Haenszel chi square statistic Holland amp Thayer 1988 Mantel amp Haenszel 1959 is distributed as chi square with one degree of freedom Critical values of this statistic are 3 84 for a Type I error rate of 0 05 and 6 63 for a Type I error rate of 0 01 Mantel Haenszel Common Log Odds Ratio MH LOR The Mantel Haenszel common log odds ratio Camilli amp Shepard 1994 Mantel amp Haenszel 1959 is asymptotically normally distributed Positive values indicate DIF in favor of the reference group and negative values indicate DIF in favor of the focal groups Standard Error of the Mantel Haenszel Common Log Odds Ratio LOR SE The standard error of the Mantel Haenszel common log odds ratio The standard error computed here is the nonsymmetric
5. a problem provided they are integer in value Any transformations must be conducted prior to importing the data into DIFAS 7 DATA FILE MUST HAVE txt OR dat EXTENSION DIFAS is designed to read in only text txt or data dat files Text files are typically maintained in a program such as Notepad 8 DATA FILES MUST BE SPACE COMMA OR TAB DELIMITED DIFAS is designed to read in files for which the columns variables or items of the data file are delimited using either a space comma or tab If the data is being maintained in a data management program such as Excel or a statistical software package such as SPSS or SAS the data must be converted to a delimited text file Here are some guidelines for creating a delimited text file from several of the more common data management and statistical programs SPSS When the desired data file is in SPSS save the data file as a tab delimited dat file and make certain that you deselect the check box labeled Write variable names to spreadsheet If this check box is not deselected the variables names will be written to the first line of the data file and DIFAS cannot read in non numeric values The resulting file is a tab delimited file having the extension dat which can be accessed by DIFAS Excel When the desired data file is in Excel save the data file as a text tab delimited file The resulting file is a text tab delimited file having the extension
6. as being data Simply delete the fist line of the dat file and save Next ensure that the type of delimiter in DIFAS is set to tab delimited Failing to do so will result in DIFAS displaying a message that an invalid entry was made In addition in order to display the dat files you must change the extension option in the top portion of DIFAS from Text txt to Data dat 3 IMPORTING DATA FROM EXCEL FILES Data from Excel can be saved as tab delimited txt file In this case ensure that in the created txt file the first row of the file consists of data not variable names Next ensure that the type of delimiter in DIFAS is set to tab delimited Failing to do so will result in DIFAS displaying a message that an invalid entry was made In addition in order to display the txt files you must ensure that extension option Text txt is selected in the top portion of the Main Window Navigating the Imported Data File Once a file has been successfully imported you may navigate through the file one cell at a time using the Navigate Data File window contained within the Data menu Opening this window you will find a central box that displays the contents of any selected element of the imported data file By default the initial element is that contained in the first row and first column of the data file By clicking Up Down Left and Right you may view successive elemen
7. third list called Select Stratifying is used to designate the variable used to stratify the individuals according to an estimate of ability Note that by default DIFAS assumes that the stratification variable is the summated test score of the items selected for the DIF analysis as noted by the selection of the Stratify by Sum option at the bottom of the Select Stratifying list However the user may select any variable contained in the imported data file to use as the stratifying variable To run DIF analyses for dichotomous items follow these steps 1 In the Select Items list the list at the far left of the Dichotomous Models Window select the items to be studied for DIF Note that by default the Range of Items option is selected In this mode a range of items can be selected by first selecting the lower item of the range and then the upper item of the range For example to select items 2 through 10 first select item 2 and then select item 10 This will cause all items between 2 and 10 to be selected To turn off the Range of Items option click on the Individual Items option 2 In the Select Groups list the list in the middle of the Dichotomous Models Window select the variable designating the groups of interest DIFAS permits only the comparison of two groups at a time Below the variable selection box specify the value designating reference group members and the value designating f
8. A MUST BE NUMERIC All data must be numeric If any nonnumeric e g text entries exist DIFAS will not be able to input the data and an error message will appear informing you of that 2 ITEM RESPONSES MUST BE CODED 0 1 2 9 All response data for items to be analyzed for DIF must have values that set 0 equal to the lowest possible response on the item and all other responses consist of integer values sequentially greater than 0 e g 1 2 3 9 Thus for multiple choice item formats the responses must be coded as 0 incorrect and 1 correct For polytomous item formats the responses must be coded as 0 1 2 k Note that the highest possible item level response category that DIFAS will permit is 9 To summarize for all item formats dichotomous and polytomous the lowest response option must be coded 0 and each successively higher response option is coded according to integers successively greater than 0 Any integer value greater than 9 can also be included in the data but the DIF analyses conducted by DIFAS will only read item level responses between 0 and 9 any value greater than 9 would be removed from DIF analyses DIFAS 5 0 User s Manual 7 3 ALL DATA MUST BE INTEGER DIFAS assumes that all data entered is integer Thus not only is it important for item responses to be integer in nature as stated in 2 above but any other variables concerning groups or external ability estimates should also consists of inte
9. AS computes the total test score any individual having a missing value is assigned a missing value for the total test score and thus is ignored in the analyses Thus if you wish a total test score to be computed for individuals having missing values be sure to code all missing values as 0 before importing the data file into DIFAS If you wish the stratifying variable to be some other variable in the data file click the option Stratify by external and then select the external variable from the Select Stratify list Once these three steps are completed click on the OK button The resulting output consists of two tables a table containing the relevant DIF statistics and a table containing the conditional differences in mean item score between the reference and focal groups at ten intervals across the stratifying variable continuum the lower and upper DIFAS 5 0 User s Manual 17 limits of each of the ten intervals are presented at the top of each interval An example of output for a DIF analysis of ten polytomous items is DIF STATISTICS POLYTOMOUS ITEMS Name Mante L A LOR LOR SE LOR Z Var 2 94 397 0 948 0 1 9 48 Var 3 0 06 0 023 0 093 0 247 Var 4 0 508 0 067 0 094 0 713 Var 5 0 748 0 079 0 093 0 849 Var 6 4 55 0 2 0 094 2 128 Var 7 1 6 0 119 0 093 1 28 Var 8 1 428 0 112 0 093 1 204 Var 9 1 50 0 114 0 094 314213 Var 10 0 543 0 069 0 093 0 742 Var 11 2 797 0 156 0 093 1 677 CONDITIONAL DI
10. FAS click the button marked Import Selected Data File in the upper right portion of the Main Window If DIFAS recognizes every element of the file as being valid i e numeric then the box labeled File Currently in Processor will list the file name and a message will appear in the output box specifying the file that was imported and the number of cases and variables in the file An example of such a message is as follows Opened the text file C playdata dif2 txt Number of Cases 400 Number of Variables 12 If an invalid element exists such as a non numeric entry then DIFAS will provide a message box stating so and listing the row and column of the file that had the invalid value Here are some guidelines to follow when opening files created by common statistical and data management programs 1 ENSURE THAT THE CORRECT DELIMITER IS SELECTED A common error that occurs in importing data into DIFAS is that the proper delimiter is not selected in the upper left portion of the Main Window and as a result DIFAS treats the incorrect delimiter as text which causes an error 2 IMPORTING DATA FROM SPSS FILES Data from SPSS can be saved as a tab delimited dat file In this case the resulting dat file will contain the variable names in the first row of the data file These names must be deleted from the file to be imported DIFAS 5 0 User s Manual 10 into DIFAS because DIFAS would mistake the first line
11. FFERENCES Intervals of size 3 Lower 0 3 6 9 12 1 5 18 21 24 27 Upper 3 6 9 12 15 18 21 24 27 30 1 Var 2 0 2 0 05 0 28 0 27 0 3 0 24 0 31 0 18 0 23 0 14 Var 3 0 11 0 03 0 13 0 02 0 11 0512 0 05 0 0 12 0 05 Var 4 0 05 0 12 0 0 08 0 04 0 14 055 1 0 02 0 01 0 29 Var 5 0 04 0 06 0 01 0 07 0 0 04 0 11 0 11 0 06 0 12 Var 6 0 05 0 05 0 1 0 02 0 1 0 01 0 09 0 02 0 04 O 1 Var 7 0 12 0 08 0 13 0 17 0 01 0 05 0 02 0 01 0 08 0 12 Var 8 0 2 0 12 0 12 0 03 0 01 0 1 0 06 0 04 0 02 O41 Var 9 0 04 0 06 0 15 0 03 0 04 0 06 0 06 0 08 0 01 0 07 Var 10 0 2 0 02 0 01 0 03 0 02 0 01 0 02 0 14 0 07 0 01 Var 11 0 07 0 02 0 04 0 06 0 09 0 08 0 02 0 01 0 04 0 04 Mantel Chi Square Mantel The Mantel chi square statistic Mantel 1963 Zwick Donoghue amp Grima 1993 Zwick Thayer amp Mazzeo 1997 is distributed as chi square with one degree of freedom Critical values of this statistic are 3 84 for a Type I error rate of 0 05 and 6 63 for a Type I error rate of 0 01 Liu Agresti Cumulative Common Log Odds Ratio L A LOR The Liu Agresti cummulative common log odds ratio Liu amp Agresti 1996 Penfield amp Algina 2003 is asymptotically normally distributed Positive values indicate DIF in favor of the reference group and negative values indicate DIF in favor of the focal groups Standard Error of the Lui Agresti Cumulative Common Log Odds Ratio LOR SE The estimated standard error of
12. W Holland amp H Wainer Eds Differential item functioning pp 337 364 Hillsdale NJ Lawrence Erlbaum Zwick R Donoghue J R amp Grima A 1993 Assessment of differential item functioning for performance tasks Journal of Educational Measurement 30 233 251 Zwick R Thayer D T amp Mazzeo J 1997 Descriptive and inferential procedures for assessing differential item functioning in polytomous items Applied Measurement in Education 10 321 334 DIFAS 5 0 User s Manual 25
13. been symbolized by T tau squared in the literature and the variance of DIF effect across mixtures of polytomous and dichotomous items has been symbolized by v nu squared in the literature If the test contains all dichotomous items then select Dichotomus Models in the Differential Test Functioning Menu option If the test contains all polytomous items or a mixture of polytomous and dichotomous items then select Polytmous Models in the Differential Test Functioning Menu option To conduct DTF analyses select the appropriate items contained in the test 1 g the items for which the DIF effect variance is desired select the grouping variable and specify the stratifying variable These steps mirror those described for DIF analyses above DIFAS will print out the following information 1 Weighted and unweighted estimates of the DIF effect variance 2 Standard error estimators of the weighted and unweighted estimates of the DIF effect variance 3 The ratio of each DIF effect variance estimate over its respective standard error estimator note that the variance estimator is not normally distributed so the interpretation of this ratio has not yet been given any guidelines in the literature Readers are referred to the papers by Camilli and Penfield 1997 and Penfield and Algina 2006 for more detailed information about the interpretation and computations of these estimators DIFAS 5 0 User s Manual 19 Differentia
14. bels variables as Varl Var2 etc and so it is often useful to have a listing of the variable ordering and or column numbers at your fingertips when using DIFAS 6 EXTERNAL ABILITY ESTIMATE CAN BE INCLUDED Because DIF analyses are predicated upon examination of a relationship between a grouping variable and item performance conditional on an estimate of ability DIFAS must be instructed as to the nature and location of the estimate of ability By default DIFAS uses the sum of the responses to the items included in the DIF analysis the summated test score as an internal measure of ability If the summated test score is the desired measure of ability DIFAS 5 0 User s Manual 8 then no other estimate of ability need be included in data file If however the measure of ability is other than the summated test score i e there is an external measure of ability then the external measure of ability must be included in the data file prior to importing the data into DIFAS Note however that DIFAS assumes all data is integer in nature and so if the measure of ability contains decimals then the ability measure would need to be transformed to eliminate the decimals For example if the ability measure consists of a standard normal variable then the user can multiply the standard variable by 1000 if 3 decimal places are desired and truncate the resulting variable accordingly to remove any resulting decimal places Negative values should not pose
15. contain a large amount of output the space is not infinite As a result you should avoid storing large amounts of output at a single time without deleting that output which is not crucial In general the output box should be capable of storing at least several thousand lines of output before experiencing capacity limitations Editing the Output The output box is a simple text editor and acts like most other data editor programs You may add text at any time to the output box For example you may want to annotate the results providing additional information concerning the data file Contents of the output box may be cut copied and pasted using either the appropriate options of the Edit Menu or using the shortcuts Control x Control c and Control v Thus you can cut contents of the output box and paste the contents into an external document Remember however that if you are including the contents of the output box in an external word processing file such as a Word or a WordPerfect file the output will only be aligned if the font is Courier Thus you may need to change the font of the output after pasting the desired output into the new file In addition to prevent output that wraps the font size may need to be changed in the new word processing file 10 point Courier font should be fine The entire contents of the output box can be erased using the Clear Output option of the Edit Menu Using this option immediately clears all contents
16. ct the appropriate analysis from the Analyses drop down menu Step 3 Examine the output of the results of analyses in the output box of the Main Window of DIFAS DIFAS adds the output of subsequent analyses sequentially to the output displayed in the output box The output can be annotated edited or saved as a text file upon request DIFAS 5 0 User s Manual 6 CHAPTER 2 Importing Data The first step in running DIF analyses using DIFAS is to import the desired data file into DIFAS DIFAS can accommodate data sets containing up to 1000 variables There is no theoretical limit that DIFAS places on the number of cases contained in the data file the limiting factor would be the memory capabilities of the computer running DIFAS DIFAS can only read in text and data files that have the extension txt or dat Because it will often be the case that the data of interest is being maintained in either a statistical software package e g SPSS or a data management package e g Excel you may have to save the data file of interest in a format that DIFAS can read There are some simple steps that can be taken to properly format the data file so that DIFAS can read its contents as outlined in the following section Preparing the Data for Import Before importing the desired data file into DIFAS it is critical that the data be formatted properly With respect to the values of the data DIFAS has the following constraints 1 DAT
17. ct the items to be studied for DIF Note that by default the Range of Items option is selected In this mode a range of items can be selected by first selecting the lower item of the range and then the upper item of the range For example to select items 2 through 10 first select item 2 and then select item 10 This will cause all items between 2 and 10 to be selected To turn off the Range of Items option click on the Individual Items option 2 In the Select Groups list the list in the middle of the Polytomous Models Window select the variable designating the groups of interest DIFAS permits only the comparison of two groups at a time Below the variable selection box specify the value designating reference group members and the value designating focal group members For example the imported data file may use a 1 to designate reference group members and a 2 to designate focal group members Any two numeric values may be used to designate the reference and focal groups 3 In the Select Stratify list the list at the far right of the Polytomous Models Window specify the stratifying or matching variable If you wish the total test score obtained from the items selected to be included in the DIF analysis to serve as the stratifying variable ensure that the option Stratify by sum is selected In this case no variable needs to be selected from the Select Stratify list Note that when DIF
18. estimator presented by Robins Breslow and Greenland 1986 Standardized Mantel Haenszel Log Odds Ratio LOR Z This is the Mantel Haenszel log odds ratio divided by the estimated standard error A value greater than 2 0 or less than 2 0 may be considered evidence of the presence of DIF Breslow Day Chi Square BD The Breslow Day chi square test of trend in odds ratio heterogeneity Breslow amp Day 1980 Penfield 2003 is distributed as chi square with one degree of freedom Critical values of this statistic are 3 84 for a Type I error rate of 0 05 and 6 63 for a Type I error rate of 0 01 This statistic has been shown to be effective at detecting nonuniform DIF Combined Decision Rule CDR The combined decision rule CDR flags any item for which either the Mantel Haenszel chi square or the Breslow Day chi square statistic is significant at a Type I error rate of 0 025 Penfield 2003 The message OK is printed if neither statistic is significant and the message FLAG is printed if either statistic is significant The ETS Categorization Scheme ETS The ETS categorization scheme Zieky 1993 categorizes items as having small A moderate B and large C levels of DIF DIF Analyses for Polytomous Items The Polytomous Models Window of the Analyses Menu conducts DIF analyses for polytomous items by computing the following statistic the Mantel chi square the Liu Agresti cumulative common log odds ratio the estima
19. ger values Non integer values can be read in by DIFAS but only the truncated values are used in analyses An important consequence of this is that any external measure of ability that is to be used for stratifying must be integer As a result ability estimates that are initially in decimal form should be transformed to an integer form as discussed below prior to importing the data into DIFAS 4 MISSING VALUES MUST BE DENOTED BY Missing values are permitted by DIFAS but must be indicated by the period symbol or the absence of any value between two delimiters For example the line 0 0 1 indicates four responses where the third response is missing Alternatively the line 0 0 1 could also be used to indicate four responses where the third response is missing Note that DIFAS will not recognize a space as a missing value because a space is used as a delimiter in space delimited files 5 DATA BEGINS ON FIRST LINE OF FILE When reading in data DIFAS assumes that the data begins in the first line of the file being read in Thus a first line containing variable names is not permitted when importing data For example a space delimited data set containing responses of three individuals on five items might look like 10110 11111 00010 The same data in comma delimited form would look like 1 0 1 1 0 1 1 1 1 1 0 0 0 1 0 And the same data in tab delimited form would look like 1 0 1 1 1 1 1 1 1 0 0 0 1 0 DIFAS la
20. ical aspects of the analysis of data from retrospective studies of disease Journal of the National Cancer Institute 22 719 748 Penfield R D 2003 Application of the Breslow Day test of trend in odds ratio heterogeneity to the detection of nonuniform DIF Alberta Journal of Educational Research 49 231 243 Penfield R D 2007a Assessing differential step functioning in polytomous items using a common odds ratio estimator Journal of Educational Measurement 44 187 210 Penfield R D 2007b April Estimating differential step functioning effects under the graded response and generalized partial credit models Paper presented at the annual meeting of the National Council on Measurement in Education Chicago DIFAS 5 0 User s Manual 24 Penfield R D amp Algina J 2003 Applying the Liu Agresti Estimator of the Cumulative Common Odds Ratio to DIF Detection in Polytomous Items Journal of Educational Measurement 40 353 370 Penfield R D amp Algina J 2006 A generalized DIF effect variance estimator for measuring unsigned differential test functioning in mixed format tests Journal of Educational Measurement 43 295 312 Robins J Breslow N amp Greenland S 1986 Estimators of the Mantel Haenszel variance consistent in both sparse data and large strata limiting models Biometrics 42 311 323 Zieky M 1993 Practical questions in the use of DIF statistics in item development In P
21. icrosoft NT or later e Pentium 90MHz or higher Running DIFAS To run DIFAS double click on the program icon DIFAS exe The program itself is contained in a single file that is about 0 22 megabytes 220 kilobytes and thus can be copied on a single floppy disk DIFAS 5 0 User s Manual 4 A Diagram of the DIFAS Main Window The DIFAS Main Window has the following components This is the box where the type of delimiter in the data file is specified This box displays the name of the currently selected file Use these boxes to This box displays the name locate desired data file of the file currently in the DIFAS processor Menu for all DIF and Use this box to view Click on this button to statistical analyses txt or dat files import the selected file into the DIFAS processor File Edit Data Analyses N elect Data EES Se Type of Delimiter E Ooo E TH File Curren TOREN in Processor i m Import Selected Data File This is the output box where DIFAS output will be displayed DIFAS 5 0 User s Manual 5 Conducting Analyses Using DIFAS Conducting analyses using DIFAS consists of three primary steps Step 1 Import the data to be analyzed Select the appropriate file to be imported the file must have either a txt or a dat extension using the top portion of the Main Window of DIFAS Imported files must be delimited as space delimited comma delimited or tab delimited Step 2 Sele
22. l Step Functioning Differential step functioning DSF pertains to invariance observed at each step underlying the polytomous response variable a polytomous item with J score levels will have J 1 steps Because DSF effects may vary in sign and or magnitude across the steps examination of DSF effects can prove useful in understanding the location of the DIF effect i e which score levels are manifesting the DIF effect and the potential causes of the DIF effect Estimates of DSF effects can be computed within the Cumulative and Adjacent Categories Windows of the Analyses Menu The cumulative approach yields odds ratio DSF effect estimators consistent with those modeled under the graded response model and the adjacent categories approach yields odds ratio DSF effect estimators consistent with those modeled under the generalized partial credit model Descriptions of these estimators can be found in Penfield 2007a 2007b To conduct DSF analyses select the appropriate items contained in the test 1 g the items for which the DSF effects are desired select the grouping variable and specify the stratifying variable These steps mirror those described for DIF analyses above For each item DIFAS will print out a table containing the following information for each step of each selected item 1 A step level log odds ratio DSF effect estimator This is denoted by CU LOR using the cumulative approach and AC LOR using the adjacent categories app
23. m width of 2 is desired then the user would change the 1 to a 2 in the Stratum Size box 5 DIFAS can print out the number of reference and focal group members located in each stratum of the stratifying variable To do so select the Print strata n check box in the lower right portion of the Dichotomous Models Window Once these three steps are completed click on the OK button The resulting output consists of two tables a table containing the relevant DIF statistics and a table containing the conditional differences in mean item score between the reference and focal groups at ten intervals across the stratifying variable continuum the lower and upper limits of each of the ten intervals are presented at the top of each interval An example of output for a DIF analysis of ten dichotomous items is DIF STATISTICS DICHOTOMOUS ITEMS Name MH CH MH LOR LOR SE LOR Z BD CDR ETS Var 2 0 01 0 052 0 233 0 223 0 034 O A Var 3 1 786 0 358 0 246 1 455 0 O A Var 4 0 189 0 128 0 23 0 557 2 776 O A Var 5 0 0 031 0 236 O LSL 0 03 OK A Var 6 0 16 O 13 05 25 0 52 0 043 O A Var 7 0 323 0 162 0 238 0 681 1 156 OK A Var 8 0 00 0 036 0 238 Ors I5 2 109 OK A Var 9 0 013 0 053 0 229 0 231 1 11 O A Var 10 0 00 0 gt 025 07252 gt 03 099 0 016 O A Var 11 0 42 0 194 0 25 0 776 0 202 O A CONDITIONAL DIFFERENCES Intervals of size 1 Lower 0 1 2 3 4 5 6 7 8 9 Upper 1 2 3 4 5 6 7 8 9 10 1 Var 2 0 07 30 51 0T 0 14 0 3 0 01 0 07
24. ning general information about the variables held in the DIFAS processor Note that by default the Range of Items option is selected in the Frequencies Window In this mode a range of variables can be selected by first selecting the lower variable of the range and then the upper variable of the range For example to select variables 2 through 10 first select variable 2 and then select variable 10 This will cause all variables between 2 and 10 to be selected To turn off the Range of Items option click on the Individual Items option DIFAS 5 0 User s Manual 13 DIF Analyses for Dichotomous Items The Dichotomous Models Window conducts DIF analyses for dichotomous items by computing the following statistics the Mantel Haenszel chi square the Mantel Haenszel log odds ratio the estimated standard error of the Mantel Haenszel log odds ratio the standardized Mantel Haenszel log odds ratio the Breslow Day chi square test of trend in odds ratio heterogeneity the Combined Decision Rule classification scheme and the ETS classification scheme These procedures are described later in this section The Dichotomous Models Window contains three lists of the variables contained in the imported data file The first list called Select Items is used to designate the items to be included in the DIF analysis The second list called Select Groups is used to designate the variable containing the grouping information The
25. ocal group members For example the imported data file may use a 1 to designate reference group members and a 2 to designate focal group members Any two numeric values may be used to designate the reference and focal groups 3 In the Select Stratify list the list at the far right of the Dichotomous Models Window specify the stratifying or matching variable If you wish the total test score obtained from the items selected to be included in the DIF analysis to serve as the stratifying variable ensure that the option Stratify by sum is selected In this case no variable needs to be selected from the Select Stratify list Note that when DIFAS computes the total test score any individual having a missing value is assigned a missing value for the total test score and thus is ignored in the analyses Thus if you wish a total test score to be computed for individuals having missing values be sure to code all DIFAS 5 0 User s Manual 14 missing values as 0 before importing the data file into DIFAS If you wish the stratifying variable to be some other variable in the data file click the option Stratify by external and then select the external variable from the Select Stratify list 4 By default DIFAS uses a stratum size of 1 You can however change this size by specifying a stratum size in the box located in the lower right portion of the Dichotomous Models Window For example if a stratu
26. of the output box and there is no way to retrieve any erased contents Thus this function should only be used if you are completely certain that the output is to be deleted DIFAS 5 0 User s Manual 22 Saving Output Files To save the contents of the output box select the Save Output option of the File Menu A window will appear that permits you to save the file The saved file is a text file and has the extension txt As a result this file can be opened in any other word processing application such as Notepad Word or WordPerfect Note however that the output will be appropriately formatted only if the font is set to Courier in the new application Opening Output Files To open the contents of the output box select the Open Output option of the File Menu A window will appear that permits you to open the appropriate file DIFAS will only open text files having the txt extension Printing Output DIFAS does not have the capability to print However output may be printed by first saving the output to an external file see Saving Output Files opening the external file in a word processing application such as Word Notepad or WordPerfect and then printing the output from the word processing application Note however that the output will be appropriately formatted only if the font is set to Courier in the new application Notepad typically uses Courier as the font by default and thus opening the saved output in N
27. or which the stratifying variable value is missing will be omitted from the entire analysis and b individuals having a nonmissing value of the external stratifying variable but a missing value on one or more particular items will be removed from the DIF analysis only for the item s for which the response is missing When a substantial number of missing responses exist the DIF analyst may find it useful to compute an external stratifying variable the controls for the impact of missing data on trait estimation One approach is to use the mean value across just the items attempted as the stratifying variable In this situation the appropriate stratifying variable would need to be computed prior to importing the data into DIFAS Also because DIFAS only stratifies on integer values the user would need to transform the obtained mean values to integer level values via some form of rounding or rescaling DIFAS 5 0 User s Manual 21 CHAPTER 4 Managing Output All output is printed in the output box on the Main Window of DIFAS DIFAS allows you to save the contents of the output box open a previously saved output file or augment the contents of the output as you deem appropriate Properties of the Output The output of DIFAS is completely text in nature The font of the output is Courier 10 point The use of the Courier font is critical so that the output is formatted in columns that are appropriately aligned Although the output box can
28. otepad provides a simple strategy for printing DIFAS output DIFAS 5 0 User s Manual 23 REFERENCES Breslow N E amp Day N E 1980 Statistical methods in cancer research Volume I The analysis of case control studies Lyon International Agency for Research on Cancer Camilli G amp Congdon P 1999 Application of a method of estimating DIF for polytomous test items Journal of Educational and Behavioral Statistics 24 323 341 Camilli G amp Penfield D A 1997 Variance estimation for differential test functioning based on Mantel Haenszel statistics Journal of Educational Measurement 34 123 139 Camilli G amp Shepard L A 1994 Methods for identifying biased test items Newbury Park CA Sage Cox D R 1958 The regression analysis of binary sequences Journal of the Royal Statistical Society B 20 215 232 Holland P W amp Thayer D T 1988 Differential item performance and the Mantel Haenszel procedure In H Wainer amp H I Braun Eds Test validity pp 129 145 Hillsdale NJ Lawrence Erlbaum Liu I M amp Agresti A 1996 Mantel Haenszel type inference for cumulative odds ratios with a stratified ordinal response Biometrics 52 1223 1234 Mantel N 1963 Chi square tests with one degree of freedom Extension of the Mantel Haenszel procedure Journal of the American Statistical Association 58 690 700 Mantel N amp Haenszel W 1959 Statist
29. roach 2 The standard error estimator of the DSF effect estimate 3 The ratio of each DSF effect estimate over its respective standard error estimator 4 The Liu Agresti common log odds ratio DIF effect estimator an estimate of the item level DIF effect and it s associated standard error estimate A sample of the DSF output for one item is shown below This item is showing large DSF effect for the first step DSF for Var 2 Step CU LOR SE Z 1 14239 0 19199 6 453 2 0 401 0 15435 2 598 3 0 23 0 21202 1 085 L A LOR 0 518 LOR SE 0 131 Readers are referred to the papers by Penfield 2007a 2007b for more detailed information about DSF and the interpretation and computations of these DSF effect estimators DIFAS 5 0 User s Manual 20 Missing Data DIFAS allows for missing data but the way in which it deals with missing data depends on how the stratifying variable is computed If the total summated score is used as the stratifying variable the default in DIFAS then DIFAS uses a listwise deletion procedure for any case containing missing data In this case DIFAS will remove any individual in the analysis having one or more missing responses The reason for this is that DIFAS assumes that the total summated score is an invalid stratifying value when missing values are present If however an external variable is used as the stratifying variable then DIFAS will use the following deletion rules a any individual f
30. scriptives function may prove useful in obtaining general information about the variables held in the DIFAS processor Note that by default the Range of Items option is selected in the Descriptives Window In this mode a range of variables can be selected by first selecting the lower variable of the range and then the upper variable of the range For example to select variables 2 through 10 first select variable 2 and then select variable 10 This will cause all variables between 2 and 10 to be selected To turn off the Range of Items option click on the Individual Items option Frequencies The Frequencies Window of the Analyses Menu computes the frequency percentage of the total sample percentage of the sample having valid data and the cumulative percentage of each outcome for each variable selected In addition the Frequencies function computes the number of cases having valid present values and the number of cases having missing values To obtain the frequencies output select the variables of interest by left clicking in the check box associated of the desired variables and then click the OK button An example of the output generated by the frequencies function is given by FREQUENCIES AND PERCENTAGES FOR Var 1 Value Freq Total Pres Cum 1 200 50 50 50 2 200 50 50 100 Present 400 100 Missing 0 0 Total 400 100 The use of the frequencies function may prove useful in obtai
31. ted standard error of the Liu Agresti cumulative common log odds ratio the standardized Liu Agresti cumulative common log odds ratio Cox s estimator of the multivariate hypergeometric noncentrality parameter the estimated standard error of Cox s estimator of the multivariate hypergeometric noncentrality parameter and the standardized value of Cox s estimator These procedures are described at the end of this section DIFAS 5 0 User s Manual 16 The Polytomous Models Window contains three lists of the variables contained in the imported data file The first list called Select Items is used to designate the items to be included in the DIF analysis The second list called Select Groups is used to designate the variable containing the grouping information The third list called Select Stratifying is used to designate the variable used to stratify the individuals according to an estimate of ability Note that by default DIFAS assumes that the stratification variable is the summated test score of the items selected for the DIF analysis as noted by the selection of the Stratify by Sum option at the bottom of the Select Stratifying list However the user may select any variable contained in the imported data file to use as the stratifying variable To run DIF analyses for polytomous items follow these steps 1 In the Select Items list the list at the far left of the Polytomous Models Window sele
32. the Lui Agresti Cumulative Common Log Odds Ratio Liu amp Agresti 1996 Standardized Liu Agresti Cummulative Common Log Odds Ratio LOR Z This is the Liu Agresti cumulative common log odds ratio divided by the estimated standard error A value greater than 2 0 or less than 2 0 may be considered evidence of the presence of DIF DIFAS 5 0 User s Manual 18 Cox s Noncentrality Parameter Estimator COX S B Cox s estimator of the multivariate hypergeometric noncentrality parameter Camilli amp Congdon 1999 Cox 1958 is distributed approximately as standard normal Positive values indicate DIF in favor of the reference group and negative values indicate DIF in favor of the focal groups Standard Error of Cox s Noncentrality Parameter Estimator COX SE The estimated standard error of Cox s noncentrality parameter Camilli amp Congdon 1999 Standardized Cox s Noncentrality Parameter COX Z This is Cox s noncentrality parameter estimator divided by the estimated standard error A value greater than 2 0 or less than 2 0 may be considered evidence of the presence of DIF Differential Test Functioning The variance of DIF effect across the items of a test or scale has been proposed as a measure of DTF for sets of dichotomous items Camilli amp Penfield 1997 and mixtures of dichotomous and or polytomous items Penfield amp Algina 2006 The variance of DIF effect across dichotomous items has
33. ts in the data file This function permits you to ensure that the contents of the data file are as expected DIFAS 5 0 User s Manual 11 CHAPTER 3 Conducting Analyses The analyses offered by DIFAS are located using the Analyses Menu of DIFAS DIFAS includes several analysis windows that can be used to conduct a variety of analyses related to DIF detection The options on the Analyses Menu are as follows gt Descriptives gt Frequencies gt Nonparametric DIF Tests Dichotomous Models Polytomous Models gt Differential Test Functioning Dichotomous Models Polytomous Models gt Differential Step Functioning Cumulative Adjacent Categories Descriptives The Descriptives Window of the Analyses Menu computes the mean standard deviation minimum maximum and number of valid cases for each variable selected To obtain the descriptive statistics select the variables of interest by left clicking in the check box associated of the desired variables and then click the OK button An example of the output generated by the descriptives function is given by DESCRIPTIVES Name Mean SD Min Max N Var 1 135 0 501 1 2 400 Var 2 03522 Ole 0 1 400 Var 3 0 507 0 501 0 1 400 Var 4 O25 0 501 0 1 400 Var 5 0 438 0 497 0 1 400 Var 6 0 522 0 35 0 1 400 Var 7 0 47 0 5 0 1 400 Var 8 0 488 0 5 0 T 400 Var 9 0 482 Ord 0 i 400 Var 10 0 472 0 55 0 1 400 Var 11 0 502 0 501 0 1 400 DIFAS 5 0 User s Manual 12 The use of the de
Download Pdf Manuals
Related Search