Home

2 - Quia

1. 61 4 Pre Scored ite MNS iiaea v 67 5 Open ended items iicicc ci odicetsnnee ces sade cet cates cascade ccccandecestuedecotcaccscasuaddicctsangecesenddzcetcoecesass 69 2003 2011 Curtin University West Australia Contents 6 Remove an ite m iie en ook asa an o FERE ga SOR aa ESAE SES eS a Oe PReRe oso ness o ProRSRS I ESNsinAOaESR IS 69 T Excluding items te 71 8 Pretest trial items odriin ainai ae aiaa Peai a Ea Paa iaiaeiaeiaa Eaa 73 9 Split half reliability eeeeeeeeen eren nnnhnn nnn nh nnne nnn nnne nn nhan 73 10 Filtering records i iere eei Use cceen detec abeeesciee Kaa KAN EANNA NAN RENTREE 73 11 HOW COs cards WOIK 5 2 ccies eaaa bene sued icaancaeiccencadecuenaatishecsgoccusncavieusseesncaeecnes 74 Part IV Toolbar and tab 78 Beet 2 e ETT EEE PERRET EPP ECCE CER EE ECE RET PECELERTCECCEECEE CECE PET PECTET ECE E EERE CECT EECE ERT TET EE 79 LAB TDGIO Ig re 80 dic cc 80 Yelow smiley FACE X 81 Datta Gtr y AIG iociccccnccniensncnanestcenssnncecsnnssdpancsosassesseaiadsiiecuscusbunisncencensuasdadddisnsansvesesteatnanssdesacraatansatussvscransedenterane 82 Sort A tO Z 2202s EE Line grapher 3 E MOMUiceicrads EE E A T T T E E T T Blanks E E E T A A eters E E A A EE ERA 87
2. 323 Apr 2004 Ver 5 4 324 Feb 2004 324 Nov 2003 324 Oct 2003 Ver 5 3 324 En mE ET 325 AUG 2003 Ver 5 2 5 m c M RBS 325 Mp T 325 JUN 20O 325 May 2003 Vier 522 pP 325 2 RESOUICES iiiin2s ced aaae aa ea aapa ae a ad aaa aaa aaaea Ei aaea Kaa aN siveaiebecscaeueeeced 326 E EIEEE A ESAE DEI 326 2003 2011 Curtin University West Australia Help file for Lertap 5 4 ROW CH EET E E TE ceeddacsteccadsicseczed cach lesalzecedasntezes 326 Index 328 2003 2011 Curtin University West Australia Welcome 1 1 Welcome X Delete fj Sort C Blank XD Interpret ill Histograms Move Q Version R Headers Elmillon iz Scatterplot License T Spread Excel a5 Copy More b Res charts Lelp Basic options New menu Run menu Graphics trio Other menus Welcome to Lertap help Lelp Lelp this document is designed to be used online in conjunction with the Lertap 5 item test and survey analysis system In particular it is meant to be used with the versions of Lertap made to work with Excel 2007 Excel 2010 and Excel 2013 Users with Macintosh computers and Excel 2011 will also find topics in this document which apply however the Excel 2011 version uses th
3. M 4 gt Statsib lt csemi Statsiul lt StatsiulCH 4 T gt Ready J 2 100 2 g Statslul reports always have a little table at the bottom such as that shown above In this case Lertap has formed five groups each with about 20 of the students The number of students in each group The mean or average of the test scores for the group avg The avg figure expressed as a percentage of the maximum possible score which was 60 in this case d The standard deviation of the scores for the group Computed as a population value If you re familiar with the equations used to calculate this statistic the one used here has n in the denominator not n 1 min The lowest or minimum score found in the group min The lowest or minimum score found in the group lowest or minimum score found in the group man The median of the group s scores the 50th percentile The median of the group s scores the 50th percentile The highest or maximum score found in the group 2003 2011 Curtin University West Australia 304 Help file for Lertap 5 Related tidbits Quintile plots have given rise to the term visual eye tem analysis as seen in this paper How to print Lertap s reports See this topic 6 4 3 3 1 With external criterion Statslul reports become ECStats1lul reports when an external criterion e analysis has been selected The d
4. 1 mas 1 Ll 2 2 3 i 3 _ 3 mao 4 0 20 awe 4 0 20 w 4 0 20 oher 3 E 0 00 oher i 2 00 oher T T T 0 00 lower 4th 3ra 2nd upper lower 4th 3rd 2nd upper lower 4th 3rd 2nd upper Key 4 Dif S0 Disc 41 Key 2 OWf 59 Disc 47 Key 1 Dif 54 Disc 47 18 19 v M4 M lt csemi lt Statsiul lt StatsiulChta PackedPlots3 Packed 4 Mm gt i Ready 73 ao oo 70 The number of blank rows or lines is controlled by Row 71 in the System worksheet If the interactive mode setting in Row 61 is set to yes then the values entered in Rows 70 and 71 are ignored and you re asked to enter them as soon as you ve activated ChartChanger3 The Page settings rows in the System sheet are there to make it easier to print packed plots You ll have to experiment with these finding the best values by using Excel s many print options and then recording the settings which seem to work best for you in Rows 64 to 69 of the System worksheet There s a special topic in the Lertap sample datasets website which discusses printing quintiles Have a look at it and then note that ChartChanger3 makes it easier to re size charts and align them to the Excel grid Much easier Much much easier After ChartChanger3 has run it will automatically select all of the packed plots and wait for you to re size them And
5. Over the years and versions Excel has at times placed a limit on the number of charts or plots which it will allow to be held in computer memory at any given time This would be one possible reason to answer No to this question but in this example a Yes answer is most appropriate so that you can see what the group response charts look like ertas xf Okay You have only two groups and are working with a cognitive subtest Want to go for a Mantel Haenszel analysis to check for differential item Functioning DIF If you say No then you ll get an ordinary group response report for each item Yes No Cancel Here the answer will be No for the moment This matter of DIF opens up a whole new topic one which will be covered laterfs Once you ve answered its questions Ibreaks gets down to work It ll make a table of results for each item and a response chart too if you asked for charts 2003 2011 Curtin University West Australia 16 Help file for Lertap 5 r a id 29 E v EjemploBangsaend4 xlsx Microsoft Excel non commercial use 5X Lertap Home Insert Page Layout Formulas Data Review View Developer Add ins ox Lertap5 item responses for score Testi grouped by Lengua 2 groups Ee ee Ing 15 30 5 9 40 Nat 39 0 876 0 39 0 49 F sig eta 0 15 0 70 0 00 I1 Ing 2096 4 10 096 T T T T T 1 A B c D
6. _D _ 0 to 9 If the option corresponds to the keyed correct answer then the option is underlined as are all the statistics for the option wt The number of scoring points given for selecting the option n The number of students who selected the option The proportion of students who selected the option Multiply this by 100 to get the corresponding percent In option The response option selected by the student Response options may be letters upper or lower case or digits from 2003 2011 Curtin University West Australia Input amp Output the example above 56 of the students selected option D on 121 pb ec The point biserial correlation with the external criterion score To compute this all those who selected the option are given a score of 1 while those who did not are given a score of 0 These scores are then correlated with the external criterion score b ec The biserial correlation with the external criterion score Note that it is possible for b ec to be greater than 1 00 avg The average external criterion score for those who ec selected the option On 121 above the 3 072 students who selected option D the keyed correct option had an average external criterion score of 27 49 The ba flag is showing because this avg ec was below the mean of the external criterion scores a fact which is also flagged by the negative z of 0 12 The aa
7. 2003 2011 Curtin University West Australia Toolbar and tab n F3 Microsoft Excel LRTPQuizWithExtraColumns xls File Edit wiew Insert Format Tools Data Window Help x amp new SS fX puny SL M l lil Moves Q License Lep D 2A1 1X 11 1X 254A1 9 0000010001000001000000000 31 0011011101111000110010000 26 0101110101011111001000100 27 1101100100011111001000000 21 1110100100101110111000110 59 0111111101111011110100111 47 1111111010000001011010101 42 1101011111101111111100111 55 1101111111111101111010101 51 1111111111111111111110111 3 0010100111110110011000100 2 2 1 3 3 1 1 I 2 ak ee m a rE AA a M 4 milf Statsof Stats2b IStats DAT Ready Notice how the Fortran format statement has changed In this case the statement is in fact correct that is as far as Bilog MG is concerned However as more columns are copied over from the Data worksheet the format statement continues to use I as a field identifier which may or may not be correct as far as Bilog MG is concerned For example we brought over a numeric column from the Data worksheet prompting the DAT file to look like this 2003 2011 Curtin University West Australia 212 Help file for Lertap 5 4 6 1 4 E3 Microsoft Excel LATPQuizWithExtraColumns xls SE File Edit View Insert Format Tools Data Window Help XxX New S9 rZ Run 21 M l l Move License Lep 0000010001000001000000000 0011011
8. m mm mmr oru uuuuuu mmm mi m r r r m D B D A C C E D D C c C B E A A B A A A A n The Data worksheet has ID information in its second column some sort of Group code 2003 2011 Curtin University West Australia 210 Help file for Lertap 5 in column 3 and item responses starting in column 4 There were 25 cognitve items each scored on a right wrong basis with one point for the right answer After going through the usual process of using the Run menu to Interpret CCs lines then Elmillon item analysis then Output item scores matrix the DAT worksheet looked like this F3 Microsoft Excel LRTPQuizWithExtraColumns xls i Ek File Edit View Insert Format Tools Data Window Help xX New S rZ pune M l fl Move Q License Lep m 2A1 1X 254A1 l 3d 26 27 59 47 42 55 1 51 20 0000010001000001000000000 0011011101111000110010000 0101110101011111001000100 1101100100011111001000000 1110100100101110111000110 Di11111101111011110100111 1111111010000001011010101 1101011111101111111100111 11011111111111011110103101 11112111111111111111710111 0010100111110110011000100 M 4 gt nif Stats2f f Statsob istats jDar Ready The Move menu was then used to Copy a Data column to the DAT worksheet as we wanted to bring over the Group code from column 3 of the Data sheet After doing this the DAT file changed as shown below
9. 2003 2011 Curtin University West Australia 246 Help file for Lertap 5 mws lines which seem to be required What you need to do is roll up your sleeves perk some fresh coffee and get into the depths of a Sub worksheet You can poke any weights anywhere once you have a look at a Sub worksheet it will hopefully be obvious what to do Here s a snippet from a Sub worksheet A B Cc D E F other 100 100 100 100 100 100 T 1 What we re looking at here is the weights array corresponding to an item from a subtest having Res A B C D E F The item could be an affective one it could be a cognitive one the format of the array is the same The numbers in the boxes all 1 00 in this case correspond to the number of points a person will get if s he chooses one of the options A through F There are four empty boxes for this item they d also have weights points if the item used more response codes What s other It s the number of points a person gets if her his response to the item is not A B C D E or F The person may not have responded at all in which case there might be a blank in the Data worksheet for this person on this item Many times scanners will record unanswered items or funny answers as an asterisk for example when a person has shaded in more than one bubble on the answer sheet Blanks and asterisks are caught by Lertap s other category in fact anything which is not o
10. CD MET ED DI Lace PS SHALS FO POMS soaks ned OtalS I RRC RETR PEET EEE TEE PEER POPES PRET E eh YE E ER S EU EIER E PEE TEP CET Full item statistics With external criterion Difficulty calculations Summary statistics The bands Brief item statistics Stats 1b plot St ts 1b cOlblmtn 2st erret remet e ara edere RT OR rete cr circa taeren oe a e a e EUUREM RDUE Withi external criterion rant E eet ai ET Mastery mode ioo e pde car eee do redde ed t e etd ere be Ree ante COnditiO Mal SEIS eec EC idi 5 import da dac tected castes ccececaudeteddauedereecigsccecsteceesaacsusedauedstecctver TDMA pere e Part VII R amp R amp R amp R 1 REVISIONS P 317 Oct 2006 Vier 5 6 8 eiii cst aaa a a o uto cni eser erase sad iso tud bce vee betes 318 UND 2OOG secs cc a N ace ge a OS 318 Ip JUTe mPc 319 Mar 2006 ec P 319 Jan 2000 Ver 5 6 nda e etie de c ie aa E uade eoi aan aaa aO 319 Lp UE 320 J ly 2005 V 6 BOSE eum 321 May 2005 E 321 Feb 2005 321 Nov 2004 322 Oct 2004 Ver 5 4 5 322 Sep 2004 923 Jul 2004
11. The item responses given by each pair of students are found under the Responses column using a format suggested in the SCheck program from Wesolowsky sed 2000 a full stop or period indicates a correct answer Each of the two students above had 20 correct answers The 2 seen at the start of each response string indicates that both students selected 2 as their response to the first item On this item both students made an error failing to find the correct answer Not only did they both make an error but they made an identical error on the item Wherever the student incorrect responses match they have an exact error in common It s pretty easy to see that the two students had nine matching errors nine exact errors in common Over all 30 of their item responses there were only two response differences The values of the Harpp Hogan measures are found under the EEIC Index and Sigma columns of the RSAcases report Briefly Harpp Hogan methods are based on 1 determining EEIC the number of exact errors found in common in student responses 2 comparing EEIC to D the total number of response differences found a comparison made by dividing EEIC by D producing the Index 3 developing a response probability measure for the pair of students and comparing it to a distribution of similar measures formed from non suspect pairs The probability 2003 2011 Curtin University West Australia 12 Help file for Ler
12. DUALUOIUWWYTO 44 DataRow46 X 1XX4 3 X XX 44XXX1 XXXXX X X XXX 24 65 DataRow67 x 1xx443 x XX2 44xxx1 2XXXXX2X X XXX424 CeA IAAU 232 X nn AAK GUIAAXI AAAXX AX X AAXX Stas Total number of cases above 57 Total number of pink cases 35 d M gt f RS sig8 RSAtable8 RSAcases8 RSAsig7 RSAtablez gt In this example the RSAcases report had 57 entries 57 paired student results Of these 35 were in the pink We might say that our RSA analysis uncovered 35 pairs whose item responses were significantly suspect or in Wesolowsky s terms excessively similar We can t yet say for sure that they cheated but we ve got reason to question their results More than one RSA analysis may be applied to the same RSAdata worksheet As discussed below there are several options which control how an RSA analysis runs it is quite common to specify an analysis which looks not at all students but only at those whose test scores fall within a certain range In some cases we might want to exclude weak test items from the analysis Note the numbering on the reports in the little example shown immediately above RSAcases8 would imply that there were at least eight RSA runs used quite unusual The RSAcases report conveys the essence of Lertap s analysis but two other reports are produced for those who care to delve further into the results 2003 2011 Curtin U
13. Lertap breaks down What s this A break down Nope just had to get your attention There s now a new option on the Run menu which will break out results by groups 2003 2011 Curtin University West Australia 320 Help file for Lertap 5 Say people have sat your test on chemistry at five different campuses of your university You have coded test venue into one of the columns on the Data sheet The new option to Breakout scores by groups will quickly produce a Breaks table with test results organized by group levels and a spiffy graph to match PlotBreaks xxx IMA Read more Plus have a look at growing whiskers immediately below Box your whiskers The Shorts menu also sports a new option Make box and whiskers from Breaks Once you ve got one of the spiffy new Breaks tables mentioned in the previous paragraph this option will make a copy of the table and reformat it so that it will suit one of Excel s built in options for plotting the performance of your stocks Don t have any stocks No matter Excel doesn t really realize what it s plotting the stock performance graph which results comes usefully close to being a real boxplot and it certainly has whiskers Burma shavel 13 The histogrammer now uses improved grammer The good old line printer compatible histogrammer from Lertap 2 days a favorite of many well at least some is now smarter being capable of plotting the scores found on three differ
14. 2003 2011 Curtin University West Australia CCs details 5 West Australian universities are truly world class Strongly agree Agree Undecided Disagree Strongly disagree Ui 4330 N H Those of us based in West Australia would want this item to be scored in a manner which gives the highest score to the first option Ordinarily the first option gets the lowest weight We could use a pol card to reverse this and most Lertappers probably would including us However we said we d step up the tempo we ve started to wade into deeper water An understanding of this example gets into the very basics of Lertap s affective item scoring The default scoring weight applied to any particular response code corresponds to the response code s ordinal position in the Res declaration If 1 or A or 5 is the first response code to appear in Res it gets a scoring weight of 1 but let s say 1 00 to emphasize that we re talking about a real number If Res 1 2 3 4 5 scoring weights are 1 00 2 00 3 00 4 00 and 5 00 If Res A B C D E scoring weights are 1 00 2 00 3 00 4 00 and 5 00 If Resz 5 4 3 2 1 scoring weights are 1 00 2 00 3 00 4 00 and 5 00 The default scoring weights have no correspondence to the actual response codes They re based entirely on the ordinal position of the response codes in the Res string The entries in the Res declaration are never read as numbers Never they re just characters If R
15. Here we have examples of two items which favour the reference group Nat The MH D DIF figure for both of these items 135 and 136 is high Whenever MH D DIF is outside of the range 1 to 1 it can be useful to get a picture which encapsulates some of the information in an IbreaksMH table the next topichs has an example Plots Above it says that asking for a DIF analysis gives two new reports IbreaksMH is one The other is a version of the usual Ibreaks reports this time its charts will come with DIF data included D E Other 14 9 68 48 2 47 71 39 DIF sig alpha delta ETS F sig eta 0 48 0 50_ 2 12 The line of DIF stats has four fields imported from IbreaksMH tables called sig MH alpha unaltered More plots Prob is here is just called alpha MH D DIF becomes delta and ETS is IbreaksMH tables given ample opportunity to make additional Excel charts Examples are given in the next topic Enhance an Ibreaks MH chart 19 2003 2011 Curtin University West Australia Toolbar and tab 185 Related tidbits There s a paper on the Lertap website with more about DIF especially as implemented in Ibreaks Please see http lertap curtin edu au Documentation GimmeABreak1 pdf For DIF references see Angoff 1993 Camilli amp Shepard 1994 Clauser amp Mazor 1998 Dorans amp Holland 1993 Dorans amp Kul
16. Minimum test score for RSA Maximum test score for RSA These two settings determine which students will be included in any RSA analysis A minimum of 0 zero and maximum of 100 will see all students included Note that experienced users of Harpp Hogan methods will often run several RSA analyses for any given test They may start with a 0 100 range for these settings or 30 100 and then reprocess the data with revised settings Allow on the fly min max test score reset If this option is set to yes then Lertap will ask you to enter the minimum and maximum test scores each time you select the Response similarity analysis RSA option from the Run menu This completely over rides the Minimum and Maximum 96 test score settings in the System worksheet Automatically exclude weak items For RSA work weak items are those where the number of students selecting the item s correct answer is less than the number selecting one of the distractors or less than the number of students who omitted the item If this option is set to no then Lertap will pause every time it encounters a weak item asking if you d like to exclude it from the RSA analysis If the option is set to yes then weak items are automatically excluded Excluding weak items is strongly recommended if a test has weak items the EEIC measure will be inflated resulting in more suspects pairs that is more student pairs whose item responses may be judged sus
17. Newmenu Run menu Graphics trio Other menus 330909 35 36 37 38 39 40 41 1 F 35 YrsComp YrsTest Type co 2 Q31 Q32 Q033 Q34 Q Type EN 4 4 4 3 2 0 3 0 1 Private EN 5 4 2 4 2 0 4 0 1 Private 993 4 4 11 4 2 2 0 4 0 2 Public 59 5 3 3 4 0 5 4 0 2 Public 7 4 4 214 1 0 2 5 1 Private 833 5 1 5 4 9 5 12 0 1 Private 03 53 1 4 4 4 5 6 0 2 Public 101 5 4 1 5 4 7 0 6 5 1 Private 7 3 S5 1 4 4 3 5 5 5 2 Public 77 4 4 1 51 4 3 0 5 5 2 Public EM n _ s gt M m mora M M Data CCs qs Scores NEM Ready XJ y In this example an original Type code of 1 has been recoded as Private The new code can be anything as long as it begins with a letter The new codes may have any length but short codes result in Ibreaks tables and charts which tend to look a bit better What s short say 8 characters or less Note the recoder also known as the Recode macro is available via the Move Menu It is also possible to exclude certain cases from Ibreaks such as for example cases with missing data Click here 22 to read more What might happen if group codes are numeric The sky will fall your bicycle tires will go flat A more substantial outcome the charts made by Excel will be wonky incorrect Okay then all s in order for an Ibreaks run Ready to answer a batch of questions Ibreaks will present several as it strives to do what you require We ll be l
18. 2003 2011 Curtin University West Australia Welcome 5 Cli gt Gordon Local Disk C ASC Lertap37x v L Organize s WM SlideShow J Burn Name a Size Type Date modified Desktop T LRTP57HHelp pdf 2 667 KB Adobe Acrobat Document 18 07 2008 9 03 AM EA LRTP57HHelp chm 1 908 KB Compiled HTML Help file 18 07 2008 9 26 AM NT E LertapRibbon xlam 57KB Microsoft Office Excel Add In 30 06 2008 11 14 Lertap W Lertap 5 User s Manual doc 6 480 KB Microsoft Office Word 97 2003 Document 5 07 2008 10 34 AM Versions ep Lertap5MacroSetA xlam 103KB Microsoft Office Excel Add In 9 07 2007 4 08 PM JermCSEM Kj Lertap5 xlsm 1231KB Microsoft Office Excel Macro Enabled Wor 21 07 2008 2 28 PM _ lertap2 If 4KB LFFile 18 04 2002 1 52 PM KEYLIB32 dll 143KB Application Extension 3 04 2007 11 41 AM _ DelsL1 isu 3KB ISUFile 11 08 2008 2 53 PM amp ISREG32 DLL 48KB Application Extension 8 04 1999 11 18 AM More _ DEISREG ISR 1KB ISRFile 11 08 2008 2 53 PM Filing Personal m J J n a p Music Folders Be sure to read about macro security e if you have problems getting Lertap to run There are two special companion files to Lertap5 xlsm LertapRibbon xlam sets up the Lertap tab Lertap5MacroSetA 23 xlam adds a series of special purpose routines or macros Both of these files are automatically loaded when Lertap5 xlsm is opened For your information the two Lertap57HHelp files chm and pdf are versio
19. Characters such as commas or tabs separate each field Fixed width Fields are aligned in columns with spaces between each field Start import at row fi File origin Ms Dos PC 8 M Preview of File C Documents and Settings nelsonl Desktop Curtin Le TECGASFC DAT 00000001001010480001Y 5327 0001 Y o000000Z001010480001Y 5327 0001 Y ooo00003001010480001Y 5327 0001 Y 00000004001010480001Y 5327 0001 Y ooo00005001010480001Y 5327 0001 Y Cancel Bacl se Finish Doesn t the Excel Import Wizard look pretty useful It is but beware it s got a problem Yes If there s a string of item responses to be imported watch out You have to make sure that the string has no blanks at the start Have a look at this screen shot 2003 2011 Curtin University West Australia Input amp Output 315 Text Import Wizard Step 1 of 3 The Text Wizard has determined that your data is Fixed Width If this is correct choose Next or choose the data type that best describes your data Original data type Choose the file type that best describes your data C Delimited Characters such as commas or tabs separate each field f hi Fields are aligned in columns with spaces between each field Start import at row ft File origin Ms Dos PC 8 m Preview of File C Documents and SettingsinelsonliDesktopiCurtiniLe TECGASFC DAT DCAADAABDCBADACBCBDCCDBBACBABBDCCDBBDAABDACCBDBCBABCDACACBDBA l DBDADCCBAABAAAABCCDBB
20. Here s an example of a Scores report It s from the LenguaBlg dataset F Fal id 9 C Be g F LenguaBigForirtp593 xIsx Microsoft Excel c3 E amp Lerta Hom Inse Page Forn Date Revi View Deve Add 9 Q o gg 2S 1 2 3 4 5 f Lertap5 Scores worksheet last updated on 3 07 2012 1 5499 5500 5501 5502 5503 5504 5505 5506 5507 n 5508 Min 5509 Median 5510 Mean 5511 Max 5512 s d 5513 var 5514 Range 5515 IQRange 5516 Skewness 5517 Kurtosis 5518 MinPos 5519 MaxPos 5520 3321 5522 3 323 5524 BEG IM 4 gt Data CCs lt Fr Scores This Scores report has four columns and over five thousand rows When Lertap creates Score reports it automatically scrolls the report so that you can see the summary statistics and correlations right away they re always at the bottom of the report Have a look at the LenguaBlg dataset in order to get an idea of how the three test scores Core Trial and All were defined see the dataset s CCs worksheet 2003 2011 Curtin University West Australia 264 Help file for Lertap 5 The statistics found at the bottom of a Scores report are explained in the table below n The number of scores corresponds to the number of students The lowest score found Media The median score There will be 50 of the scores below n the median and 50 above it The median corresponds to the 50th percentile of the scores
21. Most users 99 4496 get their Lertap by downloading it from www assess com as mentioned back sa topic or two They then use the LertapXXX WindowsTrial exe file in the download to install Lertap and its companion files XXX may be 564 581 591 593 or a number close to these Note the downloaded file will arrive on your computer as a zip file named LertapXXX WindowsrTrial zip The LertapXXX WindowsTrial exe file has to be extracted from the zip file to accomplish this it s usually possible to right click on the zip file and some sort of option to extract or unzip will appear This procedure results in a standard set of icons which may be accessed from the Windows Start button Easy peasy common stuff the normal operating method for Windows programs A small number of users get Lertap in other ways They might get a special zip file from us at Lertap headquarters and extract or unzip its files to a folder of their choice Having done this they then double click on the folder s Lertap5 xlsm file This gets Excel 2007 2010 to start up loading Lertap5 xlsm as it does Another way to run Lertap is to start Excel and then use the Office Button in Excel 2007 or the SI tab in Excel 2010 to open Lertap5 xlsm Users who have downloaded from www assess com may find the path to the Lertap5 xlsm file to be as shown at the very tippy top of this picture Gordon happened to be the name of the computer used in this example
22. import 316 import data 82 87 312 incorrect answers 237 incorrectly keyed 304 index of dependability 306 index of reliability 282 input 253 Integrity 118 interactive mode 189 internal criterion 92 271 Interpret 258 259 Interpret CCs lines 80 88 89 90 Interpret CCs lines advanced toolbar 244 intervals 160 invalid response 290 invert correlation matrix 11 lQrange 136 IRT 99 159 309 324 325 2003 2011 Curtin University West Australia Index 333 IStats 98 118 322 IStats worksheet 93 94 106 item analysis 92 item bias 151 193 Item control file 101 item difficulty 271 280 325 item difficulty bands 280 item discrimination 271 286 item IDs 255 item response charts 179 180 item responses 257 item scores 983 97 118 Item scores matrix 84 item scoring 36 item statistics 271 item weights 36 item zapper 325 ITEMAN 223 312 316 teman4 316 item component correlations 322 join columns 231 K kappa 306 keypunch 27 keys 231 knock socks off 1 known problems _ 5 6 11 KR 20 282 KR 21 282 kurtosis 263 a ie latent roots 324 latent variable 271 learning management systems 48 Lelp 4 12 13 90 325 2003 201 1 Curtin University West Australia LenguaBlg 263 Lertap 5 9 2 101 Lertap help 325 Lertap Quiz 18 67 106 Lertaptab 78 79 Lertap workbooks 87 253 Lertap5 xls file 248 Lertapb7HHelp 4 LertapbMacroSetA 150 225 2
23. 0 916 0 914 0 915 0 910 0 006 0 006 0 003 0 002 0 000 0 005 0 001 0 005 0 001 0 003 0 005 0 003 0 007 0 000 0 003 0 002 0 004 0 005 0 008 0 007 0 005 0 001 0 001 0 000 0 005 Fregs X H Ev Sls LrrtpQuizSample25June12 xisx Microsoft Ex 8 23 Lert Hom Inse Page Forn Date Revi View Deve Add Y o o ES ge of LERTAP2 created 25 06 2012 Click here z amp to read about the Stats1f output seen immediately above 2003 2011 Curtin University West Australia 6 4 3 1 1 Full item statistics Input amp Output By far the longest section in any Statsf report is that which has the detailed item statistics Here you will find a table of results for each test or subtest item x id EJ lz LrrtpQuizSample25June12 xlsx Microsoft Excel ca 5 Lertap Home Insert PagelL Formu Data Review View Develc Add Ir Y d o ep ES Lertap5 full item stats for Knowledge of LERTAP2 created 25 06 2012 Q1 c3 option wt n b r b r avg z A 1 00 26 0 43 0 66 0 83 18 15 0 79 B 0 00 25 0 42 0 57 0 72 7 92 0 68 C 0 00 9 0 15 0 17 0 26 9 78 0 41 Q2 c4 option wt n b r b r avg z A 0 00 0 07 0 01 0 01 12 50 0 02 B 0 00 12 0 20 0 41 0 58 7 00 0 81 C 0 00 7 0 12 0 29 0 48 7 00 0 81 D 0 00 8 0 13 0 27 0 42 7 88 0 68 E 1 00 29 0 48 0 66 0 83 17 66 0 72 Q3 c5 option wt n b r b r avg z A 0 00 2 0 03 0 08 0 20 9 50 0 45 n n nn 1 nn
24. 2 The beaches of Maui are better than West Australia s Strongly disagree Disagree Undecided Agree Strongly agree U1 30 N H Both of these items have five possible responses or options or alternatives and use response codes of 1 2 3 4 5 It s possible to have items with more options Lertap allows up to 10 options per item Items do not have to use the Likert style The response codes used to not have to be digits examples below Likert style items are very common another popular style is the semantic differential 2003 2011 Curtin University West Australia CCs details If we at Lertap central wanted to score these items we d be content to follow the conventional pattern of letting strongly disagree equal one point disagree two points and strongly agree five points Respondents could end up with a low total score of two 2 points and a top total score of ten 10 points They d get the low score if they answered strongly disagree on both items They d get the top score if they chose strongly agree on both items It s not really necessary to think about scores such as these but some people find them useful And if they do and if they work for WATC the West Australian Tourist Commission they d likely want to reverse the scoring for the second item Say what Reverse the scoring You bet it s a common happenstance The WATC mob would want people to strongly disagree with the se
25. 2003 2011 Curtin University West Australia 174 Help file for Lertap 5 Excel Options eS Popular a z 3 j t View and manage Microsoft Office add ins Formulas Proofing Add ins Save Name Location Advanced Active Application Add ins Conditional Sum Wizard C y SUMIF XLA Customize Lertap5Macroseta C5 acroSetA xla Addi Inactive Application Add ins wANALYS32 XL Analysis ToolPak C Trust Center Analysis ToolPak VBA CA PVBAEN XLA Custom XML Data OFFRHD DL Resources Fira Currency Tonle C OTONI YL AN Add in Conditional Sum Wizard Publisher Location C Program Files Microsoft Office Officel2 Description Helps sum data in lists Manage Excel Add ins M 4 m The Excel Add Ins from Microsoft are free and generally easy to install For assistance refer to Excel Help Lertap checks to see if you ve got the Analysis ToolPak Add In installed If you do and if Lertap and Excel are functioning as they ideally should then the histogram charts produced by the Histogram E option are supposed to appear automatically whenever you click on b the Histograms option When things are working right Lertap will make its Histo1L worksheet and then with help from the Analysis ToolPak automatically add another worksheet HistolE HistolE is supposed to contain the histogram chart seen by taking the Histogram E option discussed earlier i601 However th
26. 312 extended matching 321 external criterion 92 193 271 277 304 extra copy ofLertap 238 SE Fratio 131 141 factor analysis 93 94 111 fan out 299 304 FastTest 115 324 fax 14 FDist 131 file export 312 file import 312 fles 4 Filter 73 216 229 259 332 Help file for Lertap 5 find 216 first column blank or zero 253 flag U L disc 299 flag options 231 flagged plots 186 flagging plots 183 flags 271 277 290 294 304 font 74 font limits 198 205 font problems 87 312 format charts 147 format statement 98 formula 219 formula for new scores 323 Formula Bar 219 286 FORTRAN 98 FORTRAN format statement 209 forward scoring 23 Foxes Group 93 106 Freqs 14 61 88 90 253 258 259 full item statistics 271 G getting started 14 graph 292 graph problems 205 graphs 84 179 180 183 Group breakout 319 group differences 193 H half tests 73 halve and hold 159 325 Harpp Hogan 103 118 319 have and hold 159 header rows 88 headers 255 help file 325 H H index 118 H H sigma 118 hidden 248 hidden files and folders 223 highlighting 158 high low 265 299 Histo1E 160 173 HistoiL 160 histogram 136 Histogram E 160 histogrammer 319 histograms 160 history of Lertap 326 hover help 78 how Lertap works 19 48 258 lbreaks 150 lbreaks settings 158 lbreaks1 report 141 lbreaksMH 151 icons advanced toolbar 247 ID 255 323 identification 323 IF 216
27. 43 0 43 0 16 00 21 00 19 00 3 00 14 00 18 00 40 00 40 00 n 1 Min Median 90 Mee 3 AZ lmay M M Data CCs lt Freqs Scores lt iii 2003 2011 Curtin University West Australia 12 Help file for Lertap 5 aB 2 3 4 5 6 3 Lertap5 RSAsig a IDISZR list with EEIC min 8 pam S1 Data row a as ies SUN MN Dm DataRow3 DataRow21 3 17627 11 DataRowS6 15 DataRow60 4 3 17637 s DataRow33 11 DataRowS 7 4 1764 is DataRow41 i4 DataRow42 7 3 1765 s DataRows2 i5 DataRow60 E 3 1766 19 DataRow41 i2 DataRow47 7 3 1767 is DataRow40 16 DataRow49 4 4 1768 19 DataRow41 15 DataRow60 7 3 17697 12 DataRow47 15 DataRow60 3 3 17707 19 DataRow41 3 DataRow52 7 4 17717 io DataRow26 14 DataRow42 7 3 Suspect Not suspect Total Number of items Number of students M gt gt RSAsig1 RSAtable1 RSAcases Mi If you re looking at a Scores Breaks or RSAsig report the histogrammer will swing into action immediately after you ve clicked the histograms icon In the case of Scores and Breaks it will usually ask you to indicate which column you d like to gram the question is not required when you re on an RSAsig report as in that case there will be only one column Log PROB to plot If you click on the histograms icon whilst viewing another type of report such as Stats1b for example Lertap will take you to the Scores report
28. 7 3 7 4 Resources There are other Lertap resources There s the Lertap manual which this document has made frequent reference to There s the Lertap website at Curtin University of Technology http www lertap curtin edu au The Lertap website has screeds of additional information including sample data sets useful in measurement classes or by people just launching their Lertap careers The website also has a modest series of technical papers highlighting examples of Lertap applications and discussing current developments such as the experimental options in Lertap A history of Lertap is provided in the manual and on the website Lertap s pedigree goes back to the early 1970s You may not realise it but you could have used Lertap in the past when it was masquerading under another title And then there s always our support desk in sunny West Australia We welcome questions and comments Write to us at larry lertap com References Please refer to the list of references found at the following URL http www lertap curtin edu au References htm Rchitect Larry Nelson is not Lertap s only architect but he s been the main one having shepherded the system through several versions and numerous host institutions Larry completed a BSc in electrical engineering at Wisconsin 1964 an MSc in Educational Psychology at Wisconsin 1970 and a PhD in Educational Psychology psychometrics at Colorado in 1973 He s held
29. A second way of assessing the difficulty of a cognitive item is to simply use the item s average its mean If an item has just one correct answer and if the weight for that answer is 1 00 then the item s mean will be identical to the proportional index of difficulty 3 item mean max weight default Item means can be greater than zero Traditionally item difficulty has been measured on a scale which goes from 0 00 to 1 00 if we divide the item mean by the greatest response weight we effectively re scale the mean so that it falls back to the 0 00 to 1 00 range This method of indexing item difficulty does exactly that When there s only one correct answer to an item it yields the same result as 1 above As indicated Lertap s default method is 3 item mean divided by the maximum response weight To change it to one of the other methods do this 1 make a change in Row 19 of the System worksheetfzsl in the Lertap5 xlsm file 2 save and close the Lertap5 xlsm file Finally we should mention where cognitive item difficulties are displayed They re shown in the item difficulty bands found towards the bottom of the Stats1fl 2s report and they have their very own column in the Statsib zs report When the item difficulty calculation method has been set to 2 above Lertap s item difficulty bands can come under stress since they use a 0 00 to 1 00 scale In this case Lertap momentarily pops into the 3 method re scaling the mean so
30. B 0 60 ne B 0 60 C ne C 0 40 0 20 l D 0 00 eie D 0 20 E E 4 lower 4th 3rd 2nd upper 0 00 cher Key D Diff 56 Disc 0 other lower 4th 3rd 2nd upper Key E DIT M Disc 12 Of these four items only 115 shows a tendency to fan out 116 is very easy 123 is quite difficult 121 is goofy the trace line for the keyed correct option D decreases as we move across the plot the opposite of what would be expected On the other hand one of I21 s distractors option C increases from left to right which is also the opposite of expected This pattern is often found when an item has been incorrectly keyed It could be that an error has been made and the keyed correct option for this item should be changed to C However even in this case there would still be many students in the upper group suggesting that option D may be a plausible correct answer Indications are that 121 requires revision before being used again 2003 2011 Curtin University West Australia 306 Help file for Lertap 5 E x H I ER Sle LenguaBigForlrtp593 xisx Microsoft Excel c3 5 ES Lertz Hom Inset Page Forn Data Revii View Deve Add Y o ca pg ss 5 S mS Ira U L stats for Trial items only created CREE 0 01 0 02 0 za 0 01 0 01 0 00 0 01 0 02 0 00 erem D E 0 00 0 01 0 01 0 01 Summary group statistics n avg avg s d min mdn max upp
31. DAT file You ll be able to see the RSAdata worksheet right away as it will form part of your Excel workbook but the SCheckData DAT file becomes a separate entity a file on its own stored on your computer s hard disk Where Well if you had saved your workbook prior to taking this step it ll be saved in the same folder as your workbook otherwise you may have to dig around to find it 3 Next back to the Run menu and a click on Response similarity analysis if you want Lertap to make its RSAsig RSAtable and RSAcases reports This option may be selected more than once each time a new set of reports is created 4 f you want to use Professor Wesolowsky s SCheck exe program start SCheck exe and get it to work with the SCheckData DAT file created by Lertap Read more about SCheck by clicking herel 1031 Related comments What about selecting a subset of data records before getting into Lertap s response similarity analysis For example what if you wanted to select only those students who took the exam in the Business school s main lecture hall There are two ways you could get Lertap to cull out only the records you want One way is to use Lertap s tst card 73 on the CCs worksheet to select the desired records Of course you d have to have a column in the Data worksheet which gives exam location information Let s say this was column 3 in which case the tst card might look like this tst c3 Business Another way is to use
32. Documentation UpdatesSummaryLertap57 pdf About this document This document Lelp was made using Help amp Manual 6 a hypertext authoring system from EC Software Note that this version of Lelp applies to the Excel 2007 and Excel 2010 versions of Lertap for Windows versions 5 7 0 and up It s also appropriate for Macintosh users who have Excel 2011 H amp M lets us compile Lelp in one of several formats and we ve done so To date there are PDF and chm compilations chm help files are now found in most Windows applications they re easy to use PDF files are now a world standard usable on all sorts of computers this is nice but PDF files cannot display what are called popup topics This document contains quite a number of popups they flash up quickly in the chm version but don t show at all under PDF If you re looking at the PDF version you ll sometimes see spots which say something like click here or note but there s nothing clickable alas you ve come upon a popup that won t pop Lelp is also available as a website The address is 2003 2011 Curtin University West Australia Welcome 13 http www lertap curtin edu au HTMLHelp HTML index html Since all forms of this document are produced from the same source they re identical The chm PDF and website versions have exactly the same content However as noted the PDF version cannot display popup topics Screen shots in manuals have a habit of datin
33. Excel macro security This problem is highlighted below it s very easy to solve If you don t have the time right now to solve the macro problem ask one of the kids to do it or grandma if she s not busy making an apple pie never ever disturb someone who s making an apple pie Excel macro security Lertap s computer code written in Visual Basic is nested in a set of code modules referred to as macros by Excel In order to run Lertap Excel has to be told that it s okay to enable the macros found within Lertap If this is not done the Lertap tab for Excel 2007 2010 will not be displayed and without the tab you can t do any Lertapping The Lertap tab for Excel 2007 2010 looks something like this El 9 Verdana Lertap5 xlsm Microsoft Excel es s SA Lertap Home Insert Page Layout Formulas Data Review View Developer Add Ins X Delete TH Sort M Blank gt Interpret glij Histograms 3 Move Version Line R Headers 3 Elmillon iz Scatterplot P License fXSpread Excel Copy More b Res charts 2 Lelp Basic options New menu Run menu Graphics trio Other menus Here s what happens when you open the Lertap5 xlsm workbook without having enabled macros you get a Security Warning as seen here 2003 2011 Curtin University West Australia Lertapo xism Insert Page Layout Formulas Data Review Security Warning Macros have been disabled Options Note that there s
34. Item Item Item Item 64 25 OS 26 66 9 2 14 69 26t 22 0 64 v Di5d rrr vy O66 0 45 B us 16 0 69 0 256 srr 9 amp 35 14 45 0 45 0 06 K 62 p 2 pr 36 pp 0 62 p D 55 20 4 9 9 76 7 a Par ea ee eee 22 31 34 se 0 34 0 22 7 33 5 55 0 55 0 34 33 17 45 0 33 0 39 pere aes s BUR 16 54 0 167 D 23 8G 10 76 0 975 0 27 20 5 vB 10 Pao i c Dog uos 7 81 7 0 81 0 50 4 24 41 3 505 Jos s B B oo 65 g ww 12 tBees EPEE 1 2 3 4 5 6 3 8 9 BPR e O r 94 ee PRR bw ae The csv file above came froma Lertap Stats1b worksheet which looked like this F3 Microsoft Excel ed502 xls 8 File Edit view Insert Format Tools Data Window Documents ToGo Hep X X O9 New S PS pune shorts M fa Move License Lep 5 brief item stats for Ed 502 semester test created 7 11 00 Statsiul 2003 2011 Curtin University West Australia Toolbar and tab How did we get from the Stats1b worksheet to the csv file We followed a procedure almost identical to that described in the previous topic Creating a text file However instead of asking Excel to Save as TXT MS DOS we directed it to Save as CSV Comma delimited csv Lertap users may have a variety of needs which prompt them to save worksheets as csv files Among these would be a desire to use Lertap s st
35. What we usually want to see is a pattern like that seen for NM9 In the lower group the proportions for the options should be fairly similar while in the upper group most students will hopefully be able to identify the correct answer making the trace lines sort of fan out as they head to the right NM10 also fans out in the generally desired manner but NM8 and NM11 do not NM11 was quite easy even in the lower group It has no chance of fanning on the right as it has essentially already fanned on the left NM8 shows the desired pattern in the lower group but fails to fan on the right Graphs such as these are so popular they ve spawned numerous supporting documents Click here for more information 2003 2011 Curtin University West Australia Input amp Output 303 r x ld Ep Siz MNurForLrtp593 xlsx Microsoft Excel 5 23 Lerta Hom Inser Page Form Data Revie View Deve Add 9 Qo P zs Lertap5 U L stats for M Nur Licensing E390v6 3 created E arie u 1 dift u 1 disc 0 02 0 03 0 12 0 01 0 15 0 02 0 20 0 01 4th lower 0 04 0 08 0 07 0 12 Summary group statistics n avg avg s d min mdn max upper 353 51 1 85 2 5 48 51 60 2nd 353 45 4 76 1 5 43 45 48 3rd 357 40 7 68 1 3 39 41 43 4th 353 36 3 60 1 4 34 36 39 lower 353 29 66 49 36 288 everyone 1 769 40 6 68 PFs 2 41 60 This was an upper lower analysis with more than two groups
36. and wait for you to click on the icon again If you didn t want to plot from Scores you have the chance to click on Breaks or RSAsig instead after which you have to click on the histograms icon yet again The histogram created by the Histograms option is the standard Lertap 5 histogram dating back to Lertap 2 Its style is exemplified below 2003 2011 Curtin University West Australia Toolbar and tab 163 Basic options New menu Run menu Graphi le histogram for the score titled Comfort ag T There may not be many columns in this histogram but there s a wealth of information The z score corresponding to score Found by subtracting the overall score mean from the score and then dividing by the overall score standard deviation s d The overall mean and s d values are found in the Scores 2s worksheet In this case the overall mean was 34 48 while s d was 4 61 Thus for the first score of 26 00 z 26 00 34 48 4 61 1 84 z scores will be zero when score equals the overall mean More z stuff here These start at the lowest score found and end at the highest The frequency of the score In this case there was one score of 26 00 no scores of 27 00 and so on The most frequent score in this case referred to as the mode in the literature was 32 00 with f 8 E The percentage corresponding to f found by dividing f by the total number of scores 60 in this case
37. as you do they automatically align with the Excel grid Aligning with the grid is useful as it makes page breaks easier to adjust when it comes to printing 2003 2011 Curtin University West Australia Toolbar and tab 193 z x id 9 Ev Sis Book2 Microsoft Excel ca 5 2 Lertap Home Insert Page Layout Formulas Data Review View Developer Add Ins 9Y o c gg X3 9 aso 10 1 1 ag 1 2 2 040 2 Lj 3 3 020 3 12 4 oo 4 a0 4 other x b other lt gt b other z QS Ph LLL Key 4 Diff 50 D Key 2 Diff 59 D Key 1 Diff 54 D 14 15 17 I8 19 zr 10 T mr 20 D a 10 E M 4 gt h csemi lt Statsiul lt Statsiulchta PackedPlots4 PackedPlot 4 m i Ready EJ G 80 2 Q This snapshot shows what ChartChanger3 will sometimes do create plots which are too squished Fix them by following the comments in the next paragraph Once all the plots have been selected they should be automatically selected if not click on the small box circled in yellow slowly drag a column divider such as that circled in red above to the right to make the plots wider To make them taller or shorter slowly drag a row divider such at that circled in blue As you do these things the plots will still be aligned with Excel s underlying grid To get them to print well you ll have to adjust the
38. but not by very much Some users often find Statslf to be excessively detailed full of number and tables which make sense to experienced test developers but well couldn t there be something easier to read Yes Stats1b reports are b for briefer They are designed to be easier to understand whilst retaining the most important information from Stats1f Statslf and Stats1b are similar in that they re based on the use of correlation coefficients to reflect how items have performed When cognitive items are involved another time tested way to summarize item performance is to use upper lower U L methods sometimes referred to as high low H L methods These methods date back to pre computer days when educators endeavored to assess item performance by hand without the help of any electronic aide The idea is a simple one to see if an item is working well we ll look at results from the best test takers those with the highest test scores We will compare these results from those gathered by looking at how the weakest lowest students did If the top or upper students got the item right while the bottom lower students did not we say the item has discriminated such items are ones we ll use again in the future when we want to have a test which we know can identify the strongest and weakest students 2003 201 1 Curtin University West Australia me Help file for Lertap 5 The Stats1ul report employs the upper lower m
39. columns c28 through c37 A sub line is required for affective subtests and it must have the Aff or Affective control word on it This is seen above In this case the sub line has also been used to assign a Name and a Title What s that pol line doing To answer this question we have to return to the matter of scoring Affective items do not have a correct answer It s customary to give a certain number of points for each of an affective item s answers What were the possible response choices for this set of 10 affective items Very good question There is no Res declaration on the sub line and in this Common case Lertap assumes Res 1 2 3 4 5 Each affective item had five possible responses Unless you say otherwise Lertap will give one point if someone selects 1 as their response two points when someone selects 2 and so on This is called forward 2003 2011 Curtin University West Australia Getting started 25 scoring On a 10 item affective subtest with five response choices per item the maximum possible score would be 50 the minimum possible would be 10 The pol line allows each affective item to be reverse scored if wanted On a reverse scored item the first possible response will get 5 points not 1 The last fifth response will get just 1 point not 5 Reverse scoring for affective items is pretty common The pol line above indicates that the first item is to be forward scored while the next
40. measurement error no matter the level of the score It is found in the Summary statistics 23 sections of Stats1f reports 2003 2011 Curtin University West Australia s Help file for Lertap 5 However it has long been known that standard errors of measurement vary by ability As we go from test takers with little proficiency to those with average proficiency and then on to the strongest students respective SEMs change To reflect this Lertap also computes SEMs for various score levels using methods from a paper by Lord 1984 228 More exactly Lertap employs Lord s Method III the binomial error model and also the adjustment to Method III estimates known as Method IV A csem1 Lertap report has two parts a table followed by a plot The information seen in the sample output below is from the M Nursing dataset x id My gm ZF AL Book2 Microsoft Excel oB Lertag Home Insert Page Formi Data Revie View Devel Add I Y ca gg 2s 1 2 3 4 3 6 Pf 1 Conditional standard errors of measurement for M Nur Licensing E390v6 3 37 38 39 40 41 42 43 EE 45 46 47 48 49 50 31 52 E M 4 gt M Scor Ready 2003 2011 Curtin University West Australia Input amp Output 311 F x A ly F 7 Book2 Microsoft Excel oB Lertap Home Insert Page Lz Formul Data Review View Develo Add In YV ca gg XS 2 3
41. onya and now you ll have Stats3Bf Stats3Bb and Stats3Bul reports to look at The Stats3f 3b and 3ul reports from the last run will remain unchanged You bewdy Newk Advanced toolbar how show THIS IS AN OLD TOPIC It does not apply to the Excel 2007 version of Lertap but has been left here as the manual refers to it Note that the Excel 2003 version of Lertap is still available from www assess com and Excel 2003 will work fine even under Vista Microsoft s new operating system for Windows If you need the features offered by 2003 2011 Curtin University West Australia 28 Help file for Lertap 5 4 9 4 the advanced toolbar consider using the Excel 2003 version of Lertap How to get the advanced toolbar to show Go to the System worksheet Change the UserLevel setting in row 7 to 2 Save Close Lertap Reopen Lertap The advanced toolbar should be there Write to us if you have probs larry lertap com ring the bell if you don t System Worksheet The System worksheet is one of the worksheets found in the Lertap5 xlsm workbook you can see its tab down towards the bottom of this screen snapshop E3 Microsoft Excel Lertap5 xls File Edit View Insert Format Tools Data Window Help X New Rez Run Shorts T gt M Move Q License gt Lep To set up a new workbook use the New drop down menu option above to the right of the yellow smiling face To wark with data you ve already set
42. score was used as the external criterion and the second subtest Trial items only was selected for the analysis 2003 2011 Curtin University West Australia 278 Help file for Lertap 5 Fal H D gt Be lt LenguaBigForlrtp593 xIsx Microsoft Excel ca E amp Lertap Home Insert Pagel Formu Data Review View Develc Add Ir VY Q ca pg 2s Lertap5 external criterion stats for Trial items only created 4 07 h a I15 c19 option wt n p pb ec b ec avg ec z A 0 00 259 0 05 0 13 0 27 24 78 0 56 B 1 00 3 443 0 63 0 27 0 34 29 43 0 21 E 0 00 246 0 04 0 18 0 39 23 22 0 82 a D 0 00 1 263 0 23 0 08 0 10 27 36 0 14 F E 0 00 288 0 05 0 15 0 31 24 34 0 64 other 0 00 5 0 00 0 01 0 07 26 80 0 23 r ec 0 27 E I16 c20 option wt n p pb ec b ec avg ec z A 0 00 34 0 01 0 05 0 24 24 03 0 69 B 1 00 5 313 0 97 0 14 0 32 28 35 0 03 C 0 00 43 0 01 0 02 0 10 26 49 0 28 D 0 00 60 0 01 0 13 0 46 20 78 1 22 E 0 00 50 0 01 0 06 0 21 24 70 0 58 other 0 00 E 0 00 0 01 0 07 29 75 0 26 lt aa r ec 0 14 121 c25 option wt n p pb ec b ec avg ec z A 0 00 33 0 01 0 11 0 49 19 79 1 39 B 0 00 54 0 01 0 10 0 39 21 83 1 05 cC 0 00 2 271 0 41 0 20 0 26 29 65 0 24 lt aa D 1 00 3 072 0 56 0 13 0 16 27 49 0 12 lt ba E 0 00 Z2 0 01 0 14 0 46 20 99 1 19 other 0 00 2 0 00 0 03 0 39 19 50 1 44 r ec 0 13 v M 4M Stats3ul Histoll ECStats2f ECStats2ul 4 m gt Ready BSA 100
43. workbook formats used by Excel In SPSS 17 or later use File Open Data and select Files of Type Excel 2003 2011 Curtin University West Australia se Help file for Lertap 5 6 5 1 Lertap comes ready to export to programs such as XCALIBRE and SCheck see the discussion related to having Lertap make its own DAT file 98 ITEMAN ITEMAN is another classical item and test analysis system created by David Weiss of Assessment Systems Corporation ASC way back when When exactly As it happens about the same time as the first version of Lertap late 1960s Iteman 3 is a system which many people used for years A major update inspired in part by Lertap and in larger part by new staff at ASC came out in late 2010 It s known as Iteman 4 How does Iteman 4 compare to Lertap 5 See these papers an overview and a discussion of item flags Lertap 5 is able to import files made for Iteman 3 and it does so with real ease This could be a useful and quick way to import data from a scanner To use the Iteman importer in this manner you d first want to download the Iteman user manual from Www assess com and get an understanding of the four control lines which Iteman wants ahead of the actual data Access to Lertap s Iteman importer is via the Macs menul 23 Will this importer also work with Iteman 4 files Yes and no Yes because Iteman 4 allows for two input modes old meaning Iteman 3 and new Iteman 4 Iteman 4 use
44. 00 3 00 4 00 5 00 The weighting pattern seen in these three cards is the default pattern for items with five response codes By default is meant not requiring mention this is what I ll do unless you tell me otherwise This being the case we don t need to say anything Let Lertap apply its default weights for each item unless we say different Use mws cards to say something different Now in this case we have used two mws cards in place of one pol card That s not real efficiency The majority of Lertap users are happy with the pol card but there are some advantages in using mws cards One advantage is that there s no ambiguity with mws cards they make it absolutely clear how item responses are to be scored and explicitly indicate the items affected About that idea of looking deep into the heart of Lertap a Lertap workbook s Sub worksheets provide quite a detailed glimpse of exactly how item weights have been set up To read more about Sub sheets just click here Example A9 Here s an example which shows off a special use of the mws card col oc3 07 sub AFF mws call 5 00 4 00 3 00 2 00 1 00 mws call means we have a multiple weights specification which is to apply to all the items mentioned on the preceding col card This use of the mws card often comes into play when there are items such as the following 2003 2011 Curtin University West Australia CCs details 5 Mt Barker wines a
45. 06 2012 item difficulty bands 00 Q22 10 20 30 40 Q1 Q2 Q9 Q11 Q14 Q18 Q19 Q20 Q21 Q25 90 Q3 Q4 Q6 Q7 Q10 Q12 Q15 Q17 Q24 60 Q8 Q13 Q16 Q23 70 Q5 80 90 item discrimination bands 00 10 20 Q4 Q22 30 Q5 Q14 Q24 40 Q7 Q9 Q16 Q23 90 Q3 Q10 Q12 Q15 Q17 E 60 Q1 Q2 Q6 Q8 Q11 Q18 Q21 Q25 70 Q13 Q19 Q20 80 90 E M 4 gt Fregs lt Scores Statsif lt Statsib 4 w gt Ready 3 100 2 o The bands simply summarize item difficulty and discrimination figures In this example most items had difficulty values proportion correct lying between 0 40 and 0 60 Four items had difficulties above 0 60 but less than 0 70 One item Q5 had a difficulty greater than 0 70 while another Q22 had difficulty less than 0 10 Item discrimination values are interpreted in the same manner For this test only five items Q4 Q22 Q5 Q14 and Q24 had discrimination values below 0 40 Item discrimination figures may be negative When this happens such items will appear in the 00 band 2003 2011 Curtin University West Australia Input amp Output r X H Dv Be A 7 LrrtpQuizSample25June12 xisx Microsoft Ex E 28 Lert Hom Inse Page Forn Datz Revi View Deve Add o SS Lertap5 full item stats for Knowledge of LERTAP2 created 25 06 2012 4 alpha figures alpha 9149 without alpha
46. 2003 201 1 Curtin University West Australia 276 Help file for Lertap 5 A4 F x id 9 e J 7 LrtpQuizsample25Junel2 xlsx Microsoft Excel ca E amp Lerta Hom Inser Page Form Data Revie View Deve Add V 9o ca gg 2S Lertaps5 full item stats for Comfort with using LERTAP2 created 25 06 201 4 Q26 c28 option wt n pb r avg z 1 1 00 8 13 3 0 48 28 9 1 22 2 2 00 13 21 7 0 39 31 1 0 74 3 3 00 15 25 0 0 21 32 8 0 37 E 4 00 14 23 3 0 49 38 6 0 89 5 5 00 10 16 7 0 55 40 2 1 24 Q27 c29 option wt n b r avg z 1 5 00 3 5 0 0 36 41 7 1 56 2 4 00 14 23 3 0 40 37 9 0 73 3 3 00 22 36 7 0 05 34 8 0 06 4 2 00 21 35 0 0 57 30 9 0 78 5 1 00 0 0 0 0 00 0 0 0 00 Q28 c30 ct option wt n pb r avg z 5 00 13 21 7 0 24 32 4 0 46 4 00 27 45 0 0 46 36 8 0 51 3 00 10 16 7 0 16 32 8 0 37 2 00 8 13 3 0 08 33 5 0 21 1 00 0 0 0 0 00 0 0 0 00 other 3 00 2 4 3 0 22 29 0 1 19 Mo4 MH csemi Statsiul Stats2f Stats2b 4 ll AUNE Ready amp S E 10 y The Stats2f report seen above shows how the detailed item statistics look for an affective subtest in this case the 10 item survey included in the Lertap Quiz dataset In contrast to cognitive items where most often only one of an item s options will be scored that is have a wt which is above 0 00 each option on an affective item is usually scored The three items pictured above Q26 Q27 and Q28 all use
47. 23 11 12 13 6 4 0 0 1 876 4 Ing f 16 16 9 14 7 13 B 3 2 1 844 Ii 6 Nat diff 65 74 91 83 85 83 Fe 00 00 1 00 39 7 Ing diff 69 69 67 1 00 86 85 1 00 1 00 1 00 1 00 40 8 odds ratio gt 86 1 29 5 00 00 92 91 00 00 00 00 When a DIF analysis has been requested Lertap s Ibreaks routine creates two new reports or worksheets The snapshots above are from one of these the IbrakesMH report MH stands for Mantel Haenszel the method Ibreaks uses for its DIF analysis The Score levels row starts at the lowest test score found 4 and continues in steps of 1 one to the highest score which in this case was 39 These tables can become quite wide The screen snapshots above have captured the first ten score levels and the last ten Rows 3 and 4 give the number of students in each group at each score level These numbers start to become relatively substantial at score level 8 column 7 with 13 students in the r eference group and 15 in the f ocal group The two diff rows give the proportion of students in each group who answered the item correctly The odds ratio is a relative measure of how likely it is that a student in the reference group will get an item correct when compared to a student in the focal group Greater than one and the odds favour members of the reference group as being more likely to return the right answer Less than one and the focal group has the advantage Equal to one and it s ev
48. 239 steps to a Lertap analysis 258 string of item responses 312 string of responses 82 student ID 323 Student mode 324 student version 3 stumping 36 338 Help file for Lertap 5 Sub worksheets subtest 88 Summary group statistics Summary statistics 282 survey items 36 syntax 27 System worksheet 14 48 61 90 97 99 103 118 141 183 189 193 198 223 225 247 248 271 280 UT 88 92 245 247 258 299 templates 147 terciles 198 Test Pilot 48 53 58 60 test reliability 286 tetrachoric 93 97 106 tetrachoric correlations 325 text 82 text file 112 text importer 312 text string 98 TextEdit 112 The Spreader 82 312 325 Thompson 237 three strikes 242 time trials 106 118 159 Title 29 36 74 toolbar 1 14 78 80 82 253 toolbar advanced 244 ToolPak 173 total score 23 29 234 286 TotalTest1 234 trace lines 299 304 transfer license 238 244 transform 212 216 transform scores 323 transformation 219 trialcopy 3 trialitems 73 triangles 118 198 true scores 282 Trust Center 6 trusted location 6 tst 73 Tukey 136 TXT 112 U U L diff 299 U L diff 60 U L disc 299 U L disc 60 unanswered 46 unanswered questions underline 271 underlined 290 unequal ns 323 unidimensional 180 UniquelD field 115 unknown MinPos MaxPos unlock code 238 241 unlock Lertap 242 unscored items 73 update 14 18 36 183 update correlations 222 update note 248 updates
49. 5 you should see a message similar to this one 2003 2011 Curtin University West Australia 20 Help file for Lertap 5 Lertap 5 license information Your Lertap 5 license is fully paid up Lertap is unlocked on this computer ready and raring to go Should you encounter any problems please send an email message to support lertap com Once you have obtained a Lertap 5 license it is possible to transfer the license to another computer Information on how to do this may be found under the Setup the license transfer process 2 option 2003 2011 Curtin University West Australia Toolbar and tab 4 8 2 Apply for license A license for Lertap 5 is obtained by making application to ASC Click on the Apply for a license option and a screen similar to the following should appear Unlock Lertap 28 days left in the trial period Ea To continue using this trial version just click on the Close button below Use the Help button for more comments if wanted To place an order for a license and an unlock code please email or fax the following Session ID and Computer ID numbers to Assessment Systems Corporation Our email address is sales assess com Our fax number is 1 651 647 0412 Thank you Session ID 323189276 nv7 Computer ID 4247629 nv7 If you have purchased a license and have an unlock code carefully enter it in the box below Then click OK Unlock Code Send a
50. 74042 46X0 DataRow6 771442722 DataRow Suspect Not suspect Total Number of items Number of students EEIC minimum H H index minimum H H sigma minimum Items excluded Minimum score setting Maximum score setting The lower left portion of an RSAsig report is shown above Only one suspect student pair was found in this analysis of 5 565 total student pairings Thirty 30 items were involved in the analysis and 106 students Cutoff figures for the three Harpp Hogan criteria are shown as minimum values under the Run control heading No items were excluded from the analysis and a score range of 0 to 100 was processed Note the Number of students excludes students with perfect scores no wrong answers at all and it also excludes students with totally imperfect scores not even a single right answer 2003 2011 Curtin University West Australia Help file for Lertap 5 HK ou ewr SS Rune 2 shots di gt fl Movet License Lelp REPE FECE SE 8 9 10 31 IC min 8 created on 7 01 2006 2 ENE e x E 5564 9 0 50 15 78 4 56 5565 20 7 3 2 33 18 36 5 80 5566 20 7 3 5567 5568 5569 5570 5571 5572 as er 1 00 5573 5574 5575 5576 5577 5578 3 ound 5579 5580 5581 5582 5583 4 to 5 sigma 0 01 0 09 5584 NET NUI ER EEG M milf RSAdatal XRSAsig1 RSAtablei RS m 3 Above is a snapshot of the lower right area of an RSAsig report The
51. 82 317 updates of Lelp 14 updates of Lertap5 xlsm 12 UpdatesSummary 317 upgrade 12 upper lower upper lower groups UserLevel 247 usual subtest score 48 SM e validation validity 92 Variance components Venezuela 92 Version 5 25 29 245 26 245 321 323 208 258 265 299 324 159 306 2003 2011 Curtin University West Australia version 5 7 0 12 Xcalibre 4 1 101 version data 81 222 xls 11 version number 12 xlsx 11 Vista 244 Y Visual Basic Editor 226 T visual eye tem analysis 299 Volpi 106 Yates correction 158 W yellow smiley 81 Z7 Wainer 183 Walkenbach 147 z 53 weak items 118 z score 29 74 271 website 14 weights 23 27 29 weights items 36 weights array 245 weird responses 259 Weiss 316 Wesolowsky 103 118 Western Australia Tourist Commission 36 what is Lertap 1 whiskers 136 wizard 312 WordPad 112 worksheet names 258 wrong answer report 237 wrong item list 237 WrongltemList 237 WrongUns 237 wt 46 53 271 Wt 23 29 36 74 234 237 286 X Xicon 80 XCal 99 112 XCal worksheet 99 xCal41Data 101 XCal41ICF 101 XCALIBRE 99 112 312 324 Xcalibre 4 0 99 316 2003 201 1 Curtin University West Australia
52. E Other Ibreaks1 Statsif Ready E3 The correct answer to item I1 was E Ibreaks denotes the correct answer by underlining and by different shading The table and the graph both indicate that the two groups had rather similar response patterns for this item especially when it came to the correct answer The short row of statistics below the table provides quantitative measures of how dissimilar the groups were in their response to the correct option Ibreaks gets Lertap to undertake an analysis of variance resulting in an F ratio to provide a test of the null hypothesis The means of the two populations of test takers who have been represented by the Ing and Nat samples of this data set are equal The statistical significance of F is given by sig In this case F is 0 15 and sig is 0 70 Not statistically significant we usually want sig to be 05 or less if we re interested in the null hypothesis and statistical significance Any analysis of variance worth its salt will provide an index of practical significance as well as F and sig Lertap uses the correlation ratio for this referred to as eta It is thought by some that an eta value of 10 or more may be interpreted as indicating that the difference in sample means is practically significant possibly pointing to a difference we might refer to as important or meaningful For more comments on these statistics and on the matter of signific
53. EScuTTUI O 226 Macro SAQS m 228 b gurdizcr emDE T 229 k lnejirpce 231 TotalTest1 234 WrongltembList 2 e Ree ete detecta ee e E 237 8 License pr L 238 Dis play current StatUS 239 Apply for licensee csscsnssssssssssssssssessnecssnssnessesaoussusnssnsnasansansensansausauausauenneanensenssaauaauaceasousnsnasnasansnasansansansa 241 Unlock Ler tap mE License transfer 9 Advanced Toolbar Advanced toolbar Wy 2 s cccczecisscvecetseid 245 Advanced toolbar how USC csssccssccesssessseeenseeeesseesnseeenseeeesseeesseeenseeesseeesnaaeensneesseeessaaeenseeeeseeesseeensasens 247 Advanced toolbar how SHOW 2 leseieeseseeeessnee senes nnne sante saa sa asas DR RARI ssa SES sas DR sas DE sss sE s Ras s sas ennenen na 247 System Worksheet M 248 Part V Excel shortcuts 250 Part VI Input amp Output 253 1 Lertap workbook del ecc iere erret nnmnnn nnmnnn annnm ennn nnna 253 2 BNr E 1 2 EA A T A A EHE TETETEETHTODINE 255 2003 2011 Curtin University West Australia Contents 3 COs sheet ouniinninuienileliileollkeienliefiNilo Isi n DRIIRiQe ilERRRI E DRR n PUER T PNE
54. Ej jcro oit excel ee ein i Ele gdt View Insert Format Tools Data Window Help Type aquestionforhep B DEBS SRY ssmg y 56 o amp r A b i Bio X new S TX puny Shorts ll fal Mover License Lelp etas 00000 wecmetoetps MU R 0 1837 To set up a new workbook use the New drop down menu optior i o so 2 IK hN Comments Data CCs OldCCs Syntax Problems System 1 1000 Frequency g g 8 E gt M wf Statszb Z Stats2ul 1 2 3 4 5 M M CCs f Fregs f Scores f Statsif Statsib Statsiul Stats2f Stats2l tees im Ttetesnsesoenoenoesoeroerororonassa a e LAU Lelp Lertap 5 help Interactive PDF version 2003 2011 Curtin University West Australia Made in Western Australia Help file for Lertap 5 Table of Contents Foreword 0 Part Welcome 1 B hrudrichye R 1 2 Bequirements io Mei NE A A aN aA a ENANA AE 2 xor d4geEe c 3 4 How to license IL iiir ce ccet sete cc teaeseaets E T 4 5 HOW tO MUM A ETETE TE icctaceendescetsectanececeanchasdattaveacdascadecccueacedssccashacatadibassonutsecahsaceessecadex 4 6 Known problems i
55. Excel 2007 package If you ve installed Lertap for Excel 2007 or Lertap for Excel 2010 the installation process will have brought in the macros file When Lertap starts up in Excel 2007 2010 this macro file is automatically summoned behind the scenes How to tell what a macro does Click on it and see what it says it won t hurt you it 2003 2011 Curtin University West Australia Toolbar and tab 225 won t actually do anything until you read its wee introduction and click on some sort of run me option also see one of the titbits below some macros are discussed in this document It is entirely possible to change the macros and even to add your own Interested Look at the paper referenced under the titbit below The ability to add your own home brewed macros to Lertap opens up an expansive horizon of possibilities If you have a serious interest in tailoring Lertap macros to match local needs do one or all of these things read the titbit below read the next topic z3 write to us at larry lertap com If you have an interest in a special macro but don t have time to make it write to us to see if we might take on the task If your idea is one which could be of benefit to other users there s a fair chance we d write the macro for free especially if it S winter too cold to do much outside there s no cricket or baseball to watch and petrol is still too costly to drive north to the tropics to people in the southern hemispher
56. FE AGES a a a eaa aiara pa eaea a aa Aaaa A aa ai ee cons ade teat adar apaa iad 88 C py gt cece a aa E cae Sec ae A AAE aeee Sia 88 4 Run Menu caecos einen Da enun nea eu vet uucwuiy cu lewesancudeGuesveeweueeGacuwcedeavcevondesusutesiodcedh Guewsvenve 88 Interpret CCS NES M 89 Production Tode uoi ee evite t ced e I dT ve Ue e ect Larne E ete o edes ee dard es 90 Emillon ALIO CELLA E A AEE Y 92 I2 tddsr Ee dit ideee e e REI G 92 Output teM SCOPES 93 MGS uineis tie ed Tobin Cote eee eee 94 Tetrachoric correlations ei 5 5 Wn Ah ie nete en E ee et 97 ADATIK WOKS DGOLEc oie eatem te tre ette tty e eet eme eter Ot ce ees bae dresse e ete M 98 Xcalibre w orksheets i Xcaret E O av Acinic one tenant RSAdalaworkslieet 4i eaa ee aea ive Rai eee Inv m dates Factor analysis ert tee eee eR eed eme I II END Me Creating a csv Tiley ia 16e nette end ee e ee ime de itta ec ERROR Time trials Response similarity analysis 118 Breakouts 131 Box and w Hiskers aee edere eer e ERE Y TUAE E REA EH EE TL rs ere EIER EE CY RE ident 136 Crime E 141 breaks charts Jennie EH eer HENRI rr E EIE E Eee 147 GhartGhanger 1 2 rei e e e ER niet ee tenant d RU RR ional 150 DIF uinihinsneNUetensendlencuu
57. Ir mx X Delete f Sort Blank Interpret iu Histograms Move 3 Version ix Line 7 R Headers Elmillon Z Scatterplot License fXSpread Excel wa Copy P More b Res charts 7 Lelp Other menus Run menu Basic options New menu Graphics trio EFPl i 2 3 6 a EX 3 4 05 2000 12042997 S M 11Feb 82 S 13 0 i 4 2S 4 05 2000 9900248 S M 9 Aug 79 T 15 0 5 3S 4 05 2000 12026768 S M T Jul 81 S 12 0 1 6 4S 4 05 2000 12062759 S F 20 Sep 78 S 12 0 Ti 5S 4 05 2000 1203660 S F 28 Jun 82 S 12 0 8 6S 4 05 2000 12001804 S F 15 Aug 81 S 11 0 9 TS 4 05 2000 12049901 S F 22 Jan 79 M 13 0 8S 4 05 2000 9916591 S F 2rFeb81 S 20 0 1 9S 4 05 2000 12103166 S M 16eDec 75 O 30 0 108 14 05 2000 9915556 S M 9 Oct 81 S 18 0 118 14 05 2000 12037359 S F 16Feb 70 S 11 0 128 14 05 2000 9811683 S F 31 Oct 80 S 13 0 13S 14 05 2000 9907838 S F 23 Sep 81 S 13 0 reren 4 05 2000 12036631 S T Sep92 S b T Cts rer l Scores lt PirtEreakst Seve L In the sample above the columns labeled Degree Gender and Entry would be typical examples of columns which carry some sort of group information Note you can change the codes used in columns such as these using the Recode macro available via the Move Menu It is also possible to exclude certain cases from the breakouts such as for example cases with missing data Click here 23 to read 2003 2011 Curtin Universi
58. S576 skewness B5 kurtosis 5578 expect 5570 within 1 sigma 68 30 69 23 5580 1 to 2 sigma 26 69 l ojo 0 tn njo 5581 2 to 3 sigma 3 43 5582 3 to 4 sigma 0 52 5583 4 to 5 sigma B584 over 5 sigma 0 00 0 04 01763788 values under Found 7 values to the left of 4 expect u a a normal dist having ae 5564 cases a 5586 el m M 4 gt Mf RSAdatal XRSAsigl RSAtablel RSAcas In this case the comment informs us that seven 7 Sigma values were found to the left of 4 standard deviations compared to the 0 1763788 values which we would expect to find under a normal curve It is possible to get Lertap to graph the Log PROB values Do so by using the histogrammerl 71 routine The RSAsig report will at times differ a bit to the samples seen above There s a limit to the number of rows an Excel worksheet may have in Excel 2007 the current limit is 1 048 576 rows Whenever the number of student pairs exceeds a bit less than this number Lertap stops entering results in RSAsig but continues to compute a subset of the descriptive statistics 1 048 250 is the precise number of pairs which Lertap will presently report on corresponding to 1 448 students It then adds a small table with selected results for all student pairs as exemplified here 2003 2011 Curtin University West Australia 126 Help file for Lertap 5 7 created on 6 01 2006 8 D H Hindex Log PROB
59. T T 1 00 6 00 11 00 16 00 21 00 Knwldge The R squared value is shown to the upper right of the chart It s not easy to see there so you could drag it to the left increase its font size a wee bit and maybe change font colour And why not change the colour of the trendline so that it matches that of the R squared font colour 2003 2011 Curtin University West Australia 4 5 3 Toolbar and tab 42 00 4 40 00 4 38 00 36 00 34 00 4 Comfort 32 00 30 00 28 00 26 00 4 e f 1 00 6 00 11 00 16 00 21 00 Knwldge Chapter 10 of the manual has a small section on Scatterplots Response charts b Item response charts are made by clicking on this icon Two or three styles of charts are made depending on the type of Lertap report worksheet active when the icon is clicked on How to make a worksheet active Just get it to show this is best done by clicking on its tab for an example click here If a worksheet of the Stats1b style is active then simple item response charts are made for all the items summarised in the Stats1b report this applies to all reports of the b style such as Stats2b Stats3b and so on The items may be either cognitive or affective If a worksheet of the Stats1ul style is active then quintile style plots are made for all the items summarised in the Stats1ul report this applies to all reports of the ul style such as St
60. University West Australia Input amp Output Kl a BP Is wt 1 00 0 00 0 00 o 3 C QD P c Q2 c4 g 0 00 0 00 A B c D E Q3 c5 option wt LrrtpQuizSample25June12 xlsx Microsoft Excel ca 5 3 Pagel Formu Data Review View Develc Add Ir VY Q cg S ge of LERTAP2 created 25 06 2012 26 25 9 2 1 0 43 0 42 0 15 0 07 0 20 0 12 0 13 0 48 0 03 nno Scores Statsif b r 0 66 0 57 0 17 0 01 0 41 0 29 0 27 0 66 b r 0 08 NNE b r 0 83 0 72 0 26 avg z 18 15 0 79 7 92 0 68 9 78 0 41 option wt n p pb r b r avg z 0 01 0 58 0 48 0 42 0 83 12 50 7 00 7 00 7 88 17 66 0 02 0 81 0 81 0 68 0 72 b r 0 20 avg z 9 50 0 45 22 Click here z to bone up on this section of a Statsf report the one showing above 2003 2011 Curtin University West Australia ome Help file for Lertap 5 r K H 09 E Jie LrrtpQuizSample25June12 xlsx Microsoft Excel o B zs Lerta Hom Inser Page Form Data Revie View Deve Add YV c s Lertap5 full item stats for Knowledge of LERTAP2 created 25 06 2012 Summary statistics number of scores n lowest score found 4 0 highest score found 96 0 median 50 0 mean or average 50 5 standard deviation 27 8 standard deviation as a sample 28 0 number of subtes
61. a number of academic and non academic posts in the United States New Zealand Venezuela Thailand and Australia All have had something or other to do with applied statistics data analysis test development data banking and computers At September 2009 Larry was thought to be wearing these hats Director Lertap Project Curtin University of Technology Perth Western Australia 2003 2011 Curtin University West Australia R amp R amp R amp R Associate Professor adjunct School of Education Curtin University of Technology Perth Western Australia Professor adjunct College of Research Methodology amp Cognitive Science Burapha University Bangsaen Chonburi Thailand Driver Miss Angela s Touring Camping amp Fishing Excursions Justaboutanywhere Australia Lertap became an official project of Curtin University of Technology in 2001 Click here 14 for contact information 2003 201 1 Curtin University West Australia 32 Help file for Lertap 5 Index THHHE symbols 250 alt 27 29 231 alt new format 321 alt remove an item 69 alt special form 69 col 27 col examples 69 exc 71 286 322 key 27 29 231 294 mws 27 29 231 294 mws remove an item 69 mws special form 69 mws with other 325 mws call 67 mws other 29 36 245 pol 27 sub 27 48 231 sub with MDO 53 sub control words 74 tst 27 36 73 212 wgs 27 wts 2
62. an Res declaration on a sub card For example Res A B C D E F Res 1 2 3 4 5 6 7 and Res t f are all valid Res 2003 2011 Curtin University West Australia CCs details Now with this understanding of response codes in hand the definition of missing data can be made a bit more precise in Lertap an item response is said to be missing whenever a cell in the Data worksheet has an entry which does not match the response codes used by the item When this happens Lertap says it has encountered an other response declarations In Lertap a test may use up to 26 response codes As Lertap goes about tallying item responses it keeps track of the number of other responses in a special bin What s this bin called The other bin naturally Deep inside Lertap each and every item is assigned a storage bin for other responses The contents of an item s other bin are displayed in many of Lertap s reports As an example look at these snapshots Q11 option wt n p 1 00 28 0 47 B 0 00 D 0 00 C 0 00 13 0 22 D 0 00 13 0 22 E 0 00 0 0 00 other 0 00 6 0 10 pi Statsif Statsib Statsiul statszf f Q35 option wt n 96 1 1 00 2 3 3 2 2 00 13 21 7 3 3 00 12 20 0 4 4 00 17 28 3 5 5 00 7 a ea other 3 00 a 15 0 pi f Stats2f Stats2b Stats2ul Stats3f Stats3k In these two examples the other bin for Q11 has n 6 meaning that there were six people missing data on the item For Q
63. and is often denoted as Q in the literature The average of the test scores The highest score found wn a The standard deviation of the scores Computed as a population value If you re familiar with the equations used to calculate this statistic the one used here has n in the denominator not n 1 var The variance of the scores computed as a population value Range Computed by finding the difference between the highest and lowest scores that is in this case Max minus Min IQRan The inter quartile range computed by finding the difference between the 75th and 25th percentiles of the scores In the literature the 75th percentile is often denoted as Q while the 25th is Q Thus the IQRange is equal to Q minus Q Uu a Q Z D 2 This statistic will be zero when the distribution of scores is ess symmetric about their central value In this case the Mean and Median will be the same Negative skewness means that the scores have a tail which extends to the left of the distribution while positive skewness indicates that the tail is to the right Lertap uses Excel s SKEW function to compute this statistic Kurtosis is an indicator of how peaked the scores are when compared to the normal or Gaussian distribution The normal distribution has a kurtosis value of 3 00 Lertap uses Excel s KURT function to compute this statistic KURT is actually an index of excess kurtosis a com
64. and paste Simple How to select all of the rows in an Excel worksheet It s real easy but not obvious if you haven t done it before Just click at the top left of the worksheet where the red circle is seen in the picture below 2003 2011 Curtin University West Australia Toolbar and tab 113 X id w e mB ga a a Book2 Microsoft Excel o 5 ES File Lertap Home Insert Pagelayout Formulas Data Review View Developer Add Ins 2 ca gg X x Delete Al Sort F Blank Interpret ali Histograms Move Version fx Line Headers gg Elmillon 7 Scatterplot License Spread Excel Copy 4 Morey b Res charts Lelp Basic options New menu Run menu Graphics trio Other menus AECABBEBBDADBABBCCCBBABDC 3542335464543244334445546 Rood edges eases oes ages sh oo SCCDBABACADC A DB BDA EEBF 31 BACAABEBEDADBBDACCADBBDBE 26CEDABBABFDDDBABBAACDC BCE 27AEAABCAB ACDBABBAACCABDBB 21AECBBCABAAA BABACCCDABBDE S9BECABBEB DADBBBBCCABCBBDC 47AECABBECBADADBABBCCCBBBBC 42AEDAABEBBDA BABBCCCBABBDC SSAEDABBEBBDADBADBCCCABBB C M M esemi Statsiu Z Stats2f Statsob Statsst sta d x cM LLL v Ready 2 SA 1009 7 J 65 Another way not as easy use Excel s Save as TXT MS DOS option This will create a simple text file with an extension of TXT this file may then be renamed so as to have an extension of DAT Then here s still another way to get the job done in
65. answer is selected There s no need for mws cards for these items their scoring is standard stuff So what s special about the items in c2 and c6 They don t use one of the response codes Now this really isn t a big deal Lertap would process the c2 and c6 items even if we didn t mention the fact that these items use just three response codes Lertap s various reports would simply show that the fourth option with a response code of D in this case was not selected by anyone and the Stats1b report would flag D as a poorly performing distractor Such things as test scores and coefficient alpha will not be not affected But why not do the job right Lertap allows items to have a different number of options The alt card and the mws card both allow you to set the record right to inform Lertap that some items do not use all of the subtest s response codes Use these cards and Lertap s reports will look a bit cleaner Example C12 Still more about the mws card col c2 c sub Name Followup TV9 news quiz key ACCDB Title NewsQuiz 2003 2011 Curtin University West Australia CCs details 35 alt CDDDC mws c4 0 5 0 0 5 0 This example is quite typical The item whose responses are coded in column 4 of the Data sheet is now being double keyed If someone selects the first response they get half a point And if someone selects the third response they also get half a point There are two right answers
66. average score is found in the mean column Its standard deviation is seen under s d and the correlation of the item with the criterion score corrected for part whole inflation is under the cor column As is the case in other Lertap output the standard deviation is that for the population the equation used has n in the denominator not n 1 The correlation is a Pearson product moment coefficient Missing data can have an effect on the statistics found in many of Lertap s reports including Statsb Read more about it here 48 Related tidbits How to print Lertap s reports Not hard at all especially if you take in this topic 6 4 3 2 1 Stats1b plot A plot of item difficulty by discrimination is given at the bottom of every Statsb report for cognitive tests The screen shots found here were taken from the Excel 2010 version of Lertap J une 2011 In this case a 56 item cognitive test had been processed 2003 2011 Curtin University West Australia Input amp Output ms 0 60 0 50 0 40 0 30 0 20 Discrimination disc 0 10 0 00 4 1 T T T 0 00 0 20 0 40 0 60 0 80 1 00 lt Hard Difficulty diff Easy gt These plots of item difficulty by discrimination are made using the diff and disc figures displayed in a Statsb report As these things go Joe this plot suggests a test with high reliability almost all of the items ha
67. b r avg Z 1 1 00 2 rd D 05 30 5 D 24 2 2 00 13 24 1 0 16 30 1 D 28 3 3 00 12 22 2 0 02 J3 l 0 04 4 4 00 1 3275 0 35 37 4 D 51 5 5 00 7 13 0 0 30 40 0 0 79 other 3 00 3 10 0 0 02 32 0 0 07 If the MDO option is turned on the output will change 2003 2011 Curtin University West Australia e Help file for Lertap 5 Lertap5 full item stats for Comfort items with MDO created 30 03 2006 Q35 option wt n 926 b r avg z 1 1 00 2 3 8 D 15 30 5 D 75 2 2 00 13 25 5 D 51 29 8 0 88 3 3 00 12 23 5 D 18 32 6 D 35 4 4 00 17 33 3 0 39 ste 0 56 5 5 00 7 Lait 0 44 40 0 1 09 other 0 00 3 15 0 D 26 29 0 1 04 To grasp what Lertap has done look down the 96 column for these two reports In the top report the values and the columns to the right from pb r to z are based on n 54 the six people who did not see Q35 have been excluded from the calculations In the next report we ve got MDO operating and now we ll have n what Fifty one 51 In this case the report excludes the six did not sees and the three did not answers Right What about the corresponding Statsb reports Thought you wouldn t ask Here they be Lertap5 brief item stats for Comfort with using LERTAP2 created 30 0 496 2296 3196 1396 54 Statsib_ Stats2f Where do you stand now You see what happens or are you in the did not see group Questions Crank up your email program and point it a
68. be found by paging ahead to the time trials amp topic Related tidbit For more about these topics see Some observations on the scree plot and on coefficient alpha a 16 page Word document with lots of little tables and some wonderful screes available via the Internet click here if you re connected Factor analysis As mentioned in the previous topic when the SMC setting el is on Lertap will output a row with correlations between each item and the first principal factor Witness eigens 3 37 0 55 0 29 0 25 0 22 0 21 0 14 0 11 0 08 facti 0 39 0 26 0 06 0 24 0 29 0 01 O13 0 39 0 27 Lertap s first principal factor is exceedingly simple its initial estimate of an item s communality is the same as its final estimate the item s SMC no iterations are undertaken to improve on the SMC Some users may find Lertap s p fact1 row to be a useful tool for indicating relative item loadings on the first factor but those interested in a more complete factor 2003 2011 Curtin University West Australia 112 Help file for Lertap 5 4 4 4 8 analysis or for that matter a complete principal components analysis will want to step out to a program such as SPSS or SAS How to move beyond Lertap to get Lertap s item scores into for example SPSS Not too difficult Have a look here Creating a text file A text file sometimes referred to as an ASCII file is a file devoid of special formatting characters Exam
69. by the new MDO are almost all showcased in the Stats3b report Click here 6f to see a sample How it works You ve taken the Cook s tour Goodonya We ve got some examples coming up but first how about a quick overview of how Lertap works Lertap is used to analyse the responses people have given to a test or survey Lertap is designed to work with fixed choice items items questions For a test fixed choice usually means true false or multiple choice For a survey fixed choice means that the answers people can give to a question have been listed and enumerated For example a survey might ask people if they think beer is a good thirst quencher on a hot summer s day people can answer 1 strongly disagree a foolish response 2 disagree 3 undecided 4 agree or 5 strongly agree The answers people give are placed in the Data worksheet Each row in the Data sheet corresponds to the answers of one person Having responses recorded in the Data worksheet is good and necessary before results can be obtained but it s not all that s required Some instructions are also needed Instructions For whom For Lertap Lertap is not smart enough to be able to look at the Data sheet and figure out what s what without your help You ve got to tell Lertap which columns in the Data worksheet have the item responses you want it to look at If the items are from a test you have to tell Lertap what the right answer to each item is
70. change Qi 0 909 0 006 Q2 0 909 0 006 Q3 0 911 0 003 Q4 0 917 0 002 QS 0 915 0 000 Q6 0 910 0 005 Q7 0 914 0 001 Q8 0 910 0 005 Q9 0 914 0 001 Q10 0 911 0 003 Q11 0 910 0 005 Q12 0 912 0 003 Qi3 0 908 0 007 Qi4 0 915 0 000 Q15 0 912 0 003 Qi6 0 913 0 002 Q17 0 911 0 004 Q18 0 910 0 005 Q19 0 907 0 008 Q20 0 908 0 007 Q21 0 910 0 005 Q22 0 916 0 001 Q23 0 914 0 001 Q24 0 915 0 000 Q25 0 910 0 005 M 4 Fre Scores Statsif Statsib 4 m gt Ready HU 100 gt 4 The last table in a Statsf report indicates how the value of coefficient alpha would change if an item were removed from the test In this example with all 25 test items the value of alpha is 9149 If Q1 were removed alpha would decrease by 0 006 becoming 0 909 Since higher alpha means greater reliability we would not want to drop Q1 from the test However the situation with Q4 is different Were we to eliminate it from the test alpha would increase to 0 917 This might be cause to question the inclusion of this item the next time the test is used 2003 2011 Curtin University West Australia 288 Help file for Lertap 5 There is a relationship between item discrimination and the value of coefficient alpha Items with discrimination figures above 0 30 can generally be counted on to boost alpha Items with lower discrimination values may on the other hand serve to bring alpha down Re
71. coefficient alpha will be equal when all of the test s items are scored on a right wrong or dichotomous basis which is common for a cognitive test When one or more of a test s items have more than one correct answer or when an affective scale is in use then polytomous scoring is in effect and coefficient alpha is the preferred estimate of reliability Some third party accreditation bodies such as NCCA the National Commission for Certifying Agencies will explicitly request the value of KR 20 for a test If your test items are all dichotomous scored on a right wrong basis then write in the value of coefficient alpha for KR 20 they re the same in this case If one or more of the test items has have more than one right answer then write in the value of coefficient alpha and indicate on the form that alpha has been used as KR 20 was not appropriate KR 21 is a simplified form of KR 20 used in the very rare case when all items have the same difficulty in other words it s not much used at all in its day it was a computational shortcut to KR 20 2003 2011 Curtin University West Australia 236 Help file for Lertap 5 6 4 3 1 3 The bands Three sections of data are found at the end of a Statsf report Fr x id 9 c Ep J 7 LrtpQuizsample25Junel2 xlsx Microsoft Excel ca 5 x Lerta Hom Insei Page Form Data Revie View Deve Add VY Q ca gg 2S LertapS full item stats for Knowledge of LERTAP2 created 25
72. contained in the Data sheet Row 2 also has header information Each column has been given a header or label ID Position Experience Gender Q1 Q2 and so on For your information the CCs sheet corresponding to this workbook had these two lines col c5 c64 sub aff res 1 2 3 4 Item responses begin in column 5 of the Data sheet and continue through column 64 Lertap will use the labels found in row 2 columns 5 through 64 as item IDs That is the ID for the first item will be Q1 for the second item Q2 and Q60 for the last item not shown above Item IDs can be anything and in theory can have any length However we strongly suggest that items IDs be short not greater than 8 characters in length Valid examples of item IDs Item1 Preg 2 Soal3 Ques2b SD204 Likrt17a Having short item IDs makes parts of Lertap s output easier to read for example the Stats1f report has a section which looks like this 2003 2011 Curtin University West Australia 256 Help file for Lertap 5 item difficulty bands 00 Q22 10 20 A30 40 Q1 Q2 Q9 Q11 Q14 Q18 Q19 Q20 Q21 Q25 50 Q3 Q4 Q6 Q7 Q10 Q12 Q15 Q17 Q24 60 Q8 Q13 Q16 Q23 70 Q5 80 90 The item IDs play a prominent role in tables such as that seen above the longer the item IDs the more cluttered the tables look If item labels are not found in row 2 of the Data sheet Lertap will automatically create item IDs of this sort Item1 Item
73. declaration which sets out 20 2003 2011 Curtin University West Australia 322 Help file for Lertap 5 7 1 10 7 1 11 options Sub Res A B C D E F G H 1 J K L M N O P Q R S T Title EMQs In order to accommodate the use of EMQs we ve changed the way Lertap s alt card works It used to be that the entries on the alt card indicated how many of the Res characters were used by an item now the entries actually indicate which Res character is the last one used by an item See example C7 under the Cognitive CCs 2A topic and remember alt is used only when some items use more options than others if all items use the same number of options alt is not needed The second enhancement New options which control how quintile plots plot You can have Excel automatically attach data tables to the plots if you want and you can tell Lertap to tell Excel that you only want certain items to be quintiled not all of them To read about this you will want to click here s3 And finally the third enhancement concerns adjusting the difficulty index for cognitive items so that unanswered or omitted questions are omitted from the calculation of the index If you would not like to read about this enhancement ignore the temptation to click here 53 Nov 2004 November 2004 exc a new CCs card to exclude items Ways to quickly remove an iteml edl from its subtest is a matter discussed in a topic on its own there s even a paper o
74. descriptive statistics from minimum to kurtosis have to do with the 5 564 Log PROB and H H Sigma values found in rows 3 through 5566 of the worksheet The little expect found table is used to gain an idea of how closely the Sigma values found followed those corresponding to the normal curve Under a normal or Gaussian distribution 27 2096 of all cases will lie between one and two standard deviations on either side of the mean for the dataset above 26 6996 of actual cases were found in this region slightly less than expected It s clear that the results found for this dataset did not identically match what would have been expected under a true normal distribution but they re perhaps not too bad Small triangles to the upper right of an Excel cell signify that a comment has been attached to the cell Letting the mouse hover over such a cell will cause the comment to appear as seen below 2003 2011 Curtin University West Australia Toolbar and tab O 9 10 11 12 13 1 treated on 7 01 2006 2 H Hindex Log PROB H H sigma 5564 0 50 15 78 4 56 5565 2539 18 36 5 80 5566 2 33 18 89 6 06 5567 n 1 5 568 5 564 5568 minimum 18 89 6 06 5569 median 0 05 e tN C2 5570 mean 55g maximum 5572 d 2 07 5573 variance 4 28 l ec co nz l o Cen tM J J yb co O C J Ble ojo ojo 5574 range o e gt 5575 IOrange co H
75. each worth half a point This example exemplifies one way in which partial credit may be addressed in Lertap mws c4 1 O 1 0 Here again there are two right answers but now they re each worth one point mws c4 0 50 0 50 2 00 0 75 Things are getting real fancy now The best answer is the third one for which a whopping two points are awarded The first answer is worth half a point The second and fourth answers now have negative scoring weights a person selecting the second option loses half a point whereas someone going for the fourth option will lose three quarters of a point Example C13 Some buildings do not have a 13th floor and we don t have a 13th example either Example C14 Re scoring all items at once col c2 c6 sub Name Followup TV9 news quiz Title NewsQuiz key ACCDB mws call 1 0 1 Q The items found in all of the subtest s columns are to be scored with one point for the first and third responses with no points for the second and fourth responses This sort of scoring is not at all common for cognitive items not at all but if you want to do it you can In the world of Lertap mws cards are the most potent cards going They re dynamite They completely override whatever information has come on preceding CCs cards A special form of the mws card may be used when it s desired to quickly remove an item from a subtest Click herel 6 to read about it In Lertap Version 5 25 another
76. every time you start Lertap Unless that is you wisely decide to put your faith in The Trust Center No doubt you noticed the small offer from Excel to Open the Trust Center It s seen in the very lower left of one of the screen shots just above Looks like this Help protect me from unknown content recommended Enable this content 8 Open the Trust Center You can use the Trust Center s options to change things so that Excel considers your Lertap5 xlsm file to be in a trusted location When you open a file that s in a trusted location Excel will not give any of its security warnings This is handy It s so handy that it s worth getting the kids or grandma back in for a moment Ask them to do these things find the folder where the Lertap5 xlsm file is on your computer and then get Excel to recognize this folder as a trusted location Ry Organize v C B Date modified i Type 2LRTP57HHelp pdf 18 07 2008 9 03 AM 2 667 KB Adobe Acrobat Docum ESSE lt Local Disk C ASC gt Lertap7 Delis Bseorch 0 2 EP LRTPS7HHelp chm 18 07 2008 9 26 AM 1 908 KB Compiled HTML Help file er LertapRibbon xlam 30 06 2008 11 14 AM 57KB Microsoft Office Excel A Lertap5UserManual pdf 16 08 2008 12 10 PM 2 231 KB Adobe Acrobat Docum er Lertap5MacroSetA xlam 9 07 2007 4 08 PM 103KB Microsoft Office Excel A l LertapS xism 21 07 2008 2 28 PM 1 231 KB Microsoft Office
77. file to a Lertap DATA worksheet followed by application of The Spreader 82 in the Data worksheet s third column quickly set up the data for Lertap processing Lertap has a special ability to import the text DAT files used by the ITEMAN program This could well serve as a general means of importing data prepared by a scanner Read about it in the next topic a Are you aware of Excel s ability to dissect a text file Excel has a text import wizard a useful tool which can be of real help when you ve got to take apart a text file the DAT files output from some scanners are usually simple text files This wizard can be whizzed up in a couple of ways One way is to use Excel s File Open options and under Files of type ask for All Files then browse to the file you ve got in mind Or use Excel s Data Import External Data and follow the same steps If the file you point to is purely text Excel opens a dialog box which allows the file s contents to be cut out field by field and placed in the columns of a new Excel worksheet Here s a picture of Excel s text importer in action 2003 201 1 Curtin University West Australia a Help file for Lertap 5 Text Import Wizard Step 1 of 3 The Text Wizard has determined that your data is Fixed Width If this is correct choose Next or choose the data type that best describes your data Original data type Choose the file type that best describes your data Delimited
78. five options 1 2 3 4 5 A student who selects the first option on Q26 will get one point Selecting the first option on Q27 and Q28 on the other hand will award the student with five points These items are example of polytomous scoring To read more about how Lertap deals with items of this sort please branch to this topic s8l What happens when students don t answer questions or are not given the opportunity to answer all questions How does Lertap go about scoring in these cases See the missing datal4e topics to find out 2003 2011 Curtin University West Australia Input amp Output Finally see the manual There is much more about item statistics in the manual and it s not a difficult read at all Related tidbits Flags are also waved in Stats1b reports where they appear in the column Read more 294 The use of flags in Stats1f reports is controlled by an Annotate Stats f reports setting in the System worksheet 28 it is possible to turn them off Chapter 7 of the manual has a more in depth discussion with references to relevant literature Get a pdf copy of this chapter with a click here How to print Lertap s reports Not hard at all especially if you take in this topic 6 4 3 1 1 1 With external criterion Statsf reports become ECStatsf reports when an external criterion 92 analysis has been selected The discussion found in this topic is based on results from the LenguaBlg dataset The Core
79. flag on option C a distractor indicates that the 2 271 students who selected C with an average external criterion score of 29 65 were above average students at least as measured by their external criterion scores These are generally unwanted outcomes the flags are there to draw attention to a potential problem with 121 avg ec expressed as a z score r ec The Pearson product moment correlation between the item and the external criterion score When an item has only one option with a non zero wt value r ec will be the same as pb ec 2003 201 1 Curtin University West Australia 280 Help file for Lertap 5 6 4 3 1 1 2 F FAR En Ji LenguaBigForLrtp593 xlsx Microsoft Excel ca 5 xs Lertap Home Insert Page I Form Data Revie View Devel Add1 o c X LertapS external criterion stats for Trial items only created 4 07 h Statistics for the external criterion score Core number of scores n lowest score found highest score found median mean or average standard deviation standard deviation as a sample correlation bands with external criterion 00 121 129 135 139 149 10 116 20 115 123 I31 30 40 16 50 60 70 80 90 Ub Es 1 a Stats3ul HistoiL ECStats2f ECStats2ul 4 Ii gt Ill 4 The Statistics table for an ECStatsf report is similar to the Summary statistics found in an o
80. for every item If they re not there won t get any quintile plots at all Yes Now we can tell you this you can get into the Stats1ul report yourself and either delete the triangles or add them How Hover your mouse over the cell and right click on it 2003 2011 Curtin University West Australia 2 Help file for Lertap 5 New Run Shorts j Move License Lelp LertapS U L stats for Results from class of 84 a B c Item 1 uppa E as po D C ut I 2nps nf 0 C IA Paste BS D 4t Paste Special D lowe Insert Item 2 uppe Delete 2n Clear Contents i ta Edit Comment H a Delete Comment lowe show Comment Item 3 uppt Format Cells Pick From List 3r B Hyperlink T 4t D I4 4 MZ Fregs Scores Stat oooloococlc D C The triangles are Excel s way of indicating that cells have comments Take the Delete Comment option seen above and the triangle will disappear If a cell has no triangle the drop down menu seen above will have an option to Insert Comment In this way you can control the items which are to be quintile plotted The triangle means mark this item for a quintile No triangle no quintile A couple of questions arise If I m going to insert a comment what should it say Anything you want Love you Mom Don t forget to buy milk Hope we whomp the All Blacks Pickable for plots The comment ca
81. gt Ready 2 Bam 100 2 J 9 Three items from the maths quiz are on display in the screen snapshot above They have been selected in order to show how Lertap flags item options which may have problems There were 15 items on the quiz all scored on a right wrong basis with one point given for each correct answer The overall mean test score was 7 67 with a standard deviation of 2 88 Now if an item is meant to be one which helps us identify the strong students we expect that those who get the item right will be the best students Those who get it wrong should be the weakest students This did not happen on 111 The keyed correct response was option 1 selected by 473 students Their avg test score was 6 50 corresponding to a negative z score of 0 40 The last option 4 was selected by 459 students whose avg test score was 9 21 with a positive z score of 0 53 strong students are getting it wrong Weak students are getting 111 right while 2003 2011 Curtin University West Australia Input amp Output To draw our attention to this Lertap has flagged 111 twice We ve got a ba flag for the first option and an aa flag for the last option The ba flag appears whenever an option with a wt above 0 00 has a negative z value ba means below average This flag will wave whenever the students who select the keyed correct option have a below average test score we expect such student
82. let s start at the start The way to get to the Macs menu in the Excel 2007 2010 version of Lertap is to click on Move and then on the Macs line In early September 2010 the Macs menu looked like this 2003 2011 Curtin University West Australia Dm Help file for Lertap 5 Copy a Data column to the Scores worksheet Copy a Scores column to the Data worksheet Copy a Data column to a DAT worksheet Recode a Data column Apply a special Scores worksheet formula Update Scores worksheet correlations Lelp WrongltemList2 StatsbSortAZ NumericFilteri NumericFilter2 Itemani ChartChangerl TotalTesti AngelMaci AngelCognitive2 AngelAffectivel Keep in mind that your Macs menu might not look like this okay We may have made some changes and not yet updated this topic s displays The names of the options seen under the Macs menu are the names of macros Above the names start with WrongltemList2 and end with AngelAffectivel Macros Whatsit If you don t know what a macro is open Excel Help and toss it a search term such as about macros using macros or creating macros Macros are based on computer code they re special purpose computer programs often quite brief For your information the macros accessed via the Macs menu are stored in a special Excel file called LertapbMacroSetA xlam How to get the Lertap5MacroSetA xlam file You need not worry it s included as a standard component of the Lertap for
83. more about IStats sheets please refer to the manual You ll find an example in Chapter 7 and a fairly thorough discussion under the Item scores matrix section in Chapter 10 Note however that the SMC tetrachoric export worksheets and eigens options were added after the manual was printed SMCs An item s SMC value its squared multiple correlation indicates the proportion of the item s variance which may be linked to or predicted from the other items in the subtest As mentioned in the previous topic the SMC is sometimes used as an estimate of the amount of variance any single item has in common with the other 2003 2011 Curtin University West Australia Toolbar and tab s items For technical discussions on the SMC and its calculation see Pedhazur amp Schmelkin 1991 pp 414 417 Lord amp Novick 1968 pp 265 266 Hays 1973 pp 705 708 and Glass amp Stanley 1970 pp 186 191 or search the Internet for multiple correlation coefficient To get to the Lertap references page begin with a click here faz Lertap s IStats worksheet gives some prominence to SMC values by banding them Help your little self to a squiz of the following sample F3 Microsoft Excel Book1 8 File Edit view Insert Format Tools PopTools Data Window Help PDF Create f X X O New S rZ pune Shorts MM Move License Lep 7 LertapS IStats matrix last updated on 1 10 2004 eigens 8 62 1 87 1 61 1 42
84. more than one subtest to process no in the sense of mixing subtest types this example has a cognitive subtest and an affective subtest a mix of subtest types that s quite uncommon If you browse on into the following topics you ll see a couple of other examples Related tidbit For a really bonza example of a job which worked Lertap s CCs lines close to the limit have a look at Using Lertap in a Parallel Forms Reliability Study a 16 page Word document available via the Internet click here if you re connected The Total score a composite formed by summing subtest scores gets further mention here 234 2003 2011 Curtin University West Australia Help file for Lertap 5 2 3 4 Multiple cognitives 2 3 5 Consider these CCs lines cgld El E10 xsub name Addition title Add key DCCAB BCDDA col cl1 c20 sub name Subtraction title Sub key BBBCA DAACB col c21 c30 sub name Multiplication title Mult key CDCAB AAACC col c31 c40 sub name Division title Div key AADCC CBAAA Someone s given a maths test with four subtests Each subtest had ten items Lertap will create four subtest scores and a total score The total score will simply be the sum of the four subtest scores The maximum possible score on each subtest is 10 hence the maximum possible total score is 40 It is possible to change the number of points given for right answers by using mws and
85. multivariate scatterplot As the size of the first component comes to dwarf the others some people say there appears to be but one dimension underlying the items which in turn often leads people to say that the items are measuring the same thing Lertap will also plot the item component correlations or loadings in bands It takes the values found in the p comps1 row and makes a little table such as the one below X New S rZ pune 2 Shorts MU gt l Move License Lep LertapS IStats matrix last updated on 31 10 2004 Q26 Q27 Q28 Q29 Q30 Q31 Q32 P Comp1 bands 00 Q28 Q31 Q34 10 20 30 Q32 40 9D0 60 Q27 Q29 Q30 70 Q35 80 Q26 Q33 90 I f Stats2b Stats3f Stats3b y IStats Jp 2003 2011 Curtin University West Australia Toolbar and tab 109 The P Comp1 bands indicate that there s a group of six items Q26 Q27 Q29 Q30 Q33 and Q35 with high correlations on the first principal component If we were to create a new subtest using just these items chances are very good we d end up with a coefficient alpha value much higher than that obtained for all ten original items And speaking of alpha values did you happen to notice that one of the eigenvalues seen above the first one has a little black triangle next to it This triangle is really red not black but for some reason when we took our snapshot of the original screen
86. mws c4 0 1 2 mws c5 0 1 2 mws c6 0 1 2 mws c7 0 1 2 3 mws c8 0 1 2 3 mws c9 0 1 2 mws c10 0 1 mws cll 0 1 2 3 The mws lines indicate that the number of item points possible varied from item to item For the first two items scores were limited to O or 1 for the items found in columns 4 5 6 and 9 possible scores ranged from 0 to 2 Three items those in columns 7 8 and 11 had possible scores ranging from O to 3 Fewer lines would have been required had it been possible to get the same number of points on each item col c2 c11 sub res 0 1 2 3 name Listening AARP quiz Title L AARP mws call O 1 2 3 Here the call form of the mws line has been used call means columns all According to these CCs lines it s possible for a student to get up to 3 points on each item Related tidbit Users who often use pre scored items might want to consider qualifying for use of Lertap s advanced toolbar Why not get your mouse to click here z and have a read 2003 2011 Curtin University West Australia CCs details e 3 5 Open ended items It is possible to have Lertap score open ended items such as short answer questions and essay questions Please refer to the following URL for further information http www lertap curtin edu au Documentation ScoringNonMCitems1 doc 3 6 Remove an item It sometimes happens that users will be Lertapping along humming
87. normal curve A z score of 1 00 in a normal distribution is at the 84th percentile only 1696 of all test takers will have a z score higher than 1 00 A z score of 1 00 in a normal distribution is at the 16th percentile only 1696 of all test takers will have a z score less than 1 00 More about z here Consider now another test a 15 item mathematics quiz given to 999 high school students 2003 201 1 Curtin University West Australia Help file for Lertap 5 x id MathsQuiz25June2012 xIsx Microsoft Excel c3 B x Lert Hom Inse Page Forn Datz Revi View Deve Add YV c P zs Lertap5 full item stats for MathsQui created 25 06 2012 I11 c12 option wt n b r b r avg z 1 1 00 473 0 47 0 52 0 65 6 50 0 40 lt ba 2 0 00 18 0 02 0 10 0 29 5 61 0 71 3 0 00 30 0 03 0 10 0 25 6 03 0 57 4 0 00 459 0 46 0 49 0 62 9 21 0 53 lt aa other 0 00 19 0 02 0 18 0 53 3 95 1 29 114 c15 option wt n p pb r b r avg z 2 0 00 199 0 20 0 09 0 13 7 15 0 18 3 0 00 146 0 15 0 11 0 17 6 88 0 27 A 0 00 263 0 26 0 01 0 01 7 71 0 01 lt aa other 0 00 164 0 16 0 17 0 25 6 57 0 38 I15 c16 option wt n b r b r avg z 1 0 00 569 0 57 0 06 0 07 7 82 0 05 lt aa 2 0 00 156 0 16 0 10 0 15 7 01 0 23 3 1 00 155 0 16 0 12 0 18 9 30 0 57 4 0 00 74 0 07 0 19 0 36 5 69 0 69 other 0 00 45 0 05 0 15 0 33 5 69 0 69 v M 4 gt M Fregs lt Scores Statsif lt Statsib 1 4 Mm
88. option on the toolbar or visit us at www lertap curtin edu au Drop us a note if you have questions Our email address is given below Messages are most welcome Enjoy support lertap com 2001 2007 Curtin University of ied Ma4 Comments Data CCs Ready PQ An important component of the screen seen above is the line of worksheet tabs showing towards the bottom of the screen There are several tabs Comments Data CCs System Syntax and OldCCs Each tab corresponds to an Excel worksheet The Data worksheet has the responses of 60 people to two quizzes The CCs worksheet has a series of lines with Lertap s control syntax Each of the syntax lines gives Excel information on how to process the data found in the Data worksheet The Syntax worksheet is a quick reminder for experienced Lertappers on the correct format for CCs lines The System worksheet 2a turns on off assorted Lertap options Go ahead and look at each of the worksheets if you d like Then come back here Okay The Cook s tour starts with the Copy option on the New menu This is your first task find the New menu find its Copy option click on the Copy option Hint 2003 2011 Curtin University West Australia 16 Help file for Lertap 5 see screen snapshot below C Blank p Headers Copy New menu The Copy option causes Lertap to do a couple of somersaults and back flips it makes a copy of the Data and CCs worksheets and pl
89. owe 4 5 3 2 Quintile plots b When a Stats1ul style report is active a click or two on this icon will produce charts such as this un ded Y Mm EP 95 Verdana Excel2007Quintilesl xls M m x Lertap Home Insert Page L Formu Data Review View Develo Add in xX 4th 2rd 2nd Key C Diff 37 Disc 44 M gt r Statsiul StatsiulChta StatsiulCht i The quintile a plot pictured above shows how five item options A B C D E performed over five groups lower achievers to upper achievers with three intermediate levels of achievement The variable used to index achievement may be internal or external internal is the default and is simply the total test score The number of groups is usually five hence the term quintiles but may be less it s an option set in the System worksheet see this topic fa The other line generally represents respondents who omitted the item Wainer 1989 53 referred to plots similar to the one above as option trace lines If you haven t seen these before have a careful look at their message the green line with small triangles on it corresponds to the keyed correct answer C The proportion of people in the lower group who selected this option is 0 11 a figure which steadily increases as we get into higher levels of achievement by the time we get to the upper group the proportion able to pick out the correct opti
90. page breaks but that s fun and simple to see how take in the very bottom of this topic 4 5 3 2 2 EC quintile plots Special note the charts displayed in this topic were produced by the Excel 2003 version of Lertap It is possible to make both quintile a and quintile b plots with an external criterion 92 score When the Run menu s External criterion analysis option is selected an upper lower groups worksheet is created assuming the subtest involved is a cognitive one and the option to create upper lower analyses has been set to Yes in the System worksheet z43 2003 2011 Curtin University West Australia Dow Help file for Lertap 5 When an external criterion is used the upper lower groups worksheet produced by Lertap will be called ECStats1ul or more generally ECStatsXul where X corresponds to the subtest involved If the ECStatsXul worksheet is active clicking on the charts icon will get the quintile plots rolling Now part of the process of setting up an external criterion analysis 931 involves selecting a score from the Scores worksheet in fact it s this score which defines the external criterion Lertap will check the selected score to see if it might correspond to a categorical variable such as Gender or perhaps Region Note the Recode macrol available via the Move Menu is useful for working with categorical variables This macro will for example allow variables coded with letters to be rec
91. particularly strong on the use of effect size estimators and is certainly one of the most compelling sources when it comes to discussing the limitations of tests of statistical significance Box and whiskers Once you ve used the Run menu to Breakout scores by groups you l have a Breaks report s a worksheet whose name begins with the word Breaks And once you have a Breaks report you can use the Box and whiskers option to get a graph which looks like the following this option is found by clicking on the Run menu s More section right below Elmillon 2003 2011 Curtin University West Australia Toolbar and tab Boxplot for SelfReg 85 75 4 65 lt ERE 45 35 4 25 4 E P S The data plotted above are from the MSLQ study mentioned in Chapter 9 of your favorite read the Lertap manual Results from one of the MSLQ scales SelfReg have been plotted for three groups of student teachers ECE Early Childhood Education Primary and Secondary The top of each box corresponds to Q3 the 75th percentile the bottom of each box corresponds to Q1 the 25th percentile The line in the middle of each box represents the position of Q2 the median the 50th percentile Shouldn t the median be halfway between the 75th and 25th percentiles If the distribution of scores is symmetric yes but otherwise no Otherwise is the usual case as scores are not often exa
92. responses to be coded as letters such as the set A B C D For affective items 1 2 3 4 5 is a popular response code set If we look down the columns of the Data worksheet these response codes are what we see but of course you might say And well enough But what sometimes happens is that users mis match the codes found in the Data worksheet with the codes found in a sub card s Res declaration For example if the Data worksheet shows responses as being from the set A B C D and if Res a b c d there will be a crash the Res declaration is wrong the Data worksheet uses upper case letters Now item responses can be just about anything The answers to cognitive items can be coded as digits the answers to affective items may be coded as letters If letters are used they may be upper case or lower case But in all cases the Res declaration has to be fair dinkum if the Data sheet uses lower case letter then so must the Res declaration Finally remember the default Res assignments For cognitive tests the default is Res A B C D For affective tests the default is Res 1 2 3 4 5 If a sub card has no Res declaration on it these default settings are assumed Output What are the steps in a Lertap analysis Create a new Lertap workbookl 87 Enter item responses in the Data worksheet 5 Make up the control cards 7Y for the CCs worksheet Go to the Run 88 menu on the Lertap toolbar 7 Click on Interpre
93. scored matrix Run Help Lertap also creates worksheets with the same information as that found in the two text files One of these new worksheets is called XCal41Data the other is called XCal411CF These two worksheets are in no way critical to Lertap they may be deleted if they re of no use deleting the worksheets does not delete the corresponding text files 2003 2011 Curtin University West Australia Toolbar and tab ws 4 4 4 5 RSAdata worksheet RSA stands for response similarity analysis you d be interested in RSA if you wanted to see if the responses of any two test takers were as Wesolowsky 2241 2000 would say excessively similar In less diplomatic terms RSA is used by some to examine the possible presence of cheating in an examination environment Lertap will produce a worksheet RSAdatal and a special text file SCheckDatal DAT whenever users take the Output items scores matrix option from the Run menu and have set the RSA option to yes in the System worksheet If production modelo is on and your data set includes more than one subtest then there will be additional files RSAdata2 SCheckData2 DAT and so on one pair of files for each subtest The screen snapshot below captures the System worksheet s RSA settings as found in July 2005 note the yes setting in row 25 1 These are Lertap5 system settings Allowed Change them only if you understand them s
94. secondary worksheets may be regenerated if the primary worksheets are still on hand What the little does is allow the user to quickly delete the secondary worksheets 2003 2011 Curtin University West Australia Toolbar and tab at When this icon is clicked on a dialog box appears which informs users that all worksheets whose names begin with freqs scores sub stats and histo will be deleted if the user clicks on the OK button Why would a user want to delete the secondary worksheets Probably to save disk space The secondary worksheets can be quite large if there are many items in the data set and or if there are many records in the data set Users may also want to delete the secondary worksheets if they want to send a copy of their workbook to someone else Ordinarily no harm is done by using X Going through the process of Interpret CCs lines and Elmillon item analysis will restore the secondary worksheets at any time Users who wish to make sure their secondary worksheets will not be deleted by this icon should rename the worksheets perhaps by putting a prefix before their names For example Freqs might be renamed as OrigFregs Note that renaming Freqs as FreqsOrig would do no good as the worksheet s name still begins with Freqs a prefix is suggested not a suffix How to rename a worksheet The fastest way is to right click on the worksheet s tab at the bottom of the screen To read about the various ways w
95. sort of record ID and a string of zeros and ones representing item scores This is ordinarily sufficient for the Bilog program but users of Bilog MG often want to have more data in the DAT file For example they might want some sort of group identification code between the ID field and the string of item scores If the group identification code has been included in the Data worksheet it may be copied over to the DAT worksheet using this option In fact any column in the Data worksheet may be copied to DAT As columns are copied over they re added after the ID field Lertap makes an attempt to keep track of the format of the DAT records by including a Fortran format statement at the top of the DAT worksheet This statement will usually not be adequate for Bilog MG it s meant to be used as a guide Let s look at an example a data set having a Data worksheet as captured here when running with an earlier version of Excel Ed Microsoft Excel LRTPQuizWithExtraColumns xls File Edit view Insert Format Tools Data Window Help ux Ui Move4 o License Lelpy 6 7 8 9 10 11 12 13 14 14 OW 00 4 OY O1 A WIN O J Ch n 4 CO P2 Ln N e e e wW w e N 1H 55 12 10 51 41 an M 4 gt Data CCs Fregs Scores Ready Iu B B en 2 0 00 F O Fr 000 3 0 DD D CD CD CD D D A e E E E E E E E E D tS DOrwowowwwor te Pir ir Orrin ol gt n Statsl
96. special form of the mws card was introduced It has this form mws c12 0 1 0 1 other 1 To give credit to everyone for an item even if they didn t answer the item a card such as the following might be used 2003 2011 Curtin University West Australia Help file for Lertap 5 3 2 mws cl2 1 1 1 1 other 1 The card above gives one point for each of the item s permitted answers and it even gives people one point if they didn t answer the item Click here to read more about other 251 There are indeed times when as in Example 10 a mws card is used for each item It may be only 296 of the Lertap using world which will have an example of this sort but it does happen we ve seen it In such a case does the key card make sense No But Lertap requires each and every cognitive subtest to have a key card so put one in please Keep in mind that the manual has three chapters on CCs cards Between what s written there and what s appeared in this help topic we hope you l have an adequate to good grasp of CCs cards But drop us a note if you ve got questions lari lertap com Affective CCs This is the all you ever wanted to know page about affective control cards First some terms Let s say you had a couple of Likert style items like these uns 1 West Australian beaches are unsurpassed in the whole world Strongly disagree Disagree Undecided Agree Strongly agree Ui 430 N H
97. subtest will not be included in the total test score Now we will exemplify the use of the other control cards for cognitive tests Example C7 We ll add an alt card col c2 c0 sub Name Followup TV9 news quiz Title NewsQuiz key ACCDB alt CDDDC The alt card is optional Here it s telling Lertap that the last response code used by the first and last items is C whereas the last response code used by all other items is D Since there is no explicit Res declaration Lertap assumes Res A B C D Note that this format of the alt card differs from that shown in the manual It s a new format introduced in February 2005 In the old format this alt card would have been alt 34443 To read more about the practical effects of using alt please see the very end of this topic 25 Example C8 We ll use a wts card col c2 c6 sub Name Followup TV9 news quiz Title NewsQuiz key ACCDB wts 31121 The wts card is optional It indicates the number of points to be given for the correct answer and it s only required when some of the items are worth more than one point In this example the correct answer to the first item A is worth 3 points while the correct answer for the fourth item D is worth 2 points All other 2003 2011 Curtin University West Australia CCs details 33 items are worth one point If an item is to be worth more than 9 points a mws card has to be used mws cards are mention
98. test instead of a conventional reliability estimate such as coefficient alpha or KR 20 Three of Lertap s statistics could be used as the decision consistency estimate 1 the Index of dependability is Brennan and Kane sls24 1977 index of dependability for mastery tests 2 the Prop consistent placings is an estimate of ps a measure of the overall consistency of mastery nonmastery classifications Berk s 2000 3 is kappa an estimate of the proportion of individuals consistently classified beyond that expected by chance Berk 2000 Which of these three should be reported for NCCA Nelson 2007 suggests Prop consistent placings or the Index of dependability Berk 2000 strongly favors Po Lertap s Prop consistent placings A paper discussing the use of cutoff scores Nelson 2007 and the statistics seen in this topic may be downloaded by clicking here Examples of tests with poor reliability but good to very good classification consistency are shown here Chapters 7 and 10 of the manual have more to say about these statistics 6 4 4 Conditional SEMs Your read of the Lertap manual will have made you full bottle on the usual standard error of measurement SEM and the invaluable role it has to play in the interpretation of test scores The SEM commonly used in CTT classical test theory is an average figure one very frequently applied to each and every student s test score as an estimate of
99. that it will fall into one of the bands However the item mean will display correctly in the Stats1b report 2003 201 1 Curtin University West Australia The Summary statistics section of a Statsf report consists of two small table om Help file for Lertap 5 6 4 3 1 2 Summary statistics S E x id 9 En Ji LrrtpQuizSample25June12 xlsx Microsoft Excel ca E 9S Lerta Hom Inser Page Form Data Revie View Deve Add 9 9 o X LertapS full item stats for Knowledge of LERTAP2 created 25 06 2012 a Summary statistics number of scores n lowest score found 4 0 highest score found 96 0 median 50 0 mean or average 50 5 standard deviation 27 8 standard deviation as a sample 28 0 number of subtest items 25 minimum possible score 0 00 maximum possible score 25 00 TE reliability coefficient alpha 0 911 index of reliability 0 96 standard error of measurement 2 03 8 1 Yv M 4 M Data CCs lt Fregs lt Scores Statsif lt S 4 il gt Ready 100 gt number of This figure corresponds to the number of students scores whose item responses are found in the Datal 2s n worksheet Lertap creates a test score for each student when the Elmillon 92 option is used The actual test scores are found in the Scores ss worksheet Of all the scores this figure is the lowest one found Th
100. the Scores worksheet Lertap doesn t know what to put in the MinPos and MaxPos cells after it has been copied It writes Unknown in these cells leaving it to users to put in proper values MinPos and MaxPos values are required by some of Lertap s routines such as the external criterion 92 routine if Lertap requires these values it will ask for them Copy Scores column This option takes all the scores found in a selected column of the Scores worksheet and copies them to the first empty column found in the Data worksheet There are at least a couple of reasons why users want to do this First users may want to export the Data worksheet for use in another application such as perhaps SPSS However before doing so they d like some of the columns in the Scores worksheet to be appended to the Data worksheet At other times users may want to delete the Scores worksheet perhaps simply to save disk space Before doing this they ll sometimes copy one or two of the Scores columns to the Data worksheet 2003 2011 Curtin University West Australia Toolbar and tab mo 4 6 1 3 Copy Data to DAT The DAT worksheet 98 is a very special one It s created when the option to output item scores 93 is taken from the Run s amp menu The DAT worksheet is most likely to be used in conjunction with the Bilog and Bilog MG computer programs When it s first created the DAT worksheet contains just two real bits of information some
101. the process of saving the file as TXT MS DOS using quotation marks around the file name will allow it to be saved directly as a DAT file for example 2003 2011 Curtin University West Australia m Help file for Lertap 5 Save in E PietAbik 7 ax gt Tools E pietia txt History My Documents Favorites File name piet1a DAT My Network Places Save as type Text MS DOS txt Y Cancel z When saving TXT or DAT files in this manner Excel is likely to send a message such as this un Microsoft Excel The selected file type does not support workbooks that contain multiple sheets To save only the active sheet click OK To save all sheets save them individually using a different File name For each or choose a file type that supports multiple sheets Cancel Click OK and be prepared for Excel to say something like this Microsoft Excel piet1a DAT may contain Features that are not compatible with Text MS DOS Do you want to keep the workbook in this Format To keep this Format which leaves out any incompatible Features click Yes To preserve the features click No Then save a copy in the latest Excel Format To see what might be lost click Help This is also okay click Yes 2003 2011 Curtin University West Australia Toolbar and tab To see if the file you ve saved looks okay get out Notepad or WordPad or for Mac users Text
102. their favourite tune and suddenly wish that they could omit a single item or two from a subtest just to see how that might change things such as the value of coefficient alpha There are a variety of ways in which this may be accomplished The most obvious way is to make a new col card For example consider these CCs cards col 028 037 sub aff title Comfort pol Now suppose it was found that the item in column 36 was not correlating well with the others was serving to bring down the subtest s reliability figure coefficient alpha or for some other reason had to be removed from the subtest The following col card will do the job col c28 c35 c37 But this isn t the only change required The pol card must be changed too we ve taken an item out of the subtest and must remove the corresponding plus or minus sign from the pol card col c28 c37 sub aff title Comfort DOl tet An easier way to take the item out is to make use of a special form of the mws card col c28 c37 sub aff title Comfort pol mws c36 The mws card above has a single asterisk after the column number This is a special form of the mws card used to remove an item This special use of the mws card 2003 2011 Curtin University West Australia Help file for Lertap 5 eases the task of taking items out of a subtest there s no requirement to make corresp
103. to undertake some sort of IRT analysis At times one wants to have two samples of the original data one of these might be used to calibrate an IRT model with the second sample then used to validate the calibration 2003 201 1 Curtin University West Australia 160 Help file for Lertap 5 4 5 4 5 1 Teachers might use Halve amp Hold to demonstrate sampling variance how do Lertap s scores and item statistics vary as we compare one of the samples with the other Time trials September 2003 on a Pentium 4 running at 2 GHz with 3 000 original records the two halves were created in 18 8 seconds With a bit over 11 000 original records the two halves were ready in 4 minutes 18 4 seconds Graphics trio There are numerous areas in Lertap where charts appear or where you ll find an option to make a chart There are three chart types which are used so often that they warrant a section of their own on the Lertap tab just to make it easier to get to them ai Histograms 7 Scatterplot h Res charts Graphics trio Go ahead and click on the options above or simply page ahead to read about em Histogram a Lertap s histograms are made from the columns of one of three reports worksheets Scores Breaks and RSAsig An exemplary snapshot of each of these worksheets or reports is showing here 2003 2011 Curtin University West Australia Toolbar and tab tet Pp o o o o ojo o 4 o o 40 0
104. up use Excel s File menu to find and open the relevant workbook The File menu is usually seen on the menu bar at the very top of the screen How to use Lertap Try the resources available via the Lelp option on the toolbar or visit us at www lertap curtin edu au Drop us a note if you have questions Our email address is given below Messages are most welcome Enjoy support lertap com 2001 2002 Curtin University of Technology M HNA Comments Data CCCs Syntax Problems System Ready The Comments worksheet is displayed above To see the System worksheet just click on its tab 2003 2011 Curtin University West Australia Toolbar and tab me The System worksheet looked like this as of February 2005 X new BS rZ puny shots Ml 1 These are Lertap5 system settings Change them only if you understand them lli Move License Lelp 3 Allowed settings es no 10 to 99 4 Usual setting Data N Name of sheet where data records are found Within the data sheet the number of the first data row is Name of worksheet with Lertap5 control cards GE Should Fregs sheet be standard output highly recommended 4 User level 1 is for everyday use 2 is advanced 1 Rescale histogram when longest bar has how many cases Should brief item stats sheet be output Should upper lower stats shee
105. wee look at the two Statslul snapshots below 2003 2011 Curtin University West Australia 22 Help file for Lertap 5 IX new gt S rZ Run shots M ls b Move License Lelp ae ST 0 00 4b pI Data ae ni Fregs f Scores f Stats1f Statsib Statsiul 4 Can you spot the difference in these two Statslul reports Pat yourself on the back if you ve spotted the little triangles in the top one 2003 2011 Curtin University West Australia Toolbar and tab F3 Microsoft Excel Book2 E x File Edit View Insert Format Tools Data Window Help 8 amp X x New Run Shorts Move License Lelp LertapS U L stats for Results from class of 8 March Item 1 upper 2nd 3rd 4th TPickable For quintiles We got our mouse to hover above one of the cells with the triangle and above you see the result If at this point we click on the toolbar option to get quintile plots b we ll get a quintile plot for those items lucky enough to have a triangle Should an item not have a triangle it won t get quintiled there will be no plot Coming back to row 16 if it s set to yes all items will have triangles If it s set to no no item will have the triangle You scratching your head Does this make sense The triangles are either there or they re not there depending on the row 16 setting If they re there I ll get a quintile plot
106. where MDO is having its impact are all computed using only the responses from the 58 folks who actually answered this item In some other data analysis systems such as SPSS the correlation cor between Q28 and the criterion would be said to done on a pairwise basis only when a person has data for both variables are that person s results used in the calulations Another note about the cor values found in the Statsb reports they are corrected for part whole inflation Using the did not see option Suppose the did not see option has been turned on with X used as the did not see code The respective lines in the System worksheet would look like this 2003 2011 Curtin University West Australia CCs details e These are LertapS system settings Change them only if you understand them Bystem Settings Allowed Usual settings setting Use a did not see code Did not see code single character may be blank M N Comments Data CCs System Syntax OldCCs Problems REESE pesas Next have a look at a snippet of Freqs output c37 Q35 Option n 60 xX 6 10 096 i 2 3396 2 13 21 796 3 12 20 096 4 i7 28 396 5 7 11 796 Six people did not see Q35 three did not answer it Okay Now suppose MDO is not operating The Stats1f output for Q35 will look like this LertapS full item stats for Comfort with using LERTAP2 created 30 03 2006 Q35 option wt n 96
107. worksheet The 2003 2011 Curtin University West Australia owe Help file for Lertap 5 default action the first time you click on the icon quintile a plots are made for each item and placed in StatslulChta The next click creates quintile b plots recorded in the StatslulChtb worksheet You can reverse the order via that setting in the System worksheet To use different colours for the trace lines see the Chart colors 203 topic The Diff and Disc figures seen in the plots are taken from the b report which corresponds to the subtest involved If you re clicking off of a Statslul report to get your plots the Diff and Disc values are taken from lines in the Stats1b report Note that the Stats1ul report has its very own diff and disc values these are conventional upper lower indices in a way they re inferior to the values found in Stats1b in that they re just based on results from two groups the lowers and the uppers whereas the corresponding values in Stats1b are based on all test takers The Disc figure in the Stats1b report is a corrected point biserial correlation coefficient Note items with negative discrimination disc are flagged in the plots by the use of red coloring The item s label will be red and the statistics below the chart will also be red An example may be seen by clicking here fisa SAQ If change the number of groups maybe even to just two can I still get quintile plots even thoug
108. worksheet in one quick go from the Excel 2007 or Excel 2010 Home tab find and click on the Find amp Select option From the list which drops down choose Go to In the wee dialog box which then displays click on the Special button at the lower left Finally from the larger dialog box which will pop up or down select Objects and click OK You can quick start this process by using the F5 key at the top of your keyboard What about printing quintiles Read a how to on printing and gain more insights into using ChartChanger2 here at this topic 4 5 3 2 1 1 ChartChanger3 settings ChartChanger3 is by far the easiest way to pack your plots if you have Excel 2010 To use it you must first have a set of quintile like plots on hand we say quintile like as quintile implies five groups if you have fewer groups the plots are not really quintiles but well just play along if you would A reminder on how to get quintile plots Start by viewing a Statslulfz3 sheet Then click B the Res charts option This will result in a new worksheet or Lertap report with one plot for each item arranged in a top down manner item one at the top followed by item two below it and so on Such reports have names like StatslulChta While viewing these quintiles ChartChanger3 is then activated by clicking on the Chart changer option 2003 2011 Curtin University West Australia 190 Help file for Lertap 5 PFs g
109. worksheet correlations 15 16 17 18 19 Lelp B B D C Lertap5MacroSetA macros which may be used WrongltemList2 j StatsbSortAZ NumericFilter1 NumericFilter2 ChartChanger1 ChartChanger2 Itemani o SerunenBoss 0000000000000 AngelMaci OrPrarrrrrrewedovPr DM moon wnmnwowmorwoanrrwam OU OU rrrrrroeocv71 on rpms moro0coc co 900 gt rProarrrrrnonpr Gialong Sonnan OAOIOIOIOIDIAOIOISOISIA rPrarrrrrrYrpe BDOUPBOOCOOOUCOUm OOogcoo0oo000U0 0 o6 6 0jo6jo6j joj o0 n0 Next go to the Move menu click on Macs then click on ScrunchBoss That should do it The ScrunchBoss will join all the correct answers together and make what s called a string This string will be found in the column immediately to the right of the last correct answer Once the string has been made it may be copied and pasted into the appropriate row in the CCs worksheet What about that row seen above called Last Distractor Letter it later appears in scrunched form in the CCs sheet seen above as the alt line The alt line indicates the last option used by an item In this case the res A B C D E part of the sub line says that at least one item makes use of five possible options from A to E But here in this example not all items use all of these options The second items last option is D the 11th item s last option is also D To read more about alt why not see if your mouse or finger will click here 2
110. you return have a look at the following topics to read more about Lertap and missing data Did not see option In March 2006 new options were added to the System worksheet 23 which allow users to adjust cognitive item statistics so that they are based only on those cases students that had a chance to answer the item These are Lertap5 system settings Change them 1 only if you understand them System Settings settings setting any char The did not see option is activated by placing yes in column 2 of the appropriate row in the System worksheet in the example above this is row 40 40 Use a did not see code Did not see code single character may be blank Create an adjusted percentage score Use of this option requires a corresponding did not see code a single character recorded in the Data worksheet whenever a student did not have the chance to answer an item for whatever reason In the example above the code is a blank 2003 2011 Curtin University West Australia CCs details The code may be any single character including a blank an upper case letter from the Latin alphabet ABC XYZ a lower case letter abc xyz or a single Arabic numeral 0123456789 How does Lertap process item responses with this new option Well for any item Lertap first looks to see if a student s response corresponds to one of the response codes used by the item s option
111. zc o i uL m in 0 aN 006 6 6 c O0 9 SG uus Complete Now you can do all sorts of things with both the table and the chart shown above If you change the entries in the table the chart will change too immediately If you wanted you could delete the chart make some changes to the table and then take the Histogram E option to get the chart back For example you could maybe insert a new row in the table to indicate a cutoff point and then make the histogram chart again 2003 2011 Curtin University West Australia Toolbar and tab 169 E E NS j maf 109 12 EX uuo j EX Eu j ENS EU 5 22 12530 ENS EU J 24 19 28 ps quo ps wn EH S2 0 l 2003 2011 Curtin University West Australia 170 Help file for Lertap 5 Histogram with cutoff point in black Complete Note that while you were still trying to remember all those birth dates got into the chart and fiddled around with it changed the title and the font size of the values along the axes How d get just one bar to be black By clicking once or twice on the bar until it was the only selected one When it had obviously been selected then right clicked on it and made use of the options which Excel opened for me Complete you ask Why is the title on the x axis Complete Because that s what this test score was called all
112. 0 Total 138 11062 F ratio 1 02 363 sig eta sqrd 0 01 ANOVA tables provide information which may be used to index the extent of group differences In this regard perhaps the most critical statistic shown in the table is eta sqrd short for eta squared This statistic has a range of O zero to 1 one If the groups differ greatly with regard to the dependent variable SelfReg in this case eta sqrd will be close to its maximum possible value of 1 00 If there s little difference among the groups eta sqrd will be low as seen in this example where a 2003 2011 Curtin University West Australia 13 Help file for Lertap 5 4 4 6 1 value of 0 01 has been found Eta squared is referred to as an index of practical significance it s also commonly referred to as an effect size estimator the larger eta squared the greater the differences among the groups As Pedhazur and Schmelkin 1991 2 point out effect size estimators are often interpreted as being measures of meaningfulness the greater the effect size the more meaningful the differences among the groups The F ratio seen in the table is used to test a statistical hypothesis the so called null hypothesis the average value of the dependent variable SelfReg in the populations of people from which our groups have been sampled is the same the population groups means are equal so goes the null hypothesis The F ratio above 1 02 results from dividi
113. 101111000110010000 0101110101011111001000100 1101100100011111001000000 1110100100101110111000110 7B011111118042111011110180111 1111111010000001011010101 1101D011111101111111100111 11011111111113101111010101 1111111111111111111110111 2 0010100111110110011000100 co A 2 2 5 I 9 5 4 5 3 5 3 Ready Now the Fortran format statement is no longer correct What Lertap has called an I3 field should be F3 in Fortran I is used to denote a field containing an integer F is used to denote a real number with a floating decimal But this ain t a real problem When you save the DAT worksheet as a text file you ll end up deleting the Fortran format statement and Lertap s minor faux pas will go unnoticed How to save the DAT sheet as a text file Just click herel 98 and read on Recode Data column Suppose you had a column in the Data worksheet called Gender with entries of F for female and M for male Then suppose that for some reason you d like to instead have a code of 1 for female and a code of 2 for male The recode option would be for you Suppose you had a Data worksheet with a column for Country with entries such as NZ AU CA US DO and VZ Then suppose for some reason you d like to create a new column to be called Language with NZ AU CA and US to all be coded EN with DO and VZ to be coded SP Apologies to CA residents who speak SP The recode option would do the job for you Or suppos
114. 105 the fourth and so on Columns 5 6 and 7 indicate the number of items a student answered correctly the number answered incorrectly and the number of questions having what Lertap refers to as an other response this is usually the same as the number of unanswered questions The red triangles which may appear black on your screen or printout seen in the snapshot indicate that the worksheet cells have comments If you were to hover your mouse over one of the cells you d see that the comment is Pickable for similarity analysis RSAdata cells which have a red triangle will be included whenever the RSA similarity analysis option is taken from the Run menu This option has nothing to do with the SCheck program Besides the RSAdata worksheet Lertap will produce a companion ASCII file called SCheckData DAT If you ve saved your workbook prior to selecting the Output items scores matrix option from the Run menu Lertap will produce a message such as the following Info from Lertap x i The RSAdata worksheet is ready and a file called ScheckData DAT has been written to the same folder as your workbook D Documents and Settings 172464I My DocumentslLertaplVersionslWorking The purpose of this message is to remind you that you ve now got a brand new DAT file to work with and to tell you where to find it If you haven t saved your workbook prior to selecting the Output items scores matrix option from the Run menu t
115. 198 pickable for RSA 118 2003 2011 Curtin University West Australia pickable for similarity analysis 103 PLAB 321 plot 292 PlotBreaks1 report 131 plots 84 106 147 324 point biserial 53 115 271 points 23 27 29 polarity 27 36 290 polychotomous 280 polytomous 271 280 polytomous scoring 282 popup topics 12 PowerPoint 1 precision 282 306 pre coded items 67 pre scored items 67 pretest items 73 primary worksheets 80 89 253 258 principal component 106 322 principal factors 111 print quintile plots 183 186 printing 189 312 Prob 151 problem 160 173 problem advanced toolbar 244 problem importing data 82 312 problems 5 183 problems font 87 problems general 6 problems with charts 205 production mode 90 103 248 258 321 product moment 290 product moment correlation 277 Prop consistent placings 306 Q Q1 136 Q3 136 quantile 198 quintile chart changes 186 quintile options 198 321 336 Help file for Lertap 5 quintile plots 179 183 193 299 321 quintile printing 186 quintile problems 205 quintiles 324 E rec 277 R1C1 219 250 R1C1 referencing style 255 random samples 159 325 recalculate correlations 222 recode 73 131 141 193 212 216 318 recoder 141 record ID 323 red triangles 103 183 198 REF 222 Ref style 250 reference style 250 255 references 326 referencing style 219 refresh correlations 222 regression line 175 reli
116. 2 65 For each item or question Freqs reports on the number of times letters or digits were found in the respective item column number in the Data worksheet For Q1 there were 26 As 25 Bs and 9 Cs The c3 indicates that Q1 was found in column 3 of the Data worksheet in this example The Freqs column headed 60 indicates that a total of 60 data records were found and gives the percentage associated with each frequency For example there were 26 As for Q1 which is 43 396 of the total number of 60 As you scroll through your own Freqs sheets you may come across some rows which have a mark on the left For example 2003 2011 Curtin University West Australia Input amp Output r x id 9 lt 7 LrtpQuizsample25June12 xlsx Microsoft co E Z8 mm Lerti Hon Inse Pag Forr Dati Revi View Dev Add Y Q ca pp ES Q7 c9 Q8 c10 M gt M Data CCs Fregs J 4 m p rf Ready 0 100 2 y What Freqs is saying is that there was one response on Q7 which was not a letter or a digit You can find out what it was by going back to the Data sheet and browsing down the column with Q7 responses column 9 in this case Why doesn t Freqs show a D for Q7 Because nobody chose that option While being mindful of Freqs feelings we can point out that it s a simple no nonsense summary of response frequencies It has no pretenses it does n
117. 2 Item3 If it S desired to include ID information for the respondents such information may be recorded in any column of the Data worksheet but Lertap versions dated before July 2004 have to have the ID in either the first or second column Lertap will use the IDs to label the scores found in its Scores report providing the respective column header begins with the letters ID or id Click here for more about this Excel has two reference styles used to refer to the rows and columns of its worksheets Lertap uses what s called the R1C1 style In R1C1 notation the columns of an Excel worksheet are numbers In the other style called the A1 style columns are labelled alphabetically Excel s default referencing style is Al When Lertap starts up it automatically changes this to R1C1 Later when Lertap is closed it will set the style back to A1 if that s what was in use before Lertap was started The referencing style may be manually set by Excel s Tools Options General tab How does Lertap find the end of the Data records It thinks it s come to the end when it finds a row whose first column is empty or whose first column contains a blank or whose first column contains a zero Because of this it is generally a good idea to see that the first column of the Data worksheet is used for something other than an item response We say this as non responses to items are often recorded as a blank try to keep blanks out of the f
118. 2003 Ver 5 2 5 August 2003 what you weighted for version change to 5 25 Two accreditation professionals one in Puerto Rico one in Florida asked for an ability to credit an item even when it was not answered It was possible to do this before but now it s easier the advanced toolbarl 24 works better and a new form of the mws card has been introduced one which allows an other 2 weight to be applied 7 1 20 Jul 2003 July 2003 changed item difficulty calculations When a journal reviewer suggested alterations to Lertap s procedure for indexing the difficulty of a cognitive item we allowed our arms to be twisted and followed his advice We made it possible for a cognitive item s difficulty to be computed in one of three ways Read all about it with a click here zs 7 1 24 Jun 2003 June 2003 added an item zapper Sometimes there s a need to quickly remove an item from a subtest without having to re do a bunch of CCs cards Previous versions of Lertap allowed this to happen and we updated Version 5 2 so that it would too A wee click here 68 will explain what we did 7 1 22 May 2003 Ver 5 2 May 2003 added Lelp version change to 5 2 Lelp is of course Lertap Help It s what your peepers are feasting on at this very moment When we took the leap and installed Lelp we changed the version number to 5 2 from just 5 2003 2011 Curtin University West Australia 32 Help file for Lertap 5 7 2
119. 29 237 LertapbMacroSetA xlam 4 223 Lertap5MacroSetA xlsm 186 LertapRibbon xlam 4 Liberty Bell 244 247 license 4 238 241 license a second computer 238 license status 239 license transfer 244 Likert 36 67 180 limitation 183 limited number of charts 198 line chart 155 line graph 106 line graphs 84 loading 4 Log 118 Log PROB 118 170 Lord 1984 309 lose data 89 LRTP5HHelp chm 90 LrtpXcalData txt 101 LrtpXcallCF txt 101 M_INV 93 Macintosh 1 2 3 12 13 78 118 160 258 Macintosh problem 244 macro 150 234 237 macro example 226 macro links 228 macronames 225 macro security 6 3 Help file for Lertap 5 macros 2 4 6 186 223 319 Macs 231 Macs menu Mantel Haenszel manual 13 265 masters 306 Mastery 29 74 mastery level 306 mastery mode 306 mastery testing 306 mastery 306 matrix 93 matrix package 106 Matrix xla 106 223 226 151 155 MaxPos 208 McGill 118 MDO 18 26 36 74 93 319 MDO affective 61 323 MDO cognitive 53 58 60 321 mean 61 measurement error 282 median 136 memory limitations 198 methods 325 MH 151 M H 155 MH alpha 151 MH chi sq 151 MH D DIF 151 Michigan 147 MinPos 208 MINVERSE 93 mis keyed 271 294 missing answers 26 missing data 46 48 82 93 245 271 290 321 323 mode 160 more than one correct answer 231 move charts 147 Move Datato DAT 209 move menu 92 207 move Scores 208 Move menu 231 MSLQ 136 147 multiple copi
120. 35 there were n 9 people missing data So Are you now full bottle on Lertap s definition of missing data And an expert on Lertap s other bin Goodonyamate But wait there s more What this stuff under the wt column The answer to this question is also very 2003 2011 Curtin University West Australia s Help file for Lertap 5 3 3 1 relevant to understanding how Lertap processes missing data especially for affective tests The wt for weight indicates how many points are associated with each item option On Q11 the 28 people who chose response code A usually referred to as option A or alternative A will get 1 00 points for their answer Choosing any of Q11 s other options gets no points You guessed it Q11 is a cognitive item whose right answer is A Not answering Q11 gets no points Not answering Q11 gets no points We have repeated this as it s important an item s other bin can have scoring points attached to it Look at Q35 an affective item Each and every response code has associated scoring points and should someone not answer Q35 they will still get 3 00 points What we ve displayed in these two examples represents Lertap s default handling for cognitive and affective missing data A cognitive item s default missing data scoring action is 0 00 points an affective item s default missing data score is equal to the mean of the other scoring wts With this in mind take a break When
121. 4 3 6 E 1 Conditional standard errors of measurement for M Nur Licensing E390v6 3 h 60 61 60 62 63 Conditional SEMs 64 4 50 65 4 00 66 3 50 67 de 68 69 2 50 70 2 00 7i 1 50 72 73 74 50 y 00 3 25 2 4 6 8 10121416182022242628 3032 34 36 384042 44 46 48 5052 54 56 58 60 F True number right score 77 78 CSEM 1 cSEM2 SEM E 79 Y Id 4 gt M Scores Statsif lt Statsib csem1 lt Statsiul 4 il b Ready 2 G 100 gt g There are always three lines in the graphs The flat one the horizontal one corresponds to the usual SEM value as seen in Lertap s Statsf reports The SEM value is a constant it s the same for all test takers no matter their ability The top line a curve corresponds to standard errors computed using the binomial error model labeled as Method II in Lord 1984 The lower curve is what results when Method III estimates are adjusted using Method IV Method IV estimates will be lower than Method III s whenever the items used in the subtest have different difficulties diff values to use the parlance of Lertap s Statsbls0 reports So what does the graph tell us In this case results are from a 60 item mastery test with a cut score at 42 equal to 7096 of the maximum possible score of 60 2003 2011 Curtin University West Australia 312 Help file for Lertap 5 6 4 5 6 5 At the cut score SEM was 3 23 as it was for all
122. 5 0301 0302 0306 The NumericFilter2 macro can be used to create a new workbook for each of the six districts or for just some of them It s a way of breaking out data records into new workbooks based on the codes found in any column of a Data worksheet In this example the NumerFilter2 macro could also be used to breakout Gender data creating two new workbooks one with data only for boys another with data only for girls This would be done using the codes found in column 4 of this example where L is the code for boys and P the code for girls There are other ways to breakout or select only certain data records See for example the Recodel2 topic and also the discussion of using a tst card in the FilteringRecords 731 topic And then not to forget that there are ways to breakout results without Recodes tst and NumericFilter2 there are options on the Run Menul88 to breakout test scores by groups and also to breakout item responses by groups where groups can be gender school district brand of beer preference and so on Interested See the Breakouts 5 and Ibreaks 4 topics Note using the NumericFilter2 macro is somewhat analogous to using Select if criteria in other software packages such as SPSS However there is a difference NumericFilter2 like a classic select if filter will select only those records meeting the selection criterion but it creates a physically distinct copy of those records in an
123. 5 option wt n b r b r avg z A 0 00 1 0 02 0 03 0 08 14 00 0 20 B 1 00 30 0 50 0 53 0 66 16 67 0 58 C 0 00 13 0 22 0 36 0 50 7 92 0 68 D 0 00 16 0 27 0 33 0 45 8 81 0 55 Q16 option wt n b r b r avg z 4 0 00 T2 D 20 D 14 0 19 10 75 0 27 B 1 00 36 0 60 0 46 0 58 15 53 0 42 E 0 00 3 0 05 0 26 D0 56 4 67 co ey Ee D 0 00 6 0 10 0 29 0 50 6 50 0 88 other 0 00 3 0 05 0 23 0 49 5 67 1 00 The report above comes from Stats1f It summarizes the performance of two items Q15 and Q16 using a variety of statistics Both items use four options employing response codes A B C D A student selecting option B will get 1 00 points towards her his test score as seen under the wt column we d conclude that both items have just one correct answer as all the other wt values are zilch zero Thirty 30 students got Q15 right Sixty 60 students were involved so p for Q15 s option B is 0 50 which is of course 30 divided by 60 The pb r and b r columns are respectively point biserial and biserial correlation coefficients indexing the relationship between option selection and the criterion score If the item forms part of the criterion as it does in this case Lertap applies a part whole correction taking out the inflation the correlation coefficients would otherwise have see the manual for a more extensive discussion The avg column shows the average criterion score for those students selecting each item
124. 50 upper 2nd 3rd 4th lower The table above gives Q50 results after the MDO option has been turned on Its statistics are based only on those students who answered the item This means that all the did not see people plus the we saw it but did not answer it people have been excluded from the proportions and from the calculations underlying U L diff and 2003 2011 Curtin University West Australia CCs details ECH 3 3 5 disc The did not sees are excluded at this point as the did not see option has not yet been used the did not see code has been processed as missing data 5 U L stats for EE 101 T3 Fall 2005 DNSI on created 27 03 2006 3 4 j other U L diff U L disc In this table above the did not see option has been activated but MDO has not The other column is now indicating the proportion of students in each group from upper down to lower who were presented with Q50 by Test Pilot but did not answer it Those not presented with Q50 the did not sees have been excluded For a refresher on how the U L diff and U L disc values are calculated pay a visit to your local ice cream shoppe then have a look at Chapter 10 of the manual MDO affective The MDO control word is used on the sub card to get Lertap to exclude cases with missing data from its calculations MDO may be used with both types of test cognitive and affective The discussion found in thi
125. 7 Some readers might be wondering what happens when an item has more than one correct answer In Lertap such items are said to be multiply keyed or often double keyed and mws lines are used in the CCs sheet to define them Please refer to Example C12 in this topic 231 Finally other users might ask about the practical effects of using the alt line It s a question which is briefly discussed in the manual but we might do well to answer it 2003 2011 Curtin University West Australia 234 Help file for Lertap 5 again If Lertap knows that an item does not make use of all of the options used by other items it will make some adjustments to the Stats1b 290 and Statsif s amp l reports For example the Stats1b report has a column One of the things this column does is flag options which are not taken by anyone If an option is in fact not used by the item the alt line will effectively inform the Stats1b report that it need not should not flag the option In the Stats1f report options which are not used by an item will not be included as part of the report 4 7 6 TotalTest1 The TotalTest1 macro produces output exemplified here eg id 9 c SaaS a Gs MultipleCogSubs xlsx Micro x am Lertap Home Insert Page L Formu Data Reviev View Develc Add Ir ox x Delete vA Sort P Blank Interpret ali Histograms P oMove Version Meliney J Headers dj Elmillon 2 Scatterplot L
126. 7 29 L0 column 14 294 column Stats1b 290 mark Freqs 259 Ds 255 character limit 69 A1 referencing style 255 aa flag 271 277 about this help file 12 add anewscore 219 Add In 106 173 223 address 14 adjusted percentage score 48 advanced filter 216 advanced toolbar 244 AFF 36 74 affective 271 affective CCs 36 affective example 22 26 aid for data entry 82 algorithm 97 alpha 48 73 106 282 286 alt lines 231 analysis of variance 131 141 analysis steps 258 Analysis ToolPak 173 Angel Learning 48 223 225 Annotate Stats f reports 271 ANOVA 131 141 answer sheet 245 Appendix A 67 apple pie 6 apply foralicense 241 apply formula 219 ASC 3 112 316 324 ASC office 241 ASCII 98 112 Assessment Systems 316 Assessment Systems Corporation 241 2003 2011 Curtin University West Australia asterisks as answers 245 author 326 automatic run 90 average correlation 93 avg 583 271 avg column 294 B B mod 151 Bdisc 306 b r 53 271 ba flag 271 277 baby pram 265 bands 286 Basic options 80 bell icon 244 bewdy 247 Bilog 98 209 Bilog MG 98 112 209 325 binomial error model 309 biserial 53 115 271 blank answers 245 book 147 box and whiskers 136 319 boxplot 136 319 breakout 73 118 Breakout groups 319 breakout item responses 141 breakout scores 131 Breaks 136 Breaks worksheet 160 Breaks1 report 131 Breaksibw report 136 bub
127. 8 D 51 3 3 00 10 15 7 D 16 32 8 0 35 4 2 00 8 33 0 08 33 5 D 21 5 Oo D 0 0 0 00 0 0 0 00 other 3 00 2 3 3 D 22 29 0 1 18 Q27 option wt n 96 b r avg Z 1 5 00 3 5 0 0 36 41 7 1 57 2 4 00 14 23 3 0 33 36 8 0 60 3 3 00 22 36 7 0 05 34 1 0 07 4 2 00 21 35 0 D 51 30 2 0 70 5 1 00 D 0 0 0 00 0 0 0 00 Q28 option wt n 96 b r avg z 5 00 T3 22 4 D 20 32 2 O 2 4 00 27 46 6 0 43 36 3 D 46 3 3 00 10 n lr er 22 31 6 D 48 4 2 00 8 13 8 i 32 4 D 33 5 1 00 D 0 0 0 00 0 0 0 00 other 0 00 2 3 3 D 31 26 0 1 62 2003 2011 Curtin University West Australia CCs details e The first table above shows item stats for Q27 and Q28 without MDO while the second table reflects the results of using MDO Q27 s stats are the same in both tables are they not No one omitted this item so the statistics are unchanged aren t they No In fact they re not unchanged fooled you eh Everything s the same until we get to the pb r avg and z columns wherein some changes enter To understand why Q27 s results differ look at the other row for Q28 In the first subtest without MDO Lertap has wt 3 00 giving 3 00 points to the two 2 people who did not answer Q28 Not so in the second subtest where those two people have been stripped of scoring points There are two different scoring methods in operation here without MDO people missing an answer to an item are given points equal to the average value of the wt fig
128. A related topic having to do with assessing reliability has to do with parallel forms a situation where two tests designed to measure the same thing are developed simultaneously At times these tests will have some common test items at times they will not They re called parallel tests as they can be used ideally interchangeably An interesting example of a parallel forms development project may be seen here Of added interest in this project was the use of a mixture of test items multiple choice and constructed response Filtering records The tst control card is a special one used to have Lertap filter some of the records from whatever data set you happen to be working with Two examples of the tst card have been briefly mentioned in the topics above much more mention of tst is found in the manual In December 2004 the tst card gained some new smarts Now the filtering criteria may be more elaborate look at these examples 2003 201 1 Curtin University West Australia Help file for Lertap 5 3 11 A tst c6 CS001 B tst c6 UCS001 CS001 C tst c6 DL5 DL6 DXL7 c7 WI In example A all data records with CS001 in column 6 of the Data worksheet will be filtered out into a new data set In Example B all records with either UCSO01 or CS001 will be filtered out into a new data set In Example C all records with either DL5 DL6 or DXL7 in column 6 and WI in column 7 will be filtered out into
129. ABBDBCCBBBCCDBADADBCDCCCDBDBABCDACABABDB et DACACBADDBCBBCCBCADCDDABBBBBACCCBADDCDADCCAADBADACACACCCE 24 ACDBDCBBDCBADAACCDACBBABDACBBBDCACBAAADBBDACCBCDBBACDBCACDEDS gl gt Cancel Finish Look at row 22 above It has four blanks at the beginning probably signifying unanswered questions When the wizard is asked to import the string of responses chances are real good it ll simply lop off those four blanks shifting the string to the left This is real bad this record s test score is going to be wrong How to control for this problem Well if the file has been created by a scanner see if the scanner can t be coaxed into saving its data in an Excel ready format such as perhaps a csv file comma separated values Such files come into Excel without having a need to be converted Another useful format is the trusty old dBASE one mentioned above If you want to talk to us about this type of problem just zip off an email to larry lertap com We ll get back to you as soon as we re in from camping Update June 2011 the topics discussed here are also presented in the new samples website While talking about strings of item responses don t forget about The Spreaderl 8 It s tailor made to take strings of responses apart and it loves to be put to work As to exporting Lertap worksheets making them ready for use in another package some packages notably SPSS for Windows readily work with the xls and xlsx
130. BCBA BDAAB AAACD CBABA DABB w M M Data CCs td Dj TT rf Ready 22 SAM 100 O0 _ amp Lertap s mastery level is usually set at 70 Test scores at and above this level make it into Lertap s masters group while the rest go into a group called others It is possible to change the mastery level by using for example Mastery 80 on the sub line or by changing the default setting in Lertap s System worksheet 2 x B I C Zils MNurMasteryForLrtp593 xlsx Microsoft Excel ca E amp File Lertap Home Insert Pagel Forme Data Reviev View Devek Add S Q9 c3 pH ES Lertap5 U L stats for Mastery M Nur Licensing E390v6 3 created 9 07 2012 a 0 58 0 04 0 00 0 29 0 09 0 00 NM11 masters others 0 00 0 20 0 02 0 77 0 00 0 02 0 36 0 08 0 55 2n 0 NM12 mactorc nn E I4 4 b M Statsif Statsib Wi tatsiul a Ready 2a i 100 y z The Statslul report in mastery mode looks like the snapshot above With one exception the statistics which appear are the same as the those found when a mastery level is not used previous topic 23 The exception is the last column B disc named after Brennan 1972 28 who referred to it as a generalized upper lower 2003 2011 Curtin University West Australia 3 Help file for Lertap 5 discrimination index It is simply the difference in the proportions for the correct answer Unl
131. BIT D A D B A 58 15 BIB D B B AJC CJ A C D BJ C A 59 40 AJEJC JA JCJ JA EJIJB JAJ J D AJD B B 60 46 A EJCJAJAJBJAIB DIG B B A 61 22 BECDDADEDCBDEABCDEDAABBED2312454321 A string of 35 characters has been entered into a single cell directly on the Data worksheet When you employ The Spreader it will almost always be the case that row after row will have its item responses entered in this manner After the last row of data has been entered you scroll up to the first row click on the cell containing the string and then click on The Spreader s icon e The Spreader will dissect the string spreading the characters one by one over the cells to the right Having done this it then looks at the next row in the worksheet If there s another long string to be dissected it does so The Spreader continues to work down the rows stopping when it finds one without a string The Spreader may be stopped at any time by pressing the Esc key on the computer s keyboard What about missing data Say someone has not answered one of the items in this case leave a blank in the string pressing the space bar on the computer keyboard will produce a blank An important case arises when the string of responses begins with a digit Excel will think the entry is going to be a number and strange things can happen When the string begins with a digit it should be preceded by an apostrophe as seen here 2344133124AADDB The apostrophe tell
132. Before this option is selected it must be the case that 1 data records have been created in the Data worksheet and 2 control lines or cards have been placed in the CCs worksheet The first time this option is clicked on Lertap gets Excel to read the responses found in the Data worksheet looking in the columns specified in the col lines of the CCs worksheet If there are no errors in the CCs worksheet Lertap and Excel will produce new worksheets One of them is called Fregs 25 for frequencies Usually the Elmillon item analysis option will be taken next This results in even more 2003 2011 Curtin University West Australia Help file for Lertap 5 4 4 1 1 new worksheets with names such as Scores 23 Stats1flzssl Stats1bl 25 and so on Note added May 2005 it is now possible to get Lertap to automatically roll through from Interpret CCs lines to Elmillon item analysis without stopping as detailed in the following topic Production model 90 If the Interpret CCs lines is later selected again a warning message will appear saying that there s a possibility of losing data This is purely precautionary Lertap is about to delete Freqs Scores Stats1f and so forth but this is usually only natural new versions of these same worksheets will be generated which is almost always what users want Lertap is being overly cautious in sounding this warning the primary worksheets Data and CCs are neve
133. Edit and use the File menu to Open your work of art The screen snippet below shows an example in this case a Lertap DAT worksheet saved as Ed502 dat and viewed on a Windows computer with the Notepad program d502 dat Notepad Sele Format View Help 14A1 1xX 45A1 BAKER G 111011011010110110001111111011111110110111001 BOONKHONG 010010000010110011001111111010111111100111101 CHAMBERLAIN 000000000000001000001111100001100111111001101 CLEMENT 100001000011101100101111001111110110100101111 DEAN 10101101001110111110101110111431111113110011111 EASTABRQOK ELSCOT FALCONER GLUCINA 111110011010100111110101000111111111110110111 111011010001110111010011101111111111100111111 111011101001100110000011100111011011110111101 001101010011101010000111011001111111100111111 3 R R DEMOUCHE C 110111000010110111011111101101111100100101101 R L S D 4 4 4 9 Creating a csv file A csv file is a text file with a certain number of fields with each field representing a value of some sort or another CSV means comma separated values The records or lines in a csv file have a series of values or fields with commas used to separate them Here s an example two commas with nothing between them corresponds to an empty field 2003 2011 Curtin University West Australia ne Help file for Lertap 5 ed502 csv Notepad File Edit Format View Help Item Item Item Item Item Item Item Item Item Item
134. Elmillon Scatterplot Licer 3 More b Res charts Lelp l Use external criterion Item scores and correlations TL RSA response similarity analysis Breakout score by groups Box and whiskers Item responses by groups Enhance M H charts To halve and hold B gt kkh Hu Ex U Lelp With an IbreaksMH chart still selected click on the Enhance option The original chart is improved by adding a title 135 correct legend labels Nat r and Ing f and dinkum score values along the x axis See here I35 1 00 90 80 70 60 50 Nat r ed ng f 30 20 10 DO Ry coe uy grea mn Goa Ural eu Som ANANN sl el Gs a eae Gees net ea 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 2003 2011 Curtin University West Australia Toolbar and tab The chart for 135 reflects an item that meets the ETS criteria for large There s quite a range of scores where the Nat group consistently out performed the Ing group This is a picture of DIF This chart for item I1 reflects no DIF ETS level A neg If you go though an IbreaksMH report making plots such as these remember that you can move them to another worksheet quite easily See the Ibreaks charts topic Related tidbits For other pictures of DIF see for example Figure 4 4 in Camilli amp Shepard 1994 and FIGs 3 1 and 3 3 in Dorans amp Holl
135. Elmillon starts up what happens Quite a bit Lertap returns to the Data worksheet and reads all its records extracting the responses corresponding to the first col card in CCs forming item statistics and making a subtest score for each respondent It writes the subtest scores on the fly that is as it reads the Data records you can sometimes see it doing this Writes them to where To the Scoresl ze worksheet another new sheet which Lertap adds to the workbook Then Lertap usually creates its two main reports with item statistics Each of these reports is a new worksheet They re called Statsif a5 and Stats1b 20 respectfully containing full and brief item statistics If the subtest is a cognitive one Lertap usually creates another new worksheet Stats1ull 29 with upper lower discrimination and difficulty estimates In the process of making the Stats1ul report Lertap creates a temporary worksheet called Scratch It deletes this worksheet on completing the Stats1ul report What happens next If there s more than one col card in the CCs worksheet Lertap repeats this process Each col card defines a Lertap subtest For each and every subtest Lertap adds a score to the Scores worksheet and creates the appropriate series of Stats reports As to nomenclature the Stats reports for the first subtest are Stats1f and Stats1b for the second subtest they re Stats2f and Stats2b and so on It is possible to contr
136. English a strong widely used second language eventually resulted in high school Science and mathematics instruction switching to English Selected assessment instruments were carefully translated to English cycles of forward and back translations were used to control the process and after trials stretching over three years many of these instruments came into main stream application In this example the Nat group served as the reference group The native language version of the test was seen as the gold standard some educators thought that the switch to English was disadvantageous the focus was on students sitting the English version of the test and the question was was it a fair test Was there any evidence to suggest that the English version of the test worked against students putting them at a disadvantage when compared to those who might still get to sit the native language version of the test So it was that the answer to this Ibreaks question was No This served to define the Nat group as the reference group The Ing group became the focal group Once Ibreaks has an answer it starts to churn out results 2003 2011 Curtin University West Australia 182 Help file for Lertap 5 2 3 E 5 6 vA 8 9 10 11 12 Ing diff odds ratio gt 1 2 3 4 6 Nat diff T 8 9 2 29 30 31 32 33 34 35 36 37 38 39 1 LertapS Mantel 2 Score levels gt 30 31 32 33 34 35 36 37 38 39 in total 3 Nat r 26
137. Excel DelsL1 isu 22 08 2008 11 04 AM 3KB ISUFile 2003 2011 Curtin University West Australia 10 Help file for Lertap 5 The screen shot above shows Lertap5 xlsm nested in a folder called Lertap57 which in turn is nested within a folder called ASC for Assessment Systems Corporation a folder found on the computer s C drive Okay Now go back to Excel and take that option to Open the Trust Center The following series of screen snapshots shows what to do Trusted Publishers Warning All these locations are treated as trusted sources for opening Add ins files If you change or add a location make sure that the new location is secure ActiveX Settings Path Description Date Modified M Selti User Locations In viuo dx C MyStuff Lertap 13 05 2008 2 33 PM Message Bar C soft Templates Excel 2007 default CA e124XLSTARTY Excel 2007 default comm External Content C Excel XLSTART Excel 2007 default CA ceVTemplatesy Excel 2007 default Privacy Options s Path C MyStuff Lertap Description ified 13 05 2008 2 33 PM Allowed Add new location Remove Modify Allow Trusted Locations on my network not recommended Disable all Trusted Locations Only files signed by Trusted Publishers will be trusted OK Cancel 2003 2011 Curtin University West Australia Welcome 1 1
138. H H sigma 15 0 47 22 41 5 09 14 0 64 22 96 5 32 14 0 57 23 05 5 36 n 65 515 65 515 mean 10 45 0 00 s d 2 35 1 00 0 46 0 46 65523 s a 2 36 65524 variance 65525 __range 65526 skewness i a 3 to 4 sigma 0 26s 0 40 F ad f te Taal M 4 gt Mf RSAcases3 amp RSAsig2 RSAtable2 RS5Acases2 RS gt 3 The little table on the left has Log PROB minimum mean maximum s d variance and range data for the 220779 student pairs involved in this analysis We might now consider the 65515 cases whose statistics are given in the right most table to be a sample from the whole comparing the sample Log PROB mean and s d values 10 45 and 2 35 to those for the population 10 50 and 2 36 suggests that the sample data are representative Please note that this example is from the Excel 2003 2004 versions of Lertap where the number of rows in a worksheet was limited to 65 536 In the case of Excel 2007 the little table on the left will appear only when the number of students is greater than 1 448 corresponding to about 1 048 000 paired results To read more about response similarity analysis be sure to refer to the Related tidbits at the end of this topic Of these if you have time to read only one make it Using Lertap 5 6 to monitor cheating on multiple choice exams Lertap s RSA settings Th
139. If the items are from a survey on the other hand you have to tell Lertap to forget about having to have a correct answer for each question there aren t any You tell Lertap these things by putting your instructions in another worksheet called the CCs sheet using a special control language Once you ve got the responses in the Data sheet and your Lertap instructions in the CCs sheet away you go you just follow the same steps seen in the Cook s tour Could life be simpler Page on have a look at some examples there are more in the manual 2003 201 1 Curtin University West Australia Help file for Lertap 5 Related tidbit For more how it works insights see Lertap s Outputfs 2 3 Examples There are two primary places to visit for those interested in browsing some samples of Lertap in action The first place is right here in the very document you re looking at now Simply page ahead and you ll see The second place repeats some of the samples found here but adds more presenting a greater variety of samples largely based on real life applications of Lertap collected over the years It offers the chance to download actual Excel workbooks set up to work with Lertap and suggests practical exercises to provide more extensive insights into how Lertap may be applied Where is this second place Here 2 3 1 Cognitive example Lookit lookit lookit here s a Data worksheet Mme 2 3 4 5
140. L29 112 108 late 4 SMC bands 00 10 20 30 Q22 40 Q4 Q5 Q7 Q16 Q23 Q24 50 Q9 Q10 Q15 Q17 Q20 60 Q1 Q2 Q3 Q6 Q8 Q12 Q14 Q21 Q25 70 Q11 Q13 Q18 Q19 80 v M 4 gt vif Stats3f f Stats3b IStats wf As seen above the IStats report now has a row with SMC values just above the eigens row followed by a display of SMC bands The bands give a quick idea of the spread of SMC values we see for example that nine subtest items had an SMC value equal to or greater than 0 60 but less than 0 70 Among these nine were Q1 Q2 Q3 Q6 and Q8 you can see the actual SMC values for these five items by looking at the row of SMC values showing above the bands 2003 2011 Curtin University West Australia Help file for Lertap 5 In this case we see that Q2 s SMC was 0 67 We may interpret this as meaning that 67 of Q2 s variance can be explained by the other items in the subtest If we took the square root of Q2 s SMC we d get 0 82 a value we may interpret as indicating the correlation between Q2 and the other items In somewhat more technical lingo 0 82 is the value of the Pearson product moment correlation coefficient between the scores people earned on Q2 and a specially weighted linear composite score formed from the other items The special weights are determined via a multiple linear regression analysis as the references point out Now you ll remember that there is another Lertap rep
141. Lertap5 system settings Don t change them System Settings 1 unless you know what they do The settings below are the standard ones for the Excel 2010 Allowed Usual 2 version of Lertap settings setting 60 Automatically output packed plots for quintile plots uu Plot packing involves moving charts so that they lie more than one to a row In the example below there are four to a row 2003 2011 Curtin University West Australia Toolbar and tab ku a KA 9 C gi Maths QuizForLrtp593 xIsx Microsoft Excel of F Lertap Home Insert Pagelayout Formulas Data Review View Developer Addins v c X The number of charts in a row is controlled by System worksheet Row 70 You can set it to whatever you prefer 2 is a suggested default value It is possible to insert a blank row between the packed plots as seen here 2003 2011 Curtin University West Australia Help file for Lertap 5 r X id 2 EJ Lertap Home Insert n il 2 3 4 oher tower amp th 3rd 2nd upper Key 3 ODif 84 Disc 26 Page Layout 1 00 2 40 0 00 1 2 3 4 oher Book2 Microsoft Excel Formulas Data kwer 4th 3rd 2nd upper Key 4 Dif 97 Dic 12 ca 8 amp E 2 pe 3 oner lower 4th 3rd 2nd upper Key 1 Dif 70 Disc 35 Review View Developer Add Ins 9 o c gg x
142. MH D Dif is less than 1 00 An item with substantial DIF the C level on the ETS scale has an MH D DIF value with a magnitude of at least 1 5 that is at or above 1 5 or at or below 1 5 with MH D DIF significantly greater than 1 0 in magnitude These items are denoted in IbreaksMH tables as C large Note not shown in the examples on this page is another statistic which appears to the right of the ETS level s e the standard error of MH D DIF A concise reference for the calculations used in this part of Lertap is Michaelides 2008 see Dorands amp Kulick 2006 for a practical application of MH statistics and discussion of the ETS scale see Related tidbits below Bet you can t guess what an ETS B level item is It s one that is not in one of the other two levels It exhibits some DIF moderate DIF is the correct ETS term and IbreaksMH tables show this as B mt 2003 2011 Curtin University West Australia Help file for Lertap 5 T Lertap5 Mantel aenszel results based on score levels from Testi grouped by Lengua 9 2 Score levels gt 4 5 6 7 8 10 11 12 13 3 Nat r 1 2 4 3 13 11 24 23 37 40 4 Ing f 1 2 4 3 15 16 23 34 32 41 Nat diff Ing diff odds ratio gt 08 27 23 64 38 2 92 38 22 50 00 00 d E ci MH D DIF 2 12 MH alpha 2 47 54 Ek 2 33 MH alpha 1 68 36 38 95 54 48 1 29 4 MH D DIF 1 21 78 47 05 _ ETS level
143. ST UniquelD into it before saving as a csv file This may speed up the task of importing the item stats Is it difficult to insert a new column in the Stats1b worksheet Yep it s real tough about as hard as having to quaff a few ice cold Emu Exports on a hot summer s day Be sure to use the Shorts menu to turn column headings on first The item discrimination value seen in Lertap s Stats1b report disc is a point biserial correlation coefficient corrected for part whole inflation The manual discusses Lertap s statistics in some detail It is possible to get the Stats1b report to include the biserial equivalent something which is done by turning on Lertap s Experimental Features option Please refer to the following URL for a discussion of these features http www lertap curtin edu au Documentation ExperimentalFeatures htm 2003 201 1 Curtin University West Australia owe Help file for Lertap 5 4 4 4 10 Time trials 4 4 5 The data minuted in the secs y table below were obtained in October 2003 on a Pentium 4 running at 2 GHz N is the number of data records in the Data worksheet Nits is the number of items to be scored Item scores indicates the amount of time to produce IStats matrix of item scores No eigens indicates the total time to completion that is time to create the item scores the matrix of Pearson correlations the matrix of tetrachoric correlations and the DAT worksheet With eigens indicates h
144. Stats2f i 4 lil Ready 2 B G 100 v OF E LTM The snapshot above depicts a typical Statslul report for three items from the M Nursing dataset using the default setting of five groups with 2096 of the students in each group Results for three items are shown As you look at them remember that the figures for an item s correct answer are underlined The correct answer to the item called NM8 was B for example The figures shown in the main part of the display indicate the proportion of students in each group who selected the item s options If we want our items to be ones which are capable of discriminating among the 2003 2011 Curtin University West Australia Input amp Output 99 students picking out the strongest while simultaneously identifying the weakest then the best students should get the items right while the weakest falter This happened on two of the items seen above NM9 and NM10 On NM9 72 of the students in the upper group picked out the right answer dropping steadily to 20 in the lower group A similar pattern was noted on NM10 although the drop was not quite as marked as 4396 of the lower students got the item right NM8 was a different case There s only a small difference between the top and bottom groups wi
145. T T SA A N D SD Lertap places item response charts in new worksheets having names such as StatslbCht and Stats2bCht Item response charts are briefly discussed in Chapter 10 of the manual page 172 in the printed manual An example suggesting how item response charts for different groups of test takers may be obtained is provided in Chapter 8 of the manual pp 130 133 in the printed manual Response charts of this type differ a bit depending on the nature of the responses if they are from an affective instrument with Likert style items the charts will be similar to the ones seen above Likert items typically use strongly agree to strongly disagree response options For cognitive items response charts will resemble the one shown below 2003 2011 Curtin University West Australia Help file for Lertap 5 Q2 6 1 Ad 2 y gg Lm T A B D F The correct answer to Q2 E has been replaced by a sign in the picture above a definite plus wouldn t you say Note that problems can arise when creating these charts the number of charts which Excel can make is limited as is the number of fonts which a workbook may have the charts sometimes use a variety of fonts These problems are more likely to appear when quintile plots are made from a Stats1ul report the following topics provide more details 2003 2011 Curtin University West Australia Toolbar and tab
146. The percentage score is wrong Whenever students are presented with a different number of items or whenever items in the pool have different scoring patterns yes should be used in the Create an adjusted percentage score row So let s see what happens We put in that yes Do we still have PER on the sub card No now it s not needed there would be no harm in having it there but it is not required Here are the scores Record No EE101 10196 EE EE ERI 3 pres ar ieee EE Sa The percentage scores above have each been adjusted according to the maximum possible score a student could have obtained on the set of items s he was presented with Yes Miss A young woman in the fortieth row of the third balcony has her hand up Why wouldn t just use the adjusted percentage score option all of the time Why not put that yes where it s supposed to be and just leave it there for always A good question thank you The answer Lertap pinches extra memory from the computer whenever it has to keep track of the maximum possible score each student could have achieved given the items presented You save memory space and also a tiny bit of processing time by not adjusting the percentages If you know that each student was presented with the same number of items and all items were scored the same way then say no to the Create an adjusted percentage score option Now you know no Related tidbits Several lea
147. There s a control word MDO which may be used on the sub card to control how Lertap processes missing data MDO may be used with both types of test cognitive and affective The letters stand for missing data out meaning that records with missing data are to be excluded from Lertap s various calculations This topic discusses the effect of the MDO and did not see options on the cognitive test reports produced by Lertap a following topic does likewise for affective tests Look at these CCs lines for a cognitive test col c3 c27 sub Res A B C D E F Name Knovledge of LERTAP2 Title Enwldge Wt 0 key AECAB BEBBD ADBAB BCCCB BABDC alt CEDBC CEDFD EDCBD DCCDD DEEDF col c3 c27 sub MDO Res A B C D E F Name Knowledge with MDO Title MDOEnwl UWt 0 key AECAB BEBBD ADBAB BCCCB BABDC alt CEDBC CEDFD EDCBD DCCDD DEEDF co N00 8 n These eight cards involve the same 25 items Being the avid reader you are you know you ve seen these items before they re from the Lertap quiz 23 The cards define two subtests The only difference between the two is that the second one has the MDO option on you can see it on the 6th line Statsf reports A squiz of the Lertap s Stats1f and Stats2f reports will serve to highlight the effect of using MDO 2003 2011 Curtin University West Australia s Help file for Lertap 5 LertapS full item stats for Knowledge of LERTAP2 created 20 03 2006 Q1
148. Warning If you ake sure that the new location is secure Subfolders of this location are also trusted Description Date and Time Created 22 08 2008 1 39 PM There you go Now henceforth and forevermore Lertap will start up without Excel giving any sort of security warning Handy handy 1 6 2 Compatibility mode Excel 2007 and Excel 2010 like to run with workbooks having an extension of xlsx xlsm and or xlam Earlier versions of Excel such as the very popular Excel 2003 generally expected to create and run with workbooks having an extension of xls Excel 2007 and Excel 2010 usually have no problem at all with old xls workbooks When an xls workbook is opened with Excel 2007 or Excel 2010 Excel will say that it s running in compatibility mode The screen snapshot shown below exemplifies what s seen at the top of an Excel window when Excel 2010 is running in compatibility mode JI m d ls Book2 Compatibility Mode Microsoft Exce Lertap Home Insert Page Layout Formulas Data Review View Developer Ad X Delete tfl Sort Mi Blank Y Interoret aiii Histoarams Move At times Lertap will have a problem or two with compatibility mode For example when the Item scores and correlations option 931 is taken Lertap will sometimes fail In such cases a message such as the following may appear Sorry there s been a problem inverting the correlation matrix As far as Ler
149. When asked to do so Lertap will get all the item scores together and write them to a new worksheet named IStats An Stats worksheet has three sections The first section has rows whose cells contain the score each person earned on each item The second section includes rows of descriptive statistics for each item such as the median mean and standard deviation The third section involves the creation of inter item correlations Under normal conditions a single matrix of Pearson product moment correlation coefficients is made with a row of average correlations included at the bottom of the matrix Lertap uses standard Excel functions for all of these calculations Average correlations are computed by using the n 1 non diagonal entries in each column where n is the 2003 2011 Curtin University West Australia Help file for Lertap 5 4 4 4 1 number of items There are two settings in the System worksheet 241 which affect the information found under this third section of the IStats worksheet One of them replaces the diagonal element of the correlation matrix with the SMC the squared multiple correlation The SMC for an item is often used in factor analysis as an initial estimate of the item s common variance Prior to October 2004 Lertap used Excel s MINVERSE matrix function to invert the correlation matrix part of the process of calculating SMC values It was found however that MINVERSE would regularly fail when ask
150. a and to his web page for information about SCheck software http www business mcmaster ca msis profs wesolo wesolo htm Each row in the RSAdata worksheet contains seven columns of information The first column corresponds to the Lertap ID in use Wesolowsky generally refers to this as the student ID number but it doesn t have to be a true number it can bea name The second column is referred to as name in SCheck Lertap inserts DataRowX instead where X corresponds to the row number in the Data worksheet The third can be initials according to the SCheck exe user guide Lertap leaves this column empty The fourth column s contents has a length equal to the number of items in the subtest and indicates how each student responded to the items a full stop or period indicates that the student got the item right a dash or hyphen indicates that the student did not answer the item or had an answer not recognised by Lertap and a letter or a digit indicates which wrong answer which distractor was selected In the snapshot seen above the first student has an ID of 9 his or her complete data may be found in row 3 of the Data worksheet The student left four items unanswered there are four dashes and got only three items correct there are three full stops The student selected distractor C on the first and second items D on the third B on 2003 2011 Curtin University West Australia Toolbar and tab
151. a new data set So what are the new smarts we re talking about Well it used to be the case that the filtering criteria could consist of just a single character as in for example tst c2 W and tst c4 B G Now the criteria may be any length There may be up to 20 criteria for each column In the examples above in B there are said to be two filtering criteria on column 6 while in example C we d say that there are three criteria on column 6 In June 2006 the Move Menu was blessed with a new option Recode a Data column which provides another way to filter records Now the word Delete can be attached to certain data records as can the word Exclude In some cases this may be a very effective way to quickly remove records Click here z3 In September 2010 the NumericFilter2 23 macro was added to Lertap s special macros workbook Lertap5MacroSetA xlam This macro makes it easy to breakout or select only those Data records meeting a specified criterion It s accessed via the Macs menu 223 How CCs cards work Each subtest requires a minimum of two CCs cards Cognitive subtests Cognitive subtests must have a col card and they must also have a key card with the right answers The number of columns mentioned on the col card tell Lertap the number of items in the subtest The default response code set for cognitive items is Res A B C D The CCs card order for cognitive subtests is 2003 2011 Curtin Universi
152. a worksheet 107 TotalTesti Used to compute coefficient alpha for the Total composite 108 AngelMaci Work with cognitive items from Angel Learning 109 AngelCognitive2 Work with complex cognitive Angel items 110 AngelAffective1 Work with affective items from Angel Learning 111 112 M 4 M Comments Data CCs em lt Syntax lt OldCCs 2 c There is a one to one relationship between the macro names seen in the System worksheet and the names seen in Lertap s Macs menu What s seen in the Macs menu are the macro names found in the System worksheet In turn these names correspond to the names of the macros found in the Lertap5MacroSetA xlam file So if a user has placed MacroMio6 in row 110 of the System worksheet when the Macs menu is opened MacroMio6 will be the tenth entry When a user clicks on this entry Lertap looks in the Lertap5MacroSetA xlam file for a macro called MacroMio6 If this macro exists it starts up If it doesn t exist an error message will appear such as the one shown here Microsoft Excel E xl A The macro LertapSMacroSet4 xla MacroMio2 cannot be Found As mentioned users may either modify the macros already found in Lertap5MacroSetA xlam file or create their own saving them in the same file You give each macro a unique name and then place this name in one of the rows in Lertap s System worksheet The next time you start Lertap the Macs menu will have been update
153. ability 73 282 286 294 reliability parallel forms 73 reliability study link 23 remove an item 69 71 322 325 rename worksheets 80 replace 216 requirements 2 res 23 29 46 74 257 321 rescale 170 rescale scores 323 research 159 resize quintile charts 186 resources 326 response charts 180 response code 67 response codes 23 36 46 48 257 271 response labels 179 180 response similarity 103 response similarity analysis 118 160 response string 82 response weights 23 27 29 reverse scored 290 reverse scoring 23 36 reviewer 325 revision 18 revision note 14 36 revisions 12 82 317 ribbon 4 6 RSA 118 160 RSA settings 118 RSA similarity analysis 103 RSA steps 118 RSAcases 118 RSAdata 103 118 RSAreport 103 RSAsig 118 170 RSAsig worksheet 160 RSAtable 118 Run menu 84 88 118 325 running 4 sies s d 61 s e 151 sample data set 20 22 26 samples 20 sampling 159 SAQ 228 SAS 111 Saveas 115 scale 29 36 74 88 scaling 170 219 scanner 82 90 245 290 312 316 ScatP1 175 scattergram 175 scatterplot 175 292 SCheck 103 118 312 SCheckData DAT 103 118 score levels 151 Scores 26 88 92 93 255 263 323 2003 2011 Curtin University West Australia Scores items 36 scores subtest 36 Scores sorting 83 Scores worksheet 160 207 208 219 222 Scoring 48 69 scoring items 46 scoring essay questions 69 scoring items and subtests 23 scor
154. aces them in a new Excel workbook When it finishes this task Lertap says something like this Lertap 5 There you go this is it your new workbook with a copy of the original Data and CCs worksheets ready For you to work with You should save this new workbook soon use Excel s File Save to do so Lertap has not done this For you Don t worry about saving the new workbook for the moment Click the OK button Your next move will be to click on the Interpret option on the Run menu Interpret Elmillon More Run menu This gets Lertap to read the lines in the CCs worksheet checking to make sure they have the right syntax If they do Lertap creates some new worksheets and displays one of them Freqs The Freqs worksheet is a simple one It displays what are called response frequencies for each of the columns in the Data worksheet The little boxes above indicate that 26 people answered A on Q1 2003 2011 Curtin University West Australia Getting started The main purpose of the Freqs worksheet is to see if there may have been any errors in the preparation of the data Users generally scroll down the Freqs worksheet rather quickly looking for unexpected results For example a response of D on Q1 would be strange as Q1 allowed for just three responses A B C To continue the tour return to the Run menu now and click on the Elmillon option This causes Lertap to do quite a number o
155. acroMio2 te tH MacroMio3 MacroMio4 With great anticipation Lee clicks on the ShowDate option and voila look ETE x 10 04 2006 3 41 27 PM So there you go With our friend Lee ding the way can you now add links to your own macros Postscript your macros do not really have to be related to Lertap If you keep the Lertap toolbar on screen the Macs menu will always be available even though you may not actually be Lertapping 4 7 3 Macro SAQs SAQI1 Q How many macro links can have in the System worksheet A Ten 10 Q Do I have to have 10 A No When the Lertap5 xlsm workbook is opened it populates the Macs menu by reading down the appropriate lines in the System worksheet in August 2007 these lines started at row 101 and ended in row 110 Lertap stops reading these lines as soon as it encounters an empty cell in the first column SAQ3 Q I notice you ve left two spaces at the start of each macro name in System worksheet Why A Just because it makes the System worksheet a little neater There s no real need for the two spaces 2003 2011 Curtin University West Australia Toolbar and tab 229 SAQA Q How can I find out more about using the macros mentioned A Try them They usually have a little explanation of what they do when they start up If you have an interest in using Lertap with Angel Learning files go backl 224 a few topics and click on the URL to the paper refere
156. ad below we ve made some changes to Lertap which result in the tour producing more reports The Cook s tour is based on the Lertap Quiz data set This data set is fully described in Appendix A of the manual Briefly the Quiz consists of a test given to 60 people who had the good fortune to participate in one of the very first Lertap workshops held in Dunedin New Zealand just a couple of years ago it was 1973 for us time flies when we re fishing camping or Lertapping At the end of the workshop each participant was asked to answer 25 multiple choice questions and 10 Likert style affective questions The multiple choice questions were meant to indicate how well the participants had mastered the content of the workshop the 25 questions were a test of their knowledge of the functioning of Lertap The 10 Likert questions scored on a 5 point strongly agree to strongly disagree scale asked the participants how they felt about Lertap did they judge it to be the dynamite test and survey analysis system we know it to be Each participant was also asked to respond to two open ended questions One of these asked them to state how long they had been using computers while the other requested information on how long they had been using tests in their research or teaching The Lertap Quiz data set is included in the Lertap5 xlsm file When you start Lertap you ll see tabs at the bottom of Excel s screen one for each of the wor
157. al items often like to have negative weights corresponding to the negative side of their questions as seen above in Example A10 The set of four CCs cards above is not right the five semantic differential items will have scoring weights of 1 00 2 00 7 00 We need some mws cards col c3 c12 sub AFF Res 1 2 3 4 5 6 7 Title CapesAtt alt 55555 77777 pol ttt mws c8 3 00 2 00 1 00 0 00 1 00 2 00 3 00 mws c9 3 00 2 00 1 00 0 00 1 00 2 00 3 00 mws cl10 3 00 2 00 1 00 0 00 1 00 2 00 3 00 mws cll 3 00 2 00 1 00 0 00 1 00 2 00 3 00 mws c12 3 00 2 00 1 00 0 00 1 00 2 00 3 00 Of course we could have used mws cards for all items and in this case we d then have col c3 c12 sub AFF Res 1 2 3 4 5 6 7 Title CapesAtt mws c3 1 00 2 00 3 00 4 00 5 00 mws c4 5 00 4 00 3 00 2 00 1 00 mws c5 5 00 4 00 3 00 2 00 1 00 mws c6 1 00 2 00 3 00 4 00 5 00 mws c7 1 00 2 00 3 00 4 00 5 00 mws c8 4 3 00 2 00 1 00 0 00 1 00 2 00 3 00 mws c9 3 00 2 00 1 00 0 00 1 00 2 00 3 00 mws cl10 3 00 2 00 1 00 0 00 1 00 2 00 3 00 mws cll 3 00 2 00 1 00 0 00 1 00 2 00 3 00 mws c12 3 00 2 00 1 00 0 00 1 00 2 00 3 00 We know you can just about understand these cards but realise you might want to know what the asterisks are doing on the mws cards for the i
158. alue Int ArraySize Rnd 1 more code The Randomize function provides a seed to Excel s Rnd routine It uses the computer s clock to do this guaranteeing that the random numbers generated will differ each time Halve amp Hold is run Random numbers are generated until half of the original data records have been fingered that is identified The unfingered records are then deleted from Book1 s Data worksheet Then another copy of the original Data and CCs worksheets is made and placed in a second new workbook which we may call Book2 for purposes of this discussion Next the data records known to reside in Book1 s Data worksheet are deleted from Book2 s Data worksheet and we end up with two essentially random samples of the original data leaving the original untouched When the number of data records in the original Data worksheet is not an even number Book1 will have one more data record in it than Book2 How to generate a smaller random sample of data records Halve amp Hold always creates halves workbooks whose Data worksheets have 5096 of the records in the original Data worksheet To get a sample with 2596 run Halve amp Hold again using one of the 5096 samples for example if Book1 contains 5096 of the original Data records run Halve amp Hold with Book1 to get two new random samples each with 25 of the original Data records Who uses Halve amp Hold Researchers and teachers often people who are going on
159. alues Whenever this happens Excel s reaction is to fill fields with symbols This problem is fixed by turning the Headings option on and then widening the corresponding column column 4 in this case 2003 2011 Curtin University West Australia 252 Help file for Lertap 5 fX Spread iF x Morey k Res charts a Run menu Graphics trio Otr 3 4 1 5 6 Gridlines Eormula Bar Headings owledge of LERTAP2 created 19 PEEL 80 analysis based on a mastery cutof 6 1 Exit Excel 62 Variance components 63 df Ss MS 64 Persons 59 115 84 1 96 65 Items 24 22 62 0 94 66 Error 1416 236 50 0 17 67 68 Index of dependability 0 937 69 Estimated error variance 0 007 J0 For 68 conf intrvl use 0 085 71 The Ref style option turns R1C1 referencing on and off When R1C1 is on column heading are numbers as displayed above When it s off column headings are letters often called the Al referencing style ft Spread bf Res charts Graphics trio 2 Otr P More Run menu Copy Gridlines Formula Bar Headings Ref style Exit Excel 61 2 Variance components Sa 63 df ss MS 64 Persons 59 115 84 1 96 65 Items 24 22 62 0 94 66 Error 1416 236 50 0 17 67 68 Index of dependability 0 937 69 Estimated error variance 0 007 For 68 conf intrvl use 0 085 5 Now the column with SS figures is column D if the Ref style opti
160. ame Crit Skills A Titles CrtSklA Wts key CCACB CABAB BDBAD BBAAD ECDCE ABDDA CBCBB BDDCA exc ci c3 c6 c10 c15 c17 c31 c34 c38 c40 c46 c49 The example above shows two ways to have 12 items excluded from a 40 item subtest 2003 2011 Curtin University West Australia 3 8 3 9 3 10 CCs details The first way involves the use of multiple mws cards the second way uses a single exc card There are other ways to exclude items refer to the previous topic e8 for more details Pretest trial items Sometimes a test or subtest will include items which are not to be scored These are often referred to as pretest items or trial items They re riding in the test usually embedded amongst the normal items just to see how good they are If they qualify as okay they might appear in a subsequent version of the test and be scored that is they will move from being unscored items to normal items To see how to handle these in Lertap please refer to this topic Split half reliability Lertap uses Cronbach s coefficient alpha as its reliability estimate At times split half reliability estimates may be useful such as that found by correlating two half tests For example the half tests may be created by splitting out the odd numbered items as one half and the even numbered items as another half To see how to split a test into halves with Lertap and correlate the halves please refer to this topic
161. ample Another sample measure of how spread out the scores are number of This is the number of items in the test or subtest subtest items minimum Corresponds to the lowest possible score which a possible student could get on the test For a cognitive test score this is usually zero meaning no items correct maximum The highest possible test score On a cognitive test possible this is usually equal to the number of test items with a score student getting one point for each item answered correctly reliability Derived by applying Cronbach s method to find coefficie coefficient alpha The maximum possible value is 1 00 nt alpha A good cognitive test might be expected to have a reliability of at least 0 80 a good affective scale might be expected to have a reliability of at least 0 70 Note however that these good values are not always appropriate A mastery test for example will sometimes be acceptable even when its reliability as measured by alpha is low Affective scales frequently have alpha reliabilities below 0 70 when this happens results are often interpreted on an item by item basis instead of the overall scale score 2003 2011 Curtin University West Australia 284 Help file for Lertap 5 Notes 1 coefficient alpha may take on negative values when this happens Lertap reports 0 00 as the reliability figure 2 KR 20 another reliability estimate i
162. ance see the Analysis of variance table discussion at the end of the Breakouts 4 topic 2003 2011 Curtin University West Australia 4 4 7 1 Toolbar and tab ne c Se Sl ee O zu Ing 15 24 48 11 0 2 844 0 48 0 50 Nat 1496 9 68 8 0 1 F sig eta 71 39 0 04 On this item 135 6896 of the students in the Nat group apparently knew the correct answer compared to 4896 in the Ing group The sig value of F ratio is so small that when rounded it shows as IBO The difference in item 135 means for the two groups is statistically significant Eta at 0 04 fails to reach great heights if we applied standard guidelines for measures of practical significance we d conclude that the difference in means is not what many would term meaningful But here we would be almost foolish to think that there s no real difference Analysis of variance methods are based not surprisingly on partitioning score variances With cognitive items scored on a right wrong basis there may well not be much score variance to partition There is a difference between the groups on their answers to 135 even Blind Freddy would be likely to see it And it s a meaningful difference too An advantage of 20 points on an item difficulty scale is substantial It could be that the Nat students are more capable full stop If so the response differences seen in our little 135 plot might
163. and 1993 Refer to http lertap curtin edu au References htm 2003 2011 Curtin University West Australia 158 Help file for Lertap 5 4 4 7 4 System settings Lertap s main options are set in the Systemf zA worksheet of the Lertap5 xlsm workbook id 9 948v 7 Lertap5 xlsm Microsoft Excel non commercial use X Lertap Home Insert Page Layout Formulas Data Review View Developer Add Ins 5X 1 3 4 System Settings Allowed settings es B MER pcnc Leones aes 00 to 1 00 DEMNM RIEN yes no no These are Lertap5 system settings Change them only if you understand them 42 Create an adjusted percentage score 43 Set colours for Breaks F ratio and eta 44 Maximum Breaks F significance level for colouring 45 Minimum Breaks eta value for colouring 46 Controls for Ibreaks amp IbreaksMH reports 47 Use your own Ibreaks plot settings 48 Number of columns for plot width 49 Numbers of rows for plot height 50 Include F ratio sig and eta row in Ibreaks report 5i Maximum Ibreaks F significance level for colouring 52 Minimum Ibreaks eta value for colouring 53 Use continuity correction for Mantel Haenszel chi sq 54 Maximum chi sq significance level for colouring 55 Exclude students with scores of zero from M H calcs es no 56 L 4 gt Dl Data CCs System tax Oldccs J H8 uu The op
164. and multiplying by 100 2003 2011 Curtin University West Australia Dow Help file for Lertap 5 cf The cumulative frequency that is the number of scores at and below score c The percentage corresponding to cf found by dividing cf by the total number of scores 60 in this case and multiplying by 100 Histograms in this old format are saved in worksheets with names such as Histo1L Histo2L and so on the L is used to refer to the Lertap 2 style But wait There s more to the Histograms option A click on the arrowhead to the right will get a drop down box to show with more options See ali Histograms i Histogram E dj Scrunch em 2 Lelp The Histogram E option will get Excel to look at an L type histogram such as found on a Histo1L worksheet and make a fancier chart one which may strike you as what folks traditionally expect to see in a histogram Note some versions of Lertap automatically append the fancier chart to the output making it unnecessary to use this option Go to a Histo1L or Histo2L or so on worksheet and then click on Histogram E You should see a chart like this 2003 2011 Curtin University West Australia Toolbar and tab 165 Histogram of scores J Frequency Or NU eu ns o WO 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 Comfort This is an Excel chart of course and as such you can do what you
165. answer to the first question is B The right answer to the 6th item is C The correct answer for the 10th item is B And that s it Yes The data set is ready for Lertap At this point one would go to Lertap s Run menu and click on Interpret CCs lines Then after being Freq ed out we d go back to the Run menu and click on Elmillon item analysis Beauty 2003 2011 Curtin University West Australia 22 Help file for Lertap 5 Related tidbit The example above is based on a cognitve data set which may be seen at this URL http www lertap curtin edu au Documentation Samples TenCogs TenltemCognitive doc 2 3 2 Affective example Here s another example EE 2 3541 4 On 7 8 ee ee a ee eS 1 2 No Qi Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 010 Qii Q12 3 1 3 3 3 4 3 3 3 3 2 3 3 4 4 2 2 3 3 2 4 4 4 4 3 3 5 3 3 3 2 2 2 2 4 4 3 3 3 5 6 4 i 2 3 4 4 2 1 2 2 2 2 5 7 5 2 2 2 2 3 3 i 3 2 2 2 2 8 6 2 3 2 3 3 3 2 3 4 5 2 3 9 7 2 3 2 3 3 3 i 2 4 3 3 5 10 8 2 4 3 3 3 2 3 2 2 1 2 3 1i 9 i 3 3 3 3 2 2 3 2 2 1 5 12 10 2 4 1 1 1 1 3 2 2 2 3 1 13 11 1 3 2 2 2 2 3 2 3 4 3 4 14 12 3 2 2 2 3 2 3 3 2 3 3 4 15 13 3 3 5 1 1 1 1 3 2 3 3 4 16 14 2 2 1 1 3 3 3 2 2 3 3 3 SEM 15 3 3 3 2 3 2 4 3 3 3 2 4 Here we ve got another 15 students and this time we have responses to 12 survey items Some sort of number is found in the first column with the responses to the first question Q1 found in the 2nd column 1 PAE col C2 C13 2 Su
166. art whole inflation only for the option having the greatest wt 3 3 3 MDO cognitive Statsb The previous topic discussed how the MDO and did not see options affect the information reported in Statsf reports Now you re set to see the corresponding Statsb reports Lertap5 brief item stats for Knowledge of LERTAP2 created 20 03 2006 DOGO 2003 2011 Curtin University West Australia CCs details s DOCE JEJE The first report has an other column which indicates the number of people with missing data on each item expressed as a percentage figure The statistics in the diff and disc columns are based on calculations which include the people with missing data In the second report the other column has been replaced by the n column The entries in this column indicate how many people answered each item and the various percentage figures plus the diff and disc values are based on n that is they exclude missing data Lertap uses what is termed a pairwise exclusion rule to calculate the diff and disc values if a student is missing data for the item or did not see it s he is excluded from the calcs for that item 5196 pm aan soe a os EGEJESEI DEZIC Beno a EJEJEJES oe The report above corresponds to the Test Pilot results mentioned in the previous topic 53 The numbers seen in the n column exclude the number of cases with missing data if any as well as the
167. as easier This looks very very IFfy you say Well among all the nice things we might say about Excel one is that there are lots of resources to turn to when help is needed You can try Excel s Help look up create conditional formulas for some very helpful hints and examples Or try the internet Or the local bookstore perhaps there s now Recoding Excel Columns for Dummies Or even try us at larry lertap com 4 6 1 5 Apply a formula Lertappers sometimes want to create a new score by transforming or combining one or more of the scores found in the Scores worksheet For example let s suppose that a user wanted to apply a linear transformation to one of Scores scores of a type commonly found in texts and reference books y m b 2003 2011 Curtin University West Australia 220 Help file for Lertap 5 Here x is a score which already exists while m and b are constants The new score is y Let s suppose that m 10 and b 5 making the equation y 10x 5 Have a squiz now of the workbook below taken when running an earlier version of Excel The original score x in the equation above is called Test1 found in Scores column 2 Get out your best glasses and look carefully at the Excel Formula Bar the one which begins with R3C3 and contains a formula which a user entered 10 RC 1 5 The RC 1 is Excel s way of referring to a value found in the same row R one column to the left C 1 F3 M
168. ata set with only certain data records to be copied to the new data set s Data worksheet tst is used to set up a new Lertap workbook containing a subset of the original Data records for example just the males or only those in a specified School District This has been a quick introduction to Lertap s control cards We ll go on now to provide more specifics and we ll do this by subtest type cognitive first then affective 3 1 Cognitive CCs Before getting into the syntax for CCs cards used to analyse cognitive items let s come to terms with some terms Each cognitive item may use up to twenty six 26 response codes Response codes are also known as alternatives or as options A true false item may use T and F as response codes or t f or 1 2 A cognitive item with four possible responses may use codes of A4 B C D or a b c d or 1 2 3 4 Associated with each response code is a weight the number of points a person gets for choosing the corresponding option For example if the right answer to an item is A then people who select A will get a certain number of points people who select one 2003 201 1 Curtin University West Australia Help file for Lertap 5 of the item s other responses will usually get no points Okay Now then Let s say we ve given a 5 item cognitive test with answers appearing in columns 2 3 4 5 and 6 of the Data worksheet We d like Lertap to spin its ma
169. ation UpdatesSummaryLertap57 pdf For a complete record of changes dating back to the beginning of time please refer to the UpdatesSummary page on the website Its URL is http www lertap curtin edu au Documentation UpdatesSummary htm Most of the more salient changes are mentioned in the following topics May 2003 5 added Lelp version change to 5 2 J une 2003 3 added an item zapper July 2003 5t changed item difficulty calculations August 2003 s5 what you weighted for other version change to 5 25 September 2003 25 to halve and hold forever Bilog MG tetrachorics October 2003 eigenvalues amp SMCs version change to 5 3 November 2003 2 smiles for quintiles February 2004 21 nothing sword id about this XCALIBRE support April 2004 consolidation version change to 5 4 July 2004 23 record IDs amp formula scores September 2004 MDO now means Missing Data Out October 2004 enhanced IStats report version change to 5 4 5 2003 2011 Curtin University West Australia 318 Help file for Lertap 5 November 2004 3 exc a new CCs card to exclude items February 2005 three enhancements EMQs MDO quintile options May 2005 production mode added version change to 5 4 6 July 2005 2 beat the cheat version change to 5 5 September 2005 2 histogram charts anyone January 2006 beat the cheat 2 version change to 5 6 March 2006 t more MDO stuff plus a did not s
170. atistics with an item banking and test development system such as FastTEST from ASC Assessment Systems Corporation www assess com The latest versions of FastTEST have an Import Wizard which makes it a straightforward matter to pick up values in a csv file In the example above we d tell FastTEST to pick up csv field 7 as the P Value and csv field 8 as the ItmTtlCorr The number of columns seen in a Stats1b report depends on the number of response options or alternatives used by a subtest s items At times there will be too many columns too many fields when the worksheet is saved as a csv file for easy use with FastTEST In this case you ll want to delete some of Stats1b s columns before making the move to save as a csv file Is it difficult to delete Stats1b columns Nope it s real easy Use the toolbar s Shorts menu to Turn row and column headings on off Then get out your mouse and right Click on say column 2 Left click on Delete and guess what Bingo the column is gone Now you know how we ve been saying there may be too many Stats1b fields and how you might want to delete some if you re making a csv move to FastTEST Well come to think of it you might want to insert a new column in the Stats1b worksheet before saving it as a csv file Yes FastTEST assigns and carries a UniquelD field for each item Your work might be a bit easier if you inserted a new column in the Statslb worksheet and typed each item s FastTE
171. ats2ul Stats3ul and so on These quintile plots can greatly assist with the process of identifying how well cognitive items perform Page forward to plot your future 2003 2011 Curtin University West Australia 180 Help file for Lertap 5 4 5 3 1 Unidimensional response charts b If you have a Stats1b style report active clicking on this icon will produce charts such as the one pictured below E Q26 6 1 4 B A A N D sD Lertap s item response charts are made by Excel They may be copied and pasted to any other application such as Word They may also be extensively reformatted for help on this please refer to Excel Help The response labels showing at the bottom of the chart are as found at the top of a Stats b worksheet Here s a snapshot showing the top of a typical Stats b worksheet Lertap5 brief item stats for Comfort with usin me j2 s ej s lor ie nne ne zoe ume These response labels may be changed If they are the change will carry through to the item response charts For example 2003 2011 Curtin University West Australia Toolbar and tab ow Lertap5 brief item stats for Comfort with usin sa a mw boyjso azo om 22 257e 20m 17m Note how the original response labels of 1 2 3 45 have been changed to SAAND SD Look at what happens when item response charts are requested now Q26 6 7 4 1 ay J g 0 a T T T
172. b a If the production mode setting is set to yes then Lertap will not stop after it has been requested to Interpret CCs lines it ll power ahead automatically activating the next option on the Run menu Elmillon item analysis There are three other yes settings which may be made in these rows you see them above You can get Lertap to automatically output histograms for each subtest response charts bar charts for affective subtests quintile plots for cognitive subtests and an item scores matrix IStats This gives you time to sit back with your cup of coffee and watch the screen flash before you as Lertap goes about its tasks Of course this is not recommended Lertap ships with all the production mode options set to no Why Because sound data processing practice is always supposed to involve a data integrity check or two You want to make sure the data you re feeding into Lertap has been subjected to some quality control In Lertap 5 the main means of doing this is by having a careful look at the Freqs report the worksheet produced by taking the Interpret CCs lines option from the Run menu The Freqs report readily indicates the characters found in each of the data columns If you re running with a cognitive test whose options use the letters A B C and D then you ll want to check down the Fregs report to make sure that no other characters have crept into the scene such as perhaps lower case letters a b c
173. b Affective 3 4 M gt M Data CCs Once again there are only two CCs rows with information The first row tells Lertap that item responses are found in columns 2 through 13 of the Data worksheet The 2003 2011 Curtin University West Australia Getting started 23 second row tells Lertap that these items are affective or survey items Having seen the word affective on a sub CCs line Lertap knows that the questions do not have a correct answer they re to be scored using the default survey scoring scheme where a response of 1 equals one point a response of 2 2 points and so on This example is complete Nothing is missing At this point one would go to Lertap s Run menu and click on Interpret CCs lines Then after being Freq ed out back to the Run menu and a click on Elmillon item analysis Beauty abounds doesn t it Related tidbit The example above is based on a sample survey data set which may be seen at this URL http www lertap curtin edu au Documentation Samples CEQ CEQ1 DOC 2 3 3 Lertap data set We ve given you two really straightforward examples In our experience the two simple examples you ve seen would be very much like what maybe a quarter of Lertap users regularly live and breathe But Lertap was designed to handle more complex situations Those little CCs lines can pack more punch than what s been on display in the previous two samples As an example look at this CCs worksh
174. ble answer sheets 290 bubbled in 245 Burma shave 319 buying Lertap 3 4 C C large 151 2003 2011 Curtin University West Australia Calc 2 calculation of item difficulty 280 calibration 159 cards 27 29 36 categorical variable 193 CCs 322 CCs cards how they work 74 CCs errors 258 CCs for affective items 36 CCs for cognitive items 29 CCs lines 27 CCs lines affective example 22 26 CCs lines cognitive example 20 26 CCs lines mixed example 23 69 CCs worksheet 253 257 CFC 29 74 change chart type 147 changes 317 changing settings 248 chart 84 106 155 205 chart format 205 chart limitations 205 chart limits 198 chart limts 205 chart problems 136 205 Chart Tools 150 ChartChanger1 150 ChartChanger2 186 ChartChanger3 186 189 charts 131 141 147 150 179 324 cheating 103 118 321 chm 4 chm version 12 cigar 106 classical test theory 282 classification consistency 306 codebook 253 coefficient alpha 29 73 106 282 cognitive 271 cognitive CCs 29 cognitive example 20 26 collapsed ribbon 6 330 Help file for Lertap 5 coloring 141 colors 205 column labels 255 comments 257 common odds ratio 151 communality 111 compatibility mode 11 composite 23 234 compress 160 Computer ID 238 242 concatenate 231 conditional SEM 282 condtional standard error of measurement 309 confidence intervals 309 constructed response 73 contact 14 control wo
175. by using Excel s standard chart options of which there are many You can add titles legends and change lots off colours why you could probably spend two or three hours enhancing Lertap s initial plot ending up with a graph which when pasted into your final report is bound to bring you great kudos One thing that s really neat about these line graphs is that you can see the x and y values associated with any of the line graph s points by just letting your mouse hover right above one of the points Try it you don t need to hold down a mouse button just position the mouse pointer on top of a point and the corresponding x and y values will jump out at you Note that this won t work with the graph above as it s just a picture of Excel output not the real thing What s that you re saying You d like to know more about Excel charts Goodonyou Use Excel s Help system it s got heaps of info Heaps Line graph problems Users just starting to use Lertap s line grapher shortcut may find that it won t work as advertised here In the tests we ve done to date the problems which arise have to do with selecting cells Cells are selected in normal fashion by highlighting them with the mouse or by holding down the Shift key and using the arrows on the keyboard The cells are expected to have numbers in them however the first cell selected can have text information such as a row or column header if Excel finds the first selected cel
176. cation Its input and output files are standard Excel worksheets nested within a standard Excel workbook We have found that users will often have an Excel worksheet with data which they d like to use with Lertap If they rename the worksheet to Data and add another sheet called CCs will Lertap work Yes And no Lertap will certainly work but its output may be poorly formatted and difficult to read There are often font problems Lertap has a preferred font Verdana If the user s workbook is based on a different font Lertap s output may be adversely affected When such problems arise we suggest this use Lertap s New menu 8 to make a new blank Lertap workbook Then copy all of the data records from the original workbook and paste them into Lertap s Data worksheet This generally works without problem We have seen numerous text files imported to Lertap without problem For a fairly thorough example we recommend a visit to this URL http www lertap curtin edu au Documentation Samples MondatY 2003 2011 Curtin University West Australia Input amp Output 313 ProcessingMondaty htm Not long ago we happened to be in Central Java Indonesia where we saw some 20 000 provincial high school test results imported to Lertap froma dBASE file set up by ascanner The dBASE file had three tab delimited fields record number student ID code and a string of 80 item responses A straightforward copy and paste from the dBASE
177. cccccccescecicceveetecenneereccaceeteccascensccuseeveccneeeeccuaucheceaecetdconseeteceaaesteccacecteccaass 5 L EIcrpgjiee EE 6 Com patibility MOde 11 7 About Lertap version cecececeeeeeeeeeeeeeeeeeeeeeeeeeeeee erences eeeeeeseseseeeseeeeeeseeeseaeseeeseeneeeesees 12 8 About this dOCUMENE oeiee aeara aa a aaa aaraa aN aE ae EaD aiani 12 Differs from MANUA aaa etae cei dauceuvecesude suey sued sueedeet scvsddoderenudenescauvsenvedesuee 13 DICIMUS 14 QO GCOmMta ct ius ciccicse2ccccedce ess 14 Part Il Getting started 14 1 About the Cook s tour inier cad gatecucdencancecectscubensexcateesscudteeteecuende 18 2 HOWE WOIKS iei inne acackcanticessccsaanccezsacasdesciacceszsecesdesaiacseendqansdeasaceteasuacstcoavaccsaenss 19 Kai2cndree 20 Cognitive example Affective example Lertap data set Multiple cognitives Multiple affectives Part Ill CCs details 1 Cognitive CCS erronee enS AA KAEA ANELAR AEEA ENNAN KERRE 29 2 NAA CCS ER E E E E E E T 36 3 Missing data seereis aeria aaa EA EA a ANAA A NA RENE E Did not see option MDO cognitive Statsf MDO cognitive Stats bD cs sc csczcccecec 58 MDO cognitive Statsul 5 2 correr keer dene ota doe tency oe eee Siria iaaii aea ieta 60 M DO affe CUIVG oii c
178. chieved based on the items presented to the student Sample Scores output As you page forward to following topics you ll come across an example from an actual online testing situation one which used the Test Pilot system from McGraw Hill Over 400 students at a large North American university took an online version of a test delivered by Test Pilot Sampling from a pool of 80 cognitive multiple choice items Test Pilot served up tests whose lengths varied some students were presented with 40 items some with 43 some 45 and others 48 Please read the second tidbit below the situation was actually a bit more complicated The Lertap Data worksheet turned out to have 93 columns for item responses Why 93 2003 2011 Curtin University West Australia CCs details 51 when there were only 80 items in the pool Because a few items allowed for multiple responses see second tidbit below In this example blanks were used as the did not see code Were you to look at the item responses for any student scanning from left to right over the 93 columns allocated you d see actual responses many blanks and for those items not seen by a student but not answered a 9 We could and will entertain a variety of Lertap scoring methods which will demonstrate how the did not see options affect test scores To begin say we have the did not see option off that is we have no in the System worksheet row which says Use a did
179. ck on SETUP EXE If you do not have a program to unzip files PC users can obtain one at www winzip com m WARNING Lertap may not work properly unless it is installed by the installation program SETUP EXE that is supplied as part of the download Download the 30 day trial copy for use with Windows Excel 2010 14 2 MB 48 Note 1 please check these notes on Excel 2003 and 2007 if you re not sure which Windows version of Excel you have Excel 2010 is much like Excel 2007 for comparison purposes Note 2 this version will not work with the 64 bit version of Excel 2010 Note 3 this version will not work with Calc the OpenOffice spreadsheet program A 30 day Windows trial may be downloaded by clicking on one of the options seen immediately above where the green arrows are pointing please note that this is just a picture of the ASC page if you try and click above not much at all will happen you have to go to the actual Lertap area within the ASC website and then do your clicking The trial copy may be used for free for 30 days A license must be purchased if Lertap 2003 2011 Curtin University West Australia 4 Help file for Lertap 5 1 4 1 5 is to be used after the 30 day trial period Click here to read about licensing matters How to license it A license to use Lertap after the 30 day trial period must be purchased from ASC Please click here 241 for further comments How to run it
180. com is the best bet Getting started Be sure to also visit the What is Lertap topic 1 i for getting started suggestions It s a fairly simple matter to get started with Lertap Let us get you launched on the so called Cook s tour and you can see for yourself When you start Lertap using Excel 2007 or Excel 2010 for Windows your computer screen should bear quite a resemblance to the screen snapshot show below If it doesn t refer back to HowToRunlt 4 and MacroSecurity 6 if you re using Excel 2010 the top left of the screen will be a bit different see the Requirements 2 topic for an example 2003 2011 Curtin University West Australia Getting started Ea W By 94 verdana Lertap5 xlsm Microsoft E l Lertap Home Insert Page La Formuli Data Review View Develop Add In X Delete Al Sort Blank EO Interpret ali Histograms oF Move Version X Line IB Headers gp Elmillon Scatterplot License 7 fX Spread Excel gai Copy More bf Res charts Lelp Basic options New menu Run menu Graphics trio Other menus To set up a new workbook use the New drop down menu option above to the right of the yellow smiling face To work with data you ve already set up use Excel s File menu to find and open the relevant workbook The File menu is usually seen on the menu bar at the very top of the screen How to use Lertap Try the resources available via the Lelp
181. cond item above They d give strongly disagree a score or weight of 5 disagree a weight of 4 and strongly agree a weight of 1 This way the top scores will come from people who not only love WA s beaches but think they re better than those found around Maui Okay then we ve got some basic terms under the belt Of course if we re on one of those beaches we may not have a belt to put them under but let s proceed anyway Say we had five items of the sort shown above Say we asked 200 WA based people to respond to the items and took the trouble to fly to Hawaii paddle out to Maui and ask another 200 folks to respond to the same five questions We entered the 400 responses into a Lertap Data worksheet with some sort of ID code in column 1 a location code of W or M in column 2 and the answers to the five questions in columns 3 4 5 6 and 7 Were we to look down columns 3 through 7 we d see 1s and 2s and 3s and 4s and 5s corresponding to answers of strongly disagree through to strongly agree Having entered the data we need to go to work in Lertap s CCs worksheet Example A1 We started with this set of two CCs cards col c3 07 sub AFF These two simple cards are all that some people might use to process the results The col card gets Lertap and Excel to read information from the Data worksheet looking at five columns 3 through 7 The sub card has the AFF control word this is necessary in ord
182. condary ones The following topics get into some of the specifics of Lertap s primary and secondary worksheets Related tidbits For the definition of a Lertap workbook please click here z55 For information on how to create a new Lertap workbook simply click here 871 and you ll be whisked away to a discussion of the Lertap toolbar s New menu 6 1 Lertap workbook def A Lertap 5 workbook is an Excel workbook with the particular features mentioned here A Lertap 5 workbook will always have at least two primary worksheets one of these is named Data the other is named CCs The Data worksheet s top two rows are reserved for titles The first row may contain any text it s used as a means of briefly describing the data found in the worksheet For example the first row might say Data collected 10 March 2003 in Psych 501 2003 201 1 Curtin University West Australia 254 Help file for Lertap 5 The second row of the Data worksheet contains column headers For example if the first column of each data record is some sort of ID field then the contents of the worksheet cell formed by Row 2 Column 1 might be ID Number If responses to the first item are found in column 5 of each data record then the contents of the worksheet cell formed by Row 2 Column 5 might be Item5 Data records begin in Row 3 of the Data worksheet If the first column of any row in the Data worksheet is empty or contains a zero then that is con
183. correlated the sphere stretches out to an ellipsoid 2003 2011 Curtin University West Australia 1 Help file for Lertap 5 After the percent row comes the p comp1 row giving the correlation of each of the items with the first principal component the values found in this row are also sometimes called the loadings of the items on the first principal component The first principal component corresponds to the ellipsoid s major axis to its longest axis Each eigenvalue represents the relative length of one of the ellipsoid s axes Each of these axes is said to represent or correspond to a principal component Think for a moment of the case when n 3 If the three items are normally distributed and uncorrelated their scatterplot will have the form of a soccer ball a perfect sphere As the three items begin to correlate the soccer ball changes shape morphing into an American football and then as the correlation among the items increases into a cigar shape The shape of the scatterplot is highly related to the relative sizes of the eigenvalues if the eigenvalues are all equal the shape is a sphere If the first eigenvalue is much greater than the others the shape is a cigar and in such a case the multivariate scatterplot is said to have essentially one principal component or dimension In the 10 item example above the first principal component is said to account for 38 3 of the total variance or volume found in the
184. ct answer froma student might be said to depend on which group s he is a member of Now if it can be shown that the students in each group are of equal proficiency or ability then what might it be which would make it more likely for one of the groups to get an item correct Perhaps a difference in the two versions of the item Perhaps we have evidence of DIF differential item functioning This is often unwanted If the objective is to have a fair test test developers will generally weed out items which have such behavior see tidbits references below Angoff has a discussion of when DIF might be tolerated even expected MH D DIF is a statistic which results from converting MH alpha to a different scale MH D DIF 2 35 times the natural logarithm of MH alpha The D in D DIF stands for delta The delta metric is used by ETS the Educational Testing Service to express item difficulty Items with positive MH D DIF favour the focal group negative MH D DIF favours the reference group The ETS level for an item will be A B or C It will be A if MH D DIF is between negative one and plus one 1 00 lt MH D DIF lt 1 00 or if Prob is greater than 05 MH chi sq is not statistically significant A level items are said to indicate negligible DIF the IbreaksMH tables show this as A neg Note yes it is possible to have a statistically significant MH chi sq but still have the item falling into the ETS A level if the magnitude of
185. cters 2003 2011 Curtin University West Australia Toolbar and tab 9 E Microsoft Excel LXREd502 xls ERES File Edit View Insert Format Tools Data Window Help x e New v GER rZ Run 2l i ES i Move o License Lep A 14A1 1X 4541 1 BAKER G 111011011010110110001111111011111110110111001 BOONKHONG 5 010010000010110011001111111010111111100111101 CHAMBERLAIN 000000000000001000001111100001100111111001101 CLEMENT R 100001000011101100101111001111110110100101111 DEAN R 101011010011101111101011101111111111110011111 DEMOUCHE C 110111000010110111011111101101111100100101101 EASTABROOK R 111110011010100111110101000111111111110110111 ELSCOT L 111011010001110111010011101111111111100111111 FALCONER S 111011101001100110000011100111011011110111101 GLUCINA D 001101010011101010000111011001111111100111111 12 HAMEISTER K 000011100000101000001111000111111011110100101 n 2 3 4 5 6 Ti 8 9 10 11 HART AY 101313103430006041140000010111434130001011301111043011311 13 M 4 Mf HitolE HistolL IStats DAT a min Ready Now a program such as Bilog Bilog MG or XCALIBRE will not read data from an Excel worksheet The sheet has to be saved as a text file How How to save the DAT worksheet as a text file Click here 3 to find out or page ahead to the topic titled Creating a text file Once the DAT worksheet has been saved as a text file the first line the Fortran format statement should be deleted Bilog wo
186. ction of the Lertap tab has six components or options Lertap Home X Delete 1 Sort a Version fX Spread Excel Basic options There isn t really a theme to these options they do quite different things Click on them and you ll see Or page forward to browse each of these options one by one Delete X This icon deletes certain worksheets from a Lertap 5 workbook To understand what it does and why it is used consider a standard Lertap 5 job a user puts his her item responses and perhaps other data into a workbook s Data worksheet S he puts control statements or control cards into the CCs worksheet The user then clicks on the Run drop down menu and selects the Interpret CCs lines option What happens Lertap looks at the rows in the CCs worksheet to find out which columns in the Data worksheet the user wants to analyse and then it goes on to read data and add new worksheets to the workbook The most obvious worksheet added by this step is the one called Fregs The user pauses to scan the information found in the Freqs worksheet If all is in order the user usually returns to the Run menu and clicks on Elmillon item analysis What happens Lertap adds more new worksheets to the workbook such as Scores Stats1f Stats1b and so on These worksheets which Lertap adds are called secondary worksheets The original Data and CCs worksheets are referred to as primary worksheets At any time the
187. ctly symmetric about the median The lines sprouting from the top and bottom of each box are the whiskers The top whisker extends from the 75th percentile Q3 to the highest score which is not an outlier Similarly the bottom whisker extends from the 25th percentile Q1 down to the lowest score which is not an outlier Outliers are discussed below A plot such as the one above has a lot of information The Primary students tended to have higher SelfReg scores and the range of their scores was the greatest the P group s whiskers extend a bit further than those for the other groups The highest median score is found in the P group The SelfReg scores of the Secondary students are by and large the weakest their median score and their Q1 score are lowest of the lot How it works 2003 2011 Curtin University West Australia 13 Help file for Lertap 5 The Box and whiskers option works by first making a copy of a Breaks report If you were looking at a Breaks1 report for example then you ll get a new report called Breaks1bw with bw standing for box and whiskers This new report will have a section at the bottom with summary score data organised in the fashion shown here n Mean s d Median m Q3 Minimum Maximum 25th Pct 50 00 50 00 34 75 50th Pct 4 00 5 50 uu 75th Pct 4 00 6 50 i Top whisker 8 00 18 00 15 75 Lowliers 1 2 Highliers 0 0 Boxplots also known as box and whisker plots are an inv
188. d 0 00 0 0 highest score found 14 00 93 3 median 8 00 53 3 mean or average Gers standard deviation 88 19 296 standard deviation as a sample 2 89 19 2 8 32 number of subtest items 15 z minimum possible score 0 00 maximum possible score 15 00 reliability coefficient alpha 0 68 index of reliability 0 83 standard error of measurement 1 63 10 8 IM 4 b M Scores Statsif Statsib csem1 Ready Z EB E 100 g m On 115 the Stats1b report flagged the first option Stats1f shows that this distractor was selected by 569 students and their avg score was also above the criterion average Keep in mind that these are just flags notes created by Lertap to suggest that something might be amiss We usually do not want distractors to be selected by above average students When they are we may have some reason to suspect ambiguity the wording of the distractor may need to be improved In some cases a decision may be made to double key an item that is to score the item in a manner which gives points for more than one answer In Lertap this is done with a mws line an example which uses mws lines may be found towards the bottom of this topic Look now at I11 2003 2011 Curtin University West Australia 28 Help file for Lertap 5 X MathsQuiz xisx Microsoft Excel 1 00 0 00 0 00 0 00 0 00 The Stats1b report flagged two o
189. d Page forward for an example Mac example Please note this example uses screen shots from Excel 2003 Let s say that a user named Lee wants to create a macro which will display the date whenever it is activated 2003 2011 Curtin University West Australia Toolbar and tab He opens Excel s Visual Basic Editor inserts a new module in the Lertap5MacroSetA xlam file names the module ShowDate and creates the following little subroutine He saves the file 4B Microsoft Visual Basic LertapbMacroSetA xla ShowDate Code ae L File Edit View Insert Format Debug Run Tools Add Ins Window Help Sete d B 3 dd X a2 4 4142 1 ad See F O aa G E VBAProject Lertap5MacroSetA xla 3 Microsoft Excel Objects as ShowDate 2 Public Sub ShowDate Dim MyDate Let MyDate Nowi E Modules MsgBox MyDate A AngellrngMacs End Sub 2 ShowDate H BAProject PERSONAL XLS Properties ShowDate X ShowDate Module T Alphabetic Categorized ShowDate Next Lee puts the name of his new whiz bang subroutine into Lertap s System worksheet 248 ShowDate 102 MacroMio2 103 MacroMio3 104 MacroMio4 He saves the Lertap5 xlsm workbook closes it and then re opens it When Lee taps on the Macs menu here s what he sees 2003 2011 Curtin University West Australia 28 Help file for Lertap 5 Comments i ShowDate M
190. d amp Select options under the Editing section of the Home tab look way to the right hand side of the Home tab M zy A el g 2 at 27 eda i dA Find 23e Replace Go To Go To Special Formulas Comments Conditional Formatting Constants Data Validation Select Objects n IN Selection Pane Click on Replace and the following dialog box will pop up 2003 2011 Curtin University West Australia Toolbar and tab Find what Replace with Replace Find what z No Format Set Replace with No Format Set Within Sheet Match case Match entire cell contents Search By Rows z Lookin Formulas To give an example of using Excel to recode a column let s say that we wanted to change every occurrence of F in a given column to 1 To do so we d select the column and then fix up the dialog box so that it looks something like this 2003 2011 Curtin University West Australia 218 Help file for Lertap 5 Find and Replace Find Replace Find what F No Format Set Format v No Format Set Format v Replace with 1 Within Sheet Match case Match entire cell contents Search By Columns E E Look in Formulas Options lt lt Replace All Replace Find All If we dared to click on Replace All Excel would dutifully find and replace all Fs with 1s i
191. e the tropics lie in a northerly direction Related titbits A paper which has more on the development of macros may be admired at http www lertap curtin edu au Documentation AngelLearningLertapMacros1 doc Word file about 170 KB This paper has quite a bit to say about how to get Lertap to use data files created by Angel Learning These macros are discussed in this document ChartChanger1 758 ChartChanger2 72 NumericFilter2 223 ScrunchBossl 2s TotalTest1 2s4 and WrongltemList2 23 4 7 4 Linking to macros The previous topic mentioned that the macros activated from Lertap s Macs menu are stored in a file called LertapbMacroSetA xlam The links from Lertap to the macros in Lertap5MacroSetA xlam are made by recording the names of the macros in Lertap s System worksheet The following screen shot shows how part of the System worksheet looked in September 2010 2003 201 1 Curtin University West Australia 226 Help file for Lertap 5 4 7 2 99 100 Lertap5MacroSetA macros which may be used Text to display as tip for users 101 WrongltemList2 Will make a report of incorrect answers for each student 102 StatsbSortAZ Sort a Stats b sheet 103 NumericFilteri Use a numeric filter to pick out only certain records from thd 104 NumericFilter2 Breakout new workbooks using a column in the Data works 105 Itemani Import an ITEMAN DAT file 106 ChartChangeri Lets you quickly change all charts on
192. e 4 x4 I4 zm Key 2 Come 62 cece st Mays 4 Cee AB Clee 42 re obs sw 4 gt bl csemi Statsiul Statsiulchta PackedPlots2 PackedPloti 4 Ready 23 G sox 2 U e Note items with negative discrimination disc are flagged in the plots by the use of red coloring The item s label will be red and the statistics below the chart will also be red 111 is an example above The Disc values in the charts are brought in from Statsb z amp 9 reports If you have Excel 2010 plot packing may very readily be accomplished with an in built macro called ChartChangers It s activated by clicking on the small arrowhead next to the Res charts option Bn gis Insert Page La C Blank gt Interpret a Histograms R Headers Elmillon zi Scatterplot New menu Run menu MathsQuizForLrtp593 xlsx Microsoft Excel Cm e ER x Delete Version fk Line TZ Spread P Excel Basic options Lertap Home Formul Data Review View TH Sort 4 Move P License v Lelp Develop Add Ins o Lelp A special setting makes it possible to get normal quintile plots and ChartChanger3 s packed plots with just a single click This setting and several others pertaining to the use of ChartChanger3 are described in the next topic amp following page Another way to get packed plots is to use ChartChanger2 Users with any version of Excel are abl
193. e are a few problems which can arise when asking Lertap to ask Excel to make charts Probably the most common of these has to do with Excel running out of chart and font resources as it goes about making its plots For comments on this problem please refer to the topics immediately preceding this one At times Excel seems to gather too much speed when making charts and will forget to apply some of the formatting which Lertap has built in For example the name of the item is always supposed to be in bold face such as Item1 but Excel can at times forget this perhaps on days when it s feeling meek not bold Particularly annoying is the sometimes noted tendency of Excel to scrunch the plots to make them squatter than they re meant to be If you suspect your plots are a bit on the flat side or the fat side or the squat side then do this make them again and see if that doesn t fix things Keep in mind that you can get into Lertap s charts and change them to your little heart s content make new colours put the legend at the bottom rather than the side add or changes titles a good way to pass a rainy day or to avoid doing something more urgent but less fun Be sure to write to us if you d like to talk about charts larry lertap com is us 4 5 3 4 Chart colors It is a fairly straightforward matter to change Excel s standard colour palettes If you d like to change the colours Lertap uses to make its quintile plots histogram
194. e figure in parentheses 496 in this example expresses the lowest score as a percentage of the maximum possible score In this example the maximum possible score was 25 highest The highest score found The figure in parentheses score 9696 in this example expresses this score as a found percentage of the maximum possible score 2003 2011 Curtin University West Australia Input amp Output me median The score which corresponds to the 50th percentile In this example half of the scores are below 12 50 and half are above mean The average of the test scores also called the mean or and the arithmetic mean The mean and the median average will be equal when the scores are symmetric about the mean standard This is one measure of how spread out the scores are deviation If all the scores are the same the standard deviation will be zero If you are familiar with the equations used to calculate this statistic Lertap uses the equation with n in the denominator to derive this figure standard If the students tested are considered to be a sample deviation drawn from a larger population an unbiased estimate of as a the population standard deviation is given by this sample figure If you are familiar with the equations used to calculate the standard deviation Lertap uses the equation with n 1 in the denominator to derive this value variance The square of the standard deviation s
195. e form used to apply for a license 2003 2011 Curtin University West Australia Toolbar and tab 2 Unlock Lertap 28 days left in the trial period i To continue using this trial version just click on the Close button below Use the Help button for more comments if wanted To place an order for a license and an unlock code please email or fax the following Session ID and Computer ID numbers to Assessment Systems Corporation Our email address is sales assess com Our fax number is 1 651 647 0412 Thank you Session ID 323189276 nv7 Computer ID 4247629 nv7 If you have purchased a license and have an unlock code carefully enter it in the box below Then click OK Unlock Code OK NN nee You type the Unlock Code in the box and then click OK If the code is correct another message appears Success Done Thank you Your license is installed If your little daughter happens to sneak up from behind and tickle you causing an error in your entry of the Unlock Code give her a wee kiss and then try again But note three tickles and you re out After three unsuccessful attempts at entering the Unlock Code Lertap resets its counters and internally generates a new Session ID code You l then have to go through this whole process again sending the new Session ID and Computer ID numbers to Assessment Systems 2003 2011 Curtin University West Australia Help file for Ler
196. e old toolbar 8 not the suite of options organized as seen above Lelp exists in three distinct formats as a PDF file as a CHM file and as a website The website is here http ww w lertap curtin edu au HTMLHelp HTML index html The website is always the most up to date version of Lelp To see the date of the version you re enjoying right now be it PDF CHM or website look below next to Last update Please note although this version of Lelp is specifically designed to be a companion to the Excel 2007 and Excel 2010 versions of Lertap some of the screen snapshots found in this document show the old Lertap toolbarl78 at the top instead of the Excel 2007 2010 Lertap tab captured above Please direct questions or comments to larry lertap com Last update 20 November 2012 1 1 Whatis Lertap Lertap the Laboratory of Educational Research Test Analysis Package is a computer program used to process and analyze results from tests and surveys The fifth version of Lertap first released in 2001 is designed to work as an application running within Microsoft s Excel program This document provides some idea of what Lertap does and how to go about getting it to do what it does If you re new to Lertap you might find other resources to have a more introductory flavour For a real quickie why not try these PowerPoint slides Should open in your browser And please note these slides show screen shots from Excel 2010 Exce
197. e scores on the other subtest items the SMC based correlation will always be equal to or greater than the other correlation as it the SMC based value is derived by using the special weights resulting from the multiple linear regression analysis underpinning the calculation of the SMC What do you have to do to get Lertap to produce SMC values Nothing much really whenever you use the Run menu s option to Output item scores matrix the resultant IStats report will automatically include the row of SMC values and the little table with SMC bands as seen above What then is the SMC setting mentioned in the previous topic 93 It s a setting which determines whether or not the diagonal values of the IStats correlation matrix has 1 s or SMCs Why do some users want to have SMCs on the diagonal Often because they re thinking of using the correlation matrix as input to a factor analysis program Tetrachoric correlations Tetrachoric correlation coefficients are computed when two conditions are met 1 the tetrachorics option is set as Yes in the System worksheet zal and 2 Lertap finds that the item scores are just zeros and ones These conditions are in fact easy to satisfy The tetrachoric option s default setting in the System worksheet is No when Lertap is first installed but this may quickly be changed to Yes And cognitive test items are very often scored on just a right wrong basis with one point for a correct answer zero poi
198. e to use it including those with Excel 2011 on a Macintosh It s part of 2003 2011 Curtin University West Australia Help file for Lertap 5 the set of macros found in the Lertap5MacroSetA xlam file This file is a standard Lertap component when you start up Lertap the file is automatically opened and its macros are made ready for use by Excel and Lertap ChartChanger2 is not as simple to use as ChartChanger3 but it does have a bit more flexibility It is accessed via the Macs menu please refer to the Macs menul 223 topic for more specific details on how to get it running Here s one example of what can be done by inviting ChartChanger2 to change the layout of a set of quintiles these charts are based on the MathQuiz dataset X id Br 7 Book2 Microsoft Excel of zs Lertap Home Insert Page Layout Formulas Data Review View Developer Add Ins Y o o gg 9s Lertap5 quintile a charts for workbook Book2 E I1 I2 1 00 1 00 0 80 0 80 1 1 2 0 60 0 60 2 0 40 r 0 40 3 r 3 4 0 20 4 0 20 D gon I e De 0 00 lower 4th 3rd 2nd upper lower 4th 3rd 2nd upper Key 3 Diff 84 Disc 26 Key 4 Diff 97 Disc 12 I3 14 1 00 1 00 0 80 0 80 1 1 0 60 0 60 2 2 0 40 0 40 3 3 i 4 020 A4 0 20 U ET DD EUR E m 0 00 lower 4th 3rd 2nd upper lowe
199. e you d like to get group breakouts 133 with only CA and US selected You could use Recode entering a new code of Exclude for all records without CA or US 2003 2011 Curtin University West Australia Toolbar and tab 213 after which you d go for those breakouts Finally although you know how to use the tst cardl 24 to select only certain Data records and the NumericFilter2 2231 special macro you d like to just delete Data records with say SP in the Language column The Recode option could do it How it works You go to the Data worksheet and click on the Recode a Data column option under the Move menu You indicate the column which has the source data that is the column with the codes you want to work with The Recode macro copies the column to the far right side of the worksheet Note the use of the word copies the macro does not delete or alter the original column in any way Then the macro looks at the first entry in the copied column that is in row 3 of the newly copied column Let s say it finds a value of NZ This little snap indicates what next happens 39 40 a 42 2 0 3 0 NZ 2 0 4 0 CA 2 0 4 0 US Wwhat snew 4 What new code or value do you want to equate to the value of NZ in the original column mooo Cancel The macro adds another new column to the worksheet immediately to the right of the copied column It then asks you what NZ should become You enter EN i
200. ear immediately to the right of the Run menu as seen here Run RQ y The magnifying glass unhides or hides the Sub worksheets It s called a toggle it s an on off switch The Liberty Bell calls in Elmillon the item analysis program To use it you must have a Sub sheet open Say for example that you ve been changing weights in a Sub worksheet called Sub3 You re a good scout you ve gone to Excel s File menu and saved Sub3 after making the changes Now you click on the Liberty Bell In comes Elmillon out come your results a new column is added to the Scores worksheet and then those lovely Stats reports are created Stats3f Stats3b and if Sub3 corresponds to a cognitive test probably Stats3ul too If these Stats3 reports existed before they will be overwritten You bewdy You look at the results Hmmm maybe give more points to option D on item 21 You return to Sub3 scroll down to item 21 and increase the number of points corresponding to option D You save the worksheet You tickle the Liberty Bell Another column is added to the Scores worksheet you can compare the new scores with the last ones And once again you get all the Stats3 reports Another way to go about this make a copy of Sub3 Call the copy say Sub3B Make the changes in Sub3B Maybe even change the Subtest Title at the top so the new score will have a new moniker Then with the Sub3B sheet in focus ring the Liberty Bell You ll Scores again
201. ed below Example C9 Both alt and wts cards in use col c2 c6 sub Name Followup TV9 news quiz Title NewsQuiz key ACCDB alt CDDDC wts 31121 You understand this one don t you The right answer to the first item is A It the first item uses three response codes A B C A correct answer on the first item is worth 3 points Q if answer D on the fourth item how many points do get Two What s the maximum score can get over these five items Eight If answer C on the last item how many points do get None the right answer is B If don t answer the third item what happens get sent home early with instructions to have extra peanuts with my beer In truth nothing A non answer to a cognitive item usually gets scored as a zero Example C10 Using Lertap s Big Gun the mws card col c2 c6 sub Name Followup TV9 news quiz Title NewsQuiz key ACCDB mws c2 1 mws c3 O0 mws c4 0 mws c5 O0 j mws c6 0 1 O This example is really the same as Example 9 We want to ease you into the idea of mws cards by starting with an easy example x y f d g d OOOO C pe m O Fr Me Ys c x ox Keep in mind that the default Res A B C D applies to this example there being nothing to the contrary on the sub card The mws c2 card refers to the item whose responses are found in column 2 of the Data worksheet This is of course the first item Of the four p
202. ed by the number of items in the subtest Title Optional Provides a short label for the subtest score This will appear as a header at the top of a Scores colurm Should be no longer than 8 characters Example Name BeachSur Wt Optional Applies only when there are multiple subtests Determines how the subtest s score comes into the total test Score Example wt 0 75 The order of the control words is not important For example the following two cards accomplish the same thing 2003 2011 Curtin University West Australia Help file for Lertap 5 sub Title NewsQuiz PER SCALE sub SCALE Title NewsQuiz PER The control words may be abbreviated or expanded The following cards result in the same actions sub AFF Name Beach survey 1 Title Beachin sub Affective T Beachin N Beach survey 1 4 Toolbar and tab Here s what the Lertap toolbar looked like before Excel 2007 arrived for Windows users it still looks much like this when Lertap runs on a Macintosh computer Most Windows people now use Excel 2007 or Excel 2010 but the toolbar above is still to be seen if you want it It can be found under the Add Ins tab J ust look el 4 Verdana js Lertap5 xlsm Microsoft Excel cay e Lertap Home Insert Page Lay Formulas Data Review View Develop Add Ins Custom Toolbars You may run many of Lertap s options from this toolbar if you wish However s
203. ed to invert matrices with 50 or more subtest items and a switch was made to the M INV function found in the Foxes Group s package of matrix routines For more about the Foxes Group see the eigenvalue topic Both MINVERSE and M_INV require a scratch area to work with and Lertap uses the IStats worksheet itself for this on slow computers or cases where many items are involved you might see Lertap scratching about The standard setting for SMC calculations is off to activate SMC output change the setting seen in Row 21 Column 2 of the System worksheet If this setting is changed it takes effect immediately However this doesn t mean that any correlations matrices you may have already made will immediately change in fact they won t you ll have to delete or rename the IStats worksheet and then get Lertap to make a new Stats sheet For additional information about Lertap s calculation of SMC values just page ahead to the next topic e4l Other settings in the System worksheet determine whether or not Lertap might add a matrix of tetrachoric correlations to the IStats output and possibly make worksheets designed for export to other data analysis systems There s more about these matters in the following topics It s also possible to get your favourite item analysis package Lertap natch to make a stab at determing the eigenvalues of the correlation matrices it produces This is also discussed in an ensuing topic For
204. ee s e e x s De Le os re n e ECESESES Eus re pes e ir Eee rs ss us e Eas aes ex we n ss es aes 7e t Str ose Sen stata UJ Above average proficiency means that the students selecting the distractor had an average test or criterion score which was above the mean of all the students who sat the test Note that the criterion score may be an external one Let s look at items 114 and 115 in the Stats1f report 2003 2011 Curtin University West Australia 296 Help file for Lertap 5 XJ ld V e ay MathsQuiz xlsx Microsoft Excel c3 B 2 File Lertap Home Insert Page L Formu Data Reviev View Devei Add Ir 9 e s I15 c16 The last option for 114 4 was a distractor an incorrect answer the right answer or answers to an item are always underlined options with no underlining are the distractors This option was selected by 263 students The average criterion score for these students was 7 71 as seen under the avg column This was above the average criterion score for all students which was 7 67 as pictured below 2003 2011 Curtin University West Australia Input amp Output x id 9 Ev 3 MathsQuiz xlsx Microsoft Excel o 5 amp File Lerta Hom Inser Page Form Data Revie View Deve Add V o9 ca gg ES Lertaps5 full item stats for MathsQui created 20 06 2011 Summary statistics number of scores n 999 lowest score foun
205. ee option April 2006 let us Mac your day macros R U version change to 5 6 2 June 2006 3 a new recoder and analysis of variance October 2006 conditional standard errors of measurement now computed version change to 5 6 3 Oct 2006 Ver 5 6 3 Conditional standard errors of measurement are now estimated for cognitive subtests Read about CSEMs with a click here 9 A new supporting document related to the use of CSEMs mastery tests and cut scores is available as a Word document about 300 KB A scatterplot of item difficulty by discrimination now appears at the bottom of every Statsb report With a wee click here 24 you ll see great some examples Jun 2006 June 2006 boys will be girls and at variance Recodes recodes recodes a new option on the Move Menu will be useful when you need to change boys into girls cities into countries and all such A click here 23 will show how The breakouts report has been enhanced it now features an analysis of variance table at the end Have a look 4 2003 2011 Curtin University West Australia R amp R amp R amp R 319 7 1 3 Apr 2006 Ver 5 6 2 April 2006 how about a Big Mac No doubt you ve overheard people talking about the new macros they ve made to customize their copy of Lertap It s true too you can now get Lertap to link to your own macros Read all about it the Macs Menul2a3 7 1 4 Mar 2006 March 2006 adjustments to the MDO and s
206. eet 1 These control cards or lines set up two subtests Different background colors are used below but they re not required The first subtest has 25 cognitive items responses start in column 3 col c3 c27 sub Res A B C D E F Name Knowledge of LERTAP2 Title Knwldge Wt O key AECAB BEBBD ADBAB BCCCB BABDC alt 35423 35464 54324 43344 45546 m oe I JE There are 11 lines in use in this CCs example Four 4 of the lines are comments these are the lines which do not begin with an asterisk The use of comments is entirely optional but they can be real helpful There are two col lines above Each of these defines a group of items which will be processed together as a unit Such units are generally referred to in Lertap as subtests 2003 2011 Curtin University West Australia 24 Help file for Lertap 5 A subtest may be comprised of cognitive items or it may be comprised of affective items You ve already seen examples of the CCs lines used with these two types of subtests We ve pointed out that cognitive subtests will always have a col line and a key line while affective subtests will always have a col line and a Sub line In the example above a sub line has been used with the cognitive subtest for several reasons In this example some of the items had as many as six possible responses that is some of the multiple choice items used in this subtest had six choices or respons
207. en steven The final column of the table gives the number of students in each group 876 and 844 and then for each item the proportion of correct responses for each group over all score levels Because this version of Lertap Ibreaks looks only at items which have been scored on a right wrong basis these proportions are equivalent to classical item difficulty figures Thus for item I1 the difficulty was 39 in the reference group and 40 in the focal group Row 9 above has MH statistics for 11 MH alpha is the common odds ratio a figure derived by forming an average of the odds ratios over all score levels weighted by the number in each group at each score level In this example an MH alpha of 97 2003 2011 Curtin University West Australia Toolbar and tab 153 indicates that the odds favour the focal group but since a value of 1 00 indicates equal odds it s not a big favour MH chi sq is used to test the hypothesis that MH alpha is equal to one in the population from which the two samples of students have been drawn Prob is used to judge the statistical significance of MH chi sq We d generally say that MH chi sq is statistically significant when BOB is equal to or less than 05 and this being the case we would be tempted to say that MH alpha is in fact not equal to one in turn if this is indeed so then we have evidence suggesting that group membership makes a difference the chances of us observing a corre
208. ends on two main factors the type of subtests being processed and the complexity of the item scoring desired We ll spell out the general nature of all of the cards below The topics immediately following provide more exact details on the syntax of each card 2003 201 1 Curtin University West Australia om Help file for Lertap 5 col sub key pol alt wts The basic Lertap control card used and required by all subtests cognitive and affective Each time Lertap sees a col card in the CCs worksheet it thinks Ah ha here comes a new subtest and it expects the user to then indicate the columns in the Data worksheet which are to be processed Click herel 6 to read about a problem which can arise with very long col lines This card is optional for cognitive subtests but required for affective subtests sub cards are used to convey particular subtest characteristics to Lertap such as the name and title of the subtest and the number and nature of the response codes used by the items belonging to the subtest Gives the right answer for the items of a cognitive subtest This card is always required for cognitive subtests but it s not used at all with affective subtests pol stands for polarity that is for plus or minus Not used by cognitive subtests and optional for affective subtests When used it defines the type of scoring to be applied to affective items plus for f
209. ent Lertap reports sheets Scores Breaks and RSAsig Have a _squiz 160 Use this with the Shorts menu option to Make a histogram chart an option introduced Sep 05 220 and you ll wow your audience for sure Sep 2005 September 2005 a new way to make histograms Two options were added to the Shorts menu making it possible to change the number of bars a histogram has and enabling the creation of histograms without requiring the Analysis ToolPak Add In The new histograms referred to as histogram charts are easier to modify Note inserted August 2007 the Shorts menu does not exist in the Excel 2007 version of Lertap The histogram options referred to above are now found under the Histogram topic 2003 2011 Curtin University West Australia R amp R amp R amp R 321 7 1 7 July 2005 Ver 5 5 July 2005 response similarity analysis added version change to 5 5 Added support for those interested in investigating whether or not answer copying or sharing may have taken place during an examination Two new resources are available under the general rubric of RSA response similarity analysis As usual to find out more you d want to caress your little mouse and ask it if it wouldn t mind clicking here 7 1 8 May 2005 A production mode capability was added making it possible to roll right through non stop from the Run menu s two main options Interpret CCs lines and Elmillon item analysis Sett
210. ention of Tukey 1977 f They re a very useful way to visually compare group scores Lertap uses Excel s stacked column chart as the basis for its boxplots The core information for the chart is found in the three Pct rows 25th Pct 50th Pct and 75th Pct The length of each whisker is determined by finding the highest and lowest scores in each group which are not outliers that is not extreme scores Tukey defined the limits for outliers as 1 5 times IQR the inter quartile range Q3 Q1 Looking at the P group in the table above IQR 62 50 or 12 Multiplying this by 1 5 gives 18 scores above Q3 18 and below Q1 18 are Tukey s outliers In the case of the P group any score above 80 Q3 18 and below 32 Q1 18 will be deemed an outlier Many boxplot routines such as that found in SPSS indicate the presence of outliers by showing asterisks above and or below the whiskers Lertap does something different if there are outliers the number of them is shown in the Lowliers and Highliers rows The wee table above says for example that there are 2 Lowliers in the P group You can see them if you scroll up to the top of the worksheet as exemplified here 2003 2011 Curtin University West Australia Toolbar and tab 139 The scores of 26 and 29 are the two outliers for group P The effective lowest score for this group is 39 which is 11 points below group P s Q1 These 11 points are the length
211. er 1 100 35 9 90 1 5 34 36 40 2nd 1 100 32 0 80 1 0 30 32 34 3rd 1 104 28 8 72 1 0 27 29 30 4th 1 100 25 2 63 212 23 25 27 lower 1 100 19 1 48 3 3 amp 5 20 23 everyone 5 504 28 2 70 6 1 5 29 40 This was an upper lower analysis with more than two groups An EC external criterion score Core was used in this analysis The Summary group statistics above are for Core w M4 M Stats3ul ECStats2f ECStats2ul 4 m gt Ready 3 100 Q OFF This snapshot shows how the table at the bottom of an ECStatsul report looks when an external criterion has been used to form the groups 6 4 3 3 2 Mastery mode Mastery mode refers to the situation where some sort of cutoff score has been used to classify test scores into for example masters and non masters or pass and fail or promote and keep back The cutoff score is at times referred to as a criterion as in CRT criterion referenced testing Lertap s mastery mode is activated by using the word mastery in a CCs sub line as seen here 2003 2011 Curtin University West Australia Input amp Output 307 F KH 2 C Fil MNurMasteryForLrtp593 xIsx Microsoft Excel 5 re Lertap Home Insert Page Layout Formulas Data Review View Developer Add Ins V o c gg xs col c3 c62 sub Mastery Title Mnur_M Name Mastery M Nur Licensing E39 key BCDCA BBBCD BDCDA BDABD BBACB B
212. er to get Lertap to process the items as affective ones without this card and without the AFF control word on it Lertap would have tried to process the items as cognitive ones and would have wanted to find a key card with the right answers to each item Lertap will score each item giving a weight of 1 every time it encounters a 1 in the items Data column a weight of 2 for a response of 2 and so on The minimum score on any item is 1 the maximum is 5 2003 2011 Curtin University West Australia Help file for Lertap 5 Lertap will also make a subtest score or scale score for each respondent placing these in its Scores worksheet Such scores are just the sum of the item scores Since there are five items the minimum possible subtest score would be 5 the maximum would be what Yes 25 Not everyone is interested in these scores Many people are but some aren t Lertap makes them and you ll just ignore them if you don t like to score affective items Example A2 Example Al was too simple if we re interested in scores You ve already heard that the second item was to have reverse scoring you didn t know it but the third item was also to be reversed We have need for a pol card col c3 c7 sub AFF pol The pol card is usually optional for affective subtests It s hauled out when some of the items are to be reverse scored The pol stands for polarity The pol card above tells Lerta
213. er will probably resemble this one 2003 2011 Curtin University West Australia Toolbar and tab ome Display current license status Apply for a license Unlock Lertap Transfer the license Click on the menu options shown above for more information or just page forward Note unless you have purchased multiple licenses for Lertap your license is a single user license However you may install Lertap on one additional personal computer provided that your two Lertaps are never in simultaneous use To get a license for your second computer please write to sales assess com when you do mention when you purchased your first license and if you have it the corresponding ASC invoice number It would help if you could also send the Session ID and Computer ID codes for the second computer 4 8 1 Display current status If your computer does not yet have a license to use Lertap a message similar to the one below will appear when you click on Display current license status Lertap 5 license information x You have not yet obtained a license to run Lertap 5 at the moment you re purring along with the trial version There are 30 days remaining in your trial period y y You can apply for a license at any time Once you have one Lertap will run for an unlimited time Without a license Lertap will stop working when the trial period ends On the other hand if your computer has a valid license to use Lertap
214. ere are problems the Analysis ToolPak doesn t seem to work well with the initial release of Excel 2007 You can of course try it but lower your expectations so that you re not disappointed 2003 2011 Curtin University West Australia 4 5 2 Toolbar and tab L This icon makes it possible to compare any two columns from the Scores worksheet in a traditional scatterplot also know as a scattergraph or a scattergram A typical plot is shown below indicating the relationship between a score called Comfort and another score Kwldge Scatterplot LEN 42 00 L e e e ee 40 00 eo eoo e 38 00 4 e oo ee e e t 36 00 eo 2 e e E 34 00 e S E e E e 32 00 eo e e ji e e LJ e 30 00 e ee ee e 28 00 t oe 26 00 1 e 1 00 6 00 11 00 16 00 21 00 Knwldge Lertap s scatterplots are Excel charts They may be extensively reformatted The axes may be expanded the colours may be changed even different symbols may be selected instead of the blue circles seen above How Double click or right click on various areas of the chart when viewing it in its worksheet Or look for help on About charts by using Excel Help Excel also has some standard data analysis tools which may be quickly applied to scatterplots How about trying to superimpose a linear trendline over the chart and getting the value of R squared to display 2003 2011 Curti
215. ere s a fair smorgasbord of options which control how Lertap goes about its RSA stuff Look at the following rows from the System worksheet as captured in January 2003 2011 Curtin University West Australia Toolbar and tab X D Wew XP fX Bun Shorts o dab Mover License Lelp 2006 These are Lertap5 system settings Change them only if you understand them Refer to Lelp for assistance Lelp is online at Allowed Usual 2 www lertap curtin edu au HTMLHelp HTML index html settings setting 25 Should an RSA worksheet be created 26 Cutoff value for Harpp Hogan statistic 0 7 to 2 5 1 5 9 otoo 8 20to100 so yes 0 to 90 31 Maximum test score for RSA 10 to 100 System Settings Comments 4 Data CCs System Syntax Problems Should an RSA worksheet be created If this option is set to yes Lertap will produce a worksheet called RSAdatal whenever the Output item scores matrix option is selected from Lertap s Run menu This is the core worksheet for all of Lertap s RSA calculations If Lertap is running in production mode there will be one RSAdata worksheet for each subtest Once an RSAdata worksheet has been created the Response similarity analysis RSA option may be taken from the Run menu It is this option which produces Lertap s RSA reports Cutoff value for the Harpp Hogan statistic This refers to the H H index Harpp and Hogan suggest a minimum of 1 5 for t
216. ertap 5 4 2 5 to the workbook as a new worksheet called Sorted Then Excel s standard sort criteria box appears and the stage is set sorts are made using this new worksheet After a sort has been made may another sort be requested Yes There are a couple of ways to make an additional sort First the Sorted worksheet could be further A m H sorted by going directly to Excel It has a Ze icon found on Excel 2007 s Home tab You might want to use Excel Help if you re unsure about using Excel sort Another way to sort a second time is to delete the Sorted worksheet and then use Lertap s 44 icon again Or instead of deleting the Sorted worksheet it could be renamed after which Lertap s 4 icon will be happy to once more do its job At all times care should be taken to see that the Scores worksheet itself is never sorted Lertap needs to believe that there s a one to one correspondence between the records in the Data worksheet and the records in the Scores worksheet This will not be the case if the Scores worksheet is sorted and it s precisely because of this restriction that Lertap makes the Sorted worksheet for users to work their sorts on Line grapher ix This option is a little gem Here at Lertap HQ there s often a real urge to make line graphs from Lertap reports For example we will commonly get into a Stats1b report and graph item difficulties and also item discrimination coefficients We ll use the Run menu to O
217. es 90 multiple correct answers 231 multiple correlation 94 multiple regression 94 mws lines 231 N Name 23 29 36 74 names of worksheets 258 NBME 321 NCCA 282 306 309 Nelson 14 326 New menu options 87 new score 219 new scores 323 new variable 216 no flag 271 nomenclature 258 non response 82 nonresponses 321 normal distribution 271 NORMINV 97 NORMSDIST 97 Notepad 98 112 null hypothesis 131 number of alternatives 321 number of factors 106 number of groups 198 number of options 321 number right scoring 309 0 observed scores 282 obtaining Lertap 3 odds ratio 151 Office Button 88 omit 46 omitted items 321 323 2003 2011 Curtin University West Australia Index 335 on the fly 118 OpenOffice 2 optical scanner 290 option trace lines 183 options 248 order of CCs cards 74 ordering Lertap 241 other 103 290 299 other response 46 48 other 29 36 245 325 outlier 118 outliers 136 170 output 253 258 overview how Lertap works 19 P p 53 58 299 p 0 306 packed plots 299 PackedPlots worksheet 189 packing plots 186 page margins 189 page size 189 pairwise 58 61 parallel forms 73 282 partial credit 29 part whole inflation 61 92 271 pass fail 306 pb r 53 271 299 p comp1 106 PDF version 12 Pearson 94 Pearson product moment 277 pedigree 326 PER 29 36 48 74 percent 106 percentage 48 p factt 106 111 phone 14 pickable for quintiles
218. es The Res A B C D E F declaration tells Lertap this Without an Res declaration Lertap assumes Res A B C D the default set of item response codes for cognitive items The default set for affective items is Res 1 2 3 4 5 The key line gives the right answer for each of the 25 items in the first subtest The alt line tells Lertap that the items used a different number of the six possible responses For example the first item used just 3 responses they would be A B and C the first three characters found in the Res declaration The second item used 5 responses A B C D and E Only two of the 25 items made use of all 6 possible responses The Name declaration provides a brief description of each subtest while the Title declaration gives a short title When Name is used some of Lertap s reports display Name on their top line Title when used appears in some reports as a label for subtest scores Subtest scores What are they Well as an example on a multiple choice test people usually get one point for each right answer On a 25 item test or subtest it would be possible to get a score of 25 assuming one point for each correct answer Lertap permits the right answer to have any number of points and it even allows the wrong answers to have points too sometimes wrong answers are penalised by assigning them negative points Let s look now at the affective subtest defined above The col line points to 10 ten
219. es 5 6 7 8 9 scoring weights are 1 00 2 00 3 00 4 00 and 5 00 If Resz 1 2 3 scoring weights are 1 00 2 00 and 3 00 If Res w x y z scoring weights are 1 00 2 00 3 00 and 4 00 Example A8 The stage is set we now introduce the most powerful card in Lertap s mighty arsenal the mws card Example A2 above looked like this col c3 c7 sub AFF pol We ll knock out the pol card and instead go with these CCs cards col c3 c7 sub AFF mws c4 5 00 4 00 3 00 2 00 1 00 2003 2011 Curtin University West Australia Help file for Lertap 5 mws c5 5 00 4 00 3 00 2 00 1 00 The mws cards explicitly apply scoring weights for the two items whose responses are found in columns 4 and 5 of the Data worksheet They re saying that the first response code in the Res declaration is to get a weight of 5 00 while the last response code is to get a weight of 1 00 There isn t an Res declaration on the sub card is there No In the absence of one Lertap assumes Res 1 2 3 4 5 The two mws cards have reversed the scoring for the second and third items respectfully located in column 4 c4 and column 5 c5 of the Data worksheet Why aren t there mws cards for the items in c3 c6 and c7 There really are if you could look deep inside the heart of Lertap you d see that it s effected these statements mws c3 1 00 2 00 3 00 4 00 5 00 mws c6 1 00 2 00 3 00 4 00 5 00 mws c7 1 00 2
220. es declaration below have a look col c3 67 sub AFF Res A B C D E Title Beachin pol Lertap assumes that affective items have five options with the response codes corresponding to the options being 1 2 3 4 5 The Res declaration on a sub card tells Lertap this but when Res 1 2 3 4 5 there s no need to explicitly say so Lertap assumes Res 1 2 3 4 5 for affective items For cognitive items Lertap assumes Res A B C D If this isn t the case an Res declaration is required In this example the five affective items have five options with the response codes for the items being A B C D E It s as if the items had this sort of format 4 A few Emu Exports a day keep the doctor away A Strongly disagree B Disagree C Undecided D Agree E Strongly agree If you had no trouble with this example try the next one Sooner or later we ll stump you Example A7 Here s a big example getting into the real intricacies of Lertap and item weighting col c3 c7 sub AFF Res 5 4 3 2 1 Title WAuni pol Here we re still in a situation where the items have five options and the response codes are back to 1 2 3 4 5 However now a response of 5 is to get a weight of 1 a response of 4 a weight of 2 a response of 1 a weight of 5 It s as if the items looked like the one below where the Likert scale has been re ordered so that strongly agree is the first option very common
221. es a z score to be computed and added as a column in the Scores report Title Optional Provides a short label for the subtest score This will appear as a header at the top of a Scores column Should be no longer than 8 characters Example Name SocFinal Wt Optional Applies only when there are multiple subtests Determines how the subtest s score comes into the total test score Example wt 0 5 2003 2011 Curtin University West Australia CCs details AFF Required The appearance of the AFF control word on a sub card is the only way Lertap knows a subtest is of the affective type sub control words for affective subtests MDO Optional Gets Lertap to turn off its missing data item scoring Causes a non response to be scored as zero points A click here 48 will whisk you away to more about MDO Name Optional Whatever is found between the parentheses is used as a header on some of Lertap s reports such as Stats1f and Stats1b While the header can have any length something less than 40 characters is best Example Name WA Maui beaches survey January 2004 PER Optional Causes a percentage of maximum possible score to be created this will appear as a column in the Scores report Res Required if the default Res 1 2 3 4 5 setting is not appropriate SCALE Optional Causes a new score to be computed and added as a column in the Scores report The new score is the original score divid
222. es us with the item keys the correct answers There is one key per item of course and above each key has been recorded in a single column These keys will go into the key line 27 used in the CCs worksheet But does a key line have the correct answers spread over columns as they are above No No indeed We want to see something like the key line seen here 2003 2011 Curtin University West Australia 232 Help file for Lertap 5 col c2 c352 sub res A B C D E key CCDABADACCADCCADBABBABEADBBBDBEABBDBBCBEDCCCBDBI What do we need to do Concatenate the columns with the keys Naturally Concatenate means to join In the very technical terms sometimes found in Lertap documentation we could say that we need to scrunch the columns with the keys Enter the ScrunchBoss macro Here s how to use it 1 Last Distractor Letter E Answer Letter C l Exam Item Number 1 Student 1 A Start by selecting the first key In this case it s found in row 2 column 2 as indicated above 2003 2011 Curtin University West Australia Toolbar and tab 233 t glij Histograms Moves 1 LA Scatterplot Copy a Data column to the Scores worksheet b Res charts 7 Copy a Scores column to the Data worksheet Graphics trio Copy a Data column to a DAT worksheet eo 16 17 18 19 20 Recode a Data column Apply a special Scores worksheet formula Update Scores
223. est score found 4 2 highest score found 100 0 median 52 1 mean or average 52 5 standard deviation 28 8 standard deviation as a sample 29 0 number of subtest items minimum possible score maximum possible score iii reliability coefficient alpha index of reliability standard error of measurement 8 4 v M 4 gt Dl Statsib csemi lt Statsiul Stats2f Stats2b i 4 il gt Ready F3 EB DJ 100 2 9 45 With 24 subtest items coefficient alpha has the unrounded value seen in the Formula Bar 0 91556 We could have also excluded Q4 found in c6 of the Data sheet pertaining to this example by using exc c6 c24 Related tidbits You will no doubt have noticed the Wt 0 settings on the sub lines seen above Whenever the CCs lines reference more than one subtest Lertap is inclined to make a total score by adding up the subtest scores This Total score will then appear in the Scores 2 worksheet To keep a subtest out of the total score Wt 0 is used If all subtests have Wt 0 then the total score will not be made 2003 2011 Curtin University West Australia 20 Help file for Lertap 5 6 4 3 2 Brief item statistics Statsb reports are brief versions of the information found in Statsf reports Fr X lg 0 C Be 7 MathsQuizForLrtp593 xisx M c 8 Z E3pEE e sms os t Eee es ux t
224. ethod to index item quality When more than one subtest is involved in a Lertap analysis there will be more reports two or three for each subtest Thus we will have Stats2f and Stats2b and maybe Stats2ul corresponding to results for the second subtest and on a really lucky day with three subtests we ll be able to bask in the light of Stats3f Stats3b and well surely you get the idea The number in the report name refers to subtest number You ll now be keen to page ahead getting more detailed information for each of Lertap s Stats reports But before you do note there s a top flight manual about this stuff a real printed book whose battery lasts forever and ever something which can even be read in the brightest of sunshine withstanding coffee and tea spills without crashing You can even drop it or run over it with your cycle or the baby s perambulator Old technology but rock solid And it doesn t assume you re full bottle on statistics it holds your hand with loving humor gently nudging you along ever so gently and respectfully Read more about the manual here and do try and read it it covers Lertap s various statistics reports in more detail and has several practical examples 6 4 3 1 Statsif The Statsf reports have four sections By far the longest section is that which has the detailed item statistics Here you will find a table of results for each test or subtest item as exemplified below 2003 2011 Curtin
225. ettings Should an RSA worksheet be created Cutoff value for Harpp Hogan statistic Minimum EEIC value D to 20 Mark all records as pickable for RSA Run in production mode Include histograms in production mode 31 Include response charts in production mode 32 Include items scores matrix in production mode M 4 M N Comments Data CCs System Syntax f OldCCs Problems The snapshot below indicates how the RSAdata worksheet will generally look 2003 2011 Curtin University West Australia Dow Help file for Lertap 5 DX G New fX Rune 2 Shorts 4 is Ig Movet License Lelp 4 c 9 DataRow3 CCDBA ACA C A D BDA EEBF 3i DataRow4 B E BD D BDBE 12 13 0 Jl 26 DataRow5 C D F D AA DC CE 13 11 1 27 DataRow6 ETC IT AA CABDBB 11 13 1 21 DataRow B CA ARA A DAB E 14 10 1 5S DataRow8 B en eee ee eee 47 DataRow9 C ADA BA B C B B 14 10 1 42 DataRowiO D EE UU TI AB 20 4 1 ssiBateBusii 2 2 2 D RH 20 1 CRDPTTUISSTIEPeENUCHMMUE HMM B 24 0 20 DataRowi3 BD B C Ab e CAA Cr 12 31 2 41 DataRowi4 EC CNN EH 2T 1 23 DataRowi5 CC AA C AA B B 15 1 Af The RSAdata worksheet is made to conform to a format used by Wesolowsky s SCheck program Refer to the references for the appropriate citation to Wesolowsky s work in this are
226. evel is in use includes the extra information seen here under the Variance components section The index of dependability is a reliability like index which will have it lowest value when the cutoff score equals the mean of the test scores It is a measure of classification 2003 2011 Curtin University West Australia Input amp Output EN consistency also referred to as decision consistency approaching its maximum of 1 00 when the test is perfectly accurate making no errors in placing students in the two groups masters and others The Prop consistent placings is another measure of classification consistency or decision consistency an estimate of the proportion of students who have been correctly classified as being either a master or an other In this case it s 0 810 suggesting that 8196 of the students have been correctly placed Of course the implication is that 1996 have been misclassified we have called some students masters when they re not and vice versa As there were 1 769 students in total an estimated 1 433 have been correctly classified but 336 have not These measures of classification consistency are very practical estimates of the precision of the testing process It is possible for a test to have low reliability but yet have good classification consistency Related tidbits NCCA the National Commission for Certifying Agencies allows for decision consistency estimates to be reported for a
227. external criterion analysis Lertap s item correlation coefficients are always corrected for part whole inflation sometime referred to as part whole contamination to see what they d be without such correction define a subtest s score as found in the Scores worksheet to be the external criterion An example of the output corresponding to an external criterion analysis may be seen via a click here z7 Related tidbits See the Using an external criterion section of the manual s Chapter 8 Also see Using Lertap in a Test Validity Study a 12 page Word document available via the Internet click here if you re connected 4 4 4 Output item scores A person gets a score for each item s he answers For example for a cognitive item the custom is to give one point for a right answer and zero points otherwise For affective items each possible response usually has a certain number of points associated with it Strongly agree for example might equate to a score of 5 points while strongly disagree might be made equal to just a single point In Lertap a person gets a score on each item even when no answer is given For cognitive items a non response usually equates to zero points while for affective items Lertap will apply MDO the missing data option 48l to non responses MDO usually equals the mean of the possible scores for an item for example if the possible range is 1 00 2 00 3 00 4 00 5 00 then MDO 3 00 note
228. f 44 Disc 24 Taking the number of groups down to 3 with about 50 students in each results in smoother plots in this case with three groups these plots would be called terciles 2003 2011 Curtin University West Australia 200 Help file for Lertap 5 or tertiles i54 i55 i56 M IDEM Do 1 00 r 1 00 0 80 0 80 0 80 j 1 1 0 60 0 60 0 60 L2 4 L 2 2 3 r 0 40 3 r 0 40 3 0 40 4 0 20 4 0 20 4 0 20 other 9 00 ather L 0 00 other 0 00 lower 2nd upper lower 2nd upper lower 2nd upper Key 2 Diff 62 Disc 18 Key 4 Diff 29 Disc 24 Key 1 Diff 44 Disc 24 Lertap makes two types of quintile plots quintile a and quintile b as discussed earlier s3 The setting in row 14 of the System worksheet controls which of these plot types is produced first The row 15 option in the System sheet when set to yes gets Excel to add a data table at the bottom of each plot Here s a quintile with table 2003 2011 Curtin University West Australia Toolbar and tab Ed Microsoft Excel QuintilesNone10Feb05 xls EE The information in the data table is the same as that found in the Stats1ull 2 report for the respective item So much for the row 15 option What s this setting in row 16 Mark all items as pickable for quintile plots To understand this option have a
229. f things for example it creates test scores putting them in a new worksheet called appropriately Scores It also creates a variety of statistical reports giving these names such as Stats1f Stats1b Stats1lul Stats2f and Stats2b The partial screen snapshot below shows what the worksheet tabs will look like after this step WN Data F OCs Fi Frac z Scores x Statsif Stats 1b A Stasi Fi Stats2f Fi Stats2b J The various Stats worksheets are usually why people run Lertap they give item and test statistics Statslf provides a complete or full item and test analysis report for the first quiz or test giving quite detailed information for each test item and presenting several mini reports with overall test statistics The Stats1b report is a briefer summary of the item statistics found in Stats1f it s easier to read If the test being analyzed is a cognitive test Lertap generally produces a third report Statslul the ul means upper lower referring to a method of item analysis preferred by some Lertap produces more reports than many people want For example the b Stats sheets such as Stats1b and Stats2b are redundant in that their information is also to be found in the respective f sheets such as Statslf and Stats2f Some users make little use of the b sheets preferring to drink their cup of tea with the f sheets On the other hand some Lertap users find there to be too much information in the f sheets
230. f this item s options 1 and 4 The first of these 1 is the keyed correct answer selected by 473 students whose average criterion score avg was 6 50 well below the overall criterion average of 7 67 Option 4 a distractor was selected by 459 students and their avg score was well above the overall criterion average of 7 67 111 has been mis keyed An error was made when the key line for this test was entered in the CCs worksheet x MathsQuiz xlsx Microsoft Excel c3 B 3 col c2 c16 sub res 1 2 3 4 title MathsQuiz key 34142 13234 Qa 13 This sort of error is easy to fix In this case we d change the 1 circled in red to 4 After doing so and after once again running the Interpret and Elmillon options 111 s entry in the Statslb report was clear of flags in the column 2003 2011 Curtin University West Australia Input amp Output 2 X id 9 C EM E Book3 Microsoft Excel ca B Lert Hor Inst Pag For Dat Rev Vier Y o cg X fas oe t gt fa som es E vs ee e B Des us e us e Eu ses fa rss an rs a gt bi Statsib csemi d 2D 100 re ORE Correcting mis keyed items should increase the test s reliability estimate which it did for the test featured here the reliability went from 0 68 to 0 80 Related tidbits Flags are also waved in Stats1f reports where they appear in the right margin Read more z7 Ho
231. four items are to be reverse scored This subtest has 10 items so there are 10 c or minus symbols shown on the pol line In fact for this subtest four items are forward scored while six are reverse scored Lertap s forte is in the flexibility it provides for item scoring Any response to any item can have any weight that is any number of points In the three examples we ve presented thus far including the one above items are being scored in a conventional manner Departures from normal are supported by the use of other CCs control lines 2 such as wts and mws Are we going to get away without talking about the Wt declaration seen on the two sub lines No siree Bob here goes whenever multiple subtests are scored Lertap will add up all the subtest scores to make a Total or composite score for each person Each subtest ordinarily comes into the composite with a weight of one 1 to keep a subtest out of the total score Wt 0 is used In the example above both subtests have been given a weight of zero and Lertap will not make its Total score One final point which people often ask about there are spaces in the key line above in the alt line and also in the pol line There s a space after every five characters in each of these lines Why Simply to make the line a bit more legible The spaces are not required Is the example above a common one Yes and no Yes in the sense that Lertap users frequently have
232. g example 8 The weather in the southwest capes region of West Australia is rotten e wunderb ar There are seven blanks above If someone clicks the fourth blank the centre one their response will be entered in the Data sheet as a 4 and the mws card above will have their response scored as 0 00 points Selecting the first blank will see a 1 entered on the Data sheet because it s the first blank and a weight of 3 00 points will apply How many points if someone ticks the sixth blank 2 00 2003 2011 Curtin University West Australia Help file for Lertap 5 Example A11 Enter the alt card col c3 c12 sub AFF Res 1 2 3 4 5 6 7 Title CapesAtt alt 55555 77777 To try and explain this set of cards we ask you to imagine that we ve given a survey with ten items The first five items are of the Likert style and use response codes of 1 2 3 4 5 The last five items employ the semantic differential style and use response codes of 1 2 3 4 5 6 7 The alt card tells Lertap this It says that the first items use the first five response codes seen in the Res declaration while the last five items use seven of the response codes which of course is all of them It s still possible to add a pol card in cases like this as shown here col c3 cLT2 sub AFF Res 1 2 3 4 5 6 7 Title CapesAtt alt 55555 77777 pol ttt But there s a probable problem here Users of semantic differenti
233. g quickly the shots you see in this document will not have a 100 correspondence with the screens seen in the most recent version of the Lertap 5 system This is particularly true since Microsoft released Excel 2007 for Windows This version of Lelp the one you re looking at right now was made from the previous version Lelp for Excel 2003 Windows and Lelp for Excel 2004 Macintosh Some of the screen snapshots found in this version of Lelp display information in the format found in former versions of Excel Related tidbit Copies of the various versions of Lelp are freely available at the website http www lertap curtin edu au Documentation htm 1 8 1 Differs from manual Lelp that is this document serves a variety of purposes Above all Lelp effectively updates the manual For a summary of the major changes made since the manual was printed see Lelp s Revisions s7 topic A number of supporting documents have been made available to users since the manual came to light and Lelp has links to them often as URLs found under the Related tidbits section at the end of some topics All versions of Lelp chm PDF and website may be viewed online The manual is also available in an electronic form but unlike Lelp it s not cross referenced it has not been formatted as hypertext it s not as easy to jump from topic to topic in the manual as it is in Lelp Both the manual and Lelp have numerous examples The ones in Lelp tend
234. ge c x Delete a Version Lertap Home Vf Sort fly Line Excel _ Blank R Headers 5 Copy New menu fx Spread Basic options Run men cB Elmillon More u MathsQuizForLrtp593 xlsx Microsoft Excel Insert Page La Formule Data Review View Develop Add Ins amp XD Interpret ai Histograms 2 Scatterplot Move P License Lelp f menus What we ve just described is a two step process You can make it a single step by going to the System worksheet and changing Row 60 s Present setting from no to yes Once this is done a new worksheet PackedPlots will accompany the StatslulChta report 61 62 63 64 65 66 67 Run the plot packer in interactive mode Page settings for printing the packed plots Page size Header margin Top margin suggest 5 for Letter 1 5 for A4 Bottom margin Footer margin 68 Left margin suggest 5 for Letter 1 for A4 69 Right margin 70 Default number of charts per row 71 Default number of blank lines between rows V 4 gt b Comments Data lt CCs System Syntax Ready amp OldCCs yes no yes A4 or Letter Oto2 Oto2 Oto2 Oto2 Oto 2 2to 10 iF 4 m G GI 90 id I C gl Lertap5 xlsm Microsoft Excel oB Lertap Home Insert Page Layout Formulas Data Review View Developer Add Ins VY c gg s 1 2 3 E E These are
235. gic to analyse our data In order to do this we ll need to enter some lines in the CCs worksheet Put on a fresh pot of coffee pour yourself a cup and have a look at the examples below Example C1 This set of two CCs cards might be all that s required to get Lertap to analyse the data col c2 c6 key ACCDB Anyone who selects A on item 1 C on items 2 and 3 D on item 4 and B on item 5 will get a score of 5 one point for each answer Why There are five items the right answers the keyed correct answers are shown on the key card above Unless you say otherwise Lertap awards one point for each right answer Example C2 We ll add a sub card in order to have Lertap label some of its reports col c2 c6 sub Name Followup TV9 news quiz Title NewsQuiz key ACCDB The sub card is usually optional for cognitive subtests Here we re using one just to give a Name and a Title to the subtest The Name will appear as a heading at the top of Lertap s item analysis reports such as Stats1f 28 and Statslb The Title will appear at the top of one of the Scores s columns making it a bit easier to interpret the Scores report The Name can have any length but Title should be no longer than 8 characters If Name and or Title are not given on a sub card Lertap will create default labels Name Test 1 and Title Test1 Example C3 Next we ll use a sub card in order to turn on certain scoring options col o2 c06
236. h I no longer have true achievement quintiles Sure SAQ And can get these marvellous graphs even when my Stats1b report is based on a mastery test Yes go for it Master Related tidbits There s a paper a best seller which has a lot more to do about using quintile plots http ww w lertap curtin edu au Documentation UsingLertapQuintilePlots pdf pdf file about 400 KB How to print quintile plots is discussed in this topic 4 5 3 2 1 ChartChangers It s possible to make quite a number of changes to Lertap s quintile plots or charts An oft used built in option is a switch which turns on or turns off the data table from which the trace lines in the charts are derived How to flick this switch See the quintile options s topic Packing the plots The screen snapshot below indicates how packing the plots may lead to a useful gestalt a condensed overview of item performance This snapshot displays plots for the first twelve items from the MathsQuiz dataset 2003 2011 Curtin University West Australia Toolbar and tab 29 gi MathsQuizForLrtp593 xlsx Microsoft Excel Y Lertap Home Insert Pagelayout Formulas Data Review View Developer Add Ins 2 a ae ae ewe 4 Fs hi dw Mey 2 Cte 84 Cece 26 o B x vQ PX L J zi oe dw LST ewe zi hi sw Key 1 Dee 28 Osce 47 be 2 rs 24 cw oe oe ed wwe gt Kaye DUM 47 Diac e 22 lt e
237. hat about the joint operation of MDO and did not see What happens then Got time to look at results for two items delivered to 421 students over the internet using the Test Pilot system from McGraw Hill c10 Q50 Option c11 Q80 Option 1 2 In this example a blank was used as the did not see code and 9 as the code for missing data On Q50 three students failed to provide an answer while 106 students did not see it Test Pilot presented Q50 to 315 of the 421 students Q80 was presented to 320 students and they all answered it Look at the corresponding Statsf summaries 2003 2011 Curtin University West Australia CCs details Q50 option wt n J 0 00 114 e 1 00 94 3 0 00 62 4 0 00 42 other 0 00 3 Q80 option wt n 1 0 00 124 2 1 00 196 other LertapS full item stats for EE 101 T3 exam created 20 03 2006 b r b r 0 37 0 15 0 20 0 30 0 25 0 33 0 20 0 15 0 22 0 13 0 06 0 10 0 26 D 10 0 37 b r b r 0 39 0 27 0 35 0 61 0 18 0 23 0 24 ava 2 27 37 0 20 30 74 0 52 26 87 0 31 27 57 0 16 23 67 0 99 avg 2 26 71 0 34 29 47 0 22 The statistics for both Q50 and Q80 have been computed by excluding the students who did not see the items and by also excluding those with missing data Were you to sum the p values for Q50 s four options you d have 0 37 0 30 0 20 0 13 or 1 00 100 The 0 26 on Q50 s other line is the proportion of students wh
238. having a printed copy of the manual We used to ship Lertap with the System worksheet hidden in order to discourage less experienced users from fooling around and getting into trouble Trouble is especially likely to result if the settings in Rows 3 4 and 5 are altered 2003 2011 Curtin University West Australia 250 Help file for Lertap 5 If one of the Present Settings is changed sometimes the Lertap5 xlsm workbook has to be closed and then re opened before the change will be effected However numerous settings take effect immediately and do not require closing Lertap Among these are the settings seen above in Rows 12 through 17 and in Rows 19 through 24 if you alter the column 2 setting in one of these rows the change will take effect without having to close Lertap and then re open it But don t read too much into this If for example you change the item difficulty calculation method via row 19 the item difficulties you ve already obtained will not change until you once again use the Run menul s Note 1 inserted May 2005 a production mode capability was added to Lertap with relevant options set in rows 26 through 29 of the System Worksheet A simple click herel 941 will take to you the corresponding topic The row position of these options changed after the following was inserted Note 2 inserted July 2005 support for response similarity checking is s he cheating on you was added with relevant opti
239. he Lertap tab for Excel 2007 there s a section for Other menus Move v P License 7 Lelp Other menus Two sets of menu options are available Move and License Click on these options in the box above or simply page forward to read more about them The lelP option opens or should open the very document you re reading now but on the first pagel 1 4 6 1 Move Menu Among other things the Move menu permits columns to be copied from to Lertap s main information worksheets A column in the Data worksheet may be copied to the Scores worksheet or when it exists to the DAT worksheet The Data worksheet 2541 is created by users The Scores worksheet is created by Lertap when the Elmillon 94 program is run The DAT worksheet 981 in turn is created when the Item scores and correlations option is selected from the Run e 3 menu Shifting data around from one worksheet to another was originally the Move menu s main purpose These days the Move menu will do quite a bit more as you ll see if you click on the options seen in the box below or just browse forward Copy a Data column to the Scores worksheet Copy a Scores column to the Data worksheet Copy a Data column to a DAT worksheet Recode a Data column Apply a special Scores worksheet formula Update Scores worksheet correlations Lelp Macs gt The manual briefly discusses the Move menu in Chapter 10 in the printed manual see p 173 H
240. hich may be used to rename worksheets please refer to Excel Help 4 2 2 Yellow smiley face Use this icon to get the Lertap 5 version type and date of generation Lertap 5 item test and survey analysis system Version Mo 5 25 Version type ASC Version date Friday 08 08 2003 day month year Copyright Curtin University of Technology 2001 2002 All rights reserved Warning This computer program is protected by copyright law and international treaties Unauthorised reproduction or distribution of this program will be prosecuted to the maximum extent possible under law The screen snapshot showing above indicates that the ASC version of Lertap 5 was in use The version number is 5 25 created on the 8th of August 2003 This snapshot was taken when Lertap 5 was running with Excel 2002 working within Microsoft s XP operating system The snapshot will look a bit different when Lertap is running on other versions of Excel 2003 2011 Curtin University West Australia 2 Help file for Lertap 5 4 2 3 Lertap 5 version data may also be obtained by clicking on About this version an option which displays when you click on the LertapHelp menu drop down from Lertap s toolbar Data entry aid T this icon when clicked on activates The Spreader a powerful data entry aid To understand how it works look at the screen capture below 56 60 A EDCIBIBLBIELPBIB D B B A 57 56 AILIE CIB IGIB EI
241. his index Minimum EEIC value EEIC means exact errors in common The recommended minimum is 8 a value which may be lowered to 6 or 7 whenever the number of test items is less than 40 Minimum sigma value to be an outlier Sigma refers to how far a student pair s probability measure is from the mean of the distribution of probability measures Sigma is a z score If the probability measures are normally distributed a z score of 45 0 or 5 0 more is a very rare outcome indeed only 0 0000003 of the area under a normal distribution lies beyond a z score of 5 0 In practical terms an exam given to three thousand students will produce about five million pairings of students if the students have not colluded in their item responses only about two of the student pairs can be expected to have a sigma greater than 5 0 assuming that the distribution of probability measures follows a normal distribution 2003 2011 Curtin University West Australia 12 Help file for Lertap 5 Mark all records as pickable for RSA This option is in fact not yet an option It may be activated at a future date In the present version of Lertap students may be excluded from an RSA analysis by removing the comment the red triangle from their RSAdata records students will also be excluded if their test score does not fall within the range of scores specified by the minimum and maximum test score values set in the System worksheet see immediately below
242. his message may be a bit ambiguous but the new ScheckData DAT file will be on your computer somewhere Please note that this message will not appear if you have Lertap set to run in Production model el Lertap s SCheckData DAT file is ready for input to Wesolowsky s SCheck software The contents of this file are very similar to those of the RSAdata worksheet with commas used to separate information 9 DataRow3 CCDBA ACA C A D BDA EEBE 31 DataRow4 BA A E BDA AD BDBE 26 DataRow5 C D A F D AA DC CE 27 DataRow6 A CA AC AA CABDBB 21 DataRow7 B CA AA A DAB E The information from columns 5 6 and 7 of the RSAdata worksheet is not carried over to the SCheckData DAT records An option on Lertap s Run menu Response similarity analysis RSA will get Lertap to 2003 2011 Curtin University West Australia 1 Help file for Lertap 5 4 4 4 6 use the RSAdata worksheet to feed its own response similarity investigator To read more about this simply click here 3 To read more about the System worksheet give a wee click herel 248 Eigenvalues The eigenvalues or latent roots or characteristic roots of a correlation matrix are sometimes used as a means of estimating the number of factors or components which may underpin a test or a scale There are often times when researchers would like to be able to say that their test is unidimensional in
243. horts mentioned above have not disappeared from the Excel 2007 version of Lertap but they ve found themselves dispersed In particular the line plotter option referred to is now found under the Basic options sa section of the Lertap tab Nov 2003 November 2003 smiles for quintiles This is a BIGGIE a considerable enhancement to Lertap s processing of cognitive test items The upper lower groups analysis module has been expanded so that as many as 5 groups may be processed Two new types of charts are now available plotting item results in some remarkably revealing ways This revision is discussed under the Graphics trio 9 topic Oct 2003 Ver 5 3 October 2003 hope you like roots version change to 5 3 Added support for latent root eigenvaluel iol and SMCI 93 calculations 2003 2011 Curtin University West Australia R amp R amp R amp R 325 7 1 18 Sep 2003 September 2003 to halve and hold forever An ability to create random samples of data records was added as an option under the Run menufs This will be useful if not to the whole world then to those who might use Lertap as a precursor to some subsequent analyses such as perhaps IRT modelling To ignore this revision do not click herel 153 Another option added in September support for those who love tetrachoric correlations 9 and Bilog like data files 98 And late in the month the way The Spreader 84 operates was changed 7 1 19 Aug
244. ia CCs details For example let s say someone scored 10 on our 5 item test Divide this score 10 by the number of items 5 and SCALE 2 00 for this person This might make us think that the person s average response to our items was disagree since disagree had a scoring weight of 2 SCALE can be handy when processing an affective instrument with numerous subscales each scale having a different number of items If SCALE is used on each respective sub card then we can scan any person s SCALEd scores and quickly see their average positions on the 1 to 5 scale used to score each item The manual has a real life example of just such a situation see Chapter 8 where the MSLQ instrument is discussed Example A5 Now we will toss in two more control words for the sub card col c3 c7 sub AFF MDO Title Beachin Wt 0 pol The MDO control word will get Lertap to turn off its missing data option 40 Lertap makes a standard response substitution when people don t answer an item it gives them an item score equal to the mean of the item weights note In our example the item weights range from 1 to 5 the mean is 3 A person not answering an item gets a score of 3 someone declining to answer all items would get a score of 15 on our 5 item scale To turn off this automatic substitution use MDO on the sub card as exemplified above To have more control over how missing data are processed use mws cards w
245. icense fXSpread Excel wj Copy Morey h Res charts Lelp Basicoptions Newmenu Runmenu Graphics trio _ Other menus dm A A NE Sa YY CS 1 Lertap5 Composite Alpha Report created 9 03 2010 2 EE 1 Running Cognitive 1 0 82 Em 2 Hiking Cognitive 1 0 57 Re 3 Football Cognitive 1 1 42 6 4 Basketba Cognitive 1 1 00 B 5 Soccer Cognitive 1 1 05 a e Total Composite 8 28 Coefficient alpha for the composite 0 51657469 The CCs lines corresponding to this output are shown here 2003 2011 Curtin University West Australia Toolbar and tab 235 foal 3 5 10 c12 17 sub Res 1 2 3 4 5 Name Sample Sports Running Title Running key 43222 col c4 c7 c13 c19 c22 sub Res 1 2 3 4 5 Name Sample Sports Hiking Title Hiking key 32354 col c6 C11 c14 c15 c20 Res 1 2 3 4 5 Name Sample Sports Football Title Football key 13352 col c8 c9 c16 c18 c21 sub Res 1 2 3 4 5 Name Sample Sports Basketball Title Basketball rerum E key 51334 nm CJ Q o p c23 c27 14 sub 15 key 22141 Res 1 2 3 4 5 Name Sample Sports Soccer Title Soccer This example involves five cognitive subtests The information displayed in the Composite Alpha Report is derived from various sources The Titles are taken from the CCs lines as are the values seen in the Wt column a subtest s Wt value will equal 1 one u
246. ick 2006 Michaelides 2008 and Zieky 2003 Refer to http lertap curtin edu au References htm Note DIF is sometimes also referred to as item bias now regarded by many as an outmoded term 4 4 7 3 Enhance M H chart Hunting for pictures of DIF differential item functioning Use the IbreaksMH tables Scroll to an item of interest select that part of the scores range of interest and hunt away For example we might look further into the IbreaksMH table for item 135 The chart below was made by selecting 135 s two diff rows starting at a score level of 8 extending out to a score level of 36 Then the Line 8 option on Lertap s Excel 2007 ribbon tab was used you don t have to use that option lt Alt F1 gt is a shortcut which accomplishes the same thing with the lt Alt gt held down press on the F1 key 1 20 1 00 4 80 1 60 Series1 Serie s2 40 20 00 TT TTT TT T T TT TT T T T T T T T T T T 1 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 2003 2011 Curtin University West Australia 156 Help file for Lertap 5 What have we here A conventional Excel Line chart with two series The y axis plots over proportion correct The x axis at this point is just indicating a sequential data point number Things could be improved for sure The Enhance M H charts option on the Run menu can help out Ek Interpret a Histograms Mov dB
247. icrosoft Excel GCdata xls File Edit Yiew Insert Format Tools Data Window Help amp x X New S rZ puny Shorts M fl Move License Lep Y 10 RC 1 5 1 2 1 3 4 5 Lertap5 Scores worksheet last updated on 29 07 2004 Not everyone likes to work with the R1C1 Excel referencing style seen above in the Formula Bar Many users click on Lertap s Excel shortcuts 2s to change the referencing style so that columns are labeled with letters as shown below 2003 2011 Curtin University West Australia Toolbar and tab zm EA Microsoft Excel GCdata xls File Edit view Insert Format Tools Data Window Help A xX x New v a r Run 21 Shorts n Lee ll Move o License Lelpy C3 Y f 10 B345 2 Record No 3 3 0 70 53 00 Notice how the formula has changed to 10 B345 B3 refers to the cell where x the original score is found Both of these formulas formulae say the same thing they just use different referencing styles you can pop back and forth between the referencing styles as much as you wish Okay We ve got a user who wants to make a new score by multiplying the original score Test1 by 10 and adding 5 to the result S he begins by selecting the cell immediately to the right of the first Test1 score and then enters the formula by actually typing it in starting with the equals sign After typing the formula the user presses the Ente
248. ike the U L disc value encountered in a regular Statsiul z report the B disc value for an item depends on the where the cutoff level is it would change if for example we used 8096 as the cutoff instead of 7096 Note there is a slight difference in the way the U L diff value is calculated when a mastery level is used students who do not answer an item or have an invalid response are excluded from the calculation for that item X id te Sis MNurMasteryForLrtp593 xIsx Microsoft Excel ca B z Lertap Home Insert Pagel Form Data Reviev View Devek Add It VY o co pH ES 5 U L stats for Mastery M Nur Licensing E390v6 3 created 9 07 2012 La foe e o foreu aa on o27 ose oue oor 0 00 0 01 0 06 0 00 s 0 05 0 08 0 16 0 01 Summary group statistics n avg avg s d min mdn max masters 808 47 5 79 3 9 42 47 60 others 961 34 8 5 58 5 8 2 s 36 41 everyone 1 769 40 6 68 Jd 2 41 60 This was an upper lower analysis based on a mastery cutoff percentage of 70 Variance components df SS MS Persons 1768 1751 56 0 99 Items 59 3340 98 56 63 Error 104312 18124 56 0 17 Index of dependability 0 799 Estimated error variance 0 003 For 68 conf intrvl use 0 059 Prop consistent placings 0 810 Prop beyond chance 0 617 E ld 4 gt M Statsib lt csemi Statsiul J tj m gt Ready GA 100 2 The bottom of a Statslul report when a mastery l
249. ing short answer questions 69 Scratch worksheet 258 scree plot 106 scree test 106 screenshots 12 scrunch 231 ScrunchBoss 231 Scrunchem 160 Scrutiny 118 second license 238 secondary worksheets 80 92 253 258 sections 79 select 73 212 229 select all charts 186 189 select if 229 selecting cells 84 SEM 282 309 semantic differential 36 Session ID 238 242 set reference style 255 settings 158 248 short answer questions 69 shortcut 155 shortcuts 250 Shorts menu 115 219 324 sig 131 141 sigma 118 319 significance 131 141 similarity check 321 skewness 263 SMC 93 94 106 111 175 322 SMC bands 94 322 SMC calculations 324 SMC setting 94 2003 2011 Curtin University West Australia Index 337 smiles 324 smiley face 81 sorry message 11 Sorted worksheet 83 Sorting scores 83 special form mws alt cards 69 special macros 186 sphere 106 split data set 325 split half 73 split half reliability 282 Spreader 82 325 SPSS 61 111 208 squished 189 SSI 112 standard deviation 282 standard error 151 standard error of measurement 282 standard scores 271 starting 4 Stats 265 Stats worksheets 258 Statsib 14 58 179 180 183 265 290 292 294 Statstb column 231 StatstbCht 179 180 Stats1f 14 53 265 Stats1f sections 266 282 Statsiul 14 60 179 183 250 265 299 StatstulChta 183 StatstulChtb 183 Stats2b 265 Stats2f 265 Stats2ul 265 status of Lertap license
250. ings in this mode also make it possible to roll further automatically getting histograms response charts such as quintile plots and an item scores matrix IStats all without having to wear out your mouse with extra clicks on toolbar options Read more about itl 90 7 1 9 Feb 2005 February 2005 three enhancements EMQs may now be processed with Lertap EMQs are extended matching questions also known as EMIs where the means item A test which makes use of EMQs will typically start out by presenting a series of options up to 26 of them these turn out to be the same as the alternatives or options commonly used by MCQs multiple choice questions What makes EMQs different from MCQs Well firstly EMQs use many more options than MCQs A typical MCQ will have what Four options Five EMQs will have anywhere from 10 to 26 Then secondly EMQ using tests will have several items which use the same set of options This is why the set of options appears before the questions which use them Who uses EMQs They re popular with the National Board of Medical Examiners in the United States and they are used in exams created by the United Kingdoms PLAB the Professional and Linguistic Assessments Board We know that EMQs are also frequently used in the Caribbean particularly in Trinidad and Tobago In practical terms this means that Lertap s Res declaration may now up to 26 entries here s a little example showing an Res
251. ir statistics adjusted so that they re based only on valid item responses This is likely to be a handy revision for survey users Read more about it by clicking herel 48l As part of this revision we modified the CCs lines corresponding to the Lertap Quiz 18 so that they show off the functioning of the new MDO whenever a user takes the cook s tour 18 7 1 13 Jul 2004 July 2004 record IDs amp formula scores Prior to this revision data records with ID information had to satisfy two criteria the ID itself had to reside in either the first or second column of the Data worksheet and the column header used for the ID field had to begin with the letters ID or id or Id or iD Now the first of these requirements is gone Zapped The column with ID information may now be any column This change prompted by a request from Barbara Foster University of Texas Southwestern Medical Center will be welcomed by those who like to put the first item response in the Data worksheet s first column But a warning there s a potential problem with having the first item s responses recorded in the first column of the Data worksheet What if someone doesn t answer an item Some users let a blank or empty column represent missing responses But a blank or empty first column in the Data worksheet has a very special meaning for Lertap it indicates the end of data Users who record item responses in column 1 of the Data worksheet should use a special c
252. irst column At Lertap central when in the process of testing out large new data sets we often insert a blank row in the Data worksheet after row 52 This effectively fools Lertap into thinking there are just 50 respondents remember the first two Data rows are for header information in turn this lets us test our CCs cards faster enabling us to 2003 2011 Curtin University West Australia Input amp Output quickly see if we ve set up the cards required to get the analyses we wanted 6 3 CCs sheet The formatting of the CCs worksheet is relatively ad hoc when compared to the formatting of the Data worksheet For example there s no requirement to have rows with header information We almost always include comments in our CCs sheets rows which remind us what we ve done and when We sometimes even add some sort of colour coding to our CCs worksheets as shown below 1 These control cards or lines set up two subtests Different background colors are used below but they re not required EI 2 3 The first subtest has 25 cognitive items responses start in colum 3 4 col c3 c27 5 sub Res A B C D E F Name Knowledge of LERTAP2 Title Enwldge Wt Fas Fi 8 9 tkey AECAB BEBED ADBAB BCCCB BABDC alt 35423 35464 54324 43344 45546 The CCs sheet above has four rows with comments rows 1 2 3 and 8 Any row which does not begin with an asterisk is considered to be a comment We also like
253. is can be reset quickly as shown below col c2 c6 sub MASTERY 80 Title NewsQuiz key ACCDB The cutoff percentage has now been set to 80 More generally it is possible to have the default level of 7096 set to any value by making a change in Lertap s System worksheet Click here amp if interested Example C5 This example reflects a common situation col c2 c6 sub Res 1 2 3 4 Title NewsQuiz key 13342 The RES control word is telling Lertap that the item response codes are digits not letters Unless you tell it otherwise Lertap assumes that cognitive items have four options with response codes of A B C D If this is not the case you must use an Res declaration on a sub card as exemplified above Note that the key card has been changed if the response codes were digits then the key card will give the digit corresponding to the right answers Also note RES is the same as Res which is the same as res which can even be the same as Responses Lertap really only looks at the first letter of the control words and it doesn t care if letters are upper or lower case Here are some other examples of valid Res declarations Res T F The subtest s key card must contain Ts and Fs Res A B C D E F G H 1 J Res a b c d The subtest s key card must contain lower case letters Res 1 2 3 4 5 6 The subtest s key card must have digits Res 47 B7 E7 Py Not required This is
254. is no variance of scores within the groups no standard deviation We clicked on the charts icon once and got a new worksheet called ECStats1ulChta Next we made the ECStats1ul sheet active by clicking on its tab and then clicked on the charts icon again to get ECStats1ulChtb What did we find Most of the initial items had this sort of quintile a plot 2003 2011 Curtin University West Australia 1 Help file for Lertap 5 E QuintileTests310ct03 xls EC 1 EC 2 Key 3 Diff 55 Disc 39 1 4 2 amp 3 4 1 5 e other M mf ECStatsiulchth X ECStatsiulChta The plot above has lines which seem to be fairly horizontal we d expect horizontal lines if there were no group differences truly horizontal lines indicate that the percentage of people selecting an option is the same in each group meaning that there are no differences among the groups Some readers will recognise this discussion as being similar to that heard when folks sit down with their coffee tea to yak about item bias and DIF differential item functioning Please see the Ibreaks 4 topic for much more about DIF Above the proportion in each group selecting each of the four distractors looks to be about the same except for other the proportion of correct answers Key 3 was about the same in each group perhaps showing a slight practice effect the line rises somewhat as it move
255. iscussion found in this topic is based on results from the LenguaBlg dataset The Core score was used as the external criterion and the second subtest Trial items only was selected for the analysis Note that in this example we have a ECStats2ul report as we re dealing with the second subtest in the dataset xl ld v S cy LenguaBigForLrtp593 xlsx Microsoft Excel ca B z Lerta Home Insert Page Formi Data Revie View Devel Add1 9 o zs Il nna n 2 T2 nnnar n 16 Id 4 gt M Stats3ul ECStats2f ECStats2u Ready 7 ci tg 100 i The groups in an external criterion analysis are formed by using the external criterion score which in this case was called Core Otherwise all statistics are formed in the manner described in the previous topic fz 2003 2011 Curtin University West Australia Input amp Output 305 Note the negative U L disc value for 121 More members of the lower group got this item right than in the upper group You can see this is the corresponding quintile plot 115 116 1 00 1 00 jun A oap A 0 80 paesan B Aee E ar a d C 0 40 r C 0 40 D 0 20 D 0 20 E 0 00 E 0 00 other lower 4th 3rd 2nd upper lt other lower 4th 3rd 2nd upper Key B Diff 2 63 D 12 Key B Diff 2 97 Disc 06 121 123 1 00 1 00 Sr OO 0 80
256. isker html Much more about boxplots or box and whisker plots may be found on the internet and in many statistics texts such as Thompson s2 2006 4 4 7 Ibreaks Ibreaks refers to the process of creating a report or two which summarizes the way groups have responded to test items Ibreaks requires two things a column in the Data worksheet which identifies groups and a Scores worksheet which it can interrogate as needed Scores worksheets are created by the Elmillon e2 option while Data worksheets are created by you click herel2s3 if you need to be reminded about Data worksheets and their proper format Caveat It s entirely possible not to mention common to get Ibreaks to make response charts In the current version of Excel 2007 at September 2009 and also in the current version of Excel 2010 at February 2010 the Excel chart engine was known to have a number of quirks One which affects Ibreaks charts concerns the nature of the code used to denote groups it should begin with a letter 2003 201 1 Curtin University West Australia Dow Help file for Lertap 5 Ca s LrtpQuizNovO6a xlsx Compatibility Mode Microsoft Excel non commerci mx Lertap Home Insert Page L Form Data Reviev View Devek Add h 7 X x Delete f Sort M Slank Y Interpret ill Histograms 7i Move Version fc Line I Headers Elmillon Scatterplot License Tg Spread Excel a Copy M
257. it as seen on the Excel 2003 version in January 2005 the one you see on your computer may differ a bit that is if you try to activate it please note that Macintosh Excel Excel 2004 sometimes has a problem with this toolbar and may fail to display it X New SS puny ER 2X 2 shorts M l Moves Q License Lelp This toolbar has two special icons immediately to the right of Run The first one the magnifying glass unhides and hides the Sub worksheets Of course it s always possible to hide and unhide Excel worksheets this little button simply makes the job 2003 2011 Curtin University West Australia Toolbar and tab 2 easier The next icon to the right the bell is used to run Elmillon for just a single subtest This icon is referred to as the Liberty Bell in the manual s Chapter 10 Elmillon is usually accessed via the Run menu where it s advertised as Elmillon item analysis When you activate Elmillon via the Run menu all subtests are processed Before leading into why these two new icons are sometimes useful to some we should mention that the behaviour of the first icon on the toolbar the X to the left of the yellow smiley face is changed from normal This is the delete worksheets icon Click here 88 to be reminded of how the X normally behaves When used from the advanced toolbar the X does not delete the Fregs Subs and Scores worksheets We should also mention that the behaviour of
258. ith an other declaration as mentioned below under Example A11 Revision note 12 September 2004 the way the MDO option works has changed Now MDO effectively means missing data out when MDO is present subtest statistics are adjusted so that they exclude people who haven t answered items This is further discussed in the following topic 441 The Wt 0 declaration shown above says to Lertap As you go about summing all the subtest scores to make a total score give this subtest a weight of zilch zero What s that you say Well whenever a CCs worksheet has more than one col card we re into a situation where there are multiple subtests We are Sure each col card defines a new subtest Granted there aren t multiple col cards in this example but pretend there were Lertap s standard modus operandi is to add all the subtest scores together making a total score for each person Usually this total is just the sum of the subtest scores much as if Wt 1 had been used on each subtest s sub card The Wt declaration gives you the ability to control the way a subtest adds to the total score Since 2003 2011 Curtin University West Australia 40 Help file for Lertap 5 Wts can be negative you can even get a subtest s result to be subtracted from the scene How you doing This is all pretty straightforward is it not We ll step up the tempo a bit and get into some more advanced matters Example A6 We ve used an R
259. kely when the codes are numbers or single digits such as 1 2 3 And failure is pretty much guaranteed if one of the labels is blank If you get a message from Lertap or Excel concerning this matter go back to the corresponding Breaks report such as Breaks1 and change the codes Note that it s possible to still use digits if you convert them to text an easy way to do this is to simply place an apostrophe before the digits 1 2 3 and so forth Flexibility Once you have one of these boxplots on the screen it s very easy to alter its appearance Right click on the chart and Excel will open up numerous options To set the score range used in the plot right click on one of the scores seen along the vertical axis and then take the Format Axis options 2003 2011 Curtin University West Australia Toolbar and tab m Note that you can delete whole columns in the Breakslbw report and the corresponding box and whisker set will disappear from the plot This is useful when you ve got too many groups and want the boxplot to highlight only some of them For example if above we eliminated column 2 from the Breaks1bw report the resultant boxplot would contain just box whisker sets for the P and S groups Related tidbits More about the use of Excel stacked column charts for boxplots may be found in work by Jon Peltier Lertap s boxplots are largely based on Peltier s suggestions http peltiertech com Excel Charts BoxW h
260. ksheets contained in the Lertap5 workbook refer to the previous topic 14 for a picture If you look at the Data worksheet you ll see the actual question responses given by the 60 participants in the 1973 workshop When you took the tour your first action was to get Lertap to make a new data set for you to fool around with This you did by going to the New menu clicking on the option which directed Lertap to make a new workbook by creating a copy of the present one The present one in this case referred to the data set which comes with the Lertap5 xlsm file that is to the Lertap Quiz At the end of this step you had a new workbook which contained copies of the Data and CCs worksheets corresponding to the Lertap Quiz All subsequent actions in the Cook s tour involved the use of these copied worksheets Changes to the CCs lines In the September 2004 revision of Lertap we provided an option to process affective questions in a different manner as requested by users in Australia and the United 2003 2011 Curtin University West Australia Getting started 2 2 States The revision involves vesting more flexibility in the MDO option Read about the new MDO s capabilities by simply clicking here 4 whenever you d like To demonstrate how the revised MDO works we modified the CCs lines related to the Lertap Quiz Now the lines set out three subtests not two as was formerly the case The report goodies provided
261. l 2007 looks 2003 2011 Curtin University West Australia 2 Help file for Lertap 5 1 2 much the same There s also the knock your socks off manual and of course the main website at Curtin University There are some videos too And yes if that s not enough there s even more have a gander at our Sample Data Sets website It s a Jim dandy source for quickly getting an idea of what Lertap does and it even has some samples which may be downloaded for test drives on your computer Requirements Lertap 5 is an Excel application Your computer must be able to run Excel the spreadsheet program from Microsoft usually distributed as one of the applications in the software suite called Microsoft Office Other applications in this suite are Word and PowerPoint Yes you bet we know about the competition to Microsoft Office specifically the suite of similar applications found in OpenOffice The OpenOffice equivalent to Excel is called Calc it s good It s very good But it cannot yet run all of the computer code found in Lertap s macros We are watching as OpenOffice Calc continues to develop It may not be long at all before its macro language is on a par with that found in Excel So Excel You currently need it to run Lertap Got it Good Is it Excel 2007 or Excel 2010 If it isn t you re reading the wrong document sorry This here document is meant to be used as a helpful companion a warm hand holder as you r
262. l others a 50 item test A student s test score will depend on the number of items s he had the chance to answer Two students with the same usual subtest score say 35 will not have the same percentage score if they have been presented with a different number of items For example if Jorge got a score of 35 on a test with 50 items his percentage score would be 7096 assuming one point for each correct item Suppose it turned out that Marisol also got a score of 35 but she was presented with only 45 items her rounded percentage score will be 7896 When you ve asked Lertap to Create an adjusted percentage score the percentage test score will be adjusted according to the items a student actually saw If all students get the same number of items there is no need to create an adjusted percentage score Ah but wait a minute We should adjust this comment if all students are presented with the same number of items an adjustment is not required if all items are scored the same way When items are scored in different ways then the adjustment might again be useful For example if Item 26 is worth one point but Item 27 is worth two points then we might well want to make an appropriate adjustment to scores depending not only on how many items a student saw but also on the number of points the items are worth Lertap creates the adjusted score by dividing the usual subtest score by the actual maximum possible score a student could have a
263. l to contain text it ll use the text as a title for the line graph this can be real handy Lertap will fail to make a line graph if no cells are selected only one cell is selected or if the selected cells do not contain numbers except for what we ve just mentioned the first selected cell may contain text 2003 2011 Curtin University West Australia 4 3 4 3 1 Toolbar and tab New menu The New menu is the best way to create a new Excel workbook ready to work with Lertap It s not the only way but it s the best Why Because it results in an Excel workbook set up to display Lertap s preferred fonts and a workbook with the two core Lertap worksheets Data and CCs To find out what the New menu s options do click on the options in this little snapshot F Blank R Headers Copy New menu It is possible to create a Lertap workbook without using the New menu options All that s required by Lertap is an Excel workbook with data records in a worksheet named Data and control lines or cards in a worksheet named CCs If you presently have an Excel workbook with data records in it you can try to rename the worksheet with the data records to Data and insert a new worksheet with a name of CCs That will work However font problems may develop when Lertap s Run menu options are taken Lertap s reports such as that seen in the Stats1f worksheet may turn out to be poorly formatted If this happens a suggestion i
264. l worksheet has to be saved as a text file and Xcalibre wants the text file to have an extension of DAT You now have a need to know how to save an Excel worksheet as a text file and we ve got some comments ready for your peepers to peep click here iato jump to them or simply page ahead until you get to the Creating a text file topic 4 4 4 4 1 Xcalibre 4 1 The information in this topic has to do with getting Lertap to create the two files required by the new version of Xcalibre known as Xcalibre 4 1 At June 2011 there was only one version of Lertap which would do this It s a version for Excel 2010 known as Lertap 5 9 2 Lertap has an Xcalibre interface which works something like that for the Bilog MG program mentioned in a previous topic To activate the interface go to the System worksheet in the Lertap5 xlsm file and activate the setting in Row 24 Column 2 set it to yes Once yes is in the appropriate spot in the System sheet Lertap will spin out two new worksheets and two corresponding txt files every time the Output item scores matrix option is selected from the Run menuls8 _ the subtest selected for processing may be either cognitive or affective Xcalibre 4 1 requires a Data matrix file and an Item control file The data matrix file created by Lertap 5 9 2 is called LrtpXcalData txt It is a tab delimited text file Lertap will create this file in the same folder as the active workbook that i
265. l6 7 8 9 10 11 l Data from the ChemQuiz sample 2 ID i Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 3 Anderson DIBIBICID DIJ AICI A B 4 Baker BIB BIBIDIBICIC B B b Camberwell B B A B D A B C B D 6 Donaldson BI B CIBIDICIBIC C B 7 Eggmont BI B AIBIBIJCICIC IBI B 8 Fredicksson B A B B D C A C B A 9 Graphner DIDIAIB D I C A C BIA 10 Humphrey B B B D D C C D B B ll Invererity B BJ A JDIBICI BICI A A 12 Johnson BID BICJIAJBICIC B B 13 Klein D BiA DIC C IAN 14 Lampton B AJA C A B A 15 Mecurio B BIDIA CICIJIAI BIB 16 Nesbit ATCIATBJIBIC A D A 17 Oldfelt AJTDLATLATAICIALDCIBJI A M gt aN Data CCCs 2003 2011 Curtin University West Australia Getting started 21 What have we got here Fifteen students answered 10 multiple choice questions On the first item Q1 Anderson selected option D On the fifth item Invererity selected option B Anderson s data are found in row 3 of the worksheet The answers to Q1 are found in column 2 of the worksheet The answers to the last question are found in column 11 You can t see row 18 but it s empty And here s the corresponding CCs worksheet 1 ae 1 col c2 c11 2 key BBBBD CACBB il M Data CCs Yes the CCs worksheet has just two rows of information The first one tells Lertap that item responses are to be found in columns 2 through 11 of the Data worksheet The second line has ten letters each letter is the right answer the key for an item For example the right
266. lated tidbit This set of CCs lines could be used to compare the reliability of the whole 25 item test with that from the same test with Q22 excluded or eliminated r X I m Eni y Iz Book2 Microsoft Excel o B 98 Fite Lerta Hom Inser Page Form Data Revie View Deve Add S o pg X sleet o demEe E sub Res A B C D E F Title Knwldge wt 0 key AECAB BEBBD ADBAB BCCCB BABDC Il alt CEDBC CEDFD EDCBD DCCDD DEEDF erred Be d tests Ete i sub Res A B C D E F Title Q22 out Wt 0 key AECAB BEBBD ADBAB BCCCB BABDC walt CEDBC CEDFD EDCBD DCCDD DEEDF exc c24 M 4 M Data ccs Pa T rfl Ready E 100 gt y t ka There are two subtests defined by these lines remember that the col lines mark the start of subtest definitions A exc line has been used to exclude the item responses found in column 24 c24 of the Data worksheet This was Q22 in this example The Stats2f report corresponding to the second subtest looked like this after Elmillon was run 2003 2011 Curtin University West Australia Input amp Output 2 x id E SF 7 Book2 Microsoft Excel c3 s 9 Lertap Home Insert Page La Formule Data Review View Develop Addins YV c 53 G175 M f 0 915562369948527 x Lertap5 full item stats for Q22 out created 2 07 2012 Summary statistics number of scores n low
267. le Base 10 Display units None Alignment m x Matar tele marl himnas tti lal Above the Maximum has been set at 50 the former value was 1200 Click OK and You called tails 2003 2011 Curtin University West Australia Help file for Lertap 5 50 45 Frequency n3 n3 ta e an me a 10 Log PROB Changing the scale of the Y axis has made the left most Log PROB outliers visible Note that these outliers are also sort of visible in the corresponding HistoL sheet 2003 2011 Curtin University West Australia Toolbar and tab Distribution of Logf PROB as at 25 01 2006 Eac z score f cf c 5 15 17 00 0 0 0 2 0 096 418 1500 4 0 1 7 O99 3 70 14 00 10 0 296 17 3 22 13 00 20 0 496 L2 TSEIADO 48 0 9 Look carefully at the f column above and you ll spot the outliers below the score of 14 There aren t many but the presence of just a few can influence the interpretation of an RSA that is a response similarities analysis 4 5 1 2 The ToolPak Add In As it goes about histogram making some versions of Lertap are trained to see if you ve taken the trouble to install a standard Excel Add In called the Analysis ToolPak Add Ins are special little routines which extend Excel s capabilities In Excel 2007 the management of Add Ins is found by taking the Excel Option button after you ve clicked on the Office Button
268. lete them Note read more about making sense of Elmillon s various reports in the manual External criterion The item correlation coefficients which form part of Lertap s Stats1f and Stats1b reports are based on correlating item scores with what s referred to as an internal criterion the number produced by scoring the remaining items in the subtest or scale to which the item belongs It is possible to replace the internal criterion with another score This other score is referred to as an external criterion An external criterion may be used as part of the process of validating test items In Lertap the external criterion score must correspond to a column in the Scores worksheet The analysis begins by having the user pick the Scores column that has the score to be used as the external criterion Once this is done Lertap asks the user to pick out the subtest which has the items to be correlated with this score This is done by displaying Sub worksheets 2003 2011 Curtin University West Australia Toolbar and tab e Users of the external criterion analysis feature will often have a criterion measure which needs to be imported to the Scores worksheet In this case the criterion measure should be recorded in a column in the Data worksheet Once it s there the Move menul on the Lertap toolbar will allow the measure to be copied over to the Scores worksheet The effects of part whole inflation may be examined by using an
269. lp file for Lertap 5 E3 Microsoft Excel Book2 meax Fie Edit View Insert Format Tools Data Window Help X O nw SB YS pune 21 le li li Move License Lelp gt Ready Sum 10 00 The so called scree test for the number of factors involves nothing more than eye balling a line graph such as the one above and deciding where the scree begins In case you ve forgotten the scree is all the loose rocks at the base of the cliff your friends want to climb those pesky fallen chunks where your boot will slip in and get stuck twisting your ankle granting access to a face saving retreat to the beer tent in case you were really too chicken to climb the cliff to begin with Does the first eigenvalue dwarf the others Does our scree begin with the 2nd eigenvalue or the 5th This question will remain unanswered here many times the start of the scree is much easier to detect For references s281 on the scree test see Catell 1966 Pedhazur and Schmelkin 1991 or search the Internet Note that eigenvalues can go negative This is likely for example when SMCs are used on the diagonal of the correlation matrix when one of the items has no variance or especially when tetrachoric correlations are used Also note that it is possible for the eigenvalue extraction method used by Lertap to fail the method is an iterative one which concludes when the iteration process appears to converge Under some circumstances convergence will n
270. m has only one keyed correct option then diff will be the same as p for the correct answer in the corresponding Statsf report If an item has more than one keyed correct option the percentages for each of these options are summed to get an overall figure The disc column is the item s discrimination figure It will be the same as the pb r value seen in the corresponding Statsf report if the item has only one correct answer When an item has more than one correct answer then disc is the Pearson product moment correlation between the item and the criterion score corrected for part whole inflation The mark column is used to flag potential problems Click here z to read more about these flags r x id 0 Co Be Pls Books Microsoft Excel ca B z File Lerta Hom Inser Page Form Data Revie View Deve Add 9 Q c gb xS Fr 2a s e ne 8 Ee pre um Ee es e 9e Eme es e o Pss ors pes e En ox ae oe En fies es En E3EEESIN Pass oe n En se o zo we s it Reliability coefficient alpha 628 Statsiul Stats2f Stats2b Stats3f 4 u gt IM 4 gt bl 2003 2011 Curtin University West Australia 22 Help file for Lertap 5 A typical Statsb report for an affective test is shown above The pol column indicates the polarity of the item if the item has been reverse scored 23 then the polarity will be negative The item s
271. n 222 delete worksheets 80 89 delete worksheets advanced toolbar 244 delta 151 dichotomous 271 dichotomous scoring 282 did not see 48 53 58 60 61 319 DIF 141 155 193 DIF IbreaKsMH 151 diff 271 299 diff 58 183 290 differential item functioning 141 155 193 differs from manual 29 difficulty 271 290 292 299 difficulty calculation 325 disc 271 299 disc 58 115 183 186 290 discrimination 265 271 286 290 292 299 distractor 271 294 divide 159 documentation 325 double key 294 double keyed items 231 download 3 downloading 12 Ea ECStatsiul 193 304 ECStatsiulChta 193 ECStatstulChtb 193 ECStatsf 277 EEIC 118 effect size 131 effecting changes in settings 248 eigens 106 eigenvalue plot 106 eigenvalues 93 106 324 eliminate an item 69 ellipsoid 106 Elmillon 88 90 92 208 244 Elmillon advanced toolbar 247 2003 2011 Curtin University West Australia Elmillon item analysis 80 email 14 EMQs 321 end of data 253 255 323 endorsements 259 enhance 155 equation score 219 equivalent forms 282 errors 259 essay questions 69 etasqrd 131 141 ETS 151 examples 20 exc 71 Excel 2 Excel 2007 12 79 Excel 2007 2010 1 2 Excel 2010 1 2 101 118 Excel 2011 1 2 118 Excel limit 198 Excel shortcuts 250 exclude 212 exclude an item 69 286 322 325 exclude items 71 experimental features 115 326 export 101 111 115 208
272. n University West Australia 176 Help file for Lertap 5 as Data Review View Developer Add Ins Design Layout Format Legend 7 ida i Chart Wall l2 4i Lines t Data Labels 7 Chart Floor 21 Up Down Bars Axes Gridlines Plot Trendline Prog Data Table X Area 3 D Rotation Error Bars 7 Axes Background Analysis 5 6 7 8 9 1 cs e E o o 1 00 6 00 11 00 16 00 21 00 Knwldge A lt The Trendline option is prominent on the Layout tab Click on it and select More Trendline Options from the box which frops down frops being a new word meaning either flops or drops it s not a typo never 2003 2011 Curtin University West Australia Toolbar and tab Trendine Options Trendline Options Line Color Trend Regression Type Exponential Shadow Q Linear Logarithmic Polynomial Order 2 Power FN n Moving Average Period 2 t Trendline Name Q9 Automatic Linear Series 1 Custom a Forecast Forward 0 0 periods Backward periods E Set Intercept 7 Display Equation on chart Quicker than you can spread cold butter on cold toast you ll see the result 2003 2011 Curtin University West Australia 178 Help file for Lertap 5 42 00 40 00 38 00 36 00 34 00 Comfort 32 00 30 00 28 00 26 00 t t
273. n email message to Assessment Systems Corporation with the Session ID and Computer ID codes and ASC will respond with instructions on what to do next In some cases the Unlock Code is sent directly but usually ASC will request an order with payment before the Unlock Code is emailed back Note that it is also possible to fax the Session ID and Computer ID numbers to ASC the fax number is 1 651 647 0412 ASC s office is located in St Paul Minnesota U S A Copy ID s to Clipboard This button shows on the form above Its purpose is to save you having to copy 2003 2011 Curtin University West Australia 22 Help file for Lertap 5 down the two ID numbers If you click this button the two numbers will be copied to the Windows clipboard you can then use the Paste command in your email program to have the Session ID and Computer ID numbers automatically inserted in your message Please allow from one to five days for ASC to respond to your email or to your fax 4 8 3 Unlock Lertap A message somewhat like the one below may be expected when the Unlock Lertap option is chosen Thanks Ea o Right we ll put up the same form used to apply for a license Now you ll type in the special unlock code obtained from Assessment Systems Corporation and then that should be it your copy of Lertap should be unlocked ready to run without any time limit Click on OK and a familiar form will reappear this is the sam
274. n even be blank that is as long as Excel still leaves the little triangle in the cell s upper right corner Well great but why have a row 16 setting anyway If can insert and delete those little triangles by right clicking on appropriate cells why have the row 16 option For convenience If you re going to plot all or nearly all quintiles having row 16 set to yes saves you the onerous task of inserting lots of triangles On the other hand if you re only going to quintile a few items having row 16 set to no saves the time required to delete a swag of triangles And then again Why should worry love quintiles Can t get enough of them Triangles R Me Row 16 is yes yes yes forever But just you wait there may come the day when you find Excel refusing to give you all the plots It s happened to us 2003 2011 Curtin University West Australia Toolbar and tab 205 and it s a bother if you have lots of items there s a chance that Excel will run out of chart or font memory and you won t get all the plots Having the row 15 option set to yes for data tables increases the chance of running out of memory Perhaps keep these comments in mind Some day those little triangles may turn out to be real handy love quintiles put yeesch those colours They don t look good at all when printed on a black and white printer Not to worry help is at hand see the Chart colors 205 topic 4 5 3 8 Chart problems Ther
275. n press the Enter or Return key on the keyboard Lertap 5 There are 844 data records with a group code of Ing do you want to include these records in the analysis Yes Mo Cancel Ibreaks will step through all the distinct codes found in the Lengua column presenting a question such as this one each time it encounters a new group code In this example there are only two group codes but often there will be more Groups are included or excluded from the analysis according to the answer you give each time this questions appears At this point a click on the Yes button will make sure that the responses of the Ing group are included in the analysis Ibreaks then presents the same question this time for the Nat group Another click on the Yes button and Ibreaks moves along to its next question If the Scores worksheet contains more than one score the next question is one which asks you to pick out the score you want to work with The score you select may be from either a cognitive or an affective subtest 2003 2011 Curtin University West Australia Toolbar and tab 145 Next Lertap 5 xj What about response plots IF you d like to have a plot of item responses included this is your chance answer Yes Note that such plots can take up quite a bit of space some people prefer to have just tables without corresponding graphs If you re one of these people Yes No Cancel answer No
276. n t like it However as long as the original DAT worksheet remains part of a Lertap Excel workbook the Fortran format line should not be deleted it s possible to add more data to the DAT worksheet and the procedure which does this will cough sputter and die if the format line has gone walkabout What s this about adding more stuff to the DAT worksheet Use the Copy a Data column to the DAT worksheet option of course as found under the Moved menu 4 4 4 4 Xcalibre worksheets Xcalibre is the name of an IRT program created by Assessment Systems Corporation ASC in the United States Note 1 inserted 7 June 2011 a new version of Xcalibre 4 1 is now available from ASC The XCal worksheet described in this topic will work with Xcalibre 4 1 providing you check the box Data matrix includes an Xcalibre 1 1 Header found in the Files tab used to set up a run with Xcalibre 4 1 2003 2011 Curtin University West Australia ow Help file for Lertap 5 Note 2 inserted 7 June 2011 Version 5 9 2 of Lertap an Excel 2010 version will create the two files required by Xcalibre 4 1 Please refer to the following topic 9 for more information Lertap has an Xcalibre interface which works much like that for the Bilog MG program mentioned in the previous topic To activate the interface go to the System worksheet in the Lertap5 xlsm file and activate the setting in Row 24 Column 2 set it to yes Once yes is in the appr
277. n the little What s new box and click OK All rows whose entry in the copied column is NZ will then have EN in the adjacent column that is in the recoded column 2003 2011 Curtin University West Australia 24 Help file for Lertap 5 After this the macro returns to look at the next entry in the copied column It ll find CA according to the little snapshot above You l be asked what CA is to equate to in the new column You enter EN And so forth We might end up with rows looking like this 4 42 Country recoded NZ EN CA EN US EN NZ EN YZ SP YZ SP YZ SP YZ SP YZ SP YZ SP AU EN DO SP Got the idea It s pretty simple which will reflect poorly on you if you didn t get the idea At the end of this process the Recode macro leaves you with two new columns the copy of the original column and a column with the recoded values It s easy to see if the macro has done what you intended just scan down the columns If the result is not what you had in mind just delete these two columns and start again On the other hand if the result is in fact what you wanted you might then want to delete the column which has the copy of the original column You don t have to do this but if you do you ll save a bit of space There are after all only a certain number of columns which a worksheet may have 256 was the limit in earlier versions of Excel and this limit could be an issue at times h
278. n the selected column We could have selected more than one column this is a quick and effective way to truly recode values over a number of columns It is very possible to get Excel to do more We might have a column with numeric values such as number of years of test experience and wish to create a new column with a coded experience letter such as H for high M for medium L for low and N for none Take a deep breath and look below at the IF statement seen in the Formula Bar 2003 2011 Curtin University West Australia Toolbar and tab 219 Ej Microsoft Excel Book2 lol x 2 File Edit View Insert Format Tools Data Window Help PDF Create HBHx 1 2 4 4 4 4 3 3 5 4 2 4 2 0 4 0L 2 1 3 4 4 1 4 2 2 0 40L 3 1 2 5 3 3 4 D 5 4 0L 3 1 2 4 4 2 4 1 0 2 5 L 2 2 2 3 5 1 5 4 9 5 12 0 H 2 1 2 5 3 1 4 4 4 5 5 0 M 2 1 1 5 4 1 5 4 7 0 6 5 M 2 1 2 3 5 1 4 3 5 5 5M a gt WA Data LEG yal ling ra taf ay lof Ready The IF formula does the recoding for us It says that if YrsTest column AM has a value greater than 10 then column AN is to have an H otherwise if column AM s value is greater than 5 then column AN is to show M otherwise if column AM s value is over 0 zero show L else show N Note that we used Lertap s Excel shortcuts ss option to Change the referencing style so that the column headings are letters not numbers this can often make writing Excel formul
279. n the website which deals with the issue in somewhat extensive detail a link to this paper is found at the end of the remove an item 6s topic Now removers of items have a new tool the exc card or line which will probably be the easiest way yet to see that an item or items is are quickly excluded from a subtest Click here 7f to read about this new method Oct 2004 Ver 5 4 5 October 2004 enhanced IStats report version change to 5 4 5 More information has been packed into the IStats e3l report Its SMC values 94 are now plotted in a series of ten bands easing the task of determining the extent to which any single item relates to the others The first principal component or the first principal factor of the correlation matrix is now extracted and item component correlations are displayed in two ways as a conventional row of values and in the ten bands format similar to that used for the SMC coefficients A technical paper was added to the website to demonstrate how to interpret this new output Read more about this in the eigenvalues cs topic 2003 2011 Curtin University West Australia R amp R amp R amp R 323 7 1 12 Sep 2004 September 2004 MDO now means Missing Data Out The power of the MDO option has been increased Including MDO on an affective subtest s sub line now gets Lertap to correct its brief stats report such as Stats1b for unequal response n s Items with missing data may now have the
280. nced way at the bottom of the topic under Related titbits Thanks to phantom readers for sending in these Qs Send yours and if it s a good one we ll put it up here Send them to larry lertap com 4 7 4 NumericFilter2 Consider the following Data worksheet from an Indonesian data set x Delete VA Sort __ Blank X Interpret fig Histograms Move 7 version Me liney Headers S Elmillon A Scatterplot License fXSpread Excel Copy More b Res charts Lelp Basic options New menu Run menu Graphics trio Other menus Ip daga TEO E O EA UN Bahasa Inggris 2007 No Id Gender Ti T2 I3 14 15 I6 T7 I8 To LL 300889 L 60589 L 240689 P 141089 L 20489 L 220589 P 11188 P 151189 L P P P L P P P NJA Ov cO 10 Q4 QI H 60189 80889 180189 300489 50189 m o m p b d d AP Pp po p p p b pop p p p d p d opu op pop p op op pud 300588 Bgjgjg jgjDIjaI2I O OjO O O O 2 E p D 3j 3 3 o d od ot uot uU BgjgjgjgjDmjajD2I OI O O O 2 2 2 gjg jag jgjagIjDjiaIOI O O O O O 2 ppp ppp PP PPP Pop D 9 9 90 Ri Bienio niente Rint Bieni t tU tU tU tU tU t t9 OQ OQ P OQ 171289 A E Scores lt PlotBreaks2 lt Breaks AW Eam 100 49 d ri The Kab field in the second column is a school district code there were six districts 2003 2011 Curtin University West Australia 20 Help file for Lertap
281. ne of the six recognised response codes for this item A through F will be classed as other in Lertap Consider again the weights array above It s essentially saying that a person will get 1 00 points no matter how s he responded or non responded to this item Even unanswers get a point Crazy No not exactly There are times in the life of test scorers when it s necessary to do this Put this crazy scoring aside it s unusual yes but what we want to point out is that you get can into a Sub worksheet and poke away at the item weights Any number positive or negative may be placed in any of the weights boxes True mws cards may be used to accomplish the same thing In Lertap version 5 25 released 8 August 2003 the power of the mws card was increased so that it could be used to get weights into the other category as exemplified here mwsood2 d ky ody UY lis ll othersel An mws line such as this will result in the same weights array seen in the boxes above For more examples of other in action use the Index or Search for other This being the case it is nonetheless true that some people have a preference for launching their response weights from within a Sub worksheet The advanced toolbar 2003 2011 Curtin University West Australia 4 9 2 4 9 3 Toolbar and tab is for such users Advanced toolbar how use To review the advanced toolbar has two more icons than the normal toolbar They app
282. ng MS Between by MS Within To test the null hypot we used to refer to tables of F values these days we can simply ask the computer to see how significant the F ratio is Lertap gets Excel to do this using Excel s in built FDist function In our case FDist says that were the null hypothesis true an F Ratio equal to or greater than 1 02 would be observed 36 3 of the time given the sample sizes used in our study If you are familiar with tests of statistical significance you will know that the usual guidelines suggest that the null hypothesis will be rejected only when we find an F Ratio whose significance is 05 01 or even less Here our value referred to as sig is 363 well above the 05 level if we were really testing the null hypothesis we would not reject it in this case The problem with the F Ratio and its significance is that very small differences in means will sometimes be referred to as being significant even when the differences are meaningless this is prone to happen when sample sizes are large To circumvent this now well known widely acknowledged problem a recommended procedure is to carry along an effect size estimator such as eta squared if we find a significant F is it confirmed by a useful effect size say for example at least 10 for eta squared Refer to Thompson 2006 I 24 or Pedhazur and Schmelkin 1991 9 for more readings in this very significant area Thompson s text is
283. niversity West Australia Toolbar and tab 121 One of these is RSAtable exemplified in the screen snapshot below Lertap5 RSA table created on 7 01 2006 0 2 o os of 0 0 4 2 0 0 4 4 16 7 s 20 8 0000 0 5 9 37 5 14 s8 3 nnnunnnna 0 6 s z0 8 19 79 25 nnnnn 0 7 4 16 73 23 95 85 nuna z 0 8 0 05 23 95 8 0 9 0 0 1 0 0 05 1 1 0 0 23 95 8 1 2 0 0 23 95 8 1 3 0 0 23 95 8 1 4 0 0 23 95 8 1 5 0 05 23 95 8 1 6 0 05 23 95 8 1 7 0 0 23 95 8 1 8 0 0 23 95 8 o s PERR Il ESTAS hd RSAdatai RSAsig RSAtable1 of An RSAtable report plots the values of the Harpp Hogan Index measure for all those pairs of students having an EEIC value above the preset cutoff Most H H Index values will be less than 1 0 in magnitude To be noted is a special case the H H Index is a ratio one whose denominator D may be zero When this occurs Lertap sets H H Index equal to a value of 999 Lertap s RSAtable report is made to resemble Figures1 2 and 3 in Harpp Hogan amp ennings 1996 The RSAtable report is a hold over from Lertap 5 5 where it was used as the main indicator of potential cheating a role which has now been assumed by the RSAcases report The third Lertap RSA report is RSAsig a worksheet which contains a wealth of information RSAsig has three main areas top lo
284. nless Wt assignments are made on the sub line var values are taken from the Scores worksheet The Total score is a composite Total Sum Wt j X j where Wt j is the weight assigned to subtest j and X j is a student s score on subtest j The Sum is taken over all J subtests whose Wt is not equal to 0 zero Let V j be the variance var of subtest j and V T the variance of the Total score Then Let P1 Sum Wt j V j Let P2 J J 1 Let P3 P1 V T 2003 2011 Curtin University West Australia 23 Help file for Lertap 5 then alpha P2 1 P3 In this example alpha 0 51657469 To keep a subtest from entering the Total score set Wt 0 on the subtest s sub line col c3 c27 sub Res A B C D E F Name Knowledge of LERTAP2 Title Knwldge Wt 0 key AECAB BEBBD ADBAB BCCCB BABDC alt CEDBC CEDFD EDCBD DCCDD DEEDF In this example all subtests J 22 have Wt 0 and no Total score will be made The TotalTest1 macro will only work when two or more subtests have non zero Wt values The subtests do not have to be of the same type that is some may be cognitive while others are affective This macro was created at the request of a veteran Lertap user who knew that composite alpha used to be computed by the mainframe version of Lertap Lertap 2 The P1 P2 and P3 terms are as seen on pp 278 281 of the Lertap 2 user guide Nelson 1974 s53 2003 2011 Cur
285. no Lertap tab Not yet This is something of a problem to be sure But all is not lost Click on the Options option as seen in the Security Warning above The warning will change to an Alert displaying something this 2003 2011 Curtin University West Australia 3 Help file for Lertap 5 Security Alert Macro Macro Macros have been disabled Macros might contain viruses or other security hazards Do not enable this content unless you trust the source of this file Warning It is not possible to determine that this content came from a trustworthy source You should leave this content disabled unless the content provides critical functionality and you trust its source More information File Path C Users nelsonl Desktop StandardVersion Lertap5 xism Q Help protect me from unknown content recommended Enable this content Open the Trust Center Now if you click in the little circle next to Enable this content good as gold is what you ll be You may see the Lertap tab or you may see a collapsed ribbon one without expanded tabs Here s an example of the collapsed ribbon in Excel 2007 Formulas Data Review View Insert Page Layout Click on Lertap and away you ll be The Lertap tab will display giving access to all its wonders A mild problem we might call it a bother is that you have to go through this rigmarole 2003 2011 Curtin University West Australia Welcome 9
286. not see code Also assume we have the PER control word 2s on the sub card The Scores output will look as follows DESEE HEN EE ESI FEES E The first student had a test score of 38 Lertap says that the corresponding percentage score is 40 9 a value found by dividing the score 38 by 93 the maximum possible test score if each item is scored giving one point for a correct answer But this is not correct in this case no student had the opportunity to answer 93 items Test Pilot gave each student a random sample of approximately 40 items So we activate the did not see option by putting yes in the second column of the System worksheet row which says Use a did not see code We have no in the Create an adjusted percentage score row And we still have PER on the sub card Our Scores now look as follows _Record No EE101 EE10190 LESE a anaes ae See how the percentage scores have changed Lertap is now basing its percentage score on the maximum possible score which could have been earned on the actual items presented to the student For the first student the maximum was 48 2003 2011 Curtin University West Australia 52 Help file for Lertap 5 Dividing 38 by 48 and multiplying by 100 gives the 79 2 seen as the EE101 score for the first student Now for the second student does 30 divided by 48 equal 62 5 Yes The problem is that the second student was presented with 40 items not 48
287. not surprise us at all However this wasn t the case There was solid evidence to suggest that the Nat and Ing groups may have been equally proficient on the subject matter covered by the test The stage is set then for a DIF analysis differential item functioning Jump to the DIF topic s now if you d like to Colouring The sig value for 135 of 0 00 has been shaded above Lertap s Ibreaks routine is trained to colour sig values and also eta values whenever they equal or fall below a cutoff minimum set in Lertap s System worksheet Refer to the System settings s topic if you please Ibreaks charts The charts or plots seen in the previous topic were based on right wrong cognitive items and only two groups The plot below is from the dataset used in the Breakouts topic It used affective scales many with items having seven response options Three groups were involved 2003 2011 Curtin University West Australia 18 Help file for Lertap 5 EE E E a Se aie sec EN E 5 24 27 20 12 10 2 o 41 4 51 1 47 P 9 44 29 8 8 0 3 0 66 5 27 1 26 s 6 19 28 19 19 3 6 o 32 4 41 1 52 Q2 is an affective item from the MSLQ instrument developed at the University of Michigan MSLQ items typically ask students how often they had used a particular learning or study strategy during a given time period F
288. ns of Lelp the Lertap help file The chm file is commonly accessed from the Lelp link on the Lertap tab X Delete VA Sort Blank Interpret gi Histograms Move Q version Line iR Headers Elmillon LA Scatterplot License fX Spread Excel Bs Copy More b Res charts F9 ep D Basic options New menu Run menu Graphicstrio Other menus The chm file and the pdf file may be run on their own they don t require Lertap or Excel in order to be viewed Just find where they are on your computer and double click on their names 1 6 Known problems There are a few problems which can arise when running Lertap The two topics immediately following address two common problems macro security and matters which relate to running Excel in compatibility mode 1Y 2003 2011 Curtin University West Australia 6 Help file for Lertap 5 1 6 1 A more complete discussion of issues associated with the Excel 2007 and Excel 2010 versions of Lertap may be seen at the following URL http lertap curtin edu au Documentation UpdatesSummaryLertap57 pdf A lengthy compendium of Lertap 5 s historical development and problem fixes is found at this URL http www lertap curtin edu au KnownProblems htm Page forward to take in the macro security and compatibility mode topics Macro security Of the known problems one which warrants special mention as it crops up so often the matter of
289. nts otherwise What are tetrachoric correlation coeffcients They re estimates of what the correlation between two items would be if responses to the items had an underlying normal distribution instead of the simple right wrong dichotomy used to score the items Some researchers and test developers are at times willing to assume underlying normal distributions especially when they are interested in aspects of IRT modelling For more reading use Lertap s references sl page looking at Crocker and Algina Lord and or Glass and Stanley Or search the Internet for definitions and discussions To compute the tetrachorics Lertap uses an algorithm created by Brown 1977 see References Brown s algorithm calls for the use of two normal curve functions AlNorm and PPND Lertap uses two in built Excel functions instead NORMINV and NORMSDIST 2003 201 1 Curtin University West Australia e Help file for Lertap 5 4 4 4 3 A DAT like worksheet The Bilog and Bilog MG computer programs like to have input formatted as what their authors call a DAT file by which is meant a simple unformatted text file with fields of fixed length suitable for reading by a FORTRAN Input statement Such files are often referred to as ASCII files Bilog MG assumes that items have been scored on a dichotomous basis that is as either right or wrong with a 1 used to signify right and a 0 zero used for wrong A Bilog friendly DAT work
290. number of 2003 2011 Curtin University West Australia Toolbar and tab tor subtest items The second field must appear in character position 5 this field tells Xcalibre the code used in the data records to indicate an omitted item Lertap follows the Xcalibre convention of using the letter O for this code but you may change it as wished The N following the space after the O has to appear in character position 7 it indicates the code for items which have not been reached by a respondent The final control field appears in character positions 9 10 giving the maximum number of characters of ID information 25 for each test taker In the example used here only two ID characters were used The actual data records begin in Row 5 of the XCal sheet Each record has its ID code as the first characters followed by the item responses Note that the screen shot above has blanks in some of the records a dinkum Xcalibre user would replace these with the code used to indicate omitted items in this example that would be the letter O Xcalibre users might want to note the caution given in the Xcalibre manual about processing extremely large data sets and avail themselves of Lertap s all conquering To Halve and Hold s option an option which randomly splits a data set into halves Once Lertap has made its XCal worksheet are you all set to run the Xcalibre program No The Xcalibre program will not read an Excel spreadsheet The XCa
291. number of cases who did not see the item if any To see how many cases were missing data or did not see the item refer to the Statsf report 2003 2011 Curtin University West Australia e Help file for Lertap 5 3 3 4 which corresponds or to the Freqs report Related tidbit A reminder from Chapter 10 of the manual the Statsb reports are computed on an item level The disc value they display is a conventional product moment correlation between the item and the criterion corrected for part whole inflation MDO cognitive Statsul The two preceding topics have discussed how the MDO and did not see options affect Statsf and Statsb reports Now Lertap s third report for cognitive items Statsul what happens when the MDO and did not see options are used Lertap5 U L stats for EE 101 T3 Fall 2005 created 27 03 2006 4 other U L diff U L disc Q50 upper 2nd 3rd 4th lower First up above the standard format for Statsul what s seen before the MDO and did not see options are put to use Item Q50 is from the Test Pilot system mentioned in the previous topics Over 400 students took the EE 101 T3 exam over the internet 25 296 of the students did not see Q50 as Test Pilot did not present it to them while just under 196 of the students who saw Q50 didn t answer it LertapS U L stats for EE 101 T3 Fall 2005 with MDO created dm me a 2 a s Jomeunar usais Q
292. o did not have an answer for the item either because they did not see the item or saw the item but did not answer it Let your mouse hover over the 0 26 value and behold wt 1 0 00 2 1 00 196 case s did not see this item Q50 option wt n p pbir br avg z 1 0 00 114 0 37 0 15 0 20 E Ned 0 20 2 1 00 94 0 30 0 25 0 33 30 74 0 52 3 0 00 62 0 20 0 15 0 22 26 87 0 31 0 00 42 0 13 Proportion without an 27 57 0 16 other 0 00 3 0 26 item answer 3 case s 23 67 0 99 Q80 missing a response 106 avg 29 47 0 22 Q80 option wt n 1 0 00 124 E 1 00 196 other Q100 2003 2011 Curtin University West Australia b r b r 0 39 0 27 0 35 Proportion without an item answer 101 case s did not see this item 0 19 0 10 0 15 avg Z 26 71 0 34 29 47 0 22 s Help file for Lertap 5 See how it works When Lertap is running with both options did not see and MDO then the other line will have stats only when there were some students who did not answer the item For Q50 the avg criterion score for the three students who were missing data was 23 67 when this avg score is inserted into the distribution of criterion scores for those 312 students who did answer the item the corresponding z score is 0 99 Related tidbit As discussed in the manual when more than one option to a cognitive item has a non zero wt value the pb r and b r statistics are corrected for p
293. o nnt N15 10 0n n 22 ud KEE gt M CCs lt Fregs lt Scores Statsif Statsib 4 Il gt Ready E BO 10 g The screen snapshot above shows results for items from the 25 item cognitive subtest found in the Lertap Quiz The items in this subtest used upper case letters as response options Sixty 60 students took the test The table below provides an interpretation of the column headings seen in the report option The response option selected by the student Response options may be letters upper or lower case or digits from O to 9 If the option corresponds to the keyed correct answer then the option is underlined as are all the statistics for the option wt The number of scoring points given for selecting the option The number of people who selected the option The proportion of people who selected the option Multiply this by 100 to get the corresponding percent In the 2003 2011 Curtin University West Australia Help file for Lertap 5 pb r b r sj O avg example above 43 of the students selected option A on Q1 If the option is the keyed correct answer then p is usually referred to as item difficulty Thus the difficulty of Q1 is 0 43 In the Stats1b report p is diff Note that an item with a difficulty of 0 80 would be referred to as an easy item as 80 of the students got it right Conversely a difficulty of 0 20 would be a hard i
294. o copy the Practice column from the Data worksheet to the Scores worksheet Then we went back to the Run menu and started an External criterion analysis telling Lertap to use the Scores column with Practice values as the criterion score Lertap dutifully produced two new worksheets ECStats1f and ECStatslul We had a squiz of the latter at the very end we observed this info 2003 2011 Curtin University West Australia Toolbar and tab 195 Cli QuintileTests310ct03 xls ARR Lertap5 external criterion U L stats for Form amp MC created 4 11 2003 A68mcEC 2 0 12 0 03 0 11 I EC 0 04 0O 01 0 04 A69mcEC 2 0 05 0o 01 0 06 EC 1 Q 01 0 01 0 04 0 03 everyone This was an upper lower breakout with two groups defined by an external criterion n EC external criterion score Practice was used in this analysis The Summary group statistics above are for Practice M 4 gt n SexECStatsiulchta ECStats1f y ECStatsiul lt Lertap has denoted the two Practice values as EC 2 the Practice group and EC 1 It will always do this that is always denote the categorical variable as EC appending its various values Had there been four groups we would have had EC 4 EC 3 EC 2 and EC 1 Notice the two s d values Zero point zero and zero point zero This is correct the groups have been defined by a categorical variable all members of each group have the same score on this variable there
295. o did not answer Item 2 Yalso has not answered Item 7 10 8 Smith S B B B D D E D 1i 9 Terace T B B A D B C B C 12 10 Uptown U B D B E A B E 13 11 Virgo V D B A D C 14 12 Westphal W B A A E A 15 13 Xeno X B B D A C C A 16 14 Yalso Y A A D B A 17 15 Zenu Z A D A A A C A C There are quite a number of users who prefer to use a special code for the case of unanswered items Instead of leaving a cell empty they might use a 9 to indicate missing data there are historical reasons for this some of the most popular data analysis programs have traditionally used 9s to represent missing data When using scanners to process mark sense answer sheets the software driving the scanner may have its own missing data code such as an asterisk To fully understand how Lertap processes missing data it helps to have a good understanding of what Lertap calls response codes The letters or digits seen under a Lertap Data worksheet s item columns represent response codes In the example above it seems that items have used response codes of A B C and D this set A B C D is in fact Lertap s default response code set for cognitive items for affective items the default response code set is 1 2 3 4 5 It is common for users to have items which use other response codes Whenever a test does not use the default response codes the response codes used by that test s or survey s items are specified by using
296. o use this option after you have deleted one of the columns in the Scores worksheet When a Scores column is deleted the correlations become messy 2003 2011 Curtin University West Australia Toolbar and tab 223 4 7 with Excel tending to display REF messages or something equally ugly Updating the correlations via this option will clean things up Remember that deleting rows in the Scores worksheet is a big no no If a row is deleted Lertap loses track of what s what in terms of the data and who s who You may delete the whole Scores worksheet and you may delete one or more columns of scores without affecting Lertap s internal data structure But don t delete any of the rows okay Promise Note this option was added August 2003 becoming active in Version 5 25 Macs Menu There are quite a number of things to say about the Macs menu foremost among them this is a very volatile topic Many of the screen shots seen in this topic and those immediately following may be out of date when you get around to comparing them with what you might see on the version of Lertap you ve got running at the moment The reason for this The macros accessed via the Macs menu are wide open and subject to change at almost anytime In fact when you try the options found under the Macs menu you may very well end up getting a message such as this Microsoft Excel Bp X A The macro LertapbMacroSetA xla MacroMio2 cannot be Found But
297. ode to cover the case of unanswered items for example perhaps an x or maybe a 9 For more about IDs click here This revision also provides support for users who like to transform test scores to re scale them Any formula may be applied to any score found in the Scores worksheet Read all about it 23l 2003 2011 Curtin University West Australia 324 Help file for Lertap 5 7 1 14 7 1 15 7 1 16 7 1 17 Apr 2004 Ver 5 4 April 2004 consolidation version change to 5 4 We upped the version number to 5 4 for non student users This was done for two main reasons we d made a sufficient number of revisions to warrant a version number increase and we installed a patch for an execution problem which previously dropped users into Student mode inserting an unwelcomed and unexpected new line in their Data worksheets in the process Feb 2004 February 2004 nothing sword id about this Support for Bilog MG users was added in September 2003 Now we ve installed similar assistance for XCALIBREI 99 users XCALIBRE is an IRT program from ASC makers of the well known FastTEST item archiving and test generating system For more comments about FastTEST and Lertap wiggle your mouse and click here 3 About the same time your favourite toolbarl 78 was enhanced Yes Part of it now sports Shorts Check it out there s handy help for users who like to plot their output Note inserted August 2007 the S
298. oded with corresponding digits If the selected score is found to consist of values in the range 1 to 5 Lertap will define the number of upper lower groups as equal to the number of different values found For example if the selected score has only values of 1 and 2 Lertap will set the number of upper lower groups to 2 If the selected score has values of 1 2 3 and 4 for example Lertap will set the number of upper lower groups to 4 This action over rides the number of upper lower groups setting in the System worksheet Here s a practical example 288 junior high school students participated in a test development project which investigated the effects of coaching on test performance About half of the students sat a practice test before taking the real one Did this affect their achievement Data were entered into a Lertap workbook One of the columns in the Data worksheet indicated whether or not the student had taken the practice test this column was called Practice Practice 1 indicated the student had not taken the practice test while a Practice value of 2 indicated that the student had sat the practice test Another Data column contained a code for gender 1 for boys 2 for girls Other columns housed the student responses to the 70 test items The CCs worksheet was set up to score the 70 test items The Run menu was then used to Interpret CCs lines and to produce an Elmillon item analysis We used the Move menul2 t
299. of passes The Spreader is to make 10r2 18 use experimental features generally not recommended es no 19 Item difficul pe 1 proportion 2 mean 3 mean max wt 1323 20 Should tetrachoric correlations be output es no 21 Interitem correlation diagonal value 1 1 00 2 SMC f lor2 M 4 gt h Comments Data CCs System Syntax lt OldcCs lt J ji A Ready EJ Bm 100 The quintile options are set in rows 13 through 16 The number of groups for these plots is seen above in row 13 Its default setting is 5 a setting which gives rise to the term quintile plots the more general term applicable to any number of groupings is quantile Changing the number of groups can often result in smoother plots Ideally we d like to have say at least 50 students in each group less than this and these plots can look a bit ragged For example consider a mid term exam with 68 items and 154 students With 5 groups of about 30 students each the plots for three selected items are shown here i54 i55 i56 1 00 1 00 1 00 r 0 80 0 80 0 80 0 60 0 60 0 60 2 2 2 3 0 40 3 0 40 3 0 40 4 cf 0 20 4 0 20 4 0 20 eather L ooo ether SB d o0 ether oo0 lower 4th 3rd 2nd upper lower 4th 3rd 2nd upper lower 4th 3rd 2nd upper Key 2 Diff 62 Disc 18 Key 4 Diff 29 Disc 24 Key 1 Dif
300. of the Bottom whisker for group P A histogram would be another way to look at how outlying a Tukey outlier is If you switch over to the Breaks1 report and then take the Histograms Yl option using column 3 the column with group P s scores you ll see something like the following 2003 2011 Curtin University West Australia Dow Help file for Lertap 5 Meo o Im Ead Pind jo o dl bal eel bol o o i ele ls 3 0 3 0 3 0 3 0 3 0 3 0 3 0 3 0 ele ele e eie ojgjgm 2 82 2 2218 BISISiSls inlololo amp e 1 1 2 3 4 1 5 1 59 1 59 3 09 3 09 3 0 3 0 3 0 3 0 3 0 3 0 3 0 4 5 5 4 6 1 I a JRE 6 1 Statsib Histoll 4 The two Tukey outliers have been highlighted in yellow above and it s now possible to get another idea of how extreme these scores are notice the distance between them and the score of 39 highlighted in orange These scores are so extreme that they caused the score distribution to have negative skewing 0 37 However the boxplot indicates that once these scores are removed from the scene the skewing actually appears to be towards the high end of the distribution this is so as the top whisker in the boxplot is substantially longer than the bottom one Note a common cause of Lertap boxplot failure relates to the codes names used to label the groups In this example the labels are E P and S Failure is li
301. ol the number of reports made by Lertap The Stats1f report is always standard but the Stats1b and Statslul may be turned off How By making alterations in lines 9 and 10 of the System worksheet 28 The computational resources used to make the Stats1ul report are rather extensive turning off this report will usually save a noticeable amount of processing time especially when there are more than 500 records in the Data worksheet Freqs Scores and the suite of Stats worksheets comprise Lertap s main output But it s possible to get Lertap to deliver more For example it ll make histograms s scatterplots v3 item response charts X a worksheet with item scores 93 subtest correlation matrices 93l and even another Stats report ECStats1f resulting from an external criterion 92 analysis 6 4 1 Freqs Fregs means Frequencies When you ask Lertap to Interpret CCs lines it does a bit more it has a close look at the data columns referenced in col lines and makes a tally of response popularities or endorsements Look 2003 201 1 Curtin University West Australia 260 Help file for Lertap 5 gt Xj d Eb f 7 LrrtpQuizSample25June12 xIsx Microsoft co B 9 rie Lerti Hon Inse Pag Forr Dati Revi Viev Dev Ada v c zs Q1 c3 O E u p Q3 c5 Option 60 w M gt M Data CCs Freqs td IKI m rf Ready E EB IEI 100 2
302. ome of the options on the toolbar such as the 8 ball are not supported by Excel 2007 2010 and are likely to produce unpredictable results Better to use the Lertap tab for Excel 2007 2010 as captured below X Delete VA Sort M Blank Interpret ill Histograms gt Move Version My Line QR Headers gH Elmillon A Scatterplot License 7 fX Spread Excel gei Copy More b Res charts Lelp Basic options New menu Run menu Graphics trio Other menus Running Lertap options via the tab has some advantages such as fewer mouse clicks to get to the most popular options and on screen hover help Hover help Sure Look 2003 2011 Curtin University West Australia Toolbar and tab Interpret ah Histograms 7 f i zimil LZ Scatterplot 4 More hh Res charts Run menu Graphics trio Elmillon item analysis The routine which produces item and test statistics is called Elmillon the million for historical reasons Elmillon is used after the Interpret CCs option has produced its Freqs report Once Elmillon has run you ll see a few new worksheets added to your workbook such as Scores Statsl1f and Statslb These worksheets contain lots of information studying them should keep you quiet for an hour or so at least How to interpret these reports Have a look at the manual it s a great read jv LertapRibbon xlam Press F1 for more help Jus
303. on has increased to 0 74 The graph clearly shows an almost linear relationship between achievement and ability to identify the item s correct answer 2003 2011 Curtin University West Australia Dow Help file for Lertap 5 Distractors A and B foil fewer and fewer respondents as achievement level increases Distractor E is quite popular with the lower achievement groups falling away in the top upper group Okay Great now let s swap things around a bit and look at the data from another angle Da WI c Pais 4 Verdana js Excel2007Quintileslxls M o_o x Lertap Home Insert Page L Formu Data Review View Develo Add In m x B c D E Key C Diff 37 Disc 44 M M Statsiuichta Statsiulchtb 4 This graph a quintile b chart plots achievement groups over response options It shows in the case of item A29mc that the most popular response is in fact C the correct answer Distractor A nicely pulls off the three lowest achievement groups especially the very bottom lower one E is a fairly popular choice for the three middle achievement groups with more than 40 of the 4th group the next to lowest group distracted by it Items which are good at discriminating the knows from the know nots will have a particular quintile b profile the trace for the upper group the knows will be the highest peak above the keyed correct option and the lowest peak for all the other op
304. on scoring method is to give one point for each correct answer in this case the first score will simply be the number of items right The second score is only produced when yes is found in the System worksheet in the second column of the line which says Create an adjusted percentage score This score is formed by dividing the first score the usual subtest score by the maximum possible score which the person could have had if s he had correctly answered every presented item this figure is then multiplied by 100 to get a percentage index Coming back to the common cognitive case with one point for each correct answer the second score is the number of answers right divided by the number of items presented multiplied by 100 Note that both of these scores exclude items which the person did not see that is did not have the opportunity to answer 2003 2011 Curtin University West Australia s Help file for Lertap 5 It is not necessary to use the PER control word on the sub card in order to get the second score it s computed automatically providing yes is found in the second column of the line which says Create an adjusted percentage score Adjusted When would you want to have the percentage scores adjusted What does adjusted mean anyway Well it turns out that certain online test generators can create tests of variable length some students might see a 48 item test others a 45 item test and stil
305. on were clicked 2003 2011 Curtin University West Australia Excel shortcuts 253 again the column with SS figures would be denoted as column 4 Which referencing system is best Whichever you like the Ref style option makes no difference to the format of a worksheet nor does it affect how Lertap runs Lertap prefers the R1C1 style and will always try to turn it on if it can 6 Input amp Output Data analysis systems ask users to do at least three things input information describe how the information is to be analyzed and when ready signal that the analysis should start There are always some constraints on how things are to be done Lertap 5 is no exception it uses Excel to accomplish much of its work but not just any Excel workbook will meet Lertap s requirements Lertap wants its Excel workbook to have a worksheet named Data This is where the answers respondents have given to test or survey items are recorded Lertap wants its description of how the information is to be analyzed to be expressed as lines in another worksheet one named CCs The Data and CCs worksheets are referred to as primary worksheets It s the information from these two worksheets that enables Lertap to go about the business of creating its output the various reports found in worksheets such as Stats1f Statslb and Scores these worksheets are referred to as secondary worksheets Users create the primary worksheets Lertap makes the se
306. onding changes to other cards such as the pol card There s another way to remove items from affective subtests use asterisks on the alt card as shown in the example below col c28 c37 sub aff title Comfort pol 222 tt alt 55555 555 5b The alt card above tells Lertap that the penultimate item second to last is to be excluded from the subtest The examples above are based on an affective subtest but the special uses of the mws and alt cards shown here also apply to cognitive subtests Consider this example With all items col c3 c27 sub res A B C D E F Title Knwldge key AECAB BEBBD ADBAB BCCCB BABDC alt 35423 35464 54324 43344 45546 Using mws to remove the sixth item ool 03 627 sub res A B C D E F Title Knwldge key AECAB BEBBD ADBAB BCCCB BABDC alt 35423 35464 54324 43344 45546 mws c8 Using alt to remove the sixth item ool C3 E2 7 sub res A B C D E F Title Knwldge key AECAB BEBBD ADBAB BCCCB BABDC alt 35423 5464 54324 43344 45546 SAQ would the example immediately above actually work If copied the 16 lines and pasted them into a CCs worksheet would they actually work Yes This example is just a straightforward job with three subtests The lines which do not begin with an asterisk are comments and are not processed by Lertap Finally our examples here have discussed removing a single item from a s
307. ons set in rows 25 through 27 of the System Worksheet A simple click here will take to you the corresponding whatchamadoodle 5 Excel shortcuts Gridlines Formula Bar Headings Ref style XX Exit Excel The snapshot above shows the Excel shortcuts available from the Lertap tab as at July 2007 Experiment with these options They are toggles turning on and off corresponding options Gridlines Formula Bar Headings Ref style X Exit Excel When an option has been turned on a tick mark will appear next to it Here the Gridlines and Headings options are on 2003 2011 Curtin University West Australia Excel shortcuts 251 Lertap will commonly turn these options on and off as it thinks best depending on the worksheet in view However this is not always a good thing Look for example at the Stats ul report below Gridlines u Run menu Graphics trii Formula Bar fge of LERTAP2 created 1 Headings Ref style df SS MS Persons 59 e 1 96 Items 24 22 62 0 94 Error 1416 ziii 0 17 Index of dependability 0 937 Estimated error variance 0 007 For 68 conf intrvl use 0 085 Prop consistent placings 0 892 Pron hevond chance FEN eru M MH csemi Statslul 32 4 uu Lertap s Stats ul reports always have their Gridlines and Headings options set to off and what has happened above is not uncommon the column with SS data is not wide enough to display all v
308. ook should be saved as soon as possible The Office Button in Excel 2007 is used to save workbooks Click here for a definition of a Lertap workbookl s3 Copy This option creates a new Excel workbook which has Data and CCs worksheets copied from the original workbook If the original workbook has more worksheets they are not copied by this option only the Data and the CCs sheets get copied Given that this option only copies Data and CCs worksheets how is a complete copy of a Lertap workbook made With the original worksheet open a complete copy may be made by using the Save As option found under Excel 2007 s Office Button see previous topic Or when the workbook s name is seen in a folder listing right click on the name and then select Copy Run Menu The Run menu provides paths to Lertap s data analysis routines It s used after all data have been recorded in the Data worksheet and after control lines or cards have been typed into the CCs worksheet A typical data analysis procedure consists of two steps First users click on the Interpret option This gets Lertap to check the CCs lines for syntax errors to interpret them If no errors are found Lertap goes on to produce the Fregs ss3l report that is a worksheet which summarises the response frequencies found in the columns of the Data worksheet Which columns The ones referenced by the col lines in the CCs worksheet As part of the interpret CCs line
309. ooking at an example of a cognitive test with 40 multiple choice items scored on a right wrong basis A long used test presented for years in a country s native language was translated to English One group of high school students sat the traditional native language version of the test while another group sat the English language version The two test forms were judged to be equivalent differing only in the language used 2003 2011 Curtin University West Australia Dow Help file for Lertap 5 Ca d wv 99 E v EjemploBangsaen4 xlsx Microsoft Excel non commercial T poo Lertap Home Insert Page La Formul Data Review View Develo Add In XxX 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 1 Datos det angsaen 15 de mayo 2 009 ID Lengua I1 I2 I3 I4 I5 I6 I7 I8 I9 I10 111112113114 I15 I16T 3 Alumno 1 Ing E C E e A C A A RoR RR A RAD i 4 Alumno 2 Ing ESOR xi i 5 Alumno 3 Ing Please enter the Data sheet column number you want to work with 6 Alumno 4 Ing 7 Alumno 5 Ing 8 Alumno 6 Ing 9 Alumno 7 Ing 10 Alumno 8 Ing 11 zi man A M 4 gt I Data CCs The first question Ibreaks asks concerns the location of the column with the group code In this example groups are coded by Lengua tongue or language in column 2 A code of Ing means English while a code of Nat means native In this case the answer to the question posed is 2 enter 2 the
310. opriate spot in the System sheet Lertap will spin out a new worksheet every time the Item scores and correlations option is selected from the Run menuls8 providing that the subtest being processed is a cognitive one The worksheet will be named XCal Like Bilog MG Xcalibre wants to have its input arranged in a very specific format Here s a screen shot of a Lertap XCal worksheet one resulting from processing a 25 item cognitive subtest F3 Microsoft Excel Book2 File Edit View Insert Format Tools Data Window Documents ToGo Help amp X x New GE rZ Run 2l Shorts 7 i sc M Movet o License Lep AECABBEBBDADBABBCCCBBABDC 3542335464543244334445546 ITYYYTYYTYYTYYYYTYYTTTTTITY 9CCDBABACADC A DB BDA EEBF S3iBACAABEBEDADBBDACCADBBDBE 26CEDABBABFDDDBABBAACDC BCE 2 7 EA ABCAB ACDBABBAACCABDBB Z1AECBBCABAAA BABACCCDABBDE SSBECABBEB DADBBBBCCABCBBDC 47 RECABBECBADADBABBCCCBBBBC 42REDAABEBBDA BABBCCCBABBDC SSAEDABBEBBDADBADBCCCABBEB C The first four rows of the XCal worksheet have the control information wanted by the Xcalibre program The 2nd row has the keyed correct answers for the items the 3rd indicates how many options were used by each item and the fourth a row of Ys tells Xcalibre that all items are to be included in its analysis What about the 1st row It s really the most detailed having four fields of critical information for Xcalibre The first field characters 1 3 in the row give the
311. option On Q16 the 36 students who selected option B had average criterion scores of 15 53 As a z score 15 53 is 0 42 the manual has more to say you ought to read it some day perhaps when next at the beach Note that Q16 has an other line Three students had missing data for Q16 They were weak students their avg was low as it was for those who chose distractors C and D Okay Got it Good now have a peep at the stats for the same two items after the MDO option has been used 2003 2011 Curtin University West Australia CCs details 55 Lertap5 full item stats for Knowledge with MDO created 20 03 2006 Q15 option wt n b r b r avg z A 0 00 al 0 02 0 03 0 08 14 00 0 20 B 1 00 30 0 50 0 53 0 66 16 67 0 58 C 0 00 13 0 22 0 36 0 50 7 92 0 68 D 0 00 16 0 27 D0 33 0 45 8 81 0 55 Q16 option wt n p pbir b r avg z 0 00 12 D21 0 7 D 24 10 75 D 32 B 1 00 36 0 63 0 42 0 54 15 53 0 36 z 0 00 3 0 05 0 28 0 59 4 67 1 20 D 0 00 6 p 11 0 32 0 54 6 50 0 94 other 0 00 3 0 05 0 25 0 52 5 67 1 06 There s no change in the stats for Q15 all 60 students answered that item But Q16 has changed many of Q16 s stats above from p through z differ from the Q16 stats seen earlier Why Because the three students missing an answer to Q16 have been excluded from the calculation of the stats Look at the p column for example For Q16 s option B p is now 0 63 corresponding to 36 divided by 57 not 36 di
312. or d If your scan of Freqs reveals strange characters your task is to find them and to fix em It s only after you ve done this that you ll go on to get results by activating Elmillon But there are times when users do not concern themselves so much with Freqs For example many users have their data prepared by using a scanner Scanners can readily be trained to hoot and holler when they encounter bad data Oft times scanners will output a special character such as an asterisk to signal a questionable result For more on the System worksheet click here 43 The production mode options are all dynamic ones they take effect immediately without requiring that Lertap first be closed and then reopened Note that the yes no entry in row 26 controls the following three lines If row 26 is set to no then that s it the following three rows are ignored even if they have yes entries Finally there will be users who d like to have two versions of Lertap one as shipped from our manufacturing plant with production mode off and another version with production mode on Can do emu Yes A single computer Windows or Macintosh can have multiple copies of the Lertap5 xlsm file Of course each copy has to be in a distinct folder subdirectory but that s easy If you re going to do this and you re running under 2003 201 1 Curtin University West Australia 2 Help file for Lertap 5 4 4 2 4 4 3 Windows then
313. or example the item stem might be During the past week how often did you meet with other students to go over material presented in lectures Responses to some MSLQ items are gathered with a 7 point scale ranging for example from Every day to Never The graph for Q2 seen above suggests that the P Primary group made more use of the strategy than did students in the other groups It is easy to change Excel charts Easy Right click on one somewhere around the title Q2 above 2003 2011 Curtin University West Australia Toolbar and tab 149 Cut Copy Paste Reset to Match Style Font Change Chart Type Select Data Move Chart 3 D Rotation Group Bring to Front Send to Back Assign Macro Format Chart Area Click on Change Chart Type and say goodbye to the next two hours Below is a rather crude chart change it s not difficult to make something which is more sophisticated Let Ibreaks create its usual response charts which are in Excel chart parlance Line charts Then select any chart and modify it to your heart s content Save the result 2003 2011 Curtin University West Australia 150 Help file for Lertap 5 as a chart template The click on any other chart and apply the template to it Or use the ChartChanger1 macro to modify the chart type in one go with this special macro all you do is tizz up the first chart ap
314. ore b Res charts Lelp Basic options New menu Run menu Graphics trio Other menus EP 12003 4 5 6 7 8 9 1011 12 13 14 15 1 1 4 ui S P09 P31 P26 P27 P21 P59 P47 P42 P55 D51 M M Data CCs Fregs un Mem rip er i xe C amp In the sample above a group code Type is seen in column 2 In this case a 1 was used to denote workshop participants who were from private schools with 2 used to code participants from public schools PPOPPOUOJPDO X Dio In et NIS et LN et G B QP m mmomoo mmmmpppmp prem OQOODPUODPPUOUOJO z Pppcocn n n u DP PPPOPPOTCO nmmmmmmmmopo Be ebllePlelele SHOE EB mugggoggg gg gt p Dp These codes are not optimal As mentioned in the caveat above they should start with a letter The data recoder 23 may be used to quickly change the codes The screen snapshot below shows recoded Type values in column 41 2003 2011 Curtin University West Australia Toolbar and tab LrtpQuizNovw6a xlsx Compatibility Mode Microsoft Excel non commercial use 5 X Ca essay TE Lertap Home Insert Page L Formu Data Reviev View Develc Add Ir xX x Delete VA Sort Blank E Interpret ai Histograms Move amp Version fx Line I Headers p Elmillon Scatterplot License fXSpread Excel V3 Copy More h Res charts Lelp Basic options
315. ort which indicates how an item correlates with the other items in the subtest There is Sure Look at the item discrimination bands seen in the following screen snippet Ed Microsoft Excel Book1 File Edit wiew Insert Format Tools PopTools Data Window Help PDF Create an x X nw amp rZ pune 4 Shorts T t Mover License Lelp Lertaps full item stats for Knowledge of LERTAP2 created 1 10 2 4 90 item discrimination bands D00 10 20 Q4 Q22 30 Q5 Q14 Q24 40 Q7 Q9 016 Q23 90 Q3 Q10 Q12 Q15 Q17 60 Q1 Q2 Q6 Q8 Q11 Q18 Q21 Q25 70 Q13 Q19 Q20 80 90 alpha figures alpha 9149 without alpha change Q1 0 909 0 006 M Scores Statsif Statsib Stat 4 In Lertap the standard index of item discrimination or for affective subtests the item correlation is the correlation between the item and a person s score derived by summing over all the other items in the subtest There is more about this in the manual 2003 2011 Curtin University West Australia 4 4 4 2 Toolbar and tab Note where Q2 s discrimination index falls in the 0 60 band If we could scroll up the Statslf report or page over to the corresponding Stats1b report we d find Q2 s correlation to be 0 66 We have then two measures of Q2 s correlation with the other subtest items 0 82 and 0 66 Both figures represent the correlation between Q2 and a composite score formed by adding together th
316. orward and minus for reverse An optional card for both cognitive and affective subtests When used it indicates the last response code used by each item If this card is not used it is assumed each item uses the same number of response codes The format of this card changed early in 2005 please refer to Example C7 under the Cognitive CCs 28l topic for details An optional card for cognitive subtests not used by affective subtests This card makes it possible to quickly tell Lertap that the items of a cognitive subtest have different weights that is the right answers to the items have differing point 2003 2011 Curtin University West Australia CCs details values the first question might be worth one point for example while other questions might be worth more points This card is also known as the wgs card mws An optional card for both cognitive and affective subtests mws stands for multiple weights specification This is the most powerful control card of all it allows any weight to be applied to any item response exc An optional card for both cognitive and affective subtests exc stands for exclude This card is used to quickly remove items from a subtest exc is not mentioned in the manual tst This is a very special control card It can only be used once in any CCs worksheet and when used it has to be the very first card It s used to get Lertap to make a copy of the d
317. ot claim to be sophisticated But it s useful to be sure It quickly summarizes what went on in the data set And very importantly it s a sure fire way to see if there are any weird responses in the data columns For example if the Q7 tally had included an X that would be weird as only responses A though F were valid What if you do see weird results in Freqs and want to quickly find the responsible records in the Data worksheet Excel has a set of Data options one of them is Filter This is a powerful little option which will let you rapidly find records which have weird results Keep in mind that Excel s on line Help is there to assist you should you have questions on how to use the Eilter option 2003 2011 Curtin University West Australia 262 Help file for Lertap 5 Here s a screen snapshot of the Filter option as seen when using Excel 2010 Clear E S Reapply cece x Textto Remove em wf Advanced Columns Duplicates gt Connections A practical example of Excel s filter in action may be seen at the bottom of this topic Freqs is a worksheet produced by Lertap s Interpret option It s regarded as a crucial report one which ought to be consulted before Lertap s other reports are examined data processing errors are not at all uncommon and Fregs is in Lertap the best way to quickly spot them 2003 2011 Curtin University West Australia Input amp Output 6 4 2 Scores
318. ot occur eigenvalues will not be returned in such cases but it may be worthwhile to try again that is to return to the Run menu and 2003 2011 Curtin University West Australia 4 4 4 7 Toolbar and tab The computation of eigenvalues can be a labour intensive task for your computer If you will not be making use of eigenvalues and have no desire to become an avid scree plotter then you ll want to turn off the eigenvalue option in the System worksheet 2 amp the option s setting is found in Row 22 Column 2 set it to no again request Output an item scores matrix Note December 2004 we received a trial data set from a Lertap user with 150 cognitive items and 267 test takers Using this for some new time trials was revealing It took Lertap a total of 14 minutes to produce its IStats report for this data set of this figure fully 11 minutes were required to extract eigenvalues from the correlation matrix Note July 2010 we ran fresh tests with a data set from another user This one had 77 cognitive items and 4 700 test takers In this case it took Lertap 3 minutes to produce its IStats report with about 43 seconds required to extract eigenvalues from the correlation matrix We used Excel 2010 for this running on a Macintosh MacBook Pro 2 53 GHz 4 GB RAM with Parallels and Windows 7 if you have a more conventional Windows computer not a Mac you d likely observe better figures More timely comments may
319. otentially possible responses to this item A B C D the mws c2 card says that the first response is to get one point the second and third responses are to get zero points and the fourth response is in fact not used by this item hence the asterisk 2003 2011 Curtin University West Australia a Help file for Lertap 5 Look at the mws cards above They have the same format the column number of the item in question followed by the number of points corresponding to each of the item s response codes If the item does not use one or more of the response codes an asterisk is used Example C11 More about the mws card col c2 c6 sub Name Followup TV9 news quiz Title NewsQuiz key ACCDB mws c2 1 O0 O mws c6 0 1 0 This example is the same as the last one We hear you saying No it s not come on now The last example used five mws cards now there are only two Sure You re right What we should say is that this example accomplishes the same item scoring as the last example Look at the three cards we ve eliminated mws c3 0 0 1 O mws c4 0 0 1 O mws c5 0 0 0 1 These cards say that the items found in columns 3 4 and 5 of the Data sheet use all four response codes have one correct answer and award one point for the correct answer But this is the default Lertap assumes all items will use all response codes have one correct answer and will award one point when the correct
320. other workbook Classic select if filters operate without creating a new subset of the data records 2003 2011 Curtin University West Australia Toolbar and tab 281 4 7 5 ScrunchBoss Look at this workbook sent in by a user in November 2012 X ig J Ey Coba19Nov2012 xlsx Microsoft Excel n x Lertap Home Insert Page Lay Formula Data Review View Develop Add Ins S2 o c gg xS x 2 3 4 5 6 Fi 8 9 0 E 2 13 1 Last Distractor Letter E D E E E E E E E E D E E 2 Answer Letter C D A B A D A C A De 3 Exam Item Number 1 2 3 4 5 6 7 8 9 10 11 12 4 Student 1 A A D B A D D E D A A A 5 Student 2 E C D D A D D D D B B D 6 Student 3 C A D B D A A D G A E y Student 4 C Cc D A B A D E C C A D 8 Student 5 C E c A A A D A C C A C 9 Student 6 C C C A B A D A c D A D 10 Student 7 C A C A B A D A D C A C 11 Student 8 A c D A B A D A M D A D a5 Ce d n A C nN A n L n A e Cc A E M M Data 2 4 m mE i Ready EBEN GI 100 There are several things that need to be done before this Data worksheet is ready for Lertap processing For a reminder of how a Data sheet should be structured please have a gander at this topic zsa As far as this topic goes let us draw your kind attention to the first two rows Last Distractor Letter and Answer Letter Consider the Answer Letter row It contains the correct answer for each item that is it provid
321. other levels But the conditional standard errors of measurement at this point were greater 3 36 using Method IV CSEM2 in the graph and 3 65 using Method III CSEMI in the graph these CSEMs are a better reflection of the true state of affairs around the cut score of 42 Related tidbits NCCA the National Commission for Certifying Agencies allows for reporting of the standard error of measurement at the cut score Use CSEM2 for this Lord s Method IV Classical test theory CTT has frequently been criticized as failing to acknowledge that errors of measurement vary over score levels This alleged failure is in turn often used to highlight an alleged advantage of item response theory IRT The criticism is largely misplaced it is incorrect CTT and IRT both provide for conditional measurement error estimates Interestingly IRT estimates of measurement error are lowest in the middle of the score distribution rising to their highest values at the extremes CTT estimates are the opposite as illustrated in the chart above CTT error estimates are greatest in the center For more information please refer to this paper There s a paper another best seller which has lots more about using cut scores with several examples http ww w lertap curtin edu au Documentation ERM2007d pdf Printing Please refer to this topic for comments on printing the information produced by Lertap Import amp Export Lertap is an Excel appli
322. out how Lertap computes the Statsf figures go back for a look at the MDO cognitive Statsf 531 topic There are two main differences between the Statsf reports for cognitive and affective subtests it is rare for people omitting a cognitive item to get scoring points so the statistics for cognitive item options may not be noticeably different going from no MDO to MDO Lertap will apply a correction for inflation to the pb r and b r values corresponding to the right answer to a cognitive question but this correction is not 2003 201 1 Curtin University West Australia e Help file for Lertap 5 applied in the Statsf report for affective items but it is for Statsb see below The Statsb reports LertapS brief item stats for Comfort with using LERTAP2 created 29 03 71 ones eee re BI pese oe ase IESE B In the normal case without the MDO option the Statsb report for cognitive items has an other column which indicates the percentage of non responses to an item This column changes to n when MDO is in operation as may be seen above Q27 has no missing data its mean and s d values are the same in both tables The Q27 cor figures differ for the reason found earlier in the Statsf reports the values of the criterion measure the subtest score change as we go from no MDO to MDO Q28 s figures differ almost everywhere The percentages mean s d and cor statistics for the second table
323. ow Lertap is being told to make a new workbook whose Data records will be only those with an M in column 2 We have now set up two new data sets two new Lertap workbooks one for the respondents from West Australia and one for those from Maui For a more thorough discussion of using the tst card and breaking out subsets of records see the section titled A survey with multiple groups in Chapter 8 of the manual Good going you stuck it out and have come to the end of this topic page Lertap provides extensive support for different item scoring methods as we ve exemplified above Of course there s more material in the manual where three chapters are devoted to a discussion of CCs cards Still a few years of experience have shown us that the matter of item scoring in Lertap at times creates questions among users no matter how much documentation 2003 2011 Curtin University West Australia s Help file for Lertap 5 3 3 we might provide Should you have questions whiz off a note to us lari lertap com Missing data In Lertap missing data is a term which usually means that a person has not answered or has omitted an item If you look at a Lertap Data worksheet and see empty cells such cells usually indicate that there was no answer to the item corresponding to the cell In the screen snapshot below Virgo has not answered Item 2 nor Item 7 Westphal has left three items unanswered 2 5 and 7 Xen
324. ow much longer the job took when eigenvalues were also computed ian scores wo sigen with eigens 42 secs Note December 2004 we received a trial data set from a Lertap user with N 267 and Nits 150 It took Lertap a total of 14 minutes to produce its IStats report for this data set of this 11 minutes were required to extract eigenvalues from the correlation matrix This test used the same computer mentioned above a Pentium 4 running at 2 GHz At the end of 2004 this Pentium would be considered as being quite dated it was new in early 2002 if you re wanting IStats reports for data sets with large Nits hopefully you ll have a more powerful computer to work on Note July 2010 fresh tests with another data set This one had N 4 700 and Nits 77 In this case it took Lertap 3 minutes to produce its IStats report a figure which includes the 43 seconds required to extract eigenvalues from the correlation matrix We used Excel 2010 for this running on a Macintosh MacBook Pro 2 53 GHz 4 GB RAM with Parallels and Windows 7 if you have a more conventional Windows computer not a Mac you d likely observe better figures Mac users please note the Excel 2011 version of Lertap will compute eigenvalues without problem but is noticeably slower Response similarity analysis Response similarity analysis RSA involves getting Lertap to compare all possible pairs of students to see if their item responses might be similar This
325. owever the manual s discussion is limited to the first two Copy options seen above the other options were added after the manual was printed 2003 2011 Curtin University West Australia 208 Help file for Lertap 5 4 6 1 1 4 6 1 2 Copy Data column This option copies a designated column in the Data worksheet to the Scores worksheet The columns of the Scores worksheet may contain only numeric data Before Lertap will copy a column from the Data sheet to Scores it makes sure only numbers are found in the column to be copied Why will users want to copy columns from Data to Scores There are a few reasons One of the most common reasons is to correlate the values found in a Data column with the values found in a Scores column For example the Data worksheet may have a column with SAT test scores these are to be correlated and scatterplotted with the test scores produced by Elmillon 92 as found in the Scores worksheet Copying the respective Data column to the Scores worksheet will automatically correlate the Data column s scores with the other scores made by Elmillon and open the door to use of the Scatterplot icon 3 on Lertap s toolbar Users wanting to carry out external criterion 92 analyses sometimes have entered the criterion scores in one of the columns on the Data worksheet These scores must be moved over to the Scores worksheet before they may be used as an external criterion When a Data column is copied to
326. owever with Excel 2007 you can have over ten thousand columns Astute readers might have their hands up at this point You said you wanted to have a new column called Language but instead you have Country recoded at the top of the new column You haven t finished have you Correct ho hum We re left with the back breaking task of typing Language into the cell which presently has Country recoded 2003 2011 Curtin University West Australia Toolbar and tab 215 If your ultimate objective is to get group breakouts 4 and you d like to exclude all records with DO or VZ in the original column you d respond thusly St 1 0 2 5 NZ EN 9 5 12 0 4 5 6 0 YZ l 7 0 6 5 VZ x What new code or value do you want to equate to the value of 2 in the original column Jexclude EN You don t have to type the whole word just ex will do When the breakouts routine runs it will ignore all rows which have been excluded in this manner Similarly if there are records you want to delete enter the word delete in the little box or just del without the apostrophes Lertap will set about deleting rows from the Data worksheet once the Recode macro has worked completely down the original column Please note that Lertap will say No No if you re asking for records to be deleted from Data when your workbook also has a Scores worksheet There s a very critical correspondence between the Data and Scores work
327. p 7 ON O in iw IN ie When a student has no incorrect answers ok appears in the list Dipak got all the Soccer items correct Irawadi had no problem with the Running items something we might expect of someone named after a major river on the not ok side of the ledger results suggest that we may not want Anton on our football team Note if a subtest has Wt 0 on its sub line it will not be included when this macro goes about its business To read more about this Wt 0 thing zero in on the previous topic 253 SAQ What happened to WrongltemList1 Very good question Nate Thompson Assessment Systems Corporation wrote WrongltemsList1 as a special macro for a Chicago based Lertap site They re still using it in the Windy City and you can use it too it s in the Lertap5MacroSetA xlam file where it s simply called WrongltemList It makes some assumptions regarding the nature of Lertap subtests and may fail if your subtests use complex col lines in CCs worksheets For more assistance write larry lertap com License Menu A license in the case of Lertap 5 means a permit to use the software Without a license users can enjoy Lertap 5 for only a certain number of days For example the 30 Day Trial Copy will run for 30 days and then lock Users must purchase an unlock code in order to continue to use Lertap The License menu makes it possible to apply for an Unlock Code The License menu seen on your comput
328. p that the second and third items are to be reverse scored that s what the minus signs mean The little plus signs tell Lertap to score items 1 4 and 5 in the usual forward manner Example A3 Now we ll put in some labels to grace Lertap s reports KOOL o3 sub AFF Name Beach survey 1 Title Beachin pol The Name shown above will appear at the top of some of Lertap s reports such as Statsl1f and Stats1b while the Title will show up at the top of one of the Scores worksheet s columns The Name can be any length Title should be kept to 8 characters or less If Name and Title are not given Lertap defaults to Name Test 1 Title Test1 Example A4 Next we ll add some more control words to the sub card and then explain what they accomplish col c3 cT7 sub AFF Title Beachin PER SCALE pol t The PER control word prompts Lertap to compute a percentage score for each respondent being his or her score expressed as a percentage of the maximum possible score For example on our little test of five questions the maximum possible score was 25 someone with a score of 15 would get a PER score of 60 SCALE gets Lertap to normalise the scores It divides each person s score by the number of items This is best used when all the items have the same number of options it results in a score scaled back to the scores used at the item level 2003 2011 Curtin University West Austral
329. parative measure which indicates how the kurtosis of the scores compares to that of the normal curve Negative values mean the scores are less peaked than the normal curve 2003 2011 Curtin University West Australia Input amp Output 265 MinPo The lowest possible score on the test This will always be S less than or equal to Min the lowest score found MaxPo The highest possible score on the test Always equal to or S greater than Max the highest score found Correl These appear only when there is more than one score ations Each value is the Pearson product moment correlation between two scores The average correlation is found by computing the mean of a score s correlations with the other scores These correlation coefficients may range in value from 1 00 to 1 00 Lertap s Scatterplot 3 option may be used to graph the relationship between any pair of Scores 6 4 3 Stats reports Lertap makes two or three Stats reports for a subtest Every time you run Lertap s Elmillon option Stats1f and Stats1b reports are created If the subtest is a cognitive one a Stats1ul report is also usually created It is possible to stop Lertap from creating Statslul reports by using a setting in the System 2 amp worksheet The f in Stats1f stand for full These reports have the most detailed information for test items Statsf reports go back to Lertap s birth in 1973 they have changed a bit over the years
330. perates by changing the appropriate row in the System Worksheet 4 It s possible to tell The Spreader to forget about the first pass the one where it checks string lengths The Spreader is not slow but as you d expect it runs even faster if it doesn t have to make two passes At Lertap HQ we favour the two pass method of operation as we feel it s reassuring to check string lengths However even the two pass Spreader will not control for a nasty problem when the string has been created by importing data from a scanner made text file blanks at the start of the string may be lost Such blanks usually correspond to unanswered questions It s rare for a respondent to leave the initial questions unanswered but it does happen and when it does real care is required to make sure that the blanks remain at the start of the string If they do The Spreader will spread them If they don t woe the string will be shifted to the left and Lertap will be unaware of what s happened If a test is being scored the score will be wrong There s a bit more about this nasty in the Import amp Export s 4 topic 4 2 4 Sort A toZ 31 This icon is used in conjunction with the Scores worksheet It permits the information in the Scores sheet to be sorted according to criteria entered by you the user When a sort is requested Lertap makes a copy of the Scores worksheet and adds it 2003 201 1 Curtin University West Australia s Help file for L
331. piciously similar possibly implying cheating Is it common for tests to have weak items Yes it is difficult items with poorly functioning distractors will often fall under this definition of a weak item Note that a weak item in RSA terms does not necessarily mean a bad item bad items are generally those with a negative discrimination index it is possible for an item to be weak in RSA terms but still have an adequate discrimination figure SCheck Wesolowsky The RSA analyses mentioned above all have to do with how Lertap looks at the matter of response similarities Lertap s procedures are based on those first developed by Harpp amp Hogan amp at McGill University Canada At another Canadian university Wesolowsky 2 has developed other methods for detecting excessive response similarities Wesolowsky s SCheck program is based on them Lertap s RSA procedures will automatically produce a file which will slip right into SCheck more about this in steps 2 and 4 below Summary of RSA steps To review here are the steps required in order to have Lertap do its RSA magic 2003 2011 Curtin University West Australia Toolbar and tab 129 1 You have to say yes to RSA in the right spot in Lertap s System worksheet As this topic went to press the right spot was row 25 column 2 2 You must go to the Run menu and click on Output item scores matrix This will produce the RSAdata worksheet and also the SCheckData
332. ples of text files are those which may be processed on a Windows based computer with the Notepad or WordPad programs or on a Macintosh with a program such as TextEdit On Windows computers text files often have an extension of TXT In the data processing world data files are often text files ASCII files and they frequently have an extension of DAT Many of the programs from SSI Scientific Software International www ssicentral com and from ASC Assessment Systems Corporation www assess com enjoy receiving their input from text files saved with an extension of DAT As you now well know Lertap s repertoire of data processing capabilities includes a provision for creating Excel worksheets formatted so as to be compatible with some of the ASC and SSI programs For example Lertap s XCal worksheet is made for use with ASC s XCALIBRE program while Lertap s DAT worksheet is designed to be friendly to SSI s Bilog MG program But the ASC and SSI programs cannot yet read from Excel worksheets We require a way to save Lertap s XCal and DAT sheets so that they re text files ready for input to the other programs Can do Sure In fact there s more than one way First make sure that the DAT or XCal worksheet is the active one the one currently in view About the easiest way to prepare an ASCII file is to select copy and paste Select all of the rows in the DAT or XCal worksheet copy open the Notepad WordPad or TextEdit program
333. ply the macro and all following charts are tizzed See the following topic 10 Move em Dan O Alright you may not be a fan of Hawaii 5 O but here s another note one which will be a bit more useful when we get into the DIF topic If you might want to move charts to a worksheet of their own create a new worksheet then use the Move Chart option seen above to shift a selected chart to the new worksheet Move one two or how ever many you want and in theory they will line up well in the new worksheet Related tidbits There are many books on using Excel charts See for example John Walkenbach 2007 Excel 2007 Charts Wiley Publishing ISBN 978 0 470 04400 1 4 4 7 1 1 ChartChanger1 When you installed Lertap several files were set up on your computer One of them is the Lertap5MacroSetA xlam file This file has some special purpose macros little collections of computer code designed to meet special needs Read all about this with a clickity click 2 3 here There s a fair chance your copy of the Lertap5MacroSetA xlam file has our ChartChanger1 macro in it If it does when you follow the instructions found in the clickity click link above you will see ChartChanger1 listed as an available macro If it doesn t write to us at larry lertap com and complain we ll send out the right version of the file it s free The ChartChangerl macro operates in this way you make changes to the first chart seen on a wo
334. r key and Excels displays the value of the new score which in this example is 495 If this is what s wanted the user then selects the cell with the new formula by clicking on it cruises up to the Move menu and clicks on Apply a special Scores worksheet formula Your beloved little Lertap then applies the formula to all other original scores determines the descriptive statistics related to the new score and updates the correlation matrix found at the bottom of the Scores worksheet lo 2003 2011 Curtin University West Australia 222 Help file for Lertap 5 E3 Microsoft Excel GCdata xls SEE File Edit view Insert Format Tools Data Window Help X X New S rZ puny Shorts M l lil Move License Lelp Abb5 X f Correlations 1 Lertap5 Scores worksheet last updated on 29 07 2004 2 Record No Testi MewScore n Min Median Mean Max s d var Range IQRange Skewness Kurtosis The label given to the new score NewScore may of course be changed Don t like the results Select the new score s column delete it and start again Want to know more about working with formulas in Excel Look for assistance in Excel Help there s lots you might start by searching Excel Help for create a formula 4 6 1 6 Update correlations This option refreshes part of the Scores worksheet the part where the correlations appear which is always at the bottom of the worksheet You might want t
335. r 2004 a new CCs card or line was added to ease the task of excluding multiple items Its format is identical to that of the col card The following CCs line will remove or exclude the item in column 37 exc c37 Other examples of the use of this card exc c12 c14 c42 Removes the items found in columns 12 13 14 and 42 exc c12 c13 c14 c42 2003 2011 Curtin University West Australia Help file for Lertap 5 Also removes the items found in columns 12 13 14 and 42 exc C17 c21 c25 c27 c35 c40 Will exclude the items in columns 17 21 22 23 24 25 27 35 36 37 38 39 and 40 Both the mws and exc lines may be used to remove items from a subtest as shown in the example below mws c17 mws c27 exc c21 c25 c35 c40 The three lines above will see that the items in columns 17 21 22 23 24 25 27 35 36 37 38 39 and 40 are excluded from the subtest To see how the exc card can ease the process of removing items from a subtest look at the following example Scale 1 uses 28 of the items col ci c20 c31 c50 sub Res A B C D E Name Critical Skills Title Critical Wt 0 key CCACB CABAB BDBAD BBAAD ECDCE ABDDA CBCBB BDDCA tmws cl mws c3 mws c6 mws c10 mws c15 mws c17 muws c31 mws c34 mws c38 mws c40 mws c46 mws c49 Scale 1 uses 28 of the items col ci c20 c31 c50 sub Res A B C D E N
336. r 4th 3rd 2nd upper Key 1 Diff 70 Disc 35 Key 4 Diff 2 50 Disc 41 I5 I6 1 00 1 00 r 0 80 r 0 80 1 il 0 60 0 60 2 2 n 0 40 E 0 40 v M 4 gt M Statsif lt Statsib csemi lt Statsiul StatsiulChta 3 fla i i Ready 2 EB EI 10 C g uz In this case ChartChanger2 was directed to make a grid having two charts per row with no blank lines between the rows If you re using a large monitor with your computer you could have four or five charts per row and end up with a display which quickly captures the performance of all test items A useful tool for an item review session with colleagues 2003 2011 Curtin University West Australia Toolbar and tab 189 The ChartChanger2 macro makes it a simple matter to resize quintiles In the example above the charts were made with the Excel 2010 version of Lertap The first quintile was reduced in size the macro was run and the display above was the result The Excel 2007 version behaves the same way While we re talking about changing quintiles why not take a minute to consider the matter of deleting some of the quintiles If you d like to delete a quintile just right click on it and select Cut Bingo it s gone But doing this deleting a chart will leave a hole or a gap in the display Not to worry just run ChartChanger2 again We could also discuss how to use Excel to select all of the charts on any
337. r affected by this action For a related topic see deleting secondary worksheets 8a Chapter 9 in the printed manual also discusses these matters Production mode A production mode capability was added in May 2005 Have a look at the following snapshot of lines 35 through 38 of the System Worksheet taken in June 2009 a i Er Bi HE A gz 8 v Lertap5 xlsm Microsoft Excelnon 5 X Lertap Home Insert Page Layout Formulas Data Review View Developer Add Ins ox xX Delete 3 Sort M Blank E Interpret al Histograms 4 Move v Version Bx Line Headers gH Elmillon Scatterplot License fe Spread Excel amp 3 Copy More b Res charts Lelp Basic options Newmenu Run menu Graphics trio Other menus 1 2 3 4 System Settings Allowed Usual T settings setting es no yes no no yes no no yes no mo yes no no E These are Lertap5 system settings Change them only if ycu understand them 33 Automatically exclude weak items 34 emp 35 Run in production mode 36 Include histograms in production mode 37 Include response charts in production mode 38 Include items scores matrix in production mode 39 emp BEEN C Lu o ee M 4 gt M Comments lt Data CCs System Syntax Ready FJ 2003 2011 Curtin University West Australia Toolbar and ta
338. r of intervals by entering a number from 2 to 9 in the box below IF you enter 2 every two intervals will be combined enter a 3 and every three intervals will be joined and so on At this point you ll enter a compression factor an integer a number from 2 to 9 Let s see Lertap says that my Histo1L report presently has 94 intervals Were to make an Excel chart from the Histo1L report using all 94 intervals know from experience that the chart would either be too large or if I ve re sized it to make it small enough to print on a standard page size it ll be too busy too hard to read So I ll enter 4 in the box and click on OK Before you can recite the birth dates of all the ancestors in your family Lertap will come forth with a nifty table which looks something like this 2003 2011 Curtin University West Australia E d Fingal 2 1 11250 28 11650 22 12050 27 12450 30 12850 29 15028 NEN 140 50 is 350 2 Toolbar and tab If your luck holds good there will also be a spiffy little chart a true wonder ready for inclusion in that report you wanted to have ready for the school board meeting on Monday 2003 2011 Curtin University West Australia 168 Help file for Lertap 5 Histogram of scores e N NUW in E oOo wm gt o c o gt
339. rdinary Statsf report as seen in the next topic 2A The correlation bands plot the r ec values found for the items In this example only one item 16 had a reasonable correlation with the external criterion generally we d want these correlations to be at least 0 30 As seen in the upper part of this topic 21 s r ec was 0 13 items with negative r ec values are found in the 00 correlation band Difficulty calculations The difficulty of a cognitive item is traditionally defined as the proportion of people who answered the item correctly If for example 8096 of test takers identified the correct option to Item 1 we d say Item 1 s difficulty was 0 80 But what if there is more than one right answer to Item 1 What then What do we do 2003 2011 Curtin University West Australia Input amp Output when the scoring of a cognitive item is no longer dichotomous right wrong but instead exhibits polytomous scoring We might consider a different way of expressing item difficulty under such conditions selecting one of the following Lertap methods 1 proportional Under this method item difficulty is the sum of the people who selected one of the correct answers divided by the total number of people responding This method counts any response as being correct if its corresponding weight is greater than zero This method does not take into account any differences which may exist among response weights 2 item mean
340. rds on CCs cards 74 cook s tour 14 323 copy data 207 copy Data column 208 copy Scores 208 copying a workbook 88 cor 61 290 correct answers 231 correlation 61 correlation bands 277 correlation coefficients 92 97 correlation matrix 93 correlation ratio 131 141 correlations 106 263 correlations item component 322 correlations tetrachoric 325 correlations coefficients 265 correlations refresh 222 create a new workbook 87 Create text file 112 credit an item 29 325 criterion score 271 criterion referenced testing 306 critical note 29 257 Cronbach 73 282 cross two variables 131 CRT 306 csem 282 CSEM1 309 CSEM2 309 csem1 report 309 csem1 worksheet 309 csvfile 115 312 CTT 282 309 Curtin University 14 326 cut score 309 cut scores 306 cutoff score 306 cutoff value 103 D D 118 DAT 103 112 DAT file 93 97 98 118 209 312 Data 229 data analysis steps 88 dataentry 82 data import problem 82 data integrity 90 Data matrix file 101 datarecords 253 data tables 183 Data worksheet 131 141 207 253 255 date Lelp last updated 1 date of Lertap version 12 daughter s tickle 242 David Weiss 316 dBASE 312 decision consistency 306 default 280 default font 253 default Res 23 257 default response codes 46 definition of Lertap workbook 253 delete 212 delete columns 115 delete icon 80 2003 2011 Curtin University West Australia Index 331 delete Scores colum
341. re as fine as those of California s Napa Valley Strongly agree Agree Undecided Disagree Strongly disagree U1 330 N H The mws call card will reverse the default scoring weights equating the first response code 1 with 5 00 points and the last response code 5 with 1 00 points Astute readers might note that the scoring accomplished by the three CCs cards of this example is the same scoring achieved in example A7 In A7 we used a special Res declaration to do the job whereas now we re using mws call Examples A7 and AQ will result in the same item scoring but Lertap s Stats1f and Statslb will differ One could see the differences by running with the following set of CCs cards col c3 c7 sub AFF Res 5 4 3 2 1 Title WAuni pol zool c3 e7 sub AFF mws call 5 00 4 00 3 00 2 00 1 00 We ve combined the two examples A7 and A9 making two subtests from the same items Now the two full statistics reports Statslf and Stats2f can be compared item and test stats will be the same but the order in which item response codes are listed will differ The same will hold for the Stats1b and Stats2b reports Example A10 Item scoring weights can be any real number positive or negative col c3 c7 sub AFF Res 1 2 3 4 5 6 7 Title CapesAtt mws call 3 00 2 00 1 00 0 00 1 00 2 00 3 00 This type of weighting is sometimes seen when semantic differential items are used as in the followin
342. reteira ERE e au ien eee bt ec E ERES 151 Enhance MCHIChart eot rper tette v Meer ege dee ety eos eed eco e o n re ede 155 System settings s 158 TO pde M 159 2003 2011 Curtin University West Australia m Help file for Lertap 5 6 Graphies MiO renon EA EA T AATA Histogram Scalirig FISASIO i eR SELBE EAE The FOOIPAK AOE Iri cie rtt rp Pre RERO RUE B EROR EHE ERRAT curl Response Chart Unidimensional response charts Quintil plols ress As Ab edn Paes vase cc Tee Recreativo A ChartChangers ChartChanger3 settings EG quiritile plots o ee tecetec Ee tee t Rr LL d Quintile Options c rene PR IRR TE ODORE ERIS Ghart problem e E ChartiColore 14a certe ns endete Sdcchascesuce sath tess EA A iss ecu M destin ese esed ace datet Reine LM Nee P Copy Data COMM p Copy SCOreSCOIUMIN 5 oe SAS AAT Ete Copy Data to DA Wiss 209 Recode Data collins see Seri et He a ee es eee ae 212 Excel s recOd6rss sce el acelin eee o ege a ree adeeb 216 Apply a formula 5 xcd ete etna eer E ite niei tertiis 219 Update correlatlofis cepa tn tn tct eden e eea eiae dro on eee nda 222 7 Macs IUE E 223 LINKING ROREM cR 225 L
343. rksheet such as Ibreaks and then run ChartChanger1 Bingo All other charts on the worksheet are changed so that they match the type and style of the first chart How to change the first chart Select it by clicking on it Then use Excel s Chart Tools a suite of options which lets you change the design layout and format of a chart 2003 2011 Curtin University West Australia Toolbar and tab 151 4 4 7 2 DIF A couple of topics back at Ibreaks 4l an answer of No was entered when the following question appeared x Okay You have only two groups and are working with a cognitive subtest Want to go for a Mantel Haenszel analysis to check for differential item Functioning DIF If you say No then you ll get an ordinary group response report for each item Yes No Cancel Now an answer of Yes will be entered something which prompts the Ibreaks routine to ask one more question x Fine Now of your two groups which will serve as the reference group Answer Yes to make Ing the reference group Answer No to make Nat the reference group Yes No Cancel For this example an authentic one to be sure a professionally developed 40 item achievement test had been used for years as an important assessment tool in science education It had been presented in the country s native language that is the one most used in the general population However a push to promote the wider use of
344. rning management systems support the development and delivery of online cognitive and affective tests Some such as Angel Learning can be made to randomly sample items from a database presenting different students with different versions of a test each with the same number of items In classical test theory when certain conditions have been met these versions might be termed parallel forms or equivalent forms However as of March 2006 the data file of item responses created by Angel does not indicate which test form a student took Fortunately the Angel output is padded 2003 2011 Curtin University West Australia CCs details 53 with did not see codes so that each data record has the same length a length equal to the total number of items in the database In the Test Pilot example discussed above the test included a few items which used a Check all of the following options which would be correct format effectively turning a single item into multiple true false items Test Pilot actually served up the same number of items to each student 40 but those items which used the check all of the following format did not consistently offer the same number of options The practical end result was as described above students received a variable number of test items Did not see data will affect the calculation of alpha Lertap s reliability coefficient alpha is NOT corrected for did not see cases 3 3 2 MDO cognitive Statsf
345. rs who have their input in Iteman 3 style can use the importer Should you have an interest in using the new Iteman 4 input style with Lertap please use let us know larry lertap com Meanwhile note that it is very easy to have Lertap create files suitable for input to Iteman 4 Should this be of interest to you read this topic on Lertap and Xcalibre Iteman 4 and Xcalibre 4 share the same input files so activating Lertap s Xcalibre export feature will result in files suitable for use by Iteman 4 as well as Xcalibre 4 Related titbits A paper with more about ITEMAN and Lertap is here http www lertap curtin edu au Documentation ItemanAndLertap5 pdf pdf file about 1 5 MB A discussion of the correlation methods commonly found in item analysis programs with emphasis on why Lertap s results sometimes appear at variance http www lertap curtin edu au Documentation ItemCriterionCorrelations1 doc Word file about 190 KB 2003 2011 Curtin University West Australia R amp R amp R amp R R amp R amp R amp R Revisions The manual was printed 1 December 2000 Numerous changes have been made to Lertap since then some minor some more substantial These have been documented over the years With the spawning of the Excel 2007 version of Lertap released mid 2008 there is now a frequently revised compendium of the updates which apply version Lertap 5 7 1 and above It s here http lertap curtin edu au Document
346. s and group breakout plots you can In fact you can alter Excel s colour palettes so that the changes you make automatically apply to all charts in a workbook Once you ve coloured things in the way you like you can then pick up the colours used in one workbook and carry them over to another workbook If you don t have a colour printer you may have had the occasion to frown after trying to print the coloured charts made by Lertap another Excel application or a program such as SPSS Frown not fair friend use the procedures introduced here 2003 2011 Curtin University West Australia 206 Help file for Lertap 5 along with some experimentation to pick colours which will print well enough letting you get by until the departmental budget has enough funds to buy a colour printer Here s what to do if you re an Excel 2007 or Excel 2010 user Click on the Office Button the IF ile button in Excel 2010 Click on Excel Options a button found towards the bottom of the screen Click on Save Look for the section which says Preserve visual appearance of the workbook and then click on Colors Color ea Color Standard colors Chart fills Chart lines Copy colors from Cancel Then for more information on how to re map the colours please refer to this Microsoft document 2003 2011 Curtin University West Australia Toolbar and tab 4 6 Other menus Way to the right hand side of t
347. s If it does not then Lertap looks to see if the response matches the did not see code assuming the did not see option has been activated If it does not then the response is classed as other a response category often referred to in Lertap documentation as missing data Lertap will automatically adjust all of its item statistics so that they exclude did not see cases that is of course assuming the did not see option has been activated if the option has not been activated then what would have been a did not see response will be included in the other response category and processed as missing data Use the Fregs worksheet 14 to check on missing data and did not see cases and note if the did not see code is a blank Freqs will include did not see cases in its other line To see what the various Stats reports look like when the did not see option is on just page forward to the following topics Scoring Whenever the did not see option is in effect users have the option to have Lertap calculate two scores for every person on each subtest defined in the CCs worksheet the usual subtest score and a percentage score which is based on the maximum possible score which could have been obtained on the items actually presented The first score the usual subtest score will be the number of points earned on the number of items the person had the opportunity to answer In the cognitive case the most comm
348. s the one Lertap was working with when the Item scores and correlations option was selected 2003 201 1 Curtin University West Australia 12 Help file for Lertap 5 The item control file created by Lertap 5 9 2 is called LrtpXcallCF txt It is also a tab delimited text file and it will also be placed in the same folder as the active workbook Here s a picture of the File tab in Xcalibre 4 1 showing links to the two files from Lertap Wl xcalibre Version 4 1 0 0 lal xl File Edit r License Status QC Xcalibre 4 1 pemo RT Hem parameter calibration WWW assess com Copyright 2011 Assessment Systems Corporation Files Input Format IRT Model Calibration Estimation Output Options Data matrix file C ASC XCalibredT Lertap Files MathsQuiz LrtpXcalData tt M Data matrix file includes an Xcalibre 1 1 Header iv The data matrix file is delimited by a C Comma Tab Item control file C ASC XCalibreat Lertap Files MathsQuiz LrtpXcallCF tet Output file C ASC XCalibreaT Lertap Files Maths Quiz MathsQuizSub2 rt Run title Maths Quiz Subtest 2 with mws lines Save the item parameters in ASC format par Tab delimited format t t CSV format csv Save the IRF and IIF graphs to a separate external file Save the scored item responses Save the item control file f Include Omit codes in the scored matrix Include Not Administered codes in the
349. s Excel to consider the entry as text not a number Another way to define an entry as text is to use Excel s Format menu Cells then select Text The cells in an entire column may be formatted as text by selecting the column before using the Format menu The Spreader is a favourite tool at Lertap HQ It s powerful indeed It is often useful 2003 2011 Curtin University West Australia Toolbar and tab se when importing data from other applications including scanners The data entry aids are discussed in Chapter 3 of the manual under the section titled Entering item responses In the printed manual the reference is page 61 Update note In September 2003 The Spreader gained more smarts Its standard method of operation now involves two passes down the cells whose contents are to be spread In the first pass the length of the cell s string is compared to the length of the string in the cell above If the lengths are not the same The Spreader sounds an alarm You get the chance to stop The Spreader or to continue If you choose to stop you can then edit the cell Once The Spreader has worked its way down all the relevant cells the first time it will then ask if you re ready to truly spread cell contents You can stop at this point without anything having changed If you elect to continue The Spreader goes back up to where you last started it from and spreads things to the right You can change the way The Spreader o
350. s from left to right But now look at item A36mc 2003 2011 Curtin University West Australia Toolbar and tab QuintileTests310ct03 xls ce EC21 EC 2 Kev 2 Diff 28 Disc 23 9 1 4 2 4 amp 3 4 5 e other j ECStatsluchtb y ECStatsiulChta The proportion of people who identified the keyed correct answer 2 was noticeably higher in the practice group The lines are no longer all close to horizontal There were a few other items with patterns like A36mc s However it was a study of the quintile b plots which highlighted a major message the practice group stuck it out longer they answered more test items students without practice tended to get bogged down appearing to run out of time Look 2003 2011 Curtin University West Australia 198 Help file for Lertap 5 l QuintileTests310ct03 xls 2 3 4 5 Key 1 Diff 28 Disc 28 EC 2 mBR EC 1 M 4 gt MECStatsiulChtb ECStatsiulchta lt Other means a student omitted the item Notice that almost 4096 of the no practice group omitted item A49mc whereas the omit level in the practice group was below 1096 This pattern set in at item A49mc and continued to the end of the test without exception Well the gap did narrow somewhat after about the 65th item when the proportion of omits in the practice group began to rise rapidly Of course we didn t need plots s
351. s process Lertap also writes some intermediate worksheets with data required by the Elmillon program These are the Sub worksheets users with a keen eye can see the Sub worksheets being formed as Lertap goes about its business but then just before focus shifts to the Freqs 2003 2011 Curtin University West Australia Toolbar and tab 8 worksheet the Sub worksheets are hidden from view Users usually have little need to see them however they re not secret they can be unhidden There will be one Sub worksheet for each of the CCs worksheet s col lines The second of Lertap s usual two step data analysis process involves clicking on the Elmillon option This is the option which creates the various statistical reports which are Lertap s reason for being Each of these reports is an Excel worksheet they have names such as Stats1fl 2s Stats1b 290 and Statsullzsa1 Elmillon also produces scores one for each subtest or scale For more information on what the Run menu s options do click on the topics shown in the box below E Interpret ill Histograms z Mov Elmillon Scatterplot Licer d More b Res charts Lelp Use external criterion i2 Item scores and correlations B RSA response similarity analysis ka Breakout score by groups ir Box and whiskers Cj Item responses by groups 3 Enhance M H charts Vj To halve and hold Lelp 4 4 1 Interpret CCs lines Y Interpret
352. s the same as alpha when cognitive test items are all dichotomous see tidbits below index of In CTT classical test theory this is the correlation reliability between observed scores and true scores It is computed by finding the square root of the reliability estimate standard Often abbreviated as SEM this is a measure of error of measurement precision inversely related to reliability measurem Very reliable tests will have little measurement error ent The SEM is often used to form confidence intervals Adding and subtracting one SEM from any test score gives what is often called the 6896 confidence interval a range of scores which given certain assumptions captures the true test score with a probability of 0 68 In CTT SEM is an average figure A more accurate estimate of measurement error is found by computing the conditional standard error of measurement a figure which varies by test score Lertap s csem s3 report provides estimates of measurement error by test score 2003 2011 Curtin University West Australia Input amp Output ms Related tidbits To see how Lertap can be used to calculate split half reliability estimates have a go with the HalfTime dataset An example of equivalent forms reliability also known as parallel forms reliability may be seen here KR 20 and KR 21 are reliability estimates stemming from the work of Kuder and Richardson 1937 KR 20 and
353. s to have an above average test score The aa flag will show whenever an option with wt at or below 0 00 has a positive z value Here aa stands for above average it ll come out to flap in the breeze whenever a distractor an incorrect option is selected by above average students This is an unwanted outcome as we expect the above average students to get the item right What has happened is that 111 has been mis keyed The correct option on 111 was 4 not 1 An error was made when the key line was typed into the CCs worksheet Once this error is corrected we can expect Stats1f to display no flags at all for 111 Why has Lertap flagged 114 and 115 Because both of these items have a distractor selected by students with above average test scores The flagged distractors for these two items have z values just above zero meaning that the students who selected them were slightly above average This is unexpected It probably indicates that the items need to be reviewed what is there about these distractors which has made good students see them as plausible Asking the students is likely to uncover the underlying reasons There is one more flag which may appear in a Stats1f report It s no It will appear whenever an item option has not been selected by anyone It is possible to stop Lertap from using these flags in its Stats1f reports There is a setting in Row 58 of Lertap s System 2 worksheet which controls it
354. s to use the New menu to Make a new blank Lertap 5 workbook the Blank option Then from the original workbook copy all data records to the blank workbook s Data worksheet Click here for a definition of a Lertap workbookl zs The New Menu is discussed in Chapter 3 of the manual under the section titled Setting up a new workbook Note the examples in the manual are all based on older versions of Excel not on Excel 2007 2010 In the printed manual the reference is page 60 Blank This option opens a new Excel workbook with two blank worksheets One of the worksheets is named Data while the other is named CCs The Data worksheet is split after Row 2 the worksheet is fixed so that the top two rows always display header information Note that the new workbook should be saved as soon as possible Excel s File menu options are used for this 2003 201 1 Curtin University West Australia Help file for Lertap 5 4 3 2 4 3 3 4 4 Click here for a definition of a Lertap workbookl 23 Headers This option creates a new Excel workbook with Data and CCs worksheets which are empty of data records but otherwise identical to the original workbook The Data worksheet s first two rows the header rows will have information copied from the original workbook s Data worksheet The lines in the new workbook s CCs worksheet will be identical to those in the original workbook s CCs worksheet Note that the new workb
355. s topic assumes some familiarity with material found in the topics immediately preceding If you haven t been through them take a few minutes to read the Missing datal 461 topic followed by the topic dealing with the Did not see option 48 Then report back here Ready set go Have a look at the following CCs lines The lines above set out two subtests Both are affective as the Aff control word has been used on each of the sub cards Both subtests involve the same ten items no doubt you recognize the subtest Right 2003 2011 Curtin University West Australia Help file for Lertap 5 it s the set of Likert style Comfort questions found on the Lertap quiz 23 And no doubt you also recall that the items themselves may be see in Appendix A of that best seller the Lertap manual Very good The only differences between the two subtests are found in the sub lines The second subtest uses the MDO control word Okay rig yourself up with a refreshment of some sort polish your glasses and have a gander at Lertap s reports for these two subtests The Statsf reports LertapS full item stats for Comfort with using LERTAP2 created 2 Q27 option wt n 96 b r avg z i 5 00 3 5 0 0 36 41 7 1 56 2 4 00 14 233 0 40 37 8 Difa 3 3 00 22 36 7 0 05 34 8 0 06 4 2 00 e 35 0 D 57 30 9 0 78 5 1 00 D 0 0 0 00 0 0 0 00 Q28 option wt n b r avg z a 5 00 13 el D 24 32 4 D 46 2 4 00 27 45 0 0 46 36
356. sheet which keeps track of the location of the items comprising the subtests or scales Note that a Lertap workbook does not include the Lertap toolbar 78 It bears mentioning that the Lertap5 xlsm file is indeed a Lertap workbook but if someone uses the term Lertap workbook they are not necessarily referring to Lertap5 xlsm We could say that the Lertap5 xlsm workbook is a very special Lertap workbook Why Because the Lertap5 xlsm workbook includes the Lertap toolbar something no other Lertap workbook will have 2003 2011 Curtin University West Australia Input amp Output 255 6 2 Data sheet The name of the Excel worksheet where data are recorded for Lertap analysis has to be Data The first two rows of the Data worksheet are for header information as described in the definition 53 of a Lertap workbook Have a look at the top of a typical Lertap Data sheet 1 2 3 4 5 6 7 8 9 1l Data maa res from principals and teachers April 2003 2 Position Experience Gender Qi Q2 Q3 Q4 Q 3 5 3 2 2 4 3 4 4 4 4 B 3 1 2 2 1 3 4 3 5 C 1 3 2 4 2 4 4 4 6 D 1 2 2 3 3 3 3 3 7 E 1 3 1 3 2 4 4 4 8 F 1 3 1 2 2 3 4 4 9 G 1 2 2 4 3 2 4 3 10 H 1 3 2 4 3 3 4 4 Row 1 is a general header or title which can contain any information you wish including nothing at all Whatever is typed in this row will not appear anywhere else Lertap doesn t read this row This row is for your own use we use it to provide a brief reminder of the information
357. sheet will be created by Lertap when the appropriate option is set in row 23 of the System worksheet 2A Here s a screen shot froma Lertap DAT worksheet one froma data set having 25 cognitive items using a record ID field with a number between 15001 and 28000 E3 Microsoft Excel SetWithN2798 xIs Sle File Edit view Insert Format Tools Data Window Help X news S rZ pune ON l lll Move License Lelp H 541 1X 254A1 1 21608 1010101000100001000011000 21607 1000100100110001100100010 21605 1000100000101010000100001 21602 1101011110110101011110110 21601 1000000100011000010100100 21597 1110110110110011111110010 21596 0000100001111101000100010 21593 1100110001011001011100000 21618 0000000010111000000000000 21623 1110000000010000010100100 21626 1010000000010000011000000 21633 10nnnm anmnnmnn eammnnan 03m 0nnm1nmnnmnninan M H A Data CCs f Fregs IStats DAT Ready 2 3 4 5 6 7 8 9 10 11 m n3 Every time Lertap makes a DAT worksheet it inserts a Fortran format statement at the top The line above says the data records start with a 5 column ID field followed by a space followed by 25 item scores Here s a snippet from another DAT worksheet This one corresponds to a data set with 45 cognitive items and an ID field with student names The longest student name was 14 characters wide Lertap has right justified the names using blanks on the left side whenever the name was shorter than 14 chara
358. sheets and Lertap tries its best to see that this correspondence is not disturbed Herewith all the usual warnings about deleting records from Data you cannot recover them Best to make a copy of the workbook before deleting records something you can do by using the New menul7 If you click on OK without entering anything in the little box the Recode macro will use whatever value you last entered This makes it a bit easier to apply the same new code multiple times If the original value is blank or empty then to maintain the blank press your keyboard s space bar once and then click on OK Otherwise if you don t want blanks in the new column simply enter something in the little box and in the blank of an eye Finally as you may know Excel has its own recode facilities and they re quite 2003 2011 Curtin University West Australia 216 Help file for Lertap 5 respectable If you page forwardl2 4 to the next topic you ll see 4 6 1 4 1 Excel s recoder The Recode macro described in the preceding topic is not really a recoder It doesn t alter the contents of the original Data column instead it copies the indicated Data column and then lets you create a new column with values or codes based on those found in the original column This is much more along the lines of creating a new variable or of transforming an original variable to a new one to use terms which may be familiar to SPSS users Excel 2007 has Fin
359. sidered to be the end of data Users sometimes use this fact when they re testing their CCs lines to make the test go faster a blank line may be inserted after say 10 data records this stops Lertap from reading all the data records If the first column of a row in the CCs worksheet is blank then that is considered to be the end of the CCs lines Lertap will not read beyond this line This is useful when a user only wants to have the Freqs worksheet produced in this case the first CCs line will be col and the second line will be blank Lertap will produce its Freqs worksheet and nothing else Ideally a Lertap 5 workbook has its default font set to Verdana with the CCs worksheet being an exception in that it may at times make use of the Courier New font Lertap workbooks may have a number of secondary worksheets in addition to the primary ones Examples of secondary worksheets are Freqs Stats1f Stats1b Scores and so on Secondary worksheets are usually the result of applying a Lertap or Excel function for example the Elmillon item analysis option on the Run menu reads data records from the Data worksheet a primary worksheet and produces such secondary worksheets as Statslf and Stats1b For a related topic see Deleting secondary worksheets 80 Lertap workbooks may also have other user created worksheets For example in data sets with more than one subtest or scale users will sometimes create a codebook work
360. so MTE a a of Eus re fa Eee rs zs a e Eus aee e we ue ee we ae 7e i Reliability coefficient alpha 682 M M Scores lt Statsif Statsib A 4 ui 100 Part of a typical Statsb report for a cognitive test is shown above Cognitive tests have a second part in Statsb a plot of difficulty and discrimination see examples in the next topic z5 Results for each item are presented in a single row The percentage figures reflect the popularity of each item option that is the percentage of students who selected the option This is the same as p in the corresponding Statsf report The keyed correct option s results are underlined If an item has more than one keyed correct option results for each option are underlined The other column gives the percentage of students who did not answer the item or had an invalid response Invalid responses are for example responses which have been incorrectly entered in the Data worksheet resulting perhaps from a typing mistake Invalid responses may also result from the use of an optical scanner Bubble 2003 2011 Curtin University West Australia Input amp Output answer sheets are used with scanners students will sometimes shade in more than one bubble for an item resulting in the scanner using an invalid response code of some sort such as an asterisk or a 9 Item difficulty appears in the diff column If an ite
361. sort of analysis is generally undertaken to see if some students might have colluded in creating their answers something which is generally considered to be cheating If your data set involves N students the number of pairs to be compared will equal N N 1 2 When N 100 there will be 100 99 2 4 950 student pairs to compare 2003 2011 Curtin University West Australia Toolbar and tab 119 When N 5 000 there will be more than twelve million gt 12 000 000 student pairs to compare But fear not Lertap will crunch your pairs without a whinge asking only that you muster some patience when N gets over 800 or so see time trials below An RSAdatal 3 worksheet forms the base for similarity analyses RSAdata worksheets are made whenever the Output item scores matrix is taken from the Run menu and the RSA option has been set to yes in the System worksheet Once an RSAdata worksheet has been created another option on the Run menu Response similarity Analysis RSA will get Lertap to produce its three RSA reports RSAcases RSAtable and RSAsig The RSAcases report is the bread and butter of Lertap s RSA analysis Here s a typical sample b ad 4 gt pif IStats RSAdatal RSAsigi RSAtablei RSAcases1 gt An RSAcases report presents data for those pairs of students whose item responses have been judged to be suspect using criteria developed by Professors Harpp amp Hoganfs
362. students who selected option E the keyed correct option had an average criterion score of 17 66 In this case 17 66 is the mean test score for these 29 students the criterion is the test score The test had 25 items and all items were scored on a right wrong basis with 1 point for right O otherwise The maximum possible criterion score is thus 25 When the criterion is the test score itself it s called an internal criterion At times an external criterion measure is on hand and item performance may be assessed by correlating item results with the external criterion score Please refer to this topic 94 for more information 2003 2011 Curtin University West Australia Input amp Output Zz avg expressed as a z score Sixty 60 students took our 25 item test Their average test score that is the mean of the 60 test scores was 12 63 The standard deviation of the 60 scores was 6 95 With these two figures on hand the z score for the 29 students who selected option E on Q1 is 17 66 minus 12 63 divided by 6 95 or 0 72 z scores are standard scores No matter what the test mean and standard deviation may be when z scores are formed the mean of the z scores will always be zero 0 00 and the standard deviation will always be one 1 00 A positive z score occurs when the original score is greater than the overall average criterion score z scores are frequently interpreted by making reference to the
363. sub Title NewsQuiz PER SCALE key ACCDB Now the sub card has three control words Title PER and SCALE PER gets Lertap to create a percentage score for each test taker being the student s score expressed as a percentage of the maximum possible score For example if the maximum score is 5 and a student got three items correct PER 60 The SCALE control word adds the student s z score to the Scores 2s3 report on a test with a mean of 3 standard deviation of 1 a student test score of 4 would correspond to a z score of 1 00 Example C4 To switch Lertap into its mastery scoring and report mode include the word MASTERY on the sub card as shown here col c2 c6 2003 2011 Curtin University West Australia CCs details 31 sub Title NewsQuiz MASTERY key ACCDB Using the MASTERY control word on sub causes two things to happen Each student will have her his percentage score automatically included in the Scores 2 report just as happens when the PER control word is used More importantly the MASTERY control word gets Lertap to substantially alter one of its main statistical reports The Stats1ullsl report will include a summary group statistics table a variance components analysis and two classification accuracy indices please refer to Chapter 7 of the manual for details and also take in a 2007 journal article dealing with the use of cut scores Lertap assumes the mastery cutoff percentage to be 70 Th
364. summarised in the tables below sub control words for cognitive subtests CEC Optional Means correction for chance Adjusts subtest scores for the estimated effects of guessing Mastery Optional Gets Lertap to produce its mastery test analysis and report Also acts as if the PER control word had been 2003 2011 Curtin University West Australia Help file for Lertap 5 used causing the PER score to appear on the Scores report Note this control word is the same as using Mastery 70 a default mastery level of 70 is programmed into the System worksheet z4 Mastery Optional Lertap will produce a mastery test analysis and report using the cutoff figure found after the equals sign Also causes a PER score to be created Example Mastery 65 MDO Optional Gets Lertap to turn off its missing data item scoring Causes a non response to be scored as zero points A click here 48 will whisk you away to more about MDO Name Optional Whatever is found between the parentheses is used as a header on some of Lertap s reports such as Statslf and Stats1b While the header can have any length something less than 40 characters is best Example Name SOC 505 FINAL November 2003 PER Optional Causes a percentage of maximum possible score to be created this will appear as a column in the Scores report Res Required if the default Res A B C D setting is not appropriate SCALE Optional Caus
365. t What happens Lertap has a squiz of the CCs cards reading down the rows of the CCs sheet until it encounters a row whose first column is empty If an error is found in the CCs lines Lertap stops and makes an effort to tell you what the error is You ll need to fix the error and then click once again on the Interpret option If the CCs lines appear to be error free Lertap then starts to read the records in the Data worksheet going down the Data rows until it encounters a row whose first column is either empty or contains a zero This process initiates the production of Lertap output the secondary worksheets often referred to as Lertap reports The first of these is called Fregs s3 simultaneously Lertap makes the behind the scenes series of Sub worksheets There will be one Sub worksheet for each subtest that is for each col card found in the CCs worksheet It then hides the Sub worksheets brings Freqs to the fore and announces that it s ready for you to squiz the Fregs If you re satisfied with the squiz you return to the 2003 2011 Curtin University West Australia Input amp Output 259 Run menu and select the Elmillon option Note that there are settings in the System worksheet which get Lertap to roll from the Interpret option to Elmillon automatically non stop Lertap is said to be in production model 9 when this happens the Macintosh version of Lertap runs in production mode by default Once
366. t For cognitive tests a non answer gets a score of zilch zero However for affective items a non answer will get a score equal to the mean of the item s response weights note It is possible to defeat this scoring system by using the MDO 48 control word on a sub line Note that it is possible to achieve almost any sort of scoring for affective items or for that matter cognitive items This is done by using mws lines in the CCs worksheet 3 CCs details You ve seen that Lertap analyses start with the creation of two Excel worksheets Data and CCs The CCs worksheet contains the all important lines of Lertap syntax which effectively control how Lertap and Excel analyse the data found in the Data worksheet CCs really stands for Control Cards It used to be the case that data analysis was based on the use of punch cards Years back a typical data analysis job involved the use of a keypunch machine data were punched on cards as were the instructions which told the computer how to analyse the data The first versions of Lertap were based on the use of punch cards and the term control cards has been carried into most subsequent versions You ll see that we sometimes refer to the rows in the CCs worksheet as lines and sometimes as cards We use these terms interchangeably they mean the same thing There are a total of eight cards which may be used in a CCs worksheet The number of cards used in any given job dep
367. t larry lertap com 2003 2011 Curtin University West Australia 3 4 CCs details Pre scored items It is usually the case that the entries found in the columns of the Data worksheet correspond to the response codes selected by each person For example have a squiz at this snippet from rows 2 and 3 of a Data worksheet Q20 Q21Q22 Q23 Q24 Q25 Q28 Q27 Q28 Q29 Q30 A E E B F2 3 1 2 2 On Q20 this person selected the response which had been coded as A Apparently s he did not answer Q21 On Q22 the person selected the response coded as E The column entries change to digits from Q26 on but the meaning is probably the same on Q26 the person selected the response coded as 1 one while on the next item Q27 s he chose the response which had been coded as 3 What we re looking at here is part of the Lertap Quiz data set described in Appendix A of the manual This quiz consisted of 25 cognitive items Q1 Q25 followed by 10 affective items Q26 Q35 The cognitive items used letters as response codes while the affective items were of the Likert style with 1 the code for strongly agree and 5 the code for strongly disagree How many points did this person get for her his answer of A on Q20 We don t know we can t tell just by looking at the data above And even though there are digits in some of the columns we can t assume that a 1 for Q28 means that the person got one point for his her answer These item responses have
368. t be output for cognitive tests 5 0 7 I m Minimum percentage score for mastery level Percentage in Upper amp Lower groups P2 m M 13 Number of upper lower groups 14 Primary first quintile plot yes no no 1or2 T yes no yes m 4 ui Should quintile plots include a data table 16 Mark all items as pickable for quintile plots Number of passes The Spreader is to make 18 Use experimental features generally not recommended Item difficulty type 1 proportion 22mean 3 mean max wt 20 Should tetrachoric correlations be output 21 Interitem correlation diagonal value 1 1 00 2 SMC 22 Are eigenvalues latent roots to be extracted 23 Should a Bilog MG DAT worksheet be created n 24 Should an XCALIBRE worksheet be created 25 Empty Ziri b MN Comments Data CCs f OldCCs Syntax Problems System IET If you re looking at this page using an interactive version of Lelp not a printed PDF version you should find that some of the rows above may be clicked on many of the rows hyperlink to corresponding Lelp topic pages If the row of interest to you does not hyperlink you might try referring to the manual for a suitable discussion as an example Row 12 the percentage of N found in upper and lower groups is not discussed in Lelp see Chapter 10 of the manual look at page 166 if you have the good fortune of
369. t items 25 minimum possible score 0 00 maximum possible score 25 00 E reliability coefficient alpha 9 91 index of reliability 0 96 standard error of measurement 2 03 8 1 w M 4 gt M Data CCs lt Fregs lt Scores Statsif gi Mm gt Ready 2 G EJ 100 Q Click herel zs2 to read about the Summary statistics section of a Statsf report as shown above 2003 2011 Curtin University West Australia Input amp Output 269 r X gd De Be Pis LrtpQuizsample25June12 xsx Microsoft Excel o E X Lerta Hom Insei Page Form Data Revit View Deve Add 9 9 o pg ES LertapS full item stats for Knowledge of LERTAP2 created 25 06 2012 4 item difficulty bands 00 Q22 mI 20 oD 40 Q1 Q2 Q9 Q11 Q14 Q18 Q19 Q20 Q21 Q25 50 Q3 Q4 Q6 Q7 Q10 Q12 Q15 Q17 Q24 60 Q8 Q13 Q16 Q23 70 Q5 80 90 item discrimination bands 00 Ai 20 Q4 Q22 30 Q5 Q14 Q24 40 Q7 Q9 Q16 Q23 50 Q3 Q10 Q12 Q15 Q17 E 60 Q1 Q2 Q6 Q8 Q11 Q18 Q21 Q25 70 Q13 Q19 Q20 80 90 b les Ed ad 2003 2011 Curtin University West Australia 270 Help file for Lertap 5 File alpha without Q1 Q2 Q3 Q4 Q5 Lertap5 full item stats for Knowled ures alpha 9149 alpha change 0 909 0 909 0 911 0 917 0 915 0 910 0 914 0 910 0 914 0 911 0 910 0 912 0 908 0 915 0 912 0 913 0 911 0 910 0 907 0 908 0 910
370. t let your little mouse cursor hover above the options on the Lertap tab and a bit of help comes into view Hover help The URL below leads to a paper which has more comments on using the Lertap tab http www lertap curtin edu au Documentation Excel2007LertapPaperl pdf Page ahead to see what the Lertap tab s options do 4 1 The tab The Lertap tab for Excel 2007 and Excel 2010 has five sections Basic options New menu Run menu Graphics trio and Other menus x Delete VA Sort F Blank Interpret a Histograms 3 Move version x Line QR Headers gH Elmillon A Scatterplot License fX Spread Excel 3 Copy More bf Res charts Lelp Basic options New menu Run menu Graphics trio Other menus The three sections in the middle of the tab are used most often The New menu is used to set up a new Lertap workbook with a Lertap workbook in hand the Run menu is then used to produce results statistical summaries of item and test results and 2003 2011 Curtin University West Australia s Help file for Lertap 5 4 2 4 2 1 student scores with results in hand the Graphics trio s three options may then be used to create pictures charts or graphs of selected results Continue to page forward for a section by section discussion of what the tab s options do or click on one of the sections as displayed above to jump directly to a particular section Basic options The Basic options se
371. ta Revie View Dee adad x Csank Interpret IR Headers Elmillon LA Scatterplot P Copy More 2003 2011 Curtin University West Australia 134 Help file for Lertap 5 Tn aa EDDA EF ene ee mee tu ue we eaa 5 X Delete TH Sort 4 Moves Q Version X Line 7 Scatterplot License fX Spread Excel bb Res charts There can be up to 200 levels in the group column Values in the column may have any length and may even be numeric When there are more than 15 levels Lertap outputs a line graph instead of a bar graph 2003 2011 Curtin University West Australia Toolbar and tab 135 Average Knwidge Scores by rsComp 25 00 20 00 15 00 Kawldge 10 00 0 0 5 1 15 17 2 25 3 3 5 4 45 5 5 5 6 6 5 7 3 5 ul YrsComp It s possible to change just about everything in Excel charts Right click here and there on a chart and see what happens Change colours graph styles and maybe caffeinated coffee to decaffeinated P S we need to whisker something in your ear there s an option which will let you get a boxplot of group results Give a click about herel ise Analysis of variance table A Breaks report as seen in worksheets with names such as Breaks1 Breaks2 and so on terminates with ANOVA a small analysis of variance table rather like the one pictured below Analysis of variance at ss us Between 2 164 82 Within 136 10898 8
372. tap 5 4 8 4 4 9 License transfer License holders sometimes wish they could shift their base of operations getting Lertap to work on another computer without having to purchase another license Can do Yes is the short answer The way the license is transferred varies depending on the types of computers involved and the software they re running To get the latest information of how to transfer a license please send an email message to support assess com In your message please state when your license was purchased and under whose name It also helps to mention the type of computer you re moving from and the type you re moving to For example you might be moving from a Dell Latitude laptop running Windows XP and Excel 2003 to a Macbook equipped with Vista and Excel 2007 License transfers are generally but not always free Advanced Toolbar THIS IS AN OLD TOPIC It does not apply to the Excel 2007 version of Lertap but has been left here as the manual refers to it Note that the Excel 2003 version of Lertap is still available from www assess com and Excel 2003 will work fine even under Vista and Windows 7 Microsoft s new operating system If you need the features offered by the advanced toolbar consider using the Excel 2003 version of Lertap Other versions of Lertap such as those running under Excel 2002 2003 and 2004 a Mac version have what s referred to as an advanced toolbar Here s a picture of
373. tap 5 measure is found under the Log column with Sigma indicating how significant the pair s probability measure was EEIC Index and Sigma measures are computed for all possible pairs of students not just for those whose results come to feature in the RSAcases report When a pair s EEIC and Index measures are above preset cutoff values the pair s results are said to be suspect meaning that the pattern of their answers to test items was suspiciously similar All pairs found to be suspect are entered in the RSAcases report To these the final Harpp Hogan criterion is applied if the Sigma measure for a suspect pair is above the preset Sigma cutoff value the pair s results become significantly suspect or very suspect Their results receive special highlighting in RSAcases a pink highlight is added to their ID EEIC Index and Sigma entries It s these pairs which we might then investigate further Did they have the opportunity to cheat during the exam Were they seated close to each other Were they seen to be using mobile phones or noted to share the same eraser It is the nature of the RSA business to want to have a number representing the extent of possible cheating In Lertap s RSA analysis that number corresponds to the number of RSAcases pairs whose entries are in the pink When an RSAcases report has more than five entries a small section at the end of the report summarizes results as exemplified below eki
374. tap area mentioned above and download a trial copy to their computer As of April 2011 the ASC Lertap area had its download options located close to the bottom of the main Lertap page Scroll down on the page and eventually you ll come to a section which looks like this except it won t have the green arrows nor the green boxes nor the green background 30 Day Trial Copy for use with Windows Excel 2007 O A fully functioning 30 day trial copy is available for users of Windows Excel 2007 If you decide to purchase a Lertap license we can unlock your copy by email After downloading the file unzip it and then double click on SETUP EXE f you do not have a program to unzip files PC users can obtain one at www winzip com m WARNING Lertap may not work properly unless it is installed by the installation program SETUP EXE that is supplied as part of the download Download the 30 day trial copy for use with Windows Excel 2007 13 9 MB 18 Note 1 please check these notes on Excel 2003 and 2007 if you re not sure which Windows version of Excel you have Note 2 this version will not work with Calc the OpenOffice spreadsheet program 30 Day Trial Copy for use with Windows Excel 2010 O A fully functioning 30 day trial copy is available for users of Windows Excel 2010 If you decide to purchase a Lertap license we can unlock your copy by email After downloading the file unzip it and then double cli
375. tap goes it is best to avoid running in compatibility mode If you ve 2003 2011 Curtin University West Australia 12 Help file for Lertap 5 1 7 1 8 opened an xls workbook with Excel 2007 or Excel 2010 save a copy of it as an xlsx workbook Then close Excel completely When you return to Excel and open the saved xlsx copy all should be well compatibility mode should not be running Invisibly Lertap will thank you For more assistance with this problem please feel very free to write to lari lertap com About Lertap version We update Lertap now and then Major updates referred to as revisions or as salient changes are summarised in Lelp s the document you re reading at this very moment Examples of typical Lertap 5 version numbers are 5 6 4 5 8 2 5 9 2 and 5 10 1 The first digit the 5 denotes any Excel based version of Lertap The second digit indicates the true version number and is often related to a particular version of Excel for example 5 6 runs on Excel 2003 a Windows version and Excel 2004 Macintosh Lertap 5 8 runs on Excel 2007 Windows while 5 9 runs with Excel 2010 Windows and Excel 2011 Macintosh The third digit references a revision of the corresponding version A fourth digit as in 5 9 2 1 usually denotes bug fixes and minor tweaks You can see a summary of updates specific to the Excel 2007 and Excel 2010 versions by clicking on the link below www lertap curtin edu au
376. tem as only 2096 got it right The point biserial correlation with the criterion score To compute this all those who selected the option are given a score of 1 while those who did not are given a score of 0 These scores are then correlated with the criterion score If the option is the keyed correct answer then pb r is usually referred to as item discrimination In the Stats1b report pb r for the keyed correct answer is denoted as disc The pb r values will range from 1 00 to 41 00 Items with a discrimination of at least 0 30 are often referred to as having good discrimination Above both Q1 and Q2 would be said to have good discrimination as their pb r values for the correct option are above 0 30 Lertap corrects pb r values for part whole inflation whenever the item is part of the criterion measure The biserial correlation with the criterion score Sometimes the option selection scores of O and 1 used to compute pb r are regarded as a dichotomous code for an underlying or latent variable having a range of scores If we assume the latent variable to have a normal or Gaussian distribution then a well known equation may be applied to pb r to derive b r which is then taken as an estimate of the correlation between the latent variable and the criterion score Note that it is possible for b r to be greater than 1 00 The average criterion score for those who selected the option On Q2 above the 29
377. tem should be 0 72 0 20 0 52 but the 0 72 has been rounded up from 0 716713 you can always see the unrounded figures by turning on Excel s Formula Barl 250 Although the statistics which underlie Statslul reports are quite simple ones based on proportions nevertheless there s a lot of information to take in As an alternative to the tables Lertap is most happy to turn them into charts which are often easier to read Look 2003 201 1 Curtin University West Australia 302 Help file for Lertap 5 NMS NM9 1 00 1 00 0 80 0 80 0 60 0 60 B B 0 40 0 40 D 0 20 D 0 20 other AR 0 00 gt other rr 0 00 lower 4th 3rd 2nd upper lower 4th 3rd 2nd upper Key B Diff 33 Disc 03 Key C Diff 42 Disc 31 NM10 NM11 r 1 00 r 1 00 r 0 80 r 0 80 A A 1 0 60 0 60 B B 0 40 0 40 rC C D 0 20 D 0 20 other 0 00 other x 0 00 lower 4th 3rd 2nd upper lower 4th 3rd 2nd upper Key D Diff 62 Disc 26 Key B Diff 77 Disc 13 The lines in these graphs are referred to as trace lines as they trace how each item s options perform in each group When the Stats1ul results are based on five groups these graphs are called quintile plots The four plots above were obtained by packing the usual quintile plots in order to facilitate item performance comparisons Read more about packed plots herel l
378. tems in c3 to c7 The 2003 2011 Curtin University West Australia CCs details Ku asterisks tell Lertap that the respective items do not use the last two response codes seen in the Res declaration It is possible to use special forms of the alt and mws cards when it s required to quickly remove an item from a subtest These forms can be very handy in certain circumstances please refer to the Remove an iteml e9y topic for examples In Lertap Version 5 25 it became possible to use the other control word on a mws card as exemplified below mws call 1 2 3 4 5 other 9 The line above says that anyone not selecting one of an items five permitted responses will get a score of 9 points The use of other presents a more flexible way to handle missing affective data than does the MDO option discussed above For more about using the other control word and item weighting please click here 245 Example A12 Just one more example and this one s simple tst c2 W col c3 97 sub AFF Name Beach survey 1 Title Beachin pol This example displays use of the tst card The tst card is telling Lertap to make a new Lertap workbook one whose Data worksheet will contain copies of only those Data records which have a W in their second column We d probably then set up another Lertap analysis with these cards tst c2 M col c3 cT7 sub AFF Name Beach survey 1 Title Beachin pol N
379. th 39 of the uppers getting it right compared to 29 of the lowers A closer look reveals that the top group was quite undecided on NM8 many students in this group thought that option C was a good choice and a fair number also went for option D The other column gives the proportion of students who did not select one of the item s options or who had an invalid response which might come from shading in more than one answer of a bubble answer sheet U L diff is a measure of item difficulty corresponding to the proportion of students who got the item correct over all groups It will usually closely correspond to the diff value seen in the Stats1bl 29 report and to p in Stats1f z U L disc is an index of item discrimination designed to be easy to understand it s simply the difference between the proportions of correct answers in the upper and lower groups For NM8 the corresponding proportions are 0 39 and 0 29 giving UL disc 0 10 UL disc values will frequently differ from the disc figure found in Stats1b and from the pb r in Stats1f Stats1b and Stats1f use correlation methods to index discrimination while as we have seen Stats1ul uses simple proportions When an item s U L disc is less than zero red coloring is used to flag this unwanted outcome NM9 had the best discrimination of these three items with UL disc 0 51 all of the proportions have been rounded it would appear that UL disc for this i
380. the Recode option 2 found under the Move menu entering delete for those records of no interest The Recode option is more flexible than the tst method but it can involve more steps in some cases How about using Lertap s RSA support to simply get an estimate of the similarity problems which may pertain to a large data set Maybe there s too much data thousands or tens of thousands of students too many can we possibly get a random sample to work with But of course You d want to read about Lertap s ability to let you Halve and Hold sl Time trials Having Lertap do RSA things can take time as you might expect From Lertap s viewpoint there are usually two things to do make the RSAdata worksheet and then when requested the RSAsig RSAtable and RSAcases worksheets 2003 201 1 Curtin University West Australia 130 Help file for Lertap 5 Our preliminary tests indicate that it does not take all that much time for Lertap to create the RSAdata worksheet On a data set with 50 items and 1 400 students it took some 13 seconds to make RSAdata Doubling the number of students essentially doubled this figure 27 seconds to process 2 800 students The big crunch comes with RSAsig RSAtable and RSAcases For the data set with 50 items 1 400 students EEIC min at 6 and H H cutoff at 1 00 it took four and half minutes for Lertap to create the three worksheets Double the number of students to 2 800 and wowser twenty
381. the Run menu s Interpret CCs lines option is different when it s accessed from the Advanced Toolbar You have the chance to skip the subtest corresponding to each col line or card as shown below Lertap 5 About to parse the COL card col c3 c27 A click on the Cancel button will cause Lertap to skip to the next col card without doing any processing Please read on into the next topics to get a grasp of why some people use the advanced toolbar some of the time and how they get it to show 4 9 1 Advanced toolbar why Perhaps one of Lertap s greatest strengths lies in its ability to fairly easily attach weights or points to each one of an item s responses The mws line is the way multiple weights are usually applied mws stands for multiple weights specification mws cards live on CCs worksheets What many people do not realise no matter how often they pull the Lertap manual down from their bedside table for a relaxing read before sleep is that it s possible to change item weights without changing the control lines in the CCs worksheet Let s say this is what you ve done you ve set up a nice CCs worksheet and used the Interpret CCs lines option from the Run menu after which you run Elmillon item analysis again from the Run menu You look at the output and realise that there s a need to re weight some of the items In fact your need is so special you re not even sure how you d go about creating the
382. the colour changed When you have your own IStats screen showing find one of these triangles and let your mouse hover above it Lertap will display the alpha value for the corresponding principal component in this case the value turns out to be 0 821 it can be shown that this value 0 821 is the maximum possible value which coefficient alpha could assume for any linear combination of the items comprising the subtest Please refer to the technical paper cited below for more information and also please note that these small triangles will appear only when the corresponding alpha value is equal to or greater than 0 60 The Scree Test Plot When we think about the first eigenvalue possibly dwarfing the others we might well long for a picture of some type The scree test was invented by Cattellls241 way back in 1966 to meet these longings Cattell suggested we graph the eigenvalues from highest to lowest to see if the first eigenvalue or the first few eigenvalues dwarf the others His suggestion remains popular to this day We can graph our 10 eigenvalues using a couple of methods The plot shown below was obtained by selecting the eigenvalues and then using Excel s Insert Chart Line options An easier way to accomplish much the same thing is to use an option from the Lertap toolbar Shorts Make a line graph click here 84 for more information on this shortcut method 2003 201 1 Curtin University West Australia no He
383. the default setting for cognitive items Critical note the response codes seen in the Res declaration tell Lertap what to look for when it reads the information in the Data worksheet s rows If the response codes are upper case letters such as A B C D then Lertap will expect 2003 2011 Curtin University West Australia 32 Help file for Lertap 5 to find upper case letters in the relevant columns of the Data worksheet Nasty things can happen when for example the item responses seen in Data columns are lower case letters such as a b c d and the sub card has Res A B C D This is a mis match Res A B C D tells Lertap to look for upper case letters but none will be found Things will come a crashing There s a bit more on this towards the end of the CCs sheet s topic Example C6 Here s one more example of the sub card in action col c2 c6 sub Title NewsQuiz CFC Wt 5 key ACCDB CFC means correction for chance another scoring option entertained by Lertap This control word isn t used all that often it usually results in penalising students if they appear to be guessing see Chapter 10 of the manual for more discussion The Wt declaration applies when the CCs worksheet defines more than one subtest that is when there are two or more col cards In this case Lertap will usually generate a total test score by summing the subtest scores the Wt assignment controls how this is done If Wt 0 then the
384. the smallest 0 16 In a well conditioned correlation matrix with 1 s ones on the diagonal the sum of the eigenvalues will equal n the number of test items assuming the correlations are Pearson product moments not tetrachorics The row with the actual eigenvalues is followed by the percent row seen above The percent figures appear whenever the correlation matrix has 1 s on its diagonal when the SMC setting s3 is on and SMCs are found on the diagonal two changes are made to the table the percent figures are not created and the correlations found in the p comp1 row are replaced with correlations between the item and the first principal factor with the row s label then changing to p facti What do the percent values mean Well first note that there are ten items in this example Q26 through Q35 There are also ten eigenvalues As noted above the sum of the eigenvalues equals the number of items 10 in this example The percent value for the first eigenvalue is 100 3 83 10 or 38 396 Each eigenvalue corresponds to what s called a principal component If we could look at the multivariate scatterplot of the ten items and if each item had a distribution meeting the requirements of the normal distribution the scatterplot would have the form of an n dimensional ellipsoid where n is the number of items 10 in this case If the items are uncorrelated the ellipsoid is an n dimensional sphere If on the other hand the items are
385. the way back in the respective CCs sheet 1 col c5 ci154 2 sub name The complete test title Complete per Note that the title of this test Complete appears in quotes at the top of the table in the first row You can change the title by typing over it or more simply by just changing the title in the chart itself 4 5 1 1 Scaling RSAsig It was mentioned in the previous topic that the Histograms option will work with a Scores worksheet with a Breaks report and with an RSAsig report In this topic you ll find some special comments on histogramming the results from an RSAsig worksheet The chart below was made by first using the Histograms option when looking at an RSAsig 9 report followed by use of the Histogram E option 2003 2011 Curtin University West Australia Toolbar and tab m 1 200 1 000 800 600 Frequency 400 200 m 5S nd 52 N bs M N amp 2d 2 m 2d fv Log PROB It s often of considerable interest to have a more precise display of what s going on in the left tail of a Log PROB plot To do so right click on one of values on the Frequency axis and take the Format Axis option Format Axis P J Eaasoptons Axis Options Number Minimum Auto Fixed Fil Maximum Auto Fixed 50 Line Color Major unit Auto Fixed 5 Line Style Minor unit Auto Fixed 1 dele 7 Values in reverse order 3 D Format E Logarithmic sca
386. their needs are met by the b sheets where they find that just half a cup of tea a brief cup is all that s needed to peruse the output It s possible to get Lertap to reduce the number of reports it gives This is done by setting options in rows 9 and 10 of the Systeml 2s worksheet Why are the f sheets made so that their initial focus is well down the worksheet Stats1b and Stats1ul for example display their top rows at the top of the screen but Statslf and Stats2f do not display their upper most rows at first Why Because experience has shown that the majority of users scroll to the reliability section of the f sheets before they look at anything else A similar effect is seen in the Scores worksheet where Lertap places the initial focus well down the worksheet where the summary statistics begin For a thorough discussion of the statistics found in the Stats sheets please refer to the manual If the manual is not at hand you could look here for such a discussion it s not as complete as that found in the manual but perhaps it ll do ah wait a minute when it comes to the column found in the Stats1b report for cognitive tests it has some fairly useful comments here 2003 2011 Curtin University West Australia kw Help file for Lertap 5 2 1 About the Cook s tour We might do well to highlight some of the things that happened when you took the Cook s tour covered in the previous topicl14 And as you ll re
387. tin University West Australia Toolbar and tab 4 7 7 WrongltemList2 This macro creates the WrongUns report A typical WrongUns list looks like this Book2 Microsoft Excel non commercial use X Delete Vf Sort 4 Moves 9 Version y Line Scatterplot License fXSpread Excel b Res charts Lelp s trio Other menus In this example Jaimi missed only one question Q22 Decha missed the most questions Q24 seems to have been hard about half the students listed in the report got it wrong When there are two or more subtests results are grouped as seen here 2003 2011 Curtin University West Australia Help file for Lertap 5 4 8 Lertap5 listing of student wrong answers Anton Bisekel Running Q10 Q15 Hiking Q17 Q20 Football Q4 013 018 Bsktball Q6 Q14 Q16 Soccer Q21 Q25 Chu Running Q15 Hiking Q17 Q20 Football Q9 018 Bsktball Q16 Soccer Q25 Dipak Running Q15 Hiking Q17 Q20 Football Q4 Q18 Bsktball Q16 Soccer ok Emil Running Q3 Hiking Q11 Q17 Q20 Football Q9 Q12 Q18 Bsktball Q7 Q14 Q16 Soccer Q22 Q24 Fatima Running Q15 Hiking Q17 Q20 Football Q9 012 013 Bsktball Q6 Q16 Soccer Q21 Q24 Q25 Greg Running Q15 Hiking Q17 Q20 Football Q9 018 Bsktball Q6 Q14 Q16 Soccer Q21 9 Irawadi Running ok Hiking Q17 Q20 Football Q18 Bsktball Q6 Q16 Soccer Q21 Q22 M h Data CCs Fregs Scores WrongUns lt Statsif Statsib csemi WE m _
388. tions The more it s the highest above the key and the more it s the lowest everywhere else the more the item discriminates 2003 2011 Curtin University West Australia Toolbar and tab 185 Lertap s quintile plots are of course just Excel charts As such you can change them easily A chart may be selected in a number of ways once selected a great variety of chart options may be played with Below we ve asked Excel to include the actual data from which the quintile a chart was derived Excel refers to the table below the graph as the data table fs id 9 C By V4 Verdana s ExceDOO7QuintilesLxls M m xX x ca Lertap Home Insert Page L Formul Data Review View Develo Add In m Key C Diff 37 Disc 44 44 gt 4 StatsiulChta lt Statstulchtb NL us 0 E These plots are handy For some users they may well be more informative indicators of item functioning than any other Lertap report or graph How do you get em Simple First make sure you ve got the number of groups setting s in the System worksheet right You can have from 2 to 5 groups Then make Stats1lul or Stats2ul the active worksheet by clicking on it Next just mouse up to the Lertap toolbar and click on b Lertap will click into action opening up a new worksheet called either Stats1ulChta or StatslulChtb depending on a setting in row14 8 of the System
389. tions which pertain to Ibreaks and IbreaksMH reports are seen here in rows 47 to 55 The respective System row numbers may change over time as other options are added Ibreaks attempts to determine the optimal size for its charts However you can set your own by using the settings in rows 47 48 and 49 Rows 51 and 52 determine when Excel will use colours to highlight the sig and eta values in Ibreaks charts Row 54 does likewise for the MH Chi sq prob cell in an IbreaksMH table Row 53 turns Yates chi square continuity correction on or off In this snapshot it s set to No which means Off Row 55 s option is presently ineffective Lertap doesn t use it If scores of zero are encountered they re converted to a score of one automatically If this is done 2003 2011 Curtin University West Australia Toolbar and tab 159 Lertap will display an informative message 4 4 8 To Halve and Hold This option is used to create two random samples of data records dividing a data set into halves on a random basis How does it do it It begins by making a copy of the original Data and CCs worksheets placing them in a new workbook For convenience assume that Excel calls this new workbook Book1 Then Halve amp Hold uses two standard Excel functions to generate a set of random numbers between 1 and the number of data records in the original Data worksheet denoted as ArraySize below Randomize more code RandomV
390. to be somewhat more advanced Lelp s explanation of CCs lines and syntax is somewhat more extensive than that found in the manual Users of the Windows version of Lertap 5 will find that some of Lertap s dialog boxes and most of its menus have automatic context sensitive links to Lelp By and large these links are denoted as Lelp Macintosh users do not yet have a similar resource they may refer to the website or to the PDF version Finally this document has an invaluable educational supplement in numerous spots it 2003 201 1 Curtin University West Australia Help file for Lertap 5 1 8 2 1 9 introduces elements of Australian slang strewth in other spots it has links to selected Western Australian cultural highlights such as Emu Export Mt Barker and the Southwest Capes Updates You can see a summary of updates to Lelp here www lertap curtin edu au Documentation UpdatesSummaryLertap57 pdf This link also gives guidelines for downloading the latest copy of Lelp Contact us Lertap is a project of Curtin University Larry Nelson is the project director The project s main website is here www lertap curtin edu au Address Larry Nelson PhD Lertap Project School of Education Curtin University GPO Box U 1987 Perth Western Australia 6845 Phone 61 8 9266 2152 Fax 61 8 9266 2547 email 1 L Nelson curtin edu au email 2 larry lertap com Of the two email addresses larry lertap
391. to use a fixed pitch font such as Courier New with the CCs worksheet This makes the information in the CCs cards line up as seen above in the key and alt cards Lertap only looks at the first column of the CCs sheet any information found in subsequent columns is ignored When Lertap finds a CCs row whose first column is empty it thinks it s come to the end of the CCs lines and reads no more This can be handy we sometimes enter a single col card in the CCs worksheet followed by a blank row At other times we ll step into a CCs sheet and insert a blank row after the col card We do this as we know that this will get Lertap to make its Freqs report but nothing else Maybe with time you ll come to like the Freqs report as much as we do it provides a quick no frills look at our data We use it to rapidly get a glimpse of how people answered our questions and to see if some errors may have arisen whilst processing the data For example if the items in our test used the default response codes of A B C D we d be surprised to find Fregs reporting it found an E as one of the item responses About here we again insert a critical note regarding response codes this same 2003 2011 Curtin University West Australia 258 Help file for Lertap 5 6 4 message may be found under the Cognitive CCs 2 topic As all readers know the Data worksheet contains item responses In the case of cognitive tests it is common for item
392. two minutes Update August 2007 using Excel 2007 running RSA on several data sets each involving the same 30 item 5 option multiple choice exam 6 5 minutes with 472 students 23 5 minutes with 798 students and 48 minutes with 1 350 students Note that these tests were done with a relatively old computer having a single processor you might expect better figures than these on your own machine if it s fairly new One thing to keep in mind here it will generally not make much sense to run RSA with data sets housing students from more than one exam venue Because why Well think of what we re trying to figure out are the item responses from any given pair of students surprisingly similar If Joe sits the exam in Engineering and Sally sits the same exam in Commerce would we want to pose this question What chance do Joe and Sally have to share exam answers Maybe they ve got some sort of whiz bang radio set up which is hidden somewhere in their clothing We might have all test results in one Lertap Data worksheet true but when it comes time for RSA we d probably want to break out records according to their exam venue Interested in this idea If yes back up a few paragraphs and read about tst c3 Business Also seel the Breakout scores by groups option under the Run menu it might also be helpful Finally a closing comment the literature in this area is interesting and not ambiguous make it unnecessary to use RSA software b
393. ty West Australia 132 Help file for Lertap 5 more Lertap Home Insert Page L Formu Data Reviev View Devei Add Ir X Delete VA Sort Ll Blank Interpret al Histograms Move Version y Line Headers Elmillon 2 Scatterplot License Spread Excel Copy More Mb Res charts Lelp Basic options New menu Run menu Graphics trio a P 14 00 3 50 49 00 4 08 17 00 soo s75 seo 4 67 35 0 11 00 2 75 41 00 3 42 30 00 18 00 4 50 44 00 3 67 13 00 20 00 5 00 43 00 3 58 17 17 00 4 25 37 00 3 08 29 00 10 00 2 50 42 00 3 50 24 00 21 00 5 25 58 00 4 83 15 00 15 00 3 75 45 00 3 75 16 00 5 00 50 00 4 17 17 00 a Ea 6 8 EM 4 gt M Data i Ps Now say we had a Scores sheet such as the one above We might want to cross say Degree column 4 in the Data sheet with SelfReg column 4 in the Scores sheet We zip up to the Run menu click on More and then on Breakout score by groups asking for Data column 4 to be broken out using Scores column 4 Lertap produces a breakout report and a corresponding plot the statistics in the Breaks1 report are the same as those seen at the bottom of a Scores zs3 report 2003 2011 Curtin University West Australia Toolbar and tab we S o3 Web sar Hees sla IL si Home Insert Page Form Da
394. ty West Australia CCs details col a required card sub key a required card zalt wts mws The key alt and wts cards require a character for each item in the subtest If there are five items there must be five entries on the key card the alt and wts cards if used must also contain five entries There may be spaces between the entries We like to put a space after every five entries as that way the card is easier to read we also like to use a fixed pitch font with CCs cards such as Courier New so that when we use key alt and wts cards the entries on all cards line up mws cards are unique in that they correspond to just a single item Affective subtests Affective subtests must have a col card and they must also have a sub card with the AFF control word on it The number of columns mentioned on the col card tell Lertap the number of items in the subtest The default response code set for affective items is Res 1 2 3 4 5 The CCs card order for affective subtests is col a required card sub a required card must have AFF on it pol alt mws If used the pol and alt cards require a character for each item in the subtest If there are five items for example there must be five entries on these cards if they re used mws cards are unique in that they correspond to just a single item Control words The sub card may have a number of control words on it these are
395. ubtest To remove more than one item follow the same pattern The examples below will remove two items from their respective subtests col c28 c37 sub aff title Comfort pol mws c29 mws c36 2003 2011 Curtin University West Australia CCs details aa col c3 c27 sub res A B C D E F Title Knwldge key AECAB BEBBD ADBAB BCCCB BABDC alt 35423 5464 54324 4 344 45546 Please note the special forms of the alt and mws cards require Lertap Version 5 2 or better How to find out your version number It s simple a click here 12 will tell you how November 2004 note the new exc card was added to make it easier to remove or exc lude items Please to see the following topic 7 Related tidbit In June 2004 a special document was created to address a 255 character cell limit in Excel It contains numerous examples of using CCs lines alt mws and wts to remove items from a subtest If you re connected to the Internet why not whip out to see this Word file ExcelColumnLimitProblem1 doc 3 Excluding items The matter of removing one or more items froma subtest is discussed in the previous topicl 6 As mentioned there the quickest way to exclude a single item from a subtest is probably to use a single mws card For example the following CCs line will see that the item resident in column 37 of the Data sheet is eliminated from its subtest mws c37 In Novembe
396. uch as these to reach this finding No the numbers are all there in the source data in the ECStats1ul worksheet But you might agree that the plots are more effective in conveying the message If there s a pattern in the data plots such as these can help to uncover it 4 5 3 2 3 Quintile options There are a few options which apply to these plots Almost all of them are controlled by the System worksheet 28 found in the Lertap5 xlsm workbook Below we ve displayed part of the System worksheet 2003 2011 Curtin University West Australia Toolbar and tab 199 r 7 X id 9 En Jis Lertap5 xism Read Only Microsoft Excel oB Lertap Home Insert Page Layout Formulas Data Review View Developer Add Ins 9 o c gg X 1 2 3 4 e System Settings E Allowed Usual settings setting yes no yes es no yes 10 to 99 These are Lertap5 system settings Don t change them unless you know what they do The settings below are the standard ones for the Excel 2010 version of Lertap lal 9 Should brief item stats sheet be output 10 Should upper lower stats sheet be output for cognitive tests 11 Minimum percentage score for mastery level 12 Percentage in Upper amp Lower groups 13 Number of upper lower groups 14 Primary first quintile plot AorB 15 Should quintile plots include a data table es no 16 Mark all items as pickable for quintile plots es no 17 Number
397. un the Excel 2007 2010 versions of Lertap Mac users don t stray the Excel 2011 version of Lertap works pretty much as described in this document How to tell what version of Excel your computer is able to run If it s Excel 2007 then its upper left corner looks like this Ba Eod Microsoft Excel non commercial t E Home Insert Page Layout Formulas Data Review View If it s Excel 2010 then the top left corner will look something like this r x Lertar Lertap Home Insert Page Layout Formulas To round out this topic we should mention that there are other versions of Lertap There s the all conquering Version 5 6 4 a version which runs with almost any other Windows version of Excel such as Excel 2002 and 2003 Lertap also runs on Macintosh computers under Excel 2011 2003 2011 Curtin University West Australia Welcome 3 1 3 How to getit ASC Assessment Systems Corporation is the sole distributor of Lertap 5 ASC s website is www assess com The Lertap area within the ASC website may be reached by entering LERTAP in the Search box found towards the upper left of ASC s home page Getting yourself fixed up with a copy of Lertap is not difficult If you use an Apple computer a Macintosh of some sort then you can get a test copy of Lertap by sending a request directly to Lertap headquarters in Western Australia lari lertap com Windows users follow a different procedure they pay a visit to the ASC Ler
398. upport for did not see cases Pairwise exclusions now apply to the calculation of cognitive item correlations when MDO 52 is used on the sub card all item criterion correlations are now corrected for missing data A did not see option has been added to the System worksheet This option works in a manner analogous to MDO if a person has not been presented with the chance to answer an item for whatever reason the calculation of item statistics is adjusted accordingly A gentle click here 48 will let you see more Both of these adjustments will be of use when students see different test items In some current online testing systems for example items are sampled from an item bank with each student getting a subset of items Not only do students see a sample of test items but the items included in the sample will vary from student to student effectively presenting each student with a different test 7 15 Jan 2006 Ver 5 6 January 2006 four enhancements version change to 5 6 While most were out celebrating the arrival of yet another new year there was no rest at Lertap central Some substantial changes have been made Response similarity analysis RSA Enhanced work in this area started July last year Now Lertap produces three reports for looking at the matter of suspect cheating with an important probability index Sigma added to help put the heat on a cheat To find out more you d want your mouse to nibble here 3
399. ures for the item s options When MDO is active as in the second subtest no points are given when someone omits an item The result The subtest scores will differ Scores on the first subtest will be higher as people who miss out items are still getting points The mean average of the subtest scores on the first subtest will be higher than that for the second subtest the point biserial correlation values pb r between an option and the criterion score the subtest score are likely to differ as are the avg and z values The more missing data the greater these differences are likely to be Even though everyone answered item Q27 the criterion measure used to calculate item option statistics pb r avg and z differs from the first subtest to the second generally resulting in different values for item Q27 s output Now about Q28 As noted two people did not answer this question Compare the values found in the column for Q28 they re greater in the second subtest The figures for Q28 in the second subtest the one using MDO have been calculated with n 58 the number of people who actually answered the item In the first subtest the values were calculated with n 60 the total number of people taking the test survey In addition the pb r values for the item options seen in the second subtest have been calculated on a pairwise basis they are based only on the people who actually answered the item To read a bit more ab
400. utput an item scores matrix from which we ll plot such things as item means and variances And we often go for a scree test by plotting eigenvalues 108 The little ix icon option makes it possible to get such plots with ease How to use this option Couldn t be easier select the cells you want to plot and then just click on the ik icon That s it Yes assuming you have selected cells which have numeric data in them Excel 2007 will present its line graphing options and in another click you ll have that graph Here s an example we wanted a plot of item means as found in a Stats2b report We selected the cells of interest by running our mouse over them there are 10 means values selected in the screen snapshot seen below and note that we started selecting in the mean cell 2003 2011 Curtin University West Australia Toolbar and tab s 2 ss as ar ase 22 45 13 32 35 15 13 8 Ce pe ee 5 LL pe em E3EXESES e oe on zs Then we clicked on M Qi Headers gp au All Chart Types Excel 2007 popped up its select a chart type box seen above We clicked on the first chart type the one in the upper left This gave us what we wanted 2003 2011 Curtin University West Australia s Help file for Lertap 5 mean 4 50 4 00 3 50 3 00 2 50 2 00 1 50 1 00 0 50 0 00 T T mean You can modify the resultant Excel chart the line graph
401. ve discrimination figures above 0 20 with many going over 0 30 Coefficient alpha for this administration of the 56 items was 0 91 The item labels in the case above were i s followed by an item number i1 through i56 Lertap uses whatever labels are found in the first column of the Statsb report when these labels are longer than four characters Lertap will use only the last four For example if an item label is Ques107 the plot will show s107 Displaying labels will often make the plot crowded and a bit hard to read Fortunately the labels may be easily removed To do so right click on any one of the labels and an option to Delete the labels will appear This is what the plot looks like after the item labels have been cleared 2003 2011 Curtin University West Australia 2 Help file for Lertap 5 These plots are simply Excel charts you can reformat them in many ways right click or double left click at various spots and options will appear 6 4 3 2 2 Stats1b column The column is used to flag items which may have a problem The screen snapshots seen below were taken while using the Excel 2010 version of Lertap June 2011 2003 2011 Curtin University West Australia Input amp Output ms X id 9 Er MathsQuiz xlsx Microsoft Excel ca 23 Lert Hor Inse Pag For Dat Rev Vie De 9 Q c pp s amp s 5 brief item stats for MathsQui created 20 06 2011 ne sme
402. vided by 60 The avg values are now computed using just the criterion scores for the 57 students who answered Q16 This applies to the other line too 5 67 as a z score in the distribution of 57 scores would be 1 06 Now take a few seconds and sum down the p column for Q16 In the first report above the sum is 1 00 10096 But in the second report the sum comes to 1 05 10596 What s up The 0 05 p value for Q16 s other row is just indicating the proportion of respondents who did not answer the question The other four p values for Q16 sum to 1 00 which is what is expected when the MDO option is in use Statsf reports and the did not see option When you ve got the did not see option going the Statsf report will be quite similar to what you ve just seen immediately above Look 2003 2011 Curtin University West Australia s Help file for Lertap 5 Lertap5 full item stats for Knowledge with DNSI created 24 03 2006 Q15 option wt n b r b r avg z 4 0 00 1 0 02 0 03 0 08 14 00 0 20 B 1 00 30 0 50 0 53 0 66 16 67 0 58 G 0 00 13 0 22 0 36 0 50 7 92 0 68 D 0 00 16 0 27 0 33 0 45 8 81 0 55 Q16 option wt n b r b r avg z 4 0 00 12 21 D 17 D 24 10 75 D 32 B 1 00 36 0 63 0 42 0 54 15 53 0 36 G 0 00 3 0 05 0 28 0 59 4 67 1 20 D 0 00 6 0 11 0 32 0 54 6 50 0 94 other 0 05 There s just a wee difference in these results Can you spot it The other line is now missing most of its stats W
403. volving a single factor or construct Some feel that a test may be said to be unidimensional if it can be shown that the largest eigenvalue underlying the test s correlation matrix is so dominant that it dwarfs the others See references and discussion below Eigenvalues are computed if the System worksheet aal has yes in Row 22 Column 2 Lertap s eigenvalue extraction uses computational routines produced by Leonardo Volpi and the Foxes Group in Italy made available by the authors kind permission The Foxes Group s general matrix package Matrix xla is freely available at http digilander libero it foxes index htm Matrix xla is a powerful extensive set of matrix manipulation routines for use with Excel it includes the ability to produce a complete principal factors components analysis with Varimax rotation something Lertap users may wish to experiment with Here s a sample of Lertap s output with eigens 2003 2011 Curtin University West Australia Toolbar and tab XX Mew S YS pune Shots MM lil Move License Lelpy eigens 3 83 1 25 1 06 0 98 0 69 0 68 0 51 0 46 0 37 0 16 percent 38 396 12 59 10 696 9 8 6 9 6 896 5 1 4 696 3 7 1 696 0 85 0 61 0 14 0 60 0 68 0 02 0 34 018S ENGS 0 79 M o4 nj Statsiul Stats2f Stats2b Statsaf_ Stats3b 1Sta lt In this example the 10 item Comfort affective scale seen in the Lertap Quiz data set the largest eigenvalue was 3 83
404. w to print Lertap s reports Not hard at all especially if you take in this topic 6 4 3 3 Statstul These reports offer another way of looking at how items have performed one which does not rely at all on the use of correlation coefficients To some people they re easier to understand Statslul reports provide the gateway to Lertap s very popular quintile plots s3l The ul in Statslul means upper lower To make a Stats1ul report Lertap begins by sorting all test scores from highest to lowest It then uses the sorted scores to form groups of students The top students go into a group called appropriately enough the upper group And the bottom students They go into a group denoted as the lower group 2003 2011 Curtin University West Australia 300 Help file for Lertap 5 When this method was initially devised a long time ago before computers became commonplace it was often referred to as the high low method and only two groups were used the top 2796 and the bottom 2796 see Chapter 10 of the manual for more information Lertap has a setting in its System worksheet z which determines how many groups will be formed The minimum is two the maximum the default value is five I m if Ev VS i MNurForLrtp593 xlsx Microsoft Excel ca B 93 Lerte Hom Insei Page Form Datz Revii View Deve Add Y o c ep X LertapS U L stats for M Nur Licensing Li lower E Statsib csemi Statsiul
405. want with it It s real easy to change titles and colours and to reformat the values seen along the x and y axes x Comfort and y Frequency in this case ali Histograms l Histogram E dj Scrunch em 2 Lelp Scrunch em Sounds like it could be painful eh Not so read on It is often the case that a HistoL histogram will have too many intervals There ll be one interval for each possible score starting with the minimum score and ranging up to the maximum score If the minimum score came out to be 4 for example and the maximum 100 there would be ninety seven 97 intervals in the HistoL report Oft times some of the intervals will have no entries for example if no one got a test score of 13 there will nonetheless be an interval in the HistoL report for 13 The Scrunch em option lets you reduce the number of intervals in a HistoL type report To use it start by going to a HistoL report such as say Histo1L Then click on Scrunch em Lertap will make a copy of the Histo1L report calling the copy Histo1LCa The LCa letters mean copy a of a compressed L type histogram if you use this option again with Histo1L the next copy will be called HistollCb 2003 2011 Curtin University West Australia owe Help file for Lertap 5 You ll then see a little dialog box such as the one seen below Apply a compress to the intervals xj Copied Histo1L retaining 94 score intervals You can reduce the numbe
406. wer left and lower right 2003 2011 Curtin University West Australia 122 Help file for Lertap 5 2 5 Gnr Ala i 4017607 7704343HM 21 o 0 0 61 2 7407453 4444444 20 5 O0 27 0 00 0 68 2 73 7704343HM 77045562M 8 23 O0 25 0 00 0 90 2 63 4010714 7414167CM 15 12 0 24 0 00 0 95 2 61 7611470 4005774LI 7 23 0 26 0 00 1 02 2 57 4003420 7704343HM 19 8 1 25 0 04 1 09 2 54 7400752EL 4444444 25 5 O 26 0 00 1 17 2 50 4010112PE 4444444 25 5 0 26 0 00 1 17 2 50 7704343HM 4107475 8 19 O 26 0 00 1 18 2 50 tac M 4 Mf RSAdatal S RSAsig1 RShtablei 7 RShcases f E T E fuf ie The top of a typical RSAsig report has been captured here Such reports contain data pertaining to all student pairs whose item responses are not suspect these are all those pairs with an EEIC value and or an Index value less than respective preset cutoff figures Note to be included in the RSA analysis a student must have at least one answer wrong Students with perfect scores or totally imperfect scores not a single correct answer are excluded The entries in the RSAsig report are sorted on column 11 from highest Sigma H H sigma to lowest The Log PROB column abbreviated as Log in RSAcases is the logarithm of the Harpp Hogan response probability measure PROB described in Harpp amp Hogan lt 2 1993 2003 2011 Curtin University West Australia Toolbar and tab 123 4444444 DataRow106
407. wts lines in the CCs worksheet The sub lines do not have Res declarations so Lertap will assume Res A B C D for each subtest Multiple affectives Here s a common example of CCs lines for a survey with three subscales col c5 c20 sub aff title Anxiety pol tttt 4 4 4 4 col c21 c35 sub aff title Friends col c36 c50 sub aff title Homesick pol t t Three 15 item affective subtests are defined by these lines Two of the three subtests the first and the third have a mixture of forward and reverse scored items There is no pol line for the second subtest which means that all items for this subtest are forward scored Lertap will make three subtest scores and also a total score What about possible score ranges for this example Each subtest has 15 items 2003 2011 Curtin University West Australia Getting started There being no Res declaration on the sub lines Lertap assumes Res 1 2 3 4 5 that is five possible responses per item Lertap will score each item on a one to five point basis Why Because there are five possible responses The minimum score a person can get on an item is one the maximum is five There are 15 items in each subtest Therefore the score range for each subtest is 15 to 75 and there being three subtests the range for the total score will be 45 to 225 What happens when a person doesn t answer an item What sort of score do they ge
408. y randomly assigning students to seats in the exam venue and if possible by using different test forms with item scrambling Related tidbits For more about these topics see Response Similarity Analysis a 17 page Word document with lots of similar topics available via the Internet click here if you re connected You ll surely want to take in a journal article submitted for publication in 2006 Using selected indices to monitor cheating on multiple choice exams another Word document some 15 pages in length This article mentions other software working in the area of cheating detection such as Scrutiny Integrity and SCheck Then having looked at the journal article which was critical of Harpp Hogan methods you ll have to take in the best selling riveting sequel a paper which explains how Lertap was modified after Harpp amp Hogan revised their original guidelines in response to the 2003 2011 Curtin University West Australia Toolbar and tab 131 journal article See Using Lertap 5 6 to monitor cheating on multiple choice exams 4 4 6 Breakouts Suspected you were heading for a breakdown Lertap can help use its Breakout scores by groups option to obtain a summary table and graph comparing score results for various groups To use this option you will have a column in the Data worksheet which identifies groups on aa Se ed px Ey I Lertap Home Insert PageL Formu Data Reviev View Develc Add
409. yet to be scored Okay Consider now another case Suppose a cognitive test included the following question 33 Read the five sentences below and place a tick next to those sentences which use the pluperfect tense Student answers to a question like this one have to be scored by hand How Well if there were three pluperfect sentences and the student found and ticked each then the student would probably get 3 points If the student found two of the three s he d get 2 points A student might get 0 1 2 or 3 points on this item Look now at a snippet from another Data worksheet would you L1 L2 L3 L4 L5 L6 L7 L8 L9 L10 R1 R2 RS R4 R5 R6 R7 R8 R9 R10 1 1 O 1 1 1 2 1 1 1 1 1 2 O 1 1 1 1 1 1 In this case language students listened to an audio tape with 10 short sentences and they also read ten brief paragraphs The digits in the boxes are now real numbers 2003 2011 Curtin University West Australia e Help file for Lertap 5 they re not response codes the digits represent the number of points the student earned on each of the ten listening items L1 L10 and the number of points earned on each of ten reading items R1 R10 The items in this test are said to have been pre scored or pre coded The following CCs lines were used to process results for the ten Listening items col c2 c11 sub res 0 1 2 3 name Listening AARP test Title L AARP mws c2 0 1 mws c3 0 1
410. you might like to copy the LRTP5HHelp chm to each folder too it s the Lertap help file Lelp If you have Lertap5 xlsm in a folder without its corresponding Lelp file then Lertap help will not be available whenever you try to access it from within Lertap this comment applies only to Windows users Related tidbit For more about this topic get productive see ScannerEjemplo1 doc a marvelous captivating Word document with lots of hints for what to do on rainy days available via the Internet click here if you ve got connections Elmillon SP Elmillon Elmillon is the name of Lertap s main item analysis routine first developed for the Venezuelan Ministry of Education The name has origins in the Spanish language un millon or in English thanks a million is what the chief of the data analysis section Rogelio Blanco said when the routine was debugged and delivered The name Elmillon has been used since then Elmillon is always run after the Interpret option has been taken Elmillon reads data makes subtest and scale scores written to the Scores worksheet and goes on to produce from one to three item analysis reports The item analysis reports are Excel worksheets having names like Statslf s9 Statslb z9 and Statsull 299 Once Elmillon has been run it will be reluctant to perform again until the workbook s secondary worksheets have been deleted This isn t as difficult as it may sound selecting interpret will de

2 - Quia

Contents

Download Pdf Manuals

Related Search

Related Contents