Home

Reid, G.B., Potter, S.S. & Bressler, J.R.

1. 4 99 00 0 u 0 UR UO MO GN UR GV GU 40 99 UM ND ue ED UP UM RO 96 UD Um UB AR AM P s em wa sa MV GA UHR D s US HU UR 40 5 00 m MEAN US UB MP 69 UD RM Os 9 us s s 5 88 s dip e w
2. Dt 111 N 1 00 8 00 112 B 10 00 5 00 118 W 19 00 6 00 121 F 2 00 4 00 122 1 11 08 14 00 123 C 20 00 22 00 134 X 3 06 17 90 132 5 12 00 12 00 i33 n 21 00 25 00 211 U 4 00 7 00 212 0 13 00 3 00 213 Z 2 2 00 16 00 221 5 00 2 09 222 14 00 13 900 223 22 23 09 11 00 741 6 00 20 00 15 00 19 09 24 00 23 00 A11 16 090 10 00 sb 20 00 160 00 421 Y 1 00 17 00 1 00 ved G eL ua T 1 ito l t AD Figure 4 Printout of Entire Data Set e 34 uonnjos Surreorpug sioofqns Jo sisApeuy od oloxq a UG 42437 SL 047 65 ct 5 903136 EF 99 TT L 95 89 6 416 OT L 67 OF 95 6 66 6 a 807 847 52 V9 9g S 16 65 25 4 NNJN NIVH 253 1 08 85 09 54 9 5 RVHOOSd S 967 tv OF 097 S NI NASOOHD L 250 09 66 v NOILdG OL 09 54 5 96 002109 BE dNLas L oL 89 127 9427 647 087 OL 1 66 99 08 078 1 23 d3dAIO0J10Hd 315 135 153 Sid 351 SJL Ens 3dALOLOSd 3ONVHO 14 931539905 SLOALENS LOSCENS H VA AOA SH NOIIVI3HSNOO M SVA 3
3. ONT 1735 96 96 OL N fii3H 24 T 014 4 2 T tC Tl 1 222 222 2225 2 22 22 2 SIXV A SdJILISVdSIG SA SIXV X IVNISISNHO 1014 F2 Print Scaling Prints a hard copy of the information on this screen as well as the scaling e Information solution for this particular group or individual This is the typical method of saving a record of the analysis for future reference F3 Print All Will send the entire listing of parameters scaling history original and Iterations rescaled values and scaling solution for both algorithms to a printer If this analysis is for prototype groups or an individual this option will print this information for all prototype groups or all selected individuals Therefore this option can produce quite a large amount of information much of which is not necessary for the typical study F4 View Scaling This option will display the scaling solution for this particular group or indi Solution vidual as presented in Figure 30 Since all 27 combinations cannot fit on the screen at the same time you can use F2 to display the rest of the com binations While viewing the scaling solution an F1 will return you to this screen SCALING SOLUTION Fi RETURN TO MENU STIM LEVELS STANDARD RESCALED F2 VIEW REST OF TES SCALING SOLUTION i 111 1 200 0 2 112 777 17 2 3 113 3
4. SUMMARY OF AXIOM VIOLATIONS PROTOTYPE ANALYSIS STRESS PROTOTYPE INDEPENDENCE T INDEPENDENT OF E AND S 0 FAILURES OUT OF 108 TESTS E INDEPENDENT OF T AND S 0 FAILURES QUT OF 108 TESTS 5 INDEPENDENT OF T AND E 0 FAILURES OUT OF 108 TESTS DOUBLE CANCELLATION DOUBLE CANCELLATION IN T x E 1 FAILURES OUT OF 2 TESTS DOUBLE CANCELLATION IN E x 5 0 FAILURES OUT OF 3 TESTS DOUBLE CANCELLATION IN S x T Q FAILURES OUT OF 3 TESTS JOINT INDEPENDENCE T E INDEPENDENT OF S 8 FAILURES OUT OF 108 TESTS E S INDEPENDENT OF T Q FAILURES OUT OF 108 TESTS 5 T INDEPENDENT OF E 0 FAILURES OUT OF 108 TESTS OPTIONS PROTOTYPES GOTO NEXT PROTOYPE GO TO NEXT OPTION CHOSEN IN PROGRAM SETUP PRINT SUMMARY OF AXIOM VIOLATIONS PRINT COMPLETE AXIOM HISTORY MAIN MENU Figure 15 Summary of Axiom Violations for the Modified Stress Prototype Group Now that all of the potential problems have been addressed scaling solutions can be generated for the prototype groups Make sure to maintain the suggested prototypes as described above subject No 8 as a stress and subject No 12 omitted Then select 5 on the Program Setup screen to obtain scaling solutions for the two prototype groups The Scaling Information screen should appear as in Figure 16 for the time group and as in Figure 17 for the stress group Selecting F2 will print the vital information for permanent record of the solutions These values sho
5. oo sioofqng 4 55216 Sureos amsrq NNJN LEAN 0109 dNLaS NI N3SGHO OL 09 NOII YIOS SNTTY2S SNOILVESLI FIY lINISUd INIHd VLVG AVU SA G3TV5S38 40 1014 4 010 4 SNOILdO 253 9d 54 LA 24 S YOLIVA 404 Z 21729 LY 553915 6 8912 1 X ES BT 79 00 2 584415 8 L 8012 4 804 X 9 51 c 553815 2 96 oz 150333 9 8012 4 HOVS 40 ST 26 2 140444 S 4 ALVAIXOUddY Z 9 140333 eS 92 227 E S3HIL 29 lt 2 3 67 t ETTO P679 22 ANIL T 2 6 TAGON c LUb6 ETTO 21 3AILIGGV SAILIIGGV COtP6 PETTO TT 18 6 26110 ON NOILIVHSII NOHi GALNIYd nvl NOIIVH3II KO sg SNOILVEALI 53071 31995 JHL 31905 553815 SNI IVOS SNOIIV233II 1541 1 Section 3 EVENT SCORING e 3 1 INTRODUCTION The second phase of SWAT Event Scoring is the experiment or situation that the investigator is interested in evaluating for workload In this situation a subject performs a task and the investi gator obtains information about the
6. Ok a Oe Ot 00 Ot A mh OO a et Often have spare time Interruptions or overlap among activities occur infrequently or not at all Very little conscious mental effort or concentration required Activity is almost auiomatic requiring little or no attention High to very intense stress due to confusion frustration or anxiety High to extreme determina tion and self control required MG OS CR FER BA VIR a Oh Hn PONE OUO SH eee Hm m m E ap ORN O UP UP m Em E EAE RR Um Often have spare time Interruptions or cverlap among activities occur infrequently or not at all PAM mental effort and concentration are necessary Very complex activity requiring total attention Little confusion risk frustration or anxiety exists and can be easily accommodated 34 84 84 mmn mu 08 VM OMEN EM E ED 58 UD 48 GD D 08 19 88 00 Doa 99 UN o 68 58 DP 58 VEDO 69 8 18 0 8850 08 EU 08 A VUE P M Em E a Almost never have spare time Interruptions or overlap among activities are very frequent or occur all the time M
7. Human Factors Society Crabtree M S Bateman R P and Acton W H 1984 Benefits of using objective and sub jective workload measures in Proceedings of the Human Factors Society 28th annual Meeting pp 950 953 Santa Monica California Human Factors Society Detro S D 1985 Subjective assessment of pilot workload in the advanced fighter cockpit in Proceedings of the Third Symposium on Aviation Psychology Columbus Ohio The Ohio State University Aviation Psychology Laboratory Dodge D C Wong T J and Brown K W 1984 Boom control system improvement study Phase II Supplemental indication system Report No MDC J9732 Douglas Aircraft Company McDonnell Douglas Eggemeier T Considerations in the application of subjective measures of workload in Proceedings of the Ninth Congress of the International Ergonomics Association pp 115 117 Bournemouth England Taylor and Francis 1985 Eggemeier F T 1984 Workload metrics for system evaluation in Proceedings of the Defense Research Group Panel VIII Workshop Applications of System Ergonomics to Weapon System Development pp C 5 C 20 Shrivenham England Eggemeier T 1987 Properties of workload assessment techniques in P Hancock and N Mishkate Eds Human Mental Workload Amsterdam The Netherlands North Holland 1988 Eggemeier F T and Amell J R 1986 Visual probability monitoring Effects of display load and signal di
8. Moderate conscious mental effort or concentration required Complexity of activity is moderately high due to uncertainty unpredictability or unfamiliarity Considerable attention required Moderate stress due to confusion frustration or anxiety noticeably adds to workload Significant compensation is required to maintain adequate performance 86 tO Occasionally have spare time Interruptions or overlap among activities occur frequently Extensive mental effort and concentration are necessary Very complex activity requiring total attention high to very intense stress due to confusion frustration or anxiety High to extreme determina tion and self control required 59 48 49 08 0 RENE SODAS SSNS SSeS SOA 06 GS 28 OD SASS 58 58 06 28 58 60 80 28 00 08 OD Often have spare time Interruptions or overlap among activities occur infrequently or not at all Extensive mental effort and concentration are necessary Very complex activity requiring total atten
9. The criteria for deriving either a group scaling solution or a prototyped scaling solution is based upon the value of the Kendall s Coefficient of Concordance which is listed immediately before the prototyping section This coefficient is an index of the degree of intersubject agreement within the card sort Our experience indicates that a value of approximately 75 and above indicates rela tively homogeneous group of subjects 1 only one scaling solution is necessary to capture the 28 TABLE 1 RANK ORDERING OF THE SIX PROTOTYPE GROUPS AS IDENTIFIED BY THE RANDOMLY ASSIGNED LETTERS ON THE SWAT CARD DECK 1 1 2 4 3 7 4 5 5 6 8 7 3 8 6 9 9 TACOS KO TABLE 2 PROTOTYPE ANALYSIS OF SWAT CARD SORT DATA SPEARMAN RANK CORRELATION RS FOR EACH PROTOTYPE Subject TES TSE RS 0 62 0 65 No 2 0 86 0 88 No 3 0 68 0 80 0 77 Highest correlation Second highest correlation ETS 29 EST SET 0 72 0 81 0 79 0 65 0 72 0 77 0 48 0 84 0 92 0 76 0 68 subjects composite view of workload A Kendall s Coefficient below 75 usually requires that a separate scaling solution be developed for each of the three main prototype groups and sometimes e for additional groups as previously described This is needed in order to capture the differential weighting of the three SWAT dimensions that the groups have revealed through their ordering of the de
10. higher workload Remember you provide the event something with which you are familiar If we specified events for you to rank we very likely would describe 76 something with which you are not familiar This would make your judgments of the relative workload less valid This process of recalling events helps to establish a scale that is representative of this group of subjects opinion Some of the combinations may not remind you of a particular event It may be very difficult to think of how you could have the highest level of one dimension while having the lowest level on the other two dimensions It is true that in most cases the levels of the dimensions will go in the same direction In other words as Time Load becomes greater then both Mental Effort Load and Psychological Stress Load are probably going to be increasing also and conversely On the other hand as we have discussed this with people they have gener ally been able to think of events where the more extreme combinations have existed We would suggest therefore that these combinations do exist but are not the most frequently occurring type of events If you simply cannot think of an event that a particular combination of descriptors would be appropriate for then you should think if an event did exist where would it fall in this order Also remember that we are asking you to judge how much work is associated not which would you like to have It might be clear
11. 8 DENN 2 2 4 g 5 lt 5 B E o 73 workload while other people will say that the only thing that is important to them is to be able to manage the psychological stress and others think that difficulty of the required tasks is most important Definition of Dimensions As a start you need to understand a little better what we mean by these three dimensions Time Load 1 Description Time Load is the amount of time pressure experienced in performing your task This includes the fraction of total available time that you are busy and the degree to which different aspects of the task overlap or interfere with one another Under high Time Load you are unable to complete the task due to a shortage of time or interference created by overlap of activities 2 Example For example in an emergency situation especially in a situation with multi ple emergencies the required actions may be relatively simple and well prac ticed The only real problem may be that things happen so fast that you just cannot get everything accomplished before things go from bad to worse B Mental Effort Load 1 Description Mental Effort Load is the amount of attention and or concentration required to perform a task Things that are considered mental effort include recalling things from long term memory decision making performing calculations storing and retrieving things from short term memory
12. NAME OF FUNDING SPONSORING us ae 9 PROCUREMENT INSTRUMENT IDENTIFICATION NUMBER applicable F33615 85 C 0541 Bc ADDRESS City State and ZIP Code 10 SOURCE OF FUNDING NUMBERS PROGRAM PROJECT TASK WORK UNIT ELEMENT NO NO NO ACCESSION NO 62202F 7184 14 07 11 TITLE Include Security Classification Subjective Workload Assessment Technique SWAT A User s Guide U PERSONAL etd Ber otter Scott S Bressler Jeine 134 TYPE OF REF ORT 13b TIME COVERED 14 DATE OF REPORT Year Month Day 15 PAGE COUNT Interim FROM 6 86 10 88 1989 Jul 115 16 SUPPLEMENTARY NOTATION Systems Research Laboratories Inc 2800 Indian Ripple Road Dayton Ohio 45440 17 COSAT CODES 18 SUBJECT TERMS Continue on reverse if necessary and identify by block number Mental Workload Conjoint Measurement SWAT 7 T 0 Subjective Workload Conjoint Scaling BER ONE ee I I2 01 2 1 Scale Development Event Scoring 19 ABSTRACT Continue on reverse if necessary and identi This report 15 to serve as a user s guide to accompany software Version 3 1 for the Subjec tive Workload Assessment Technique SWAT a tool for measuring mental workload The guide has been developed as a how to manual for implementing SWAT Specifically this report begins with a general overview of workload describes in detail all aspects of the scale development phase provides general information relating to the event scoring phase an
13. The equipment specification will direct the system to the appropriate disk drive You will be advised that the file already exists and asked to verify with a Yes or No that you wish to use data from a previous file This is a safeguard to prevent accidentally writing over an existing data file The screen will change to an updated version which includes the information previously saved in that file as depicted in Figure 23 The third function of this screen is to select the program options With the selection of either a new or an existing file you will be asked to choose one of the following options to direct the program to specific operations 88 3 dtues 022125 NW 201811 1937135 V d lds EJ KULNA Zd NOILVDISIDAdS LNAHdINDI 54 SINJHNOD 14 NIVH oc pesn sie Sour 3uemwoo 3 ATTeOTdAL 1 9 FO 02 2 ay aq SIL LNSNWOD 5103 815 40 KVH 5 9 XVM SHS3IOVHVHO O7 3IdNVS 3NHYN 3114 135 VLVd 9n 34VN AGGLS 44 L8 0 EO0 1 S AVGOL eseesee ANNAN SIHSHHOO OPTIONS F1 Edit Comments This
14. 1984 A comparison of projected and measured workload ratings using the subjective workload assessment technique SWAT in Proceedings of the IEEE 1984 National Aerospace and Electronics Conference pp 827 831 Dayton Ohio IEEE Eggleston R G and Kulwicki P V 1984 Estimating the value of emerging fighter attack system technologies in Proceedings of the Defense Research Group Panel Workshop Applications of System Ergonomics to Weapon System Development pp 1 20 Shrivenham England Eggleston R G and Quinn T J 1984 A preliminary evaluation of a projective workload assessment procedure in Proceedings of the Human Factors Society 28th Annual Meeting pp 695 699 Santa Monica California Human Factors Society Haskell B E and Reid G B 1987 The subjective perception of workload in low time private pilots Journal of Aviation Space and Environmental Medicine pp 1 12 109 Haworth L Bivens C Shively J and Delgado D 1987 Advanced cockpit and control configurations for single pilot helicopter nap of the earth flight Paper presented at the American Helicopter Society Forty Third Annual Forum and Technology Display Kuperman G G 1985 Pro SWAT applied to advanced helicopter crewstation concepts in Proceedings of the Human Factors Society 29th Annual Meeting pp 398 402 Santa Monica California Human Factors Society Masline P J 1986 A comparison oi the sensitivit
15. As previously described SWAT incorporates both MONANOVA and NONMETRG into the scaling solution MONANOVA is performed first and produces a set of rescaled values for each of the three levels of the three dimensions To aid in efficiency these rescaled values are then used as input for the NONMETRG algorithm After both solutions have been calculated the SWAT pro gram chooses the best solution from the two algorithms This is based on a rule which assumes that a difference of less than six units between the rescaled values of any two levels of a dimension implies essentially equivalent levels This occurrence is best fitted by the MONANOVA algorithm If the differences between each pair of all three levels of any given dimension are greater than six the program will choose the solution from the NONMETRG algorithm The selected solution 22 automatically will be displayed by the program However both solutions may be examined by selection of the proper options in the program as described in Appendix F 2 3 5 4 Subscales Whichever scaling solution is used the result is rescaled values with interval level properties for each of the three levels of the three dimensions Typically these are combined to form the 27 com binations of workload levels However it should be noted that these subscale values as we call them can also be used independently as a possible diagnostic approach to workload assessment Studies have been performed to inve
16. DD Re SRS SO SSN RO 1 MM uM EHE M NNNM DN Often have spare time Interruptions or overlap among activities occur infrequently or not at all Very little conscious mental effort or concentration required Activity is almost automatic requiring little or no attention Moderate stress due to confusion frustration or anxiety noticeably adds to workload Significant compensation is required to maintain adequate performance C Often have spare time Interruptions or overlap among activities occur infrequently or not at all Moderate conscious mental effort or concentration required Complexity of activity is moderately high due to uncertainty unpredictability or unfamiliarity Considerable attention required High to very intense stress due to confusion frustration or anxiety High to extreme determina tion and self control required Almost never have spare time Interruptions or overlap among activities are very frequent or occur all the time Very little conscious mental effort or concentration required Activity is almost automatic requiring little or no attention High to very intense stress due to confusion frustration or anxiety High to extreme determination and self control required ee Ne EE EN EE
17. The program will include only their data in the analyses If you make a mistake and enter an incorrect subject number reenter that number and they will not be included subjects to be included are indicated by highlighting Upon pressing RET the program will proceed with the analyses previously chosen on this Pro gram Setup screen Due to the criticality of the prototype correlations section it will ALWAYS be 95 performed even if it was not selected If no changes have been made however the program will not have to recalculate this section e PROTOTYPE ANALYSIS This screen is presented in Figure 26 and displays the Spearman s rank correlations for each sub ject s data with the six perfect prototype rankings the suggested prototype groupings based on these correlations and the Kendall s Coefficient of Concordance for agreement among the entire group of subjects in this analysis There are several options for this screen F1 Change It may be desired to group the subjects differently based on the experi Prototype menter s evaluation of the pattern of correlations To do this use the up and down arrow keys to move the cursor to the appropriate subject s prototype and then enter the new prototype T E or S The highlighting will change in accordance with the new prototype To leave a particular subject out of the analysis simply change that subject s suggested prototype to 1 The program will ignore this subject s d
18. UR UP UD UR dE UD s s P s s AM Me um ENS e e Fm m SSR s 58 59 5 58 58 49 A am SSS GR APO This appendix describes the requirements and methods for analyzing SWAT scale development data using the microcomputer based software The computer system requirements for running the program are IBMPC or compatible system 512K internal memory Two floppy disks or one hard disk and one floppy disk The software requirements are SWATPRGM BAT This is the current version 3 1 of the SWAT program which will handle MAIN EXE SWAT DAT ANSI SYS Optional Math Coprocessor Appendix F INFORMATION FOR PROGRAM OPERATION up to 30 su
19. 2 1 2 2 2 3 2 4 2 5 INTRODUCTION DESCRIPTION OF THE DIMENSIONS 2 2 1 Time Load 2 2 2 Mental Effort Load 2 2 3 Psychological Stress Load THE CONJOINT MEASUREMENT MODEL 2 3 1 Introduction 2 3 2 Axiom Tests 2 3 2 1 Independence 2 3 2 2 Joint Independence 2 3 2 3 Double Cancellation Scale Development Data Collection Axiom Testing Programs Scaling 2 3 5 1 MONANOVA 2 3 5 2 Johnson s Algorithm 2 3 5 3 Final Scaling Solution 2 3 5 4 Subscales CARD SORT PROCEDURE Nyy Uy 7 gt 2 4 1 Introduction 2 4 2 Timing of Card Sorts 2 4 3 Additional Comments CARD SORT ANALYSIS 2 5 1 Introduction 2 5 2 Prototyping 2 5 3 Sample Data Analysis EVENT SCORING 3 1 3 2 3 3 INTRODUCTION TASK DEFINITION 3 2 1 Laboratory Tasks 3 2 2 Operational Evaluations DATA ANALYSIS TABLE OF CONTENTS continued Section 4 SUMMARY AppendixA CARD DECK FACSIMILE Appendix B OUTLINE FOR VERBAL CARD SORT INSTRUCTIONS COLLEGE STUDENTS Appendix C OUTLINE FOR VERBAL CARD SORT INSTRUCTIONS PILOT POPULATION AppendixD WRITTEN CARD SORT INSTRUCTIONS AppendixE SAMPLE CARD SORT DATA SHEET AppendixF INFORMATION FOR PROGRAM OPERATION Appendix G EVENT SCORING INFORMATION BIRLIOGRAPHY REFERENCES 55 56 6 71 78 83 85 105 107 114 N co 11 13 14 LIST FIGURES Comments and Main Menu for Sample Data Set Indicating Utilization of an Existing F
20. 3 GROUP SCALE 4 PROTOTYPE AXIOMS 5 PROTOTYPE SCALE 6 INDIVIDUAL AXIOMS 7 INDIVIDUAL SCALES ESC MAIN MENU OPTIONS CHOSEN Figure 25 Program Setup Screen The prototype correlations analysis performs a Spearman s rank order correlation on each of the subject s rank order data with the rankings associated with the six possible prototype groups The pattern of correlations indicates the relative importance a subject places on the three dimensions of 94 the SWAT definition of workload The Kendall s Coefficient of Concordance is an index of the degree of agreement among the group of subjects about the ordering of the 27 cards A high Kendall s indicates substantial agreement about the order and therefore about the relative impor tance of the three dimensions The axiom testing section performs the axiom tests for independence joint independence and dou ble cancellation This is done to check for violations of these axioms which may invalidate the additive model as being a suitable model to use for the conjoint scaling routine Only a summary of the three axiom tests is automatically displayed but the complete history of results for all the tests may be viewed or printed by selection of the appropriate option Because the scaling solution requires data from both the prototype correlations routine and the axiom testing routine these functions will by default be performed by the program if you are working on an initial analysis The
21. 311 20 9 312 45 2 313 72 3 321 28 5 322 52 9 323 79 9 331 48 6 332 73 0 333 100 0 54 Section 4 SUMMARY If reviewed in its entirety this user s guide should provide 1 an overview of the tenet of mental workload 2 a step by step description of how to implement SWAT as a measurement tool and 3 step by step instructions for running the analysis software The additional information on the conjoint measurement model including axiom testing and scaling was introduced to provide the technical background and reasoning underlying the application of conjoint measurement to SWAT Appendix A CARD DECK FACSIMILE eo 8 06 28 58 08 08 29 8 08 mG ON GD ea 50 a ed 98 10 Almost never have spare time Interruptions or overlap among activities are very frequent or occur all the time Moderate conscious mental effort or concentration required Complexity of activity is moderately high due to uncertainty unpredictability or unfamiliarity Considerable attention required Moderate stress due to confusion frustration or anxiety noticeably adds to workload Significant compensation is required to maintain adequate Performance mrem mme mo
22. 5 00 5 00 123 C 9 00 10 00 22 00 4 00 20 00 18 00 20 00 22 00 8 00 7 00 131 X 6 02 12 00 7 00 7 00 3 00 8 00 3 00 8 00 6 00 6 00 132 8 00 11 00 16 00 8 00 12 00 12 00 12 00 12 00 7 00 10 00 133 21 00 23 00 25 00 9 00 21 00 16 00 23 00 23 00 9 00 12 02 2110 2 00 4 00 2 00 10 00 4 00 2 00 4 00 2 00 10 00 8 00 212 G 3 00 5 00 11 00 11 00 13 00 9 00 13 00 5 00 12 00 9 00 2132 15 00 13 00 20 00 12 00 22 00 20 00 21 00 13 00 13 00 13 00 221 V 4 00 6 00 5 00 13 00 5 00 7 00 5 00 3 90 11 00 13 00 222 Q 13 00 14 00 14 00 14 00 14 00 11 00 14 00 20 00 14 00 14 00 223 22 14 00 15 00 23 00 15 00 23 00 25 00 22 00 14 90 15 00 18 00 231 11 00 16 00 8 00 16 00 6 00 10 00 7 00 10 00 16 00 16 00 232 E 25 00 17 00 17 00 17 00 15 90 15 00 16 00 21 00 17 00 19 00 233 20 00 24 00 26 00 18 00 24 00 21 00 24 00 26 00 18 00 22 00 311 16 00 7 00 3 09 19 00 7 00 13 00 6 00 4 00 19 00 15 00 312 P 17 00 18 00 12 00 20 00 16 00 17 00 15 00 15 00 20 00 17 00 313 D 19 00 25 00 21 00 21 00 25 00 23 00 25 00 19 00 24 00 23 00 321 Y 23 60 19 00 6 00 22 00 8 00 14 00 8 00 17 00 21 00 20 00 322 A 18 00 20 00 15 02 23 00 17 00 19 00 17 00 16 00 22 00 21 00 323 26 00 26 00 24 00 24 00 26 00 26 00 26 00 24 00 25 00 26 00 331 L 22 00 21 00 9 00 25 00 9 00 22 00 9 00 18 00 20 00 24 00 392 T 24 00 22 00 18 00 26 00 18 00 24 00 18 00 25 00 26 00 25 00 333 I 27 90 27 00 27 00 27 00 27 00 27 00 27 00 27 00 27 00 27 99
23. 9 FAILURES OUT OF 0 TESTS JOINT INDEPENDENCE T INDEPENDENT OF S 23 FAILURES OUT OF 108 TESTS E x 8 INDEPENDENT OF 21 FAILURES OUT OF 108 TESTS S x T INDEPENDENT OF E 30 FAILURES OUT OF 108 TESTS OPTIONS PROTOTYPES Fi GOTO NEXT PROTOYPE F2 GO TO NEXT OPTION CHOSEN IN PROGRAM SETUP PRINT SUMMARY OF AXIOM VIOLATIONS F4 PRINT COMPLETE AXIOM HISTORY ESC MAIN MENU e Figure 7 Summary of Axiom Violations for the Two Effort Prototype Subjects 37 anantun SUMMARY OF AXIOM VIOLATIONS PROTOTYPE ANALYSIS STRESS PROTOTYPE INDEPENDENCE T INDEPENDENT OF E AND Q FAILURES OUT OF 108 TESTS E INDEPENDENT OF T AND 5 Q FAILURES OUT OF 108 TESTS S INDEPENDENT OF T AND E Q FAILURES OUT OF 108 TESTS DOUBLE CANCELLATION DOUBLE CANCELLATION IN x E Q FAILURES OUT OF 3 TESTS DOUBLE CANCELLATION IN E 5 0 FAILURES OUT OF 3 TESTS DOUBLE CANCELLATION IN S x T 0 FAILURES OUT OF 3 TESTS JOINT INDEPENDENCE T x E INDEPENDENT OF S 0 FAILURES OUT OF 108 TESTS E x S INDEPENDENT OF T 0 FAILURES OUT OF 108 TESTS 8 x T INDEPENDENT 0 E 0 FAILURES OUT OF 108 TESTS OPTIONS PROTOTYPES GOTO NEXT PROTOYPE F2 GO TO NEXT OPTION CHOSEN IN PROGRAM SETUP F3 PRINT SUMMARY OF AXIOM VIOLATIONS F4 PRINT COMPLETE AXIOM HISTORY ESC MAIN MENU Figure 8 Summary of Axiom Violations for the Four Stress Prototype Subjects 38 Return to the Program Setup screen and choose No 6 Individu
24. 958 Santa Monica California Human Factors Society Emery D and Barron H 1979 Axiomatic and numerical conjoint measurement An evaluation of diagnostic efficacy Psychometrika 44 19 210 Green and Rao V 1971 Conjoint measurement for quantifying judgmental data Journal of Murketing Research 8 pp 355 363 Green P and Srinivasan V 1978 Conjoint analysis in consumer research Issues and out look Journal of Consumer Research 5 pp 103 123 Holt J and Wallsten S 1974 A user s manual for CONJOINT A computer program for evaluating certain conjoint measurement axioms Technical Report No 42 University of North Carolina L L Thurstone Psychometric Laboratory Johnson R 1973 Pairwise nonmetric multidimensional scaling Psychometrika 38 pp 11 18 114 Krantz D H and Tversky A 1971 Conjoint measurement analysis of composition rules in psychology Psychological Review 78 pp 151 169 Kruskal J 1965 Analysis of factorial experiments by estimating monotone transformations of the data Journal of the Royal Statistical Society Series B 27 pp 251 263 Notestine 7 C 1984 Subjective workload assessment and effect of delayed ratings in a proba bility monitoring task in Proceedings of the Human Factors Society 28th Annual Meeting pp 685 690 Sunta Monica California Human Factors Society Nygren E 1982 Conjoint Measurement and
25. Amalberti R and Poyot G 1987 Evaluation de la charge de travail du pilote induite par un systeme d arme guide laser Ministere de la Defense Centre d Etudes et de Recherches de Medecine Acrospatiale Laboratoire d Etudes Medicophysiologiques 16 330 110 Potter S S 1986 Subjective workload assessment technique SW AT subscale sensitivity to variations in task demand and presentation rate unpublished masters thesis Wright State University Dayton Ohio Potter S S and Acton W H 1985 Relative contributions of SWAT dimensions to overall subjective workload ratings in Proceedings of the Third Symposium on Aviation Psychology pp 231 238 Columbus Ohio The Ohio State University Aviation Psychology Laboratory Reid G B 1982 Subjective workload assessment A conjoint scaling approach Annual Scientific Meeting Aerospace Medical Association pp 153 154 Bal Harbour Florida Reid G B 1985 The systematic development of a subjective measure of workload in Proceedings of the Ninth Congress of the International Ergonomics Association 109 111 Bournemouth England Reid G B 1985 Current status of the development of the subjective workload assessment technique in Proceedings of the Human Factors Society 26th Annual Meeting pp 220 223 Santa Monica California Human Factors Society e Reid Eggemeier T and Nygren E 1982 An individual differences approach to
26. Conjoint Scaling A Users Ciuide Technical Report AFAMRL TR 82 22 Wright Patterson Air Force Base Ohio Air Force Aerospace Medical Research Laboratory Nygren E 1985 An examination of conditional violations of axioms for additive conjoint measurement Applied Psychological Measurement 9 pp 249 264 Potter 5 and Acton W H 1985 Relative contributions of SWAT dimensions to overall subjective workload ratings in Proceedings of the Third Symposium of Aviation Psychology pp 231 238 Columbus Ohio The Ohio State University Aviation Psychology Laboratory Reid G B Eggemeier F T and Nygren T E 1982 An individual differences approach to SWAT scale development in Proceedings of the Human Factors Society 26th Annual Meeting pp 639 642 Santa Monica California Human Factors Society Sheridan T B and Simpson R W 1979 Toward the Definition and Measurement of the Mental Workload of Transport Pilots FTL Report R79 4 Cambridge Massachusetts Massachusetts Institute of Technology Flight Transportation Laboratory Ullrich J R and Cummins D E 1973 PCJM A program for conjoint measurement analysis of polynomial composition rules Behavioral Science 18 pp 226 227 1 15 Government Printing Office 1989 748 002 00218
27. EE EEE EE EN EO 56 SEES NS EE REN OHS RRS SS MNES SOON TTS 9 Occasionally have spare time Interruptions or overlap among activities occur frequently Extensive mental effort and concentration are necessary Very complex activity requiring total attention Moderate stress due to confusion frustration or anxiety noticeably adds to workload Significant compensation is required to maintain adequate performance 88 88 SSS 68 00 OS 0 04 RRS SORES 08 08 OR NSN 34 84 88 24 OD 00 08 08 88 08 ERN MM ME EHE n 50 58 68 39 Often have spare time Interruptions or overlap among activities occur infrequently or not at Moderate conscious mental effort or concentration required Complexity of activity is moderately high due to uncertainty unpredictability or unfamiliarity Considerable attention required Little confusion risk frustration or anxiety exists and can be eas
28. Program Setup screen where you will select options that permit you to specify the analysis you want per formed See the description of the Program Setup screen for more details This option allows you to bypass the Data Entry screen if data have already been entered F4 Equipment If you change systems this option can be used to change the specifications Specification to correspond to the present system F5 End the Program After all analyses have been completed this option allows you to exit to the system e 90 a Id As can be seen this screen allows you access to all parts of the program Consequently you will be returning here frequently and this is where an ESC keypress will take you DATA ENTRY SCREEN Upon selection of F2 Data Entry on the Main Menu screen a formatted screen as depicted in Fig ure 24 will be displayed that allows you to enter or modify data The number of subjects that was previously entered on the Main Menu will be displayed at the top of the screen and sufficient space for data for this number of subjects will be created While decimals are not typically used space is included for special applications which require their use Normally only the integers 1 through 27 will have to be entered The following options then exist F1 Enter Edit Data If you select this option the cursor automatically moves to the position for subject No 1 card 1 1 1 and data may be entered or edit
29. SWAT scale development in Proceedings of the Human Factors Society 26th Annual Meeting pp 639 642 Santa Monica California Human Factors Society Reid G B and Nygren T E 1988 The subjective workload assessment technique A scaling procedure for measuring mental workload in P Hancock and N Mishkate Eds Human Mental Workload Amsterdam The Netherlands Elsevier Reid B Shingledecker C and Eggemeier F T 1981 Application of conjoint measure ment to workload scale development in Proceedings of the Human Factors Society 22th Annual Meeting 522 526 Santa Monica California Human Factors Society Reid B Shingledecker Hockenberger L and Quinn T J 1984 A projective application of the subjective workload assessment technique in Proceedings ot the IEEE National Asrospace and Electronics Conference pp 824 826 Dayton Ohio IEEE 111 Reid B Shingledecker C A Nygren T E and Eggemeier T 1981 Development of multidimensional subjective measures of workload in Proceedings of the IEEE International Conference on Cybernetics and Society pp 403 406 Schick F V and Hahn R L 1987 The use of subjective workload assessment technique in a complex flight task AGARD AG 282 Advisory Group for Aerospace Research and Develop ment pp 37 41 Schlegel E and Gilliland K 1987 Evaluation of the criterion task set Unpublished Contract
30. a subject to refer to the card at the time of giving event ratings the card can be used as a last minute refresher just prior to beginning the experiment When referring to the SWAT dimensions or obtaining ratings on the dimensions the order of Time Load Mental Effort Load and Psychological Stress Load should always be maintained This convention has been established in order to minimize the mental load associated with communi cating about the dimensions Such standardization allows the subjects to devote less attention to deciding which dimension to evaluate first second and third thereby reducing the chances of the rating task being intrusive on primary task performance Acton and Colle 1984 addressed the issue of order effects by investigating the relationship between dimension order and ratings They found that the interaction between order of dimensions used to obtain ratings and the workload level presented to the subject was not statistically significant This result cun be interpreted to indi cate that the order of presentation of dimensions does not affect the ratings ussigned to task levels o 50 3 2 TASK DEFINITION 3 2 1 Laboratory Tests Experiments in the laboratory are generally divided into a series of trials t is then a simple matter to obtain ratings between trials The exact method of obtaining the ratings may vary to fit the con straints of each experiment Ratings may be spoken written or entered
31. and Reid B 1986 A comparative evaluation of two subjective workload measures SWAT and the modified Cooper Harper scale Paper presented at the e Symposium on Psychology in the Department of Defense USAFA Colorado Springs Colorado 113 REFERENCES Acton W and Colle A 1984 The effect of task type and stimulus pacing rate on subjec tive mental workload ratings in Proceedings of the IEEE 1984 National Aerospace and Electronics Conference pp 818 823 Dayton Ohio IEEE Arbak C J Shew R L and Simons J C 1984 The use of reflective SWAT for workload assessment in Froceedings of the Human Factors Society 28th Annual Meeting pp 959 963 Santa Monica California Human Factors Society Eggemeier F T Crabtree S and LaPointe 1983 The effect of delayed report on subjective ratings of mental workload in Proceedings of the Human Factors Society 27th Annual Meeting pp 139 143 Santa Monica California Human Factors Society Eggemeier McGhee J Z and Reid B 1983 The effects of variations in task loading on subjective workload rating scales in Proceedings of the IEEE 1983 National Aerospace and Electronics Conference pp 1099 1105 Dayton Ohio IEEE Eggemeier T Melville B E and Crabtree M S 1984 The effect of intervening task per formance on subjective workioad ratings in Proceedings of the Human Factors Society 28th Annual Meeting pp 954
32. and problem solving High levels of Mental Effort Load are required in a situation which demands total concentration to perform while during lower levels of mental effort your mind may wander or attention may easily be shared with several relatively easy tasks 74 2 Example Mental Effort Load could involve such things as remembering a radio fre quency that must be dialed in after passing some navigation point or having to make a decision regarding which of several potential targets should be attacked and what direction to approach a target from on each pass Another example might be the memory load associated with remembering a complex procedure needed to activate a particular piece of equipment This situation might be intensified if employment of this piece of equipment is a rare event and therefore not as thoroughly learned as more frequently occurring event C Psychological Stress Load 1 Q Description Psychologicul Stress Load refers to the presence of confusion frustration and or anxiety which hinders completion of your task Psychological stress refers to the feelings of apprehension and tension one usually thinks of when you hear the term stress In addition other factors such as fatigue motivation und low levels of physical stressors may also contribute to the feeling of psychologicul stress load ixample It is well known that physical stressors such as G forces vibration tempera ture and
33. applicable to most work situations where mental workload is an anticipated problem The dimensions are defined in the following paragraphs 11 2 2 1 Time Load The Time Load dimension depends on the availability of spare time and the overlap of task activi ties This is closely associated with the use of time line analysis as a primary method of evaluating whether or not a person should be able to accomplish a task Time Load may be experienced as the rate that events occur or the speed of a system The three levels are 1 Often have spare time Interruptions or overlap among activities occur infrequently or not at all 2 Occasionally have spare time Interruptions or overlap among activities occur frequently 3 Almost never have spare time Interruptions or overlap among activities are frequent or occur all the time 22 2 Mental Effort Loud Mental Effort Load is an indicator of the amount of attention or mental demands that are required to accomplish a task independent of the number of subtasks or time limitations With low Mental Effort Load the concentration and attention required by a task are minimal and thus performance 18 almost automatic As Mental Effort Load increases so does the amount of concentration and atten tion required Generally this is due to the complexity of the task or the amount of information which must be processed by the operator in order to perform adequately High demand for mental effort req
34. by the combina tion represented in each card Arrange the cards from the lo vest workload condition through the highest condition You may use any strategy you choose to order the cards One strategy that is useful is to arrange the cards into three preliminary stacks representing high moderate and low workload Individual cards may be exchanged between stacks if necessary and then rank ordered within stacks Stacks can then be recombined and checked to be sure that they represent your ranking of lowest to highest workload However the choice of strategy is up to you and you should choose the one that works best for you 80 There is no school solution to this problem There is no correct order The correct order is what in your judgment best describes the progression of workload from lowest to highest for a general case rather than any specific event That judgment differs for each of us The letters you see on back of cards are to allow us to arrange cards in a previously randomized sequence so that everyone gets the same order If you examine your deck you will see the order on the back runs from A through Z and then ZZ Please remember 1 The card sort is being done so that a workload scale may be developed for you This scale will have a distinct workload value for each possible combination of Time Load Mental Effort Load and Psychological Stress Load The following example demonstrates the r
35. dimensions Note that this independence is really a monotonicity property and not a statistical independence property Hence theoretically it would be quite possible for example to find Time Load independent monotonic of Effort and Stress but not Effort indepen dent of Stress and Time or Stress independent of Effort and Time To the extent that the mono tonicity or simple independence property holds we have support for an additive model Every test of independence of A with B and C requires a 2 x 2 x 2 matrix with two levels of factor and two combinations of B x C Thus the total number of possible tests of the property in this case would be 108 for the 3 x 3 x 3 design 2 3 2 2 Joint Independence A second form of independence can also be examined in our three factor model The property known as joint independence states that A and B are jointly independent of C whenever al bl cl gt a2 b2 c1 if and only if al b1 c2 gt a2 b2 c2 7 16 Joint independence of A and B with respect to C indicates that if one combination of and B is greater than another at a fixed level of C i e 1 1 gt a2 b2 at c1 then the ordering should be preserved for any other level of the third factor c2 If joint independence holds for all pairs of factors then this implies that independence holds for a single factor However the converse is not necessarily true If simple independence holds for all factors this doe
36. noise can when existing in sufficient magnitude interfere with task performance At low levels these stressors may not produce interfer ence but will produce enough of an annoyance that after a certain amount of time some of the person s capacity may be expended just to keep the irrita tion pushed into the background This level of capacity expenditure we would attribute to the psychological stress dimension of workload 4 Description of Levels Within the Dimensions Now that we have some idea what we mean by the three dimensions notice in our cube that each dimension has three levels Level one is associated with the lowest degree of each dimension level three is associated with the highest degree of each dimension and level two is associated with a moderate degree of load for each dimension Verbal descriptors have been written to precisely define each of the levels for each of the dimen sions The numbers for these three dimensions will be used by you later when you are doing the event scoring The descriptors have already been introduced to you when you read the written instructions Now as you arrange the card deck in order from the lowest to the highest workload situation you will read each of these descriptor many times This will help you to become familiar with the meaning associated with each level of each dimension You are asked to try to think of the wording of the descriptors as you do your ordering rather than trying to u
37. of the sort to think of the of descriptors on each card as representative of all workload events like this It is important that the subject thinks of e events that he she has experienced so that judgments of the joint effects can be made between the different combinations Their sort should not be based on the anticipation of the upcoming study or experiment Subjects are permitted to write on the cards if they desire However you should request that they try to place the cards in order based upon the word descriptions rather than by creation of an algorithm based upon the level numbers Occasionally the data have the appearance of having been generated by application of an algorithm and marks on the cards usually support this observation Talking is permitted during the session but you should request that they express their own opinion in their ordering rather than attempting to develop a consensus sort should be pointed out that cards have letters the back which have been randomly assigned The deck is placed in alphabetical order so that all subjects have the same starting configuration The subjects should be warned against trying to use the letters as indicators of which is higher or lower good conscientious effort usually takes at least 20 minutes and may take as long as an hour Be suspicious of not convincing the subject of the importance of the sort if he she finishes too quickly In some insta
38. option allows you to change any or all of the information on the first e screen including the date study name file name number of subjects or additional comments This can be used to retrieve a different already existing data set for example by changing the file name Doing this will cause the program to search the disk for the newly entered file name Once the program finds the file a yes response to the prompt that asks whether to use the existing file will cause all of the stored information from that file to be loaded into the program and displayed in place of the current file When you have the tile you want in memory you may use the Edit Com ments option to change any piece of information on this screen This is useful if you are doing additional analyses to a file previously analyzed or if you are adding subjects If you wish to add subjects simply edit the Num ber of Subjects specified on this screen This more recent version of the data set can be saved under a different name if both an old and new version are desired through the Data Entry screen with the Save Data option If you change any character on a line you must reenter the rest of the line as e the program will not save the characters to the right of the cursor when you press RETURN F2 Data Entry This option takes you to the Data Entry screen to enter add subtract or otherwise modify data Program Setup Selection of this option takes you to the
39. order relations are necessary to adequatelv test these properties Thus it is sufficient to require each subject to merely present rank order judgments for each of the stimulus combinations generated by combining levels of the factors In SWAT the method for obtaining this information is the card sort During the scale development phase each subject is given a deck of 27 cards A facsimile of these cards has been included in Appendix Each of the cards has three statements on it representing one of the possible combinations of the levels of Time Load Mental Effort Load and Psychological Stress Load Therefore there is a total of 27 possible combinations of the descriptors Subjects are 17 asked to sort the cards so that the 27 combinations are rank orccred to reflect the degree of subjec tive workload from lowest to highest represented by each combination A card deck facsimile Appendix A has been included as both an example and a master to be used in making up your own card decks The letter printed on each card actually should be placed on the back of the cards when they are printed Typed capital letters are easier to distinguish than lower case letters If lower case letters are used a line should be placed under each letter to aid in discriminating those letters which might be confused if they are turned upside down e g an upside down u frequently looks like an n Each letter has been specifically assigned to each ca
40. prototype information will then automatically be displayed for the user but the axiom test results will not The prototype information which is displayed will be put into a data file and used by the program for subsequent analyses This saving of data from the prototype routine will speed up further analyses which you may require The scaling solution produces the scale values that result from the conjoint scaling routine These values are then used as the workload scores in subsequent analyses of results from the Event Scoring phase There are three methods for handling the scale development data group solutions prototyped solutions and individual solutions The determination of which solution is best is based on study objectives and the results of the correlations Kendall s Coefficient and axiom tests For a more detailed explanation of interpreting these analyses refer to Section 2 5 2 Prototyping and Section 2 5 3 Sample Data Analysis Upon choosing a group solution the program will automatically include all subjects in further analyses If you specify prototyped solutions the program will use the suggested groupings provided in the next screen to be described A separate solution will be provided for each of the three groups If individual axioms or scaling solutions are chosen the program will ask for specific individual subjects you desire to analyze separately This is done by simply entering the subject number s and pressing RETURN
41. questions 77 Appendix D WRITTEN CARD SORT INSTRUCTIONS SWAT CARD SORT INSTRUCTIONS FOR SUBJECTS During the course of this experiment you will be asked to quantify the mental workload required to complete the tasks you will be performing Mental workload refers to how hard you work to accomplish some task group of tasks or an entire job The workload imposed on you at any one time consists of a combination of various dimensions which contribute to the subjective feeling of workload The Subjective Workload Assessment Technique SWAT defines these dimensions as 1 Time Load 2 Mental Effort Load and 3 Psychological Stress Load For the purposes of SWAT the three dimensions have been assigned three levels The dimensions and their levels are described in the following paragraphs Time Load Time Load refers to the amount of spare time that you have available fraction of the total time that you are busy When Time Load is low sufficient time is available to complete all of your mental work with some time to spars As Time Load increases spare time drops out and some aspects of performance overlap and tasks interrupt one another This overlap and interruption can come from performing more than one task or from different aspects of performing the same task At higher levels of Time Load several aspects of performance often occur simultaneously you are busy and interruptions are very frequent Time Load may be rated on the t
42. rm mue 80 88 84 09 68 08 00 89 08 DOREM ESE REESE M MER MN NM RM EHE UA 00 64 18 06 EMT C um Occasionally have spare time Interruptions or overlap among activities occur frequently mental effort and concentration are necessary Very complex activity requiring total attention Little confusion risk frustration or anxiety exists and can be easily accommodated ananman a 68 6 88 8 28 08 08 98 DOMINE M MEME FER 8 Pm m dh 06 08 40 08 m 5 OD mee Almost never have spare time Interruptions or overlap among activities are very frequent or occur all the time EXC mental effort and concentration are necessary Very complex activity requiring total attention Little confusion risk frustration or anxiety exists and can be easily accommodated Often have spare time Interruptions or overlap among activities occur infrequently or not at all Extensive mental effort and concentration are necessary Very complex activity requiring total attention High to very intense stress due to confusion frustration or anxiety High to extreme determina tion and self control required Often have spare time Interruptions or overlap among activities occur infrequently or not at all Very
43. that the sorts be made prior to Ev Scoring participating in the experiment because of the training value associated with repeated reading of the descriptors However as far conjoint measurement theory is concerned the sort can be obtained at any time There are no known requirements regarding elapsed time between card sorts and Event Scoring In past 24 experiments the same subjects have performed sorts as far apart as a year Over 80 percent of the subjects produced sorts correlating 90 and above Less than 5 percent gave sorts whose order reflected different combination rules Thus the indication is that if the person is really trying to give an honest evaluation of how these dimensions combine his her judgment remains relatively stable for substantial time periods Conversely requiring subjects to repeat the card sort several times could result in annoying them and cause them to be careless in the ordering simply to complete the task as quickly as possible The result would be an ordering of the cards that does not faithfully represent the subjects com position rule and a potentially inconsistent ordering which could not be modeled In general it is recommended that the subjects be scheduled for a 1 hour scale development ses sion either individually or groups This session should be dedicated time free from other time constraints The experimenter should provide a complete explanation about the card sort and ma
44. to obtain data concerning how these dimensions combine to create each individual s personal impression of workload The Event Scoring phase is the experiment or test situation where the investigator is interested in obtaining information about the workload associated with task performance This SWAT User s Guide is to accompany Version 3 1 of the software used in the scale devel opment phase of SWAT and is intended to be a how to manual Discussion of technical issues and research related to the application of SWAT will be a limited part of this manual A SWAT bibliography is included for readers interested in a more in depth treatment of these subjects Before we begin there is one additional point that needs to be made As is well known there are at least two types of workload mental workload and physical workload There is extensive history of physical measurement and physiological measurement that addresses physical workload We have chosen to use SWAT as a measure of mental workload and in general omit considera tion of physical workload Some physical components that are within normal tolerance ranges but provide a source of irritation may be considered and this will be explained in a later section In the event that a large physical component of workload is expected to be present in a study the experi menter should plan to obtain appropriate measures to reflect it Despite the fact that SWAT is a measure of mental workload only through
45. workload as you see it You may prefer a 2 2 2 situation instc d of a 1 1 1 situation However you should still realize that the 1 1 1 situation imposes less workload on you and leaves a greater reserve capacity The card sort procedure will probably take 30 minutes to an hour Please feel free to ask questions at any time Thank you for your cooperation 82 Appendix E SWAT CARD SORT DATA SHEET ee s em UU 96 QU GM UP UD SD UD UD UR WR UP GR UD Gn UR 99 m 08 sa Q 88 s er ery A Ub UD UR UD UR ND UD 09 US D UR 99 UD UP te UB MD UD MD UD M MN UM UO UD MU 00 QA Ub MM AD 09 ONU D WP UD ND NA UD GN GP UD GD UD RS GR UD UD GE QM UD UG UD 96 us Vs WR ad MM 83 SWAT CARD SORT DATA SHEET continued re ILL II 89
46. y n USE A SEPARATE DATA DISK FOR EACH STUDY Figure 1 Comments and Main Menu for Sample Data Set Indicating Utilization of an Existing File esc COMMENTS AND MAIN MENU ue TODAY S DATE 02 26 87 mm dd yy FILE NAME teat STUDY NAME gample data 8 CHARS MAX 20 CHARACTERS MAX NUMBER OF SUBJECTS 12 COMMENT Thies im a sample data set included with the SWAT program COMMENT to demonstrate the procedures for performing conjoint COMMENT analysis Refer to the User s Guide for instructions MAIN MENU Fi EDIT COMMENTS F4 EQUIPMENT SPECIFICATION F2 DATA ENTRY FS END THE PROGRAM F3 PROGRAM SETUP MAKE A SELECTION Figure 2 Comments and Main Menu Displaying Previously Saved Information 31 In most applications the first task is to enter data The Data Entry screen is obtained by entering an F2 When creating a new data set enter F2 to begin entering data refer to Appendix F for a complete description of the data entry features Whether creating a new data file or using e after entering data the Data Entry screen should match that of Figure 3 One valuable feature of this program is a data checking algorithm which is invoked as you exit from entering editing data To demonstrate this when using TEST DAT enter F2 to enter edit and then enter an F1 to exit from entering data As you see there is a possible error in one of the sub jects data Typically it is easier to proof the dat
47. you place on the three dimensions You will tell us this by taking this deck of cards each of which has a combination of the three dimensions on it and order them from that combination which represents the lowest workload to the combina tion which represents the highest workload Do this by imagining a situation that you have experienced which could be described by the combination of the dimensions on a particular card and making a relative judgment about the workload associated with accomplishing this task and rank the cards accordingly As you can see with three levels of each dimension there is a total of three times three times three or 27 possible combinations which could be given In this way 27 cards are created which you must rank order Notice that in doing this ranking procedure the difficult part involves the trade off decisions which have to be made Suppose for example that you are comparing card 1 2 1 for time effort and stress respectively with card 1 1 2 Now in one situation the effort is higher while in the other situation stress is higher You must decide which situation yor would choose as lower in workload To do this you need to decide which dimension effort or stress has a greater impact on the overall workload to you Similar situations will arise with different combinations of all three dimensions There is no right or wrong answer to these decisions since each person feels differently about the importance of the d
48. 0 1014 14 SNOILdO 99 bE 9 553815 6 2661 ZO 2 559915 8 S9 T 553815 2 ZE 1404435 9 20 2 430443 S Ob T 150339 5 22 SHIL bT sSHIL T G3 IVOS3H 3 IHVISVA SAILIqQGV SAILIGGV ON NOILVSSLI NOYA G31NINHd 3 SNOILVH3II JHL JOJ SANIVA 37725 3 WIS ONIIVIS 3 S 012 4 JOJ X 2569 3 012 4 X 101 1 012 3 X LTTE FT EG HNOLO2VJ HOV3 40 32NVISOdNI 3AILIVT3H 3l1VWHIXOHddV 469 125610 16 166107 L690 75610 SSGIQ 169 TvS610 VISHL SNOIIVHAILI Tx ct TT oT 6 8 NOZIVH3II S 1571 e Bled MEY 10512 5 30304 60 211314 4469 ST 900 1556 2969 9 86139 0052 42 6046 Z 9 29 6 I Z9 91 EA ee te 414 4 2 817 4 9 gr I vPO T 69 o 68 SL Vi 77 S 7 19 Lb Lb a 2 o 0 7 ZE e T vo eo 00 YvO 6E 6E K ONAN 253 49 9 NOILVAYOINI
49. 0 12 00 0 ST 00 0079 OEZ 90 11 90 002 o s eo ec 00 22 00 6 00 G6 2 80 F 20 9 90 ZT 20 ZT 00 8 00 00 092 005 00 TT o z 0079 00 6 28 00T p 1 9 s 5123 5 ST T 91 00 00 ctt ee OO IT 09 OT 2036 9079 00 2 007 90 Ss 0079 00 80 2 ee 1 00 792 90721 2038 0 OO 0076 eo oc 80 2 00 SZ 20 91 pe ze 00 eov 99 29 NI VLVG 103 8 5 H3lN3 00 T 99 9T PI 0079 20 20 S OO gt oo EZ OO Qo 6 e 20 8 2 990 00 52 OO 90 00 eo TZ 2079 9079 9076 00 eo S 00 OT ee 20 1 1 N 04 lt mQ x Li m O x n gt O gt G N x i EEZ eec TEZ EZZ 2446 Tec 212 Tic EET ZET TET EZT ETT 211 TTT 33 12 SUBJECTS a 2 a 4 5 6 7 8 9 1e 111 N 1 00 1 00 1 00 1 00 1 00 1 00 1 00 1 00 1 00 1 00 112 B 7 00 2 00 10 00 2 00 10 00 4 00 10 00 3 00 2 00 2 00 113 10 00 8 00 19 00 3 00 19 00 6 00 19 00 7 00 3 00 4 00 121 5 00 3 00 4 00 6 00 2 00 3 00 2 00 6 00 4 00 3 00 122 J 12 00 9 00 13 00 5 00 11 00 5 00 11 00 11 00
50. 01 36 5 4 121 804 16 1 5 122 380 33 3 e 123 096 52 6 7 131 32 3 8 132 019 49 5 9 133 495 68 8 12 211 883 12 9 11 212 459 30 1 12 213 017 49 4 13 221 486 29 0 14 222 062 46 2 15 223 414 65 5 16 231 286 45 2 17 232 337 62 4 18 2 3 3 813 81 7 19 311 433 31 2 20 312 010 48 3 e Figure 30 Scaling Solution Screen 103 F5 Go to Next Option Chosen Program Setup ESC Escape Continues with any additional analyses which have been chosen in the program setup screen If no additional options were selected this option e will return you to the Main Menu screen Returns you to the Main Menu screen 104 Appendix G EVENT SCORING INFORMATION REFRESHER SUMMARY This summary outlines the purpose of the Subjective Workload Assessment Technique SWAT and the procedure for giving SWAT ratings SWAT is a quantitative method for measuring mental workload using subjective ratings Remember that this method uses the ranking information which you provided with your card sort to create a workload scale This scale has a distinct workload value for each possible combination of Time Load Mental Effort Load and Psychological Stress Load the three dimensions which comprise SWAT The definitions of the dimensions are as fol lows Time Load refers to the amount of task interruption or overlap Mental Effort Load is the amount of attentiori or concentration required to perform a tas
51. 20NVGQHOJNOO 40 INSISTAAIOD S TIVGNAN Wivd 5 103 9 5 40 SISATVYNV 44 SUMMARY OF AXIOM VIOLATIONS 8 446 PROTOTYPE ANALYSIS TIME PROTOTYPE INDEPENDENCE T INDEPENDENT OF E AND S 0 FAILURES OUT OF 108 TESTS E INDEPENDENT OF T AND S 0 FAILURES OUT OF 108 TESTS S INDEPENDENT OF T AND E 0 FAILURES OUT OF 108 TESTS DOUBLE CANCELLATION DOUBLE CANCELLATION IN T x E 0 FAILURES OUT OF 3 TESTS DOUBLE CANCELLATION IN E 5 1 FAILURES QUT OF 2 TESTS DOUBLE CANCELLATION IN S x T 0 FAILURES OUT OF TESTS JOINT INDEPENDENCE T x E INDEPENDENT OF S 2 FAILURES OUT OF 108 TESTS E x 8 INDEPENDENT OF T 7 FAILURES OUT OF 108 TESTS S x T INDEPENDENT OF E 2 FAILURES OUT OF 108 TESTS OPTIONS PROTOTYPES GOTO NEXT PROTOYPE F2 GO TO NEXT OPTION CHOSEN IN PROGRAM SETUP PRINT SUMMARY OF AXIOM VIOLATIONS F4 PRINT COMPLETE AXIOM HISTORY ESC MAIN MENU Figure 6 Summary of Axiom Violations for the Six Time Prototype Subjects SUMMARY OF AXIOM VIOLATIONS eee e PROTOTYPE ANALYSIS EFFORT PROTOTYPE INDEPENDENCE T INDEPENDENT OF E AND 5 28 FAILURES OUT OF 108 TESTS E INDEPENDENT OF T AND S 16 FAILURES OUT OF 108 TESTS S INDEPENDENT OF T AND E 26 FAILURES OUT OF 108 TESTS DOUBLE CANCELLATION 1 DOUBLE CANCELLATION IN x E 1 FAILURES OUT OF 2 TESTS DOUBLE CANCELLATION IN E x 5 1 FAILURES OUT OF 2 TESTS DOUBLE CANCELLATION IN 5 T
52. 6 99 L 85 96 897 V6 65 95 L6 5 L 84 447 647 59 5 T 46 9 65 NIVU 253 1 98 8456 09 V9 40135 5 09 96 tb 99 NI N3SO00HOD 42 2 909 66 NOILdO OL 09 54 S 96 0 T 09 tb 40155 1 89 TL 4 64 OL 6S7 195 99 69 24 415 135 153 513 351 SdALOlO0Hd 931539015 zt SIOSFrrns St t 46 66 P9 62 99 531 5 H VA 04 SH NOIIVTISHHOO 45 9S M 25 40 14319133305 S TIVGN33 JHL ViVa 5 103 05 H VA JO SISATIVNV 34 101084 N dade 43 MH N m n O Q sioofqng ad amp ioj014 ou Surpnpou 10319209 Jepu2 jo uonenoe2os Sureorpu sis euy 2303024 anst 1 86 19 54 0 7 257 1 92 71 967 99 TT L 85 49 95 89 Z6 OT L GE OF 95 16 66 6 1 gt 82 447 564 v9 v9 g 9031 46 SP EE 667 fINSH NIVH 253 4 98 86 097 53 527 9 401345 00 7 967 tb 9 t 5 NI N3SOOHO L iP 46 09 667 009011 NOILdO LX3N OL 09 vd 1 96 90931 09 tb
53. 89 3 84 68 28 9 99 5 56 66 19 67 667 FP 9 19 59 667 5 6 66 145 p GG 2 L tb 99 96 3 96 0 09 NIVM 253 L tv 09 36 081 dNLIS 9S7 09 68 06 904 S7 NI 3 G9 6 95 09 NOILdO OL O9 54 S GL 908 99 6v OF PE 5 66 46 9G WVH9OHd Ol NH ISH 64 3 06 69 56 95 14 S3dALOLOSd 315 195 153 5 351 531 S3dALOlOHd 4315399105 5104 5 LISCENS BOA SH NOILV I3HHOOD NVHHVddS 9702 2 M SVA 40 44319143305 5 1 4 1 4 5 123 8 5 HOV3 40 SISAT1VNV 3dAlOlONd HH m 97 Kendall s Coefficient you decide to choose different configurations for axiom testing and scaling solutions you can cancel any additional selections and choose again For example say you origi nally had chosen prototype correlations group axioms and group scale Now you observe that the subjects are not homogeneous with a Kendall s Coefficient of 72 You may wish to simply proceed with separating the subject population into prototype groups rather than a single group solution In either case the program will not recalculate the correlations expediting continuation throug
54. AAMRL TR 89 023 e AD A215 405 SUBJECTIVE WORKLOAD ASSESSMENT TECHNIQUE SWAT A USER S GUIDE U Gary B hala me FILE ARMSTRONG AEROSPACE MEDICAL RESEARCH LABORATORY Scott S Potter Jeine R Bressler D T SYSTEMS RESEARCH LABORATORIES INC ELECTE DEC O 51989 J e JULY 1989 INTERIM REPORT FOR PERIOD JUNE 1986 OCTOBER 1988 Approved for public release distribution is unlimited HARRY G ARMSTRONG AEROSPACE MEDICAL RESEARCH LABORATORY HUMAN SYSTEMS DIVISION AIR FORCE SYSTEMS COMMAND WRIGHT PATTERSON AIR FORCE BASE OHIO 45433 6573 55 01 922 NOTICES When US Government drawings specifications or other data are used for any purpose other than a definitely related Government procurement operation the Government thereby incurs no responsibility nor any obligation whatsoever and the fact that the Government may have formulated furnished or in any way supplied the said drawings specifications or other data is not to be regarded by implication or otherwise as in any manner licensing the holder or any other person or corporation or conveying any rights or permissior to manufacture use or sell any patented invention that may in any way be related thereto Please do not request copies of this report from the Armstrong Aerospace Medical Research Laboratory Additional copies may be purchased from National Technical Information Service 5285 Port Roy
55. BE 41135 0 7 89 142 4 GL 98 c 01 NHfiL3H 1 9 65 19 39 08 08 1 24 315 135 153 513 351 531 ans SdALOLl0Hd 39NVHO 14 931533915 5104 8 5 23 5 H YA HOA 58 NOILVIdJHNOO XNVH 5 GESS A SVA 32NYQNOONOO JO 13121443035 S rIVGN3X JHL 9194 S LOd3fHf l5 2 dO SISA IVNV 34 101084 8 120fqns sioo qng 55205 20 1031217207 5 9 jo 4 p 1 419 SL 0 55 5 00 7 967 EF 99 TT 1 85 84595 95 89 b6 416 1 6b OF 46 66 6 S bL 942 982 62 9 59 g S O T 46 6S Cb 4 nN3SH NIVH 2535 08 85 0997 64 9 dNLAS 5 O T 96 097 EV S NI N3SOOHOD 19 4242 99 S6 9071 v NOILdO 1 01 09 74 S 96 0031 09 dfiLds QA 89 14 527 64 2 OL 64 65 I9 99 68 09 T Zd 4 101044 315 135 153 513 3451 531 815 SdALOLlOHd 14 aarisagons 21 5193 5 123 56 HOA SH NOILIV OI3HNHOO STT6 SVM 40 LINSIOIJJ3OD S TIVONSY JHL WLiVd S LoO3rdnS HOV3 10 SISA IVNY 344 101089
56. DO ONOWY dv 1H3AO0 BO SNOI1dflHH31NI 34995 3AVH ATIVNOISVODO 14 LON HO ATLNANOSHSNI HNDDO ODNOWY dY1H3JAO 4 3HVdS 3AVH N313O Qvolawit 1 67 3012025 6 210814 31VOS 1VMS 001 00 69 6564416 IVONO lOHOASd 130333 IW1INAW c 1 853915 180333 L 6 Describe How Dimensions Combined Into One Dimensional Scale Another interesting aspect of SWAT is that the procedure provides a mathematical technique for combining your ratings on the three dimensions into a single workload scale similar to a one to ten scale described earlier As can be seen in these two situa tions the 1 1 1 combination is translated into naturally the lowest possible scale value of 0 The 2 1 3 combination in the right scene is translated to a 69 for example Later you will perform a certain task and give a rating for each of the three dimensions at the end of each task trial These three numbers will then be combined as in this illus tration into a value from 0 to 100 This will then serve as your workload score for that particular trial How Does This Relate to What We re Doing Today For now we need to find out information from you so that we can develop the mathematical model which will be used to combine the ratings you will give later The information we are looking for concerns the importance
57. Data Entry Screen Program Setup Screen Prototype Analysis Screen Axiom Tests Summary Screen Scaling Information Screen Plot of Rescaled Versus Raw Data Screen Scaling Solution Screen 92 94 97 99 101 102 103 LIST OF TABLES Number Page 1 RANK ORDERING OF THE SIX PROTOTYPE GROUPS AS IDENTIFIED BY THE RANDOMLY ASSIGNED LETTERS ON THE SWAT CARD DECK 29 2 PROTOTYPE ANALYSIS OF ONE SUBJECT S DATA 29 3 EXAMPLE OF HOW ASSIGN A RESCALED VALUE SUBJECT S GIVEN RATING 54 Section 1 INTRODUCTION The Subjective Workload Assessment Technique SWAT has been developed in response to a need for a workload measure with known metric properties that is useful in operational or real world environments Maximum effort has been expended to keep the SWAT data collection as unintrusive as possible The principal way this has been accomplished is through the application of a scaling procedure known as conjoint scaling This approach allows responses to be made in the operational setting using only three simple descriptors for each of three factors that have been used to operationally define workload This approach also minimizes the amount of time required to make responses by keeping down the number and complexity of descriptors that an operator must memorize SWAT is divided into two distinct phases Scale Development and Event Scoring The Scale Development phase is used to train the subjects on the use of the descriptors and
58. Report Wright Patterson Air Force Base Ohio Air Force Aerospace Medical Research Laboratory Schlegel R E Gilliland K and Schlegel B 1986 Development of the criterion task set performance data base in Proceedings of the Human Factors Society 30th Annual Meeting pp 58 60 Santa Monica California Human Factors Society Shingledecker C 1983 and m teli workload metrics for environ ments in Procee AE 338 6 1 6 8 Skelly J J and Purvis D 1985 B 52 wartime mission simulation Scientific precision in workload assessment in Proceedings of the 1985 Air pp 105 109 Thiessen S Lay J E and Stern J 1988 Neuropsychological workload test battery validation study Unpublished Contract Report Wright Patterson Air Force Base Ohio Air Force Aerospace Medical Research Laboratory Vidulich A and Tsang S 1985 Assessing subjective workload assessment A son of SWAT and the NASA BIPOLAR methods in Proceedings of the Human Factors Society 29th Annual Meeting pp 71 75 Santa Monica California Human Factors Society Warr D 1986 A comparative evaluation of two subjective workload measures Subjective workload assessment technique and the modified Cooper Harper scale unpublished masters thesis Wright State University Dayton Ohio Warr D Colle A
59. UCTIONS COLLEGE STUDENTS I Introduction 1 Describe Concept of Workload When we speak of mental workload we are clearly referring in some sense to mental effort Our ideas about mental workload are affected by our experiences with physical workload We can easily think of the effort one must expend to lift a heavy object dig a ditch or participate in our favorite sport Physical work has been quantified in many ways including carbon dioxide production heart rate or amount of work performed in a unit of time Mental workload on the other hand has proven to be more difficult to measure because it is something that occurs within the person and isn t directly observable We might think of mental workload as the amount of concentration required to write a paper work simple addition problems or solve complex algebra problems There is probably unanimous agreement that the amount of work required to solve complex algebra problems v ould be greater than the amount of work required to solve simple addition problems While sorne mental tasks are clearly harder than others mea sures which quantify this phenomenon have been difficult to develop and validate 2 Describe Purpose of This Experiment The experiment you are participating in is concerned with mental not physical work load and we will deal with methods of measuring the amount of workload experienced while performing a computer controlled laboratory task 3 Performance
60. UGH FOR EVERYONE TO READ THE INSTRUCTIONS For the purposes of SWAT workload has been defined as being composed primarily of three things Time Load Mental Effort Load and Psychological Stress Load Each of these three factors or dimensions has had three levels defined Therefore for the pur poses of SWAT workload can be represented by the cube shown in Figure 20 You can see that there ure three levels of each of the three dimensions All possible combina tions of these dimensions comprise the 27 cells of this larger cube that we are calling workload Your task today is through a card sort procedure to help us determine how these dimensions combine to create your conception of workload The deck of cards in front of you has a card for each of the cells in this cube Each card has three descriptors written on it one for time load one for mental effort load and one for psychological stress load By arranging this deck in an order that represents which combination you think describes the lowest workload condition to the combination that you think repre sents the highest workload condition and the 25 steps in between you are helping us create a scale that will reflect the way you people in this test think these dimensions combine to create the impression of workload This is not the same for everyone Some people think that time is the only element that has any importance for determining 72 SS E r 8 421 2
61. a by printing it out so that the entire data set can be viewed at once The entire data set as it should appear by printing it out is presented in Figure 4 Press F3 to print the data and find the error As a rule for checking data no matter what order the data are in every number 1 through 27 should appear only once and there should be no numbers out of this range Once you have spotted the error go back into the data set and make the correc tion Then as after any changes save the data set At this point you can go directly to the Program Setup screen F4 The first analyses typically per formed the correlations and Kendall s Coefficient of Concordance The results of these will dictate the direction of further analyses Choose a 1 on the Program Setup screen and proceed The results of the prototype correlations and Kendall s coefficient are presented in Fig ure 5 This screen indicates several important points First of all a single group scale would not be appropriate since the Kendall s Coefficient of Concordance is 7380 below the criterion of 75 8 Secondly it indicates a potential problem with the effort group There are only two subjects with highest correlations in the effort group and their correlations are not very high Generally a sub ject with a highest correlation below 80 is cause for further investigation especially if he she is in a small group of subjects To investigate further ret
62. al Axioms to investigate subjects No 8 and No 12 individually Choose these two subjects on the subject selection menu and pro ceed Figure 9 shows the axiom violations for subject No 8 and indicates that the analysis for this subject falls right on the border of acceptability Pressing F1 will display axiom inforination for subject No 12 as in Figure 10 This analysis far exceeds the criterion of axiom violations as indi cated by a maximum of 40 violations Now the question is how do we develop a scale for these two subjecis An additive solution definitely is not feasible to represent subject No 12 s data therefore this sub ject would either have to be dropped from the study or asked to provide additional information Refer back to Section 2 4 3 for further information on handling this type of problem However even though there is considerable inconsistency in subject No 8 s ordering an additive model will appropriately represent this data set The only problem is that if No 12 is dropped this leaves No 8 alone in the effort group Typically the goal is to try to have as many subjects in a group as possible since averaging tends to eliminate any random error that may be present in the data How ever referring back to Figure 5 indicates that No 8 had an equally high correlation with the stress group which suggests the solution of changing his her prototype membership to stress and repeating the analysis Figure 11 indicates the pro
63. al Road Springfield Virginia 22161 Federal Government agencies and their contractors registered with the Defense Technical Information Center should direct requests for copies of this report to Defense Technical Information Center Carneron Station Alexandria Virginia 22314 TECHNICAL REVIEW AND APPROVAL AAMRL TR 89 023 This report has been reviewed by the Office of Public Affairs PA and is releasable to the National Technical Information Service NTIS At NTIS it will be available to the general public including foreign nations This technical report has been reviewed and is approved for publication FOR THE COMMANDER CHARLES BATES JR Director Human Engineering Division Armstrong Aerospace Medical Research Laboratory UNCLASSIFIED REPORT DOCUMENTATION PAGE iHa Ove Olah 1 REPORT SECURITY CLASSIFICATION 16 RESTRICTIVE MARKINGS Unclassified 2a SECURITY CLASSIFICATION AUTHORITY 3 DISTRIBUTION AVAILABILITY OF REPORT Approved for public release distribution is 2b DECLASSIFICATION DOWNGRADING SCHEDULE gt unlimited PERFORMING ORGANIZATION REPORT NUMBER S AAMRL TR 89 023 NAME OF PERFORMING ORGANIZATION Armstrong Aerospace Medical if applicable Research Laboratory AAMRL HEG ADDRESS City State and ZIP Code 7b ADDRESS City State and ZIP Code Wright Patterson Air Force Base Ohio 45433 6b OFFICE SYMBOL 7 NAME OF MONITORING ORGANIZATION
64. amount of workload associated with task accomplishment When SWAT is used this type of information is obtained by applying the sume set of descriptors that the subject has becorne familiar with during the Scale Development phase After completion of a task or task segment the subject is asked to give a rating for Time Load Mental Effort Load and Psychological Stress Load The subject responds by giving either a 1 2 or 3 for each of the three dimensions The three levels are defined in the same way that they were for the composition of the cards used in Seale Development The difference is that in the Scale Development phase the sub ject had to recall some event from his or her past experiences for which the particular combination of descriptors was representative On the other hand in the Event Scoring phase the subject experiences the event task and must select the correct set of descriptors that describes the work load created by the event If a considerable amount of time has elapsed between the Scale Development card sort and the study it is important to provide refresher instructions to reacquaint the subjects with the descrip tors that are used to rate the events during the experiment An example of refresher instructions is included in Appendix G Also included in Appendix G is a SWAT Dimensions card Copies of this card may be used by subjects as a quick reference card during event scoring If experimental conditions do not permit
65. ance of the three dimensions included in SWAT The Prototype screen displays the results of how the individual subjects prototype A subject who prototypes time considers the Time Load dimension to contribute the heaviest to his perception of workload A subject who prototypes stress would consider the Psychological Stress Load dimension to contribute the heaviest These are just two examples of the three main prototyping groups The six possible groups are listed in the output as TES TSE ETS EST SET and STE 27 To calculate prototype group membership every subject s data are correlated against the six differ ent strings of data that represent these respective prototype groups which are presented in Table 1 The pattern of correlation coefficients of the six groups determines to which group a subject belongs As shown in Table 2 subject No 1 correlates the highest with SET and STE and so he is considered a stress subject In this manner nearly every subject can be labeled as either a time effort or stress subject more detailed description of this procedure can be found in Reid Eggemeier and Nygren 1982 While a large majority of subjects normally fall into one of the three main prototypes this is not always the case For some subjects as with subject No 4 in Table 2 the two highest correlation coefficients do not both point to a single prototype group For this subject the highest correlation is with TES but the
66. are not a sim ple function of one another Therefore Kendall s Tau is included as another goodness of fit indi cator in the NONMETRG solution In SWAT for example for a set of ranks that is perfectly additive Tau will be 1 0 indicating that all 351 pairs of estimated scale values are in the same order as the 351 pairs of ranks For data containing a slight error it is still possible for THETA to be 0 0 by producing tied scale values but for Tau not to be equal to 1 0 It may at first seem redundant to perform two scaling procedures in SWAT since both will yield identical results for perfectly additive data and generally very similar results However the differ ences in handling of inconsistencies error provide a complementary approach to the conjoint scaling problem MONANOVA interprets inconsistencies in the data as error and the order from the card sort is maintained The scale values are forced to be different resulting in a scale that would not properly reflect the two level model in the previous example On the other hand NONMETRG will give a better solution when a subject s card sort conforms to the typical model with three levels of each of the three dimensions but has a number of inconsistencies that are in fact random error If the program cannot find a pattern of ties in the ordering that explains incon sistencies then they are assumed to be error and NONMETRG is the selected scaling solution 2 3 5 3 Final Scaling Solution
67. as a Measure There are several ways in which your mental workload could be measured the first of which is to assess your performance on the task For example if the task is driving a car the precision of following a desired track or reaction time to something suddenly entering your visual field etc could be an indicator of your workload Frequently however there isn t any change in this type of observable performance although two e 62 people or the same persun under two different conditions may experience differing e degrees of effort expenditure to achieve this performance 4 Discuss Subjective Measures A Unidimensional Another way to attempt to measure your mental workload would be to simply ask you to rate your workload on a scale say from one to ten for whatever task you are performing If daydreaming was labeled as 1 and intense concentration as 10 you could probably give a rating corresponding to the workload you were experiencing in performing a task However this approach does not give us much information about WHY you gave a particular rating and we would not be sure that each person intended to describe the same level of workload even though the numbers used were the same B Multidimensional Another approach is to break up mental workload into several dimensions or fac e tors which are generally considered to comprise workload In this approach you would not actually be giving ratings on wor
68. ata This approach can be used in a case where a subject belungs to a particular prototype group but these data also contain a large number of axiom test violations indi e catir z an unacceptable amount of error in the data In this situation it may be desirable to exclude this subject from the scaling solution or attempt to remove the ambiguity in the data through additional information obtained from the subject This is also very useful in examining the Kendall s Coefficient for a particular prototype group To do this for the time group for example change all the effort and stress subjects suggested prototypes to L s and press again to end the changes The Kendall s Coefficient will automatically be recalculated and displayed F2 Print Sends the correlation matrix prototypes and Kendall s Coefficient to a printer F3 Return to This option allows you to return to the Program Setup screen and choose Program Setup additional or different analyses This option is used in two situations First of all if prototype correlations was the only option previously chosen in the Program Setup F3 allows you to choose additional analyses based on the results of the prototype correlations and Kendall s Coefficient of Concor dance Second if based on the results of the prototype correlations and 96 mang siskeuy 9c 5 18 84 562 TL 04 59 396 66 5 86 56 25 O 297
69. bjects data To start program operation type SWATPRGM and then return This file is provided on the SWAT software disk This is the executable program which is called by SWATPRGM This file is provided on the SWAT software disk This is a data file which provides parameter information to the program This file is included on the SWAT software disk This is a functional device driver that enables the program to use special character sequences The ANSI driver MUST be added to the system s linked list of standard device drivers by inserting a DEVICE ANSISYS in the CONFIG SYS file in the root directory Of course the ANSI SYS file must be in the root directory as well Once you have done this ANSI SYS will be loaded into memory whenever you boot up your system ANSLSYS is part of the MS DOS operation systems and therefore is purchased by each user The 8087 math coprocessor chip will speed up program execution but is not necessary to run the program on systems based on the 8086 or 8088 family of processors The SWAT program will run on systems based on the 80286 family of processor chips as long as the system has a math coprocessor included in the system configuration SWAT will not run on a 80286 based computer system that does not have a math coprocessor During program operation one file is written to the program disk Because of this the original disk is not write protected It is advisable to make backup disks immediately
70. c that could affect the amount of a person s capacity that is required to maintain acceptable task performance 71 In this study we are going to measure workload through the use of a scaling approach called the Subjective Workload Assessment Technique or SWAT This is a technique that has been developed and extensively tested at the Armstrong Aerospace Medical Research Laboratory at Wright Patterson Air Force Base and has been successfully used in a number of simulation tests flight tests and OT amp Es This technique is differ ent from most scaling procedures in that there are two parts to it The first part is called Scale Development and is what we are going to do today and the second part is called Event Scoring and is the part where you will give workload ratings in the simulator One of the primary objectives of this technique is to create as little interference as possi ble during task performance while getting the best quality data possible Before I start a more detailed explanation of the procedure I would like for you to quickly read over the written instructions that you have been provided Don t labor over these instructions since I am going to repeat much of it anyway I want you to read the instructions to be sure I don t forget something important and by getting a preview it will help you follow what I am trying to tell you later and hopefully provide you with a better understanding of the procedure PAUSE LONG ENO
71. cedure for doing this Return to the Prototype Correlations screen and enter F1 to change the prototype Do this by moving the cursor to the desired subject and entering the proper letter To exclude a subject enter an L as the prototype group Notice that subject 12 has been excluded by this technique Upon pressing F1 again the program will recalculate the Kendall s Coefficient and the correlations as indicated by Fig ure 12 As is evident in this figure the Kendall s increased to 758 with the deletion of No 12 the group Kendall s does not change by changing prototype group as for subject No 8 After changing the groupings there are two other aspects which need to be checked First of all the Kendall s Coefficient for each of the prototype groups in this case only time and stress should be calculated The procedure for doing this is as just described delete all subjects which are notin a particular prototype group The results for the time group are shown in Figure 13 This figure indicates a Kendall s of 8835 well above the criterion Similarly for the stress group Figure 14 shows that the Kendall s is 9119 Secondly since subject No 8 has been added to the stress group the axiom violations for this new group should be checked This is done by selecting 4 on the Program Setup screen As presented in Figure 15 adding this extra subject only slightly changes the axiom violations and the solution for this group is certainly s
72. d finally goes into a step by step explanation of each manu and screen of the program Various topics are covered including descriptions of the three SWAT dimensions use of con joint measurement and scaling card sort procedures and analysis methods of prototyping event scoring and data analysis Each how to section that the user will need has been provided in an appendix so that the appropriate section can be separated from the rest of the report for handy reference iss 20 DISTRIBUTION AVAILABILITY OF ABSTRACT 21 ABSTRACT SECURITY CLASSIFICATION G uNcLAssrigD UNLIMITED JsAME 5 Cloricusers Unclassified 22 549 RESPONSIBLE INDIVIDUAL 223 TELEBBONE Area Code ARE ey em DD Form 1473 JUN 86 Previous editions are obsolete SECURITY CLASSIFICATION OF THIS PAGE UNCLASSIFIED 5 MONITORING ORGANIZATION REPORT NUMBER S SUMMARY This user s guide along with its appendices is the source document for Version 3 1 of the Subjec tive Workload Assessment Technique SWAT a procedure for measuring mental workload This particular version of SWAT was developed to be more accessible in terms of both equipment requirements and user friendliness than the previous versions that required a mainframe computer The basic equipment needed to implement Version 3 1 includes a microcomputer running the MS DOS operating system 512K internal memory and two floppy disks or one hard disk and one floppy dis
73. ects and the levels of the indepen dent variables are obtained Thus there are effectively two separate and independent processes in the conjoint measurement methodology and hence two such processes in SWAT First one attempts to find the appropriate combination rule and then assuming the rule is valid finds numeri cal functions that best fit the observed order of the joint effects in the data while conforming to the specified rule The general theory as outlined by Krantz and Tversky 1971 provides for a series of axioms which when tested with a set of data aid in discriminating among four simple polynomial models to determine which of ther best fit the set of data For example let f1 T1 f2 E1 and f3 S1 represent the subjective scale values associated with these levels for a given individual We could postulate that the levels of the three factors combine to form an overall judged value for perceived workload f T1 El S1 via either an additive model as previously described if El 51 f1 T1 1 f3 S1 2 14 a multiplicative model if f T1 E1 1 f1 T1 f2 E1 f3 S1 3 a distributive model if E1 81 f2 E1 f3 81 4 or a dual distributive model if f T1 51 f1 T1 f2 E1 f3 S1 5 Note that in the latter three models the overall value of the combined effect of the three factors f T1 E1 S1 could be completely erased if one of t
74. ed The rank assigned to card 1 1 1 by this subject should be entered and then two options exist for entering the data to the computer s memory Pressing RET or down arrow will move the cursor vertically one position down while pressing the right arrow key will move the cursot to the right one posi tion The program is set up to accept either the ranks assigned to all 27 cards by the first subject moving downward or the ranks assigned by all subjects for a particular card moving to the right In either manner the program will step through all 27 cards for the specified number of subjects At the end of a row or column of data use the appropriate arrow keys to move the cursor to the next position on the data entry screen Every third tow on this screen is highlighted to make it easier to keep track of where you are us you scroll through the data As you reach the end of the dis played data the screen will scroll to the next position to allow continuation of data entry Because the scrolling is rather slow the system is capable of storing up to 11 key presses This allows you to continue to enter data und then let the system catch up at the end of a row Due to the delays involved in entering data across rows it is recommended that if you have 20 or more subjects enter the data down the columns For editing purposes use the arrow keys to move the cursor to any position on the screen and make the appropriate change Pressing either RET o
75. elationship between the card sort and the resulting workload scale Time Effort Stress Workload Scale 1 1 1 00 3 3 3 100 0 2 When performing the card sorts use the descriptors printed on the cards Please remember not to sort the cards based on one particular task such as flying an airplane or what you anticipate that you will be doing in this study Sort the cards according to your general view of workload and how important you consider the dimensions of Time Load Mental Effort Load and Psychological Stress Load 10 be Base these decisions on all types of experiences and ask situations 3 During the actual expenment you will accomplish the desired task Then you will provide a SWAT score base on your opinicn of the mental workload required to perform the task This SWAT score will consist of one number from each of the three dimensions For exam ple a possible SWAT score 15 1 2 2 This represents for Time Load 2 for Mental Load and 2 for Psychological Stress Load 81 4 We are not asking for your preference concerning Time Load Mental Effort Load and Psy chological Stress Load Some people may prefer to be busy rather than idle in either Time Load Mental Effort Load or Psychological Stress Load dimension We are not con cerned with this preference We need information on how the three dimensions and the three levels of each one will affect the level of
76. f each of the dimensions Identification of Other Dimensions While it van be seen that these three dimensions contribute to mental workload you may be able to think of other dimensions that may have an effect on the workload involved in performing a task While this may be true we believe that these three dimensions can be used to cover most of what most people are referring to when they speak of workload more precise breakdown into more fundamental components would complicate the rating proces and possibly interfere with the rater s ability to perform the assigned task thus con ibuting to workload Examples of the Inte tions Between the Three Dimensions Figure 19 provides an example of two situations and how workload may be affected In the left scene we have a pilot approaching an airfield on a clear sunny day Let us presume that he she has made this landing many times before In this situation for this pilot the task might get a rating of 1 for Time Load 1 for Mental Effort Load and 1 for Psychological Stress Load In contrast on the right side the situation has changed and the approach is being attempted in adverse weather with visibility obscured Now the pilot might give a rating of 2 for Time Load since he is not able to see what is hap pening and is not able to perform the necessary actions as early as in the previous situa tion He may give a rating of 1 again for Mental Effort Load since the decisions and amount of co
77. factors when they are at relatively low levels but create enough of an irritant that individuals must draw on resources in order to prevent interference with task performance The specific levels for the Psychological Stress Load dimension are 1 Little confusion risk frustration or anxiety exists and can be easily accommodated 2 Moderate stress due to confusion frustration or anxiety noticeably adds to workload Significant compensation is required to maintain adequate performance 3 High to very intense stress due to confusion frustration or anxiety High to extreme determination and self control required 2 3 THE CONJOINT MEASUREMENT MODEL 2 5 1 Introduction There are many composition rules that describe how complex multifactor or multidimensional judgments are formed One simple yet psychologically useful rule is the additive rule which suggests that independent variables combine in an independent additive fashion to produce an overall joint psychological effect This additive rule is the model that underlies SWAT For example let T1 be a level of the Time factor E1 be a level of the Effort factor and S1 be a level of the Stress factor We might hypothesize that the joint effects of these three factors which we call workload could be described as f T1 E1 1 f1 T1 1 f3 S1 1 13 where f 1 f2 and 3 are separate and identifiable numerical functions Krantz and Tversky 1971 Additive models
78. for operational meaningfulness is that each of the tasks may have a different duration In the previous example the landing task might have a duiation of approximately 45 seconds while lift off to passing through 10 000 feet might be over 10 minutes Some psychometricians would say that because of the difference in duration measures of these tasks are not comparable While there is undoubtedly noise injected into these data due to this difference such noise very likely is less severe than the noise injected by artificially segmenting real world tasks As in laboratory tasks ratings can be obtained either verbally or through some automated data collection process Like wise the cue to prompt the subject to give a rating can be either an inquiry by the experimenter or a signal of some type that is under equipment control In some cases there ure capabilities inherent in the operational equipment that can be adapted to this purpose 51 A primary concern in operational tests is to obtain data without interfering with the operator s abil ity to perform the task Most likely if the entire complex task or job is considered a very large number of task segments will exist If the investigator requires a response for every identifiable task segment then the objective of not interfering with the operator s task performance will proba bly not be attained Therefore it is important for the investigator to identify the crucial segments to provide
79. gh Psychological Stress Load and have plenty of spare time between relatively simple tasks Since the three dimensions contributing to workload are not necessarily correlated please treat each dimension individually and give independent assessments of the Time Load Mental Effort Load and Psychological Stress Load that you experience in performing the following tasks One of the most important features of SWAT is its unique scoring system SWAT uses a procedure to find separate scoring weights for each level of a dimension Then it determines a distinctive workload scale for each person or group This scaling system greatly improves the precision of the workload ratings you will give later In order to develop your individual scale we need information from you regarding the amount of workload you feel is imposed by various combinations of the dimensions described above We get this information by having cu rank order the workload associated with each of the combinations In order for you to rank order the workload for each of the combinations you have been given a set of 27 cards with the combinations from each of the three dimensions Each card contains a dif ferent combination of levels of Time Load Mental Effort Load and Psychological Stress Load Your job is to sort the cards so that they are ranked according to the level of workload represented by each card In completing your card sorts please consider the workload imposed on a person
80. gram will go back to the Main Menu F2 Print Summary This will print a hard copy of just this summary of the axiom test violations of Group Axioms Print Complete This will print a hard copy of the results of the entire set of axiom tests Axiom History which includes more detail on the results of the axiom tests Typically this 98 ww SUMMARY OF AXIOM VIOLATIONS GROUP ANALYSIS INDEPENDENCE T INDEPENDENT OF E AND S E INDEPENDENT OF T AND S 5 INDEPENDENT OF T AND E DOUBLE CANCELLATIO DOUBLE CANCELLATION IN T x E 0 FAILURES OUT OF 1 TESTS DOUBLE CANCELLATION IN E x S 1 FAILURES OUT OF 3 TESTS DOUBLE CANCELLATION IN S x T Q FAILURES OUT OF 1 TESTS JOINT INDEPENDENCE T x E INDEPENDENT OF S 4 FAILURES OUT OF 108 TESTS E x S INDEPENDENT OF T 11 FAILURES QUT OF 108 TESTS 5 x T INDEPENDENT OF E x 4 FAILURES OUT OF 108 TESTS FAILURES OUT OF 108 TESTS FAILURES OUT OF 108 TESTS FAILURES OUT OF 108 TESTS un Zoos OPTIONS GROUP 60 TO NEXT OPTION CHOSEN IN PROGRAM SETUP F2 PRINT SUMMARY OF AXIOM VIOLATIONS F3 PRINT COMPLETE AXIOM HISTORY o ESC MAIN MENU Figure 27 Axiom Tests Summary Screen level of detail is not necessary for evaluation of adequacy of an additive model as the summary of the axiom tests provides the critical information WARNING This printout is at least six pages long and once F3 is selected there is no way to abort the print Be sure you want the e
81. gt G EEZ ZEZ TEZ EZZ 242 ETZ 12 EET ZET TET EZT IZT ETT ZTT TTT 92 F2 Save Data F3 Print Data F4 Program Setup ESC Escape arrow key will make the change in the computer s memory Y ou must select the save data F2 option at the end of the data entry edit session to make the changes on disk Four special function keys exist to aid in data entry These allow quick movement to the top bottom left and right of the data matrix avoiding excessive scrolling Typical use of these keys would occur during data entry Upon reaching the end of a row of data use the left function key to move back to the beginning of the row However when entering data across rows for 20 or more subjects the use of the right and left keys leave several columns in the center of the data matrix inaccessible In this situation it would be more efficient to enter the data down the columns As you exit from entering data by pressing F1 a data checking algorithm is invoked which will indicate possible errors in the data This algorithm checks the sum of the columns and is able to indicate which columns have probable errors Usually the error can be found by comparing the desig nated column with the same column on the data entry sheet When you select this option the program saves the data currently on the data entry screen to a diskette or hard disk file with either the nume you
82. h the program As mentioned earlier though you are able to obtain a Kendall s Coefficient for a particular group F4 Go to Next This option will display the results of the next analysis previously chosen Option Chosen in either axiom tests or scaling solution If no other options were selected Program Setup control will be taken back to the Main Menu ESC Escape This option returns you to the Main Menu in case you run into problems and need to start over SUMMARY OF AXIOM TEST VIOLATIONS When option F4 of the previous screen Go to Next Option Chosen in Program Setup is selected and you had previously chosen axiom tests as an option in the Program Setup screen the screen shown in Figure 27 will be displayed This screen presents a summary of the violations for the simple independence axiom tests which are the most critical axioms for acceptance of an additive model 45 well as a summary of the violations of double cancellation and joint independence The screen heading will state whether the summary is for a single group a prototype group or an individual subject If the summary is for the entire group of subjects the following options exist F1 Go to Next This option will calculate and display the scaling solution for the group of Option Chosen subjects if scaling was chosen in the Program Setup screen or whichever in Program Setup other option was selected in the Program Setup screen If no other options were selected then the pro
83. he month date and year separate The study name which can be used as a reference to the data set can be up to 20 characters The comment lines can be used for additional information such as subjects names or other identifying comments The second func tion of this screen is to specify the name of the file and number of subjects to be used to later iden tify the data set When you assign the file name on this screen it becomes the file the program uses when the data set is stored to disk Consequently the file name must be eight characters or less The file name should be different from the study name or some variant of the study name to allow for multiple analyses you may do with the same data As always if you want to re zieve a data file for additional work the file name you use must be typed in exactly as before For exam ple you may want to access a previous data file to modify it by adding subjects The number of 87 COMMENTS AND MAIN MENU TODAY S DATE mm dd yy FILE NAME STUDY NAME 8 CHARS MAX 20 CHARACTERS MAX NUMBER OF SUBJECTS COMMENT COMMENT COMMENT USE A SEPARATE DATA DISK FOR EACH STUDY Figure 22 Comments Section of Main Menu Screen subjects will specify the allotment of space in the data entry screen If you desire to work with an existing data file press RETURN for all entries on this screen EXCEPT filename When you enter a file name of an existing file
84. he multiplicative factors has a zero level In this case it would not matter what the levels of the other factors were For an additive model of course this is not the case since a zero level of a factor would make only that factor irrelevant for the combined stimulus effect Since in this and many other applications one would not expect to find a multiplicative factor with this zero level property most theoretical and empirical research in conjoint measurement has focused on the additive model 2 3 2 Axiom Tests The Krantz and Tversky 1971 axioms define five ordinal properties that are useful in differen tiating among the models in equations 2 through 5 In addition all are necessary although not sufficient for the additive model These are single factor independence joint factor independence double cancellation distributive cancellation and dual distributive cancellation It is clear from the results of a recent Monte Carlo study Nygren 1985 that the critical axioms that are used to assess additivity are independence joint independence and double cancellation Hence these axioms are used in the SWAT analysis to determine if an additive model exists in the data These three axioms are summarized below Distributive cancellation which is similar to double cancellation but with four antecedent condi tions was found to be extremely weak as a diagnostic tool in determining additivity In fact it was found that even for random no
85. hers may be needed While these instructions are intended to be given verbally some investigators may wish to have only a written set for their subjects In this case a modification of these appendices may be used in conjunction with Appendix D Appendix D or a modification of Appendix D is a set of instructions which nearly all subjects should read before beginning the card sort These instructions are designed to reemphasize points made in the briefing Appendix B or C and pro vide additional points comments which are necessary Appendix D may be copied and made available to all subjects Appendix E is a master card sort data sheet which may be used to record the subjects card sort data t also should be copied prior to use The procedure for recording data is as follows Take a deck of sorted cards identify which card represents the lowest workload combination typically this is card N Find this letter on the data sheet and assign it a rank of one 1 Take the next card from the deck find its corresponding letter on the data sheet and assign it a rank of two 2 Con tinue in this manner until all 27 letters on the data sheet have been assigned a rank Perform this same operation for each subject s card deck This matrix of numbers will be the input data for the analysis program The procedure for entering data into the program will be described in Appen dix F Data Analysis Procedures 2 4 2 Timing of Sort Cards It is preferable
86. hm this routine starts by finding a set of estimates of the stimulus scale values For efficiency it uses the final estimates found by the STRESS based procedure as these starting values If the data do in fact perfectly conform to an additive model the procedure stops after one iteration since the scale values have already been determined If the data are not perfectly additive as is usually the case then the badness of fit measure THETA is computed by summing the differences in scale values for all pairs of stimuli for which the original ranks are not in the same order as the estimated scale values This sum is then normalized by dividing by the sum of all differences in scale values and taking the square root The numerator of this term and 21 thus THETA will be zero if all pairs of ranks and pairs of estimated scale values are in the same order As in the case of STRESS the partial derivative of THETA actually THETA squared is taken with respect to each scale value in order to find new estimates that will minimize the differ ences in scale values for which there are incorrect pairwise orderings The iterative procedure is then continued until no significant improvement in the estimated scale values i e that will mini mize THETA can be found Following the last iteration as in MONANOVA the scale values are normalized to a range of 0 to 100 This THETA measure is strongly related to Kendall s Tau coefficient although they
87. hree point scale below 1 Often have spare time Interruptions or overlap among activities occur infrequently or not at all 2 Occasionally have spare time Interruptions or overlap among activities occur frequently 3 Almost never have spare time Interruptions or overlap among activities are very frequent or occur all the time 76 Mental Effort Load As described above Time Load refers to the amount of time one has available to perform a task or tasks In contrast Mental Effort Load is an index of the amount of attention or mental effort required by a task regardless of the number of tasks to be performed or any time limitations When Mental Effort Load is low the concentration and attention required by a task is minimal and per formance is nearly automatic As the demand for mental effort increases due to task complexity or the amount of information which must be dealt with in order to perform adequately the degree of concentration and attention required increases High Mental Effort Load demands total attention or concentration due to task complexity or the amount of information that must be processed Mental Effort Load may be rated using the three point scale below 1 Very little conscious mental effort or concentration required Activity is almost automatic requiring little or no attention 2 Moderate conscious mental effort or concentration required Complexity of activity is moder ately high due tu uncertai
88. ile Comments and Main Menu Displaying Previously Saved Information Data Entry Screen for Sample Data Set Printout of Entire Data Set Pres Analysis of All 12 Subjects Indicating the Need for Prototyped olution Summary of Axiom Violations for the Six Time Prototype Subjects Summary of Axiom Violations for the Two Effort Prototype Subjects Summary of Axiom Violations for the Four Stress Prototype Subjects Summary of Axiom Violations for Subject No 8 Individually Summary of Axiom Violations for Subject No 12 Individually Prototype Analysis Screen Indicating Procedure for Excluding Subject No 12 Prototype Analysis Screen Indicating Recalculation of the Kendall s Coefficient Excluding Subject No 12 Prototype Analysis Screen Indicating Recalculation of the Kendall s Coefficient Including Only the Time Prototype Subjects Prototype Analysis Screen Indicating Recalculation of the Kendall s Coefficient for the Stress Prototype Subjects and Subject No 8 Summary of Axiom Violations for the Modified Stress Prototype Group Scaling Information for the Six Time Prototype Subjects Scaling Information for the Five Stress Prototype Subjects SWAT Rating Scales Event Scoring Scale Development Equipment Specification Screen Comments Section of Main Menu Screen Main Menu Screen with Sample Comments 31 31 33 34 35 36 37 38 40 41 42 43 45 46 48 49 67 24 25 26 27 28 29 30 LIST OF FIGURES continued
89. ily accommodated 18 48 49 48 49 86 48 5 OE ON 89 0 0998 84 68 08 9 48 GRD OD At GD an aD Occasionally have spare time Interruptions or overlap among activities occur frequently Very little conscious mental effort or concentration required Activity is almost automatic requiring little or no attention e Moderate stress due to confusion frustration or anxiety noticeably adds to workload Significant compensation is required to maintain adequate performance Os 16 Os PL 4 GA 08 08 08 08 04 46 90 88 16 Ok 30 2304 48 08 Almost never have spare time Interruptions or overlap among activities are very frequent or occur all the time Very little conscious mental effort or concentration required Activity is almost automatic requiring little or no attention Little confusion risk frustration or anxiety exists and can be easily accommodated WP SAO 48 aa EE 16 84 80 8 88 38 SS 49 08 56 08 SSE 28 08 00 OD 06 18 30 88 OG 08 80 13 88 86 An 06 mm SSE HSER 00 m m m Pn m e he Almost ne
90. imensions Some people feel that time has the greatest impact othets feel it is effort and still others feel that stress is the most important 69 Because we are trying to determine what constitutes mental workload for you we would rather not supply you with examples of workload that are represented by each combination We ask that you supply your own examples This is necessary because a situation that is very demanding for one person might be very easy for another Likewise since we all have different backgrounds we may or may not be able to relate to a specific example For exam ple if I give examples that are related to flying an airplane that could be very meaningful to a skilled and experienced pilot while most of you in this room might not have any such experi ence Therefore your impressions of the workload involved in such a task would be depen dent upon impressions provided by other people To avoid this you are asked to read the descriptors from a card and try to think of something you have experienced that this set of descriptors would have accurately described Then take another card and repeat the procedure By comparing the events which vou have recalled determine which of the events had the highest workload for you Repeat this process until you arrive at an order for the 27 cards that begins with the description of the lowest workload event and ends with the description of the highest workload event As you try to i
91. into the experimental apparatus The cue for when the subject is to respond may be provided by the experimenter or automatically presented by the apparatus Since laboratory tasks tend to be abstractions of real world tasks it is important to give subjects enough training in the range of tasks to be used in the experiment so that he or she has some idea of the meaning of the various levels of the task Prac tice in giving ratings should also be given to eliminate interference or erratic ratings due to unfamili arity with the experimental procedures 3 2 2 Operational Evaluations Operational evaluations generally do not have the convenient break characteristic of laboratory trials Operational tasks are generally continuous in many cases the evaluation will last several hours In these cases component tasks or subtasks will most likely be of interest to the investi gator The investigator should identify the subtasks of interest through either a formal or informal task analysis Tasks should be identified in a way that is meaningful to the operators to facilitate their ability in identifying the task that they are trying to evaluate In other words a pilot would have very little problem in relating to the kind of load he was under from the time he passed the middle marker to touchdown while he would have a very difficult time relating what kind of load he was under for 3 minutes after being presented a tone One weakness in selecting the tasks
92. ited amount of time e g 30 minutes Notice that we are not considering anything about how much effort is involved in solving the problems or the stress level involved in this situation B Mental Effort Load 1 Description Mental Effort Load refers to the amount of attention and or concentration required to perform a task Tasks that require Mental Effort Load include storing and recalling things from memory decision making calculations and problem solving High levels of Mental Effort Load are required in situations that demand total concentration whereas lower levels of Mental Effort Load are required when your mind wanders or your attention is dis tributed over more than one easy task component 64 2 Example in Everyday Life Mental Effort Load could involve memorizing items performing calcula tions on numbers concentrating on listening to a speaker for important points or making very difficult decisions In the test situation the prob lems to solve could be very difficult requiring you to remember a formula conversions and complicated solution procedures Or they could be very easy with the solution to the problem being immediately obvious The diffi culty of the problems is not necessarily related to the amount of time pro vided to complete the test C Psychological Stress Load 1 2 Description Psychological Stress Load refers to the presence of confusion frustration and or anxiet
93. k along with the user s guide The program can analyze scale development data for up to 30 subjects The guide has been written as a how to manual for the first phase of SWAT Scale Development but some explanations concerning the second phase Event Scoring are included The report begins with a general overview of workload and proceeds to various topics encompassing the implementation of this particular subjective workload measurement technique These include sec tions on descriptions of the three SWAT dimensions use of conjoint measurement and scaling card sort procedures and analysis methods of prototyping event scoring and data analysis Appendix F is a step by step explanation of how to use the analysis program including system requirements information for getting started and a description of each menu and screen This section and each of the other how to sections have been provided as appendices so that the appro priate part can be separated from the rest of the report and used independently as needed As an aid to users who need more information on mental or subjective workload or studies where SWAT has been used an extensive bibliography has been included LES Accesign For NTIS PREFACE This report was written as part of an ongoing effort by the Workload and Ergonomics Branch of the Harry G Armstrong Aerospace Medical Research Laboratory AAMRL to facilitate the use and ease of impleme
94. k and Psychological Stress Load refers to the degree of confusion anxiety or frustration involved in performing a task As you may recall each dimension has three levels of verbal descriptors low moderate and high The levels of each dimension can be thought of as a three point scale with level 1 being the lowest point or least amount of a dimension and level 3 being the highest point or greavest amount of a dimension Remember that as you give your ratings for a task do so in the order of Time Load then Mental Effort Load and finally Psychological Stress Load For example if you decide on a workload rating of 1 2 3 for a trial which you just completed this indicates that there was enough or extra time to complete the task 1 the task required moderate mental effort 2 and you experienced a high degree of stress 3 DO YOU HAVE ANY QUESTIONS ABOUT SWAT I TIMELOAD 1 Often have spare time Interruptions or overlap among activities occur infrequently or not at all 2 Occasionally have spare time Interruptions or overlap among activities occur frequently 3 Almost never have spare time Interruptions or overlap among activities are very frequent or occur all the time 105 IJ MENTAL EFFORT LOAD 1 Very little conscious mental effort or concentration required Activity is almost automatic requiring little or no attention 2 Moderate conscious mental effort or concentration required Complexity of activity is modera
95. ke every effort to obtain a sort that accurately reflects the subjects opinions regarding the combination of the dimensions The experimenter can then use the subjects same card sorts data for more than one experiment or for several sessions of the same experiment There are of course certain experi mental questions that may require additional sorts from subjects The general rule is meant to apply to studies such as system evaluations 2 4 3 Additional Comments The following is a list of points which commonly arise concerning SWAT They are answers to the most commonly asked questions concerning card sort procedures 1 The card sorts can be obtained in group sessions or individua sessions or some of both as study constraints dictate 2 Astrategy that is suggested to the subject for accomplishing the card sort is to make three stacks of approximately nine cards each One stack represents relatively low workload the middle stack represents moderate workload and the other stack represents relatively high workload Each stack is then ordered 1 to 9 for low to high workload within a stack The stacks are then put together and the order rechecked 3 Subjects should be assured that there is not a correct order to the card sort What is desired however is that the order should represent their opinion as to what constitutes workload to them personally and that it may differ from person to person They are instructed for purposes
96. kload but you would rate the amount or degree of each factor that exists in a given task situation An application of this approach has been developed and is called the Subjective Workload Assessment Technique SWAT This approach has an added feature of obtaining information from the people using the scale about how the identified factors go together to cre ate their perception of mental workload II Describe SWAT 1 Basic Concept The technique describes subjective workload as being composed primarily of three dimensions Time Load Mental Effort Load and Psychological Stress Load 63 Brief Description of the Dimensions It is important that you understand the meaning associated with the three dimensions and how they relate to the definition of workload Let s go into a little more detail about the dimensions Time Load 1 Description Q Time Load refers to the amount of time pressure experienced in performing your task This includes the fraction of total available time that you are busy and the degree to which different aspects of the task overlap or interfere with one another Under high amounts of time load you are unable to com plete the task due to a shortage of time or interference created by the overlap activities Example in Everyday Life In a classroom test situation there could be a high degree of Time Load caused by having a large number of problems e g 100 versus 10 to solve and in a very lim
97. lated night as well as day conditions and under three different kinds of simulated weather Performance will be recorded in terms of the precision with which you hold the assigned altitudes and course track control actions ordnance expended and target scores Jn addition to the performance data we will be collecting data to determine the e amount of workload associated with mission accomplishment 2 Describe Concept of Workload You are probably quite familiar with the concept of mental workload This is a concept that has become increasingly important in modern high technology aircraft When we speak of mental workload we are referring to some sense of mental effort The basic idea is that we have a finite capacity for performing mental work and if we exceed this capacity then we will begin to make a large number of errors or experience total per formance breakdown We can think of situations where very little effort is required thus leaving us with considerable spare capacity for work Likewise we can think of situations that require substantial effort leaving us with little or no spare capacity Exam ples of performing the same task with varying amounts of workload include employing different systems to perform the same task performing the same task with different levels of experience or skill or performing the same task under different levels of fac tors related to the task such as fatigue weather conditions system mulfunctions et
98. like the three factor model illustrated in equation 1 have been and continue to be an important part of many psychological theories Until recently however even for this simple model there has not been a satisfactory means by which one could simuitaneously estimate all four functions f1 f2 f3 and f Conjoint measurement theory provides a means to accomplish this objective Just as important however is the aspect of the theory which indicates that only ordinal relations are required among the data points to produce resultant scales that possess interval proper ties The implications of this result will become more apparent following the presentation of an introduction to the basic theory of conjoint measurement Prior to an introduction to the mathematical foundations of conjoint measurement it is useful to define two terms that are generally distinguished in the literature Emery and Barron 1979 Green and Rao 1971 Green and Srinivasan 1978 First we define conjoint measurement as the pro cedure whereby we specify for a given combination rule the conditions under which there exist measurement scales for the dependent and independent variables such that the order of the joint effects of the independent variables in the data is preserved by the numerical composition rule We then define conjoint analysis sometimes referred to as numerical conjoint measurement as the pro cedure whereby the actual numerical scale values for the joint eff
99. little conscious mental effort or concentration required Activity is almost automatic requiring little or no attention Little confusion risk frustration or anxiety exists and can be easily accommodated 58 Almost never have spare time Interruptions or overlap among activities are very frequent or occur all the time Moderate conscious mental effort or concentration required Complexity of activity is moderately high due to uncertainty unpredictability or unfamiliarity Considerable attention required High to very intense stress due to confusion frustration or anxiety High to extreme determina tion and self control required Almost never have spare time Interruptions or overlap among activities are very frequent or occur all the time Very little conscious mental effort or concentration required Activity is almost automatic requiring little or no attention Moderate stress due to confusion frustration or anxiety noticeably adds to workload Significant compensation is required to maintain adequate performance Occasionally have spare time Interruptions or overlap among activities occur frequently
100. lts compared with the enclosed figures since it is essential to become familiar with the data entry procedures Alterna tively the enclosed file may be retrieved and used for tutorial purposes In either approach the results can be compared with the subsequent figures to assure proper program functioning begin operation type SWATPRGM If this the first time the program has been used the Equipment Specification screen will appear After making the appropriate selections for the system being used the Main Menu will appear For a new data file enter the appropriate information on the Main Menu making sure to enter a file name different than TEST DAT This information will be saved for future reference when the data are saved To use TEST DAT which exists on disk bypass all information on the Main Menu screen by entering all RETURN s except for the file name For this case enter TEST DAT as the file name The program will ask whether you want to work with the existing file to which you should answer yes This sequence is presented o 30 in Figure 1 After responding yes the information which was previously saved will appear on this screen as shown in Figure 2 COMMENTS AND MAIN MENU TODAY S DATE mm dd yy FILE NAME test STUDY NAME 20 CHARACTERS MAX 8 CHARS MAX NUMBER OF SUBJECTS COMMENT COMMENT COMMENT WARNING FILE EXISTS Work with the existing file
101. magine situations which could be described by the combinations on the cards there may be combinations for which you have a hard time imagining a situation that fits In these cases we could provide you with an appropriate situation but it is more beneficial if you assume that such a situation or event does exist and try to determine where it would rank in relation to the other situations Pay attention to the verbal descriptors on the cards to make your judgments as it is important that vou become comfortable with the levels of the dimensions This will help you later when you make your ratings Now read the written instructions which will describe in more detail the dimensions and the sorting task which you will do 70 Appendix C e OUTLINE FOR VERBAL CARD SORT INSTRUCTIONS PILOT POPULATION I Introduction 1 Describe Purpose of This Experiment The experiment you are about to participate in is to evaluate the new XYZ all weather terrain following radar It is very important for us to get the best data that we possibly can if we are to get this system delivered in a configuration that will require few modifi cations once it becomes operational Considerable time and resources have already been expended to get these systems to this point In this test all of you will fly simula tions over three different courses using both candidate XYZ systems as well as one sortie that you will fly manually These flights will be flown under simu
102. nadditive data the property will be satisfied on the average about 75 percent of the time Dual distributive cancellation is a very complex property requiring five antecedent conditions from a 5 x 5 x 5 design to be met in order for the test to even be 15 possible It like distributive cancellation is not able to reject additivity Therefore these two axioms are not performed in this application of conjoint measurement For a more detailed e description and explanation of these properties the interested reader is referred to Nygren 1982 1985 2 3 2 1 Independence We begin with the fundamental property of independence which can be checked separately for each of the three factors In general we say that A is independent of B and C whenever 41 61 1 gt a2 b1 c1 if and only if a1 b2 c2 gt 2 52 2 6 where A B and C represent the three dimensions and a1 a2 and a3 represent three levels within the first dimension Similarly b1 b2 and b3 represent the three levels of the second dimension likewise c1 c2 and c3 for the third dimension Thus independence of A asserts that if a2 gt al for some combination of levels of factors B and C then this relation will hold for any other com bination of levels of B and C For SWAT this axiom can be translated as stating that the ordering e of any two levels of the time load dimension for example will remain consistent for all combina tions of the other two
103. ncentration are very similar to the first situation However the Psycho logical Stress Load rating may increase to a 3 since there could be increased anxiety about the landing due to the obscured visibility Note that these ratings are intended to illustrate what a pilot might say and that for any given task any other combination of ratings is possible depending upon the precise task conditions and pilot related factors such as amount of training 66 5 25 TJ VAS 781 10B1NOO 313S NOILIVNINH3130 01 HOIH ALZIXNY HC 01 O1 553915 3SN31NI AH3A HDIH 30NYAWEtOdJH3d NIVINIYW 1 Si LINYOLIJINDIS Oi SGGW AL3IXNY 1501 4 NOISNSNOD 01 553615 JiVH3GOW 72 Ga1VOOWMOOOV NYO AI3IXNY HO 51 NOISNANOD 31117 1 QvO1 SS3H1S TVOIDOIOHDAS 11 19101 ALIAILOV AH3A AHYSS3O3N NOLIVEIN3ONOO 160333 vIN3N 3AISNZIX3 DuviWOLnv LISONIY 51 ALIAILOV G3HINOSY 1 iVINZAW 50012550 QVO1 130353 1 1 y YNDDO HO iN3nO3H4 AH3A 83 LIALLOY ONOWY dV1H3AO HO 3HYdS BAYH 1S0 1v E YND
104. nces a subject s card sort may contain an unacceptable number of axiom test violations as described in Section 2 3 2 and to be described in Section 2 5 3 In the event of this occurrence there are two alternatives First if the subject s data exhibit an identi fiable pattern such as Mental Effort Load most important Time Load second most impor tant and Psychological Stress least important but an unacceptable number of axiom test violations it is possible that when this subject s data are averaged with others the group errors will be acceptable due to the strength of multiple subjects However if this is not the case another sort may be needed It is important that the subject not feel that his her sort was wrong but that since this is a communication process there was a possible break down in the exchange cf information emphasize that you may not have explained the task very well Then reiterate the main points and try to elicit questions from the subject before starting the new card sort It is advisable that the additional sorts be performed one on one rather than in groups 26 If the errors are not too numerous or if another sort is not possible due to study constraints it may be possible to reach an acceptable solution through the use of paired comparisons In this approach subjects are presented with pairs of cards which violated the predominant rule in their sort and asked to compare each pair independently by stati
105. nd objective workload correlates in Proceedings of the IEEE National Aerospace and Electronics Conference Dayton Ohio IEEE Arbak C J Shew R L and Simons J C 1984 The use of reflective SWAT for workload assessment in Proceedings of the Human Factors Society 28th Annual Meeting pp 959 963 Santa Monica California Human Factors Society Bates e Jr and Vikmanis M 1985 Some quantitative for cockpit design in CP 387 pp 9 1 9 10 Beare and Dorris E 1984 The effects of supervisor experience and the presence of shift technical advisor on the performance of two man crews in a nuclear power plant simulator in Proceedings of the Human Factors Society 28th Annual Meeting pp 242 246 Santa Monica California Human Factors Society Beville Engineering Inc 1986 Human factors analysis of refinery consolidation Boyd S P The use of conjoint analysis for interval subjective scaling of mental workload Unpublished Masters Thesis Virginia Polytechnic Institute Blackburg Virginia 107 Boyd 5 P 1983 Assessing the validity of SWAT as a workload measurement instrument in Proceedings of the Human Factors Society 27th Annual Meeting 124 128 Santa Monica California Human Factors Society Courtright J F and Kuperman G 1984 Use of Swat in USAF system T amp E in Proceedings of the Human Factors Society 28th Annual Meeting pp 700 703 Santa Monica California
106. ned by the program In this way you can always get back to a place where you know your way out and avoid losing data EQUIPMENT SPECIFICATION af this is the first time the program is used an equipment specification screen as shown in Fig ure 21 will be displayed The three options are 1 Two floppy disk drives and B 2 One floppy drive and one hard drive A and E respectively 3 One floppy drive and one hard drive A and C respectively Make the selection which corresponds to the system being used You will then be asked whether a printer is attached to the computer Once this information has been entered subsequent uses of the 86 4 EQUIPMENT SPECIFICATION 9 OPTIONS 1 2 FLOPPY DISK DRIVES A and B mource in drive A and data in drive B 2 1 FLOPPY DISK DRIVE A 1 HARD DISK DRIVE E in drive E and data in drive A 3 1 FLOPPY DISK DRIVE A 1 HARD DISK DRIVE gource in drive C data in drive A 4 EXIT WITH NO CHANGES MAKE A SELECTION Figure 21 Equipment Specification Screen program will not invoke this menu Changes in the equipment specifications can be made how ever using option 4 of the Main Menu MAIN MENU As can be seen in Figure 22 this screen has three main functions The first is to allow entering of date study name and additional comments for labeling a data set When entering the date include slashes as displayed to keep t
107. ng this question Eggemeier Crabtree and LaPointe 1983 found that SWAT ratings delayed 15 minutes after completing a short term memory task did not differ significantly from immediate ratings Eggerieier Melville and Crabtree 1984 investigated the effect of an intervening task on delayed SWAT ratings in a short term memory task Their results indicate that while there was not a significant effect of a 14 minute delay or intervening task on SWAT ratings there was a tendency for ratings to be affected by performing a very difficult intervening task Notestine 1984 found that a delay of 30 minutes did not significantly affect SWAT ratings in a probability monitoring task In these last two experiments liowever subjects tended to report lower SWAT ratings after a delay than imme diately after task completion In many situations such as some flight tests it may be necessary to obtain all of the data after the entire task or mission has been completed In these debriefing ses sions questionnaires are frequently used to obtain operator opinion data If the ratings MUST be obtained post hoc the best prompts available should be used to help the operator recall what he she was experiencing at the moment of interest Arbak Shew and Simons 1984 In some cases the operators have been shown videotapes of the events and asked to give the ratings in this prompted post hoc fashion This approach is probably the best of these prompts If the investigat
108. ng which of the two is higher in workload Typically these pairs are indicated by violations of the indepen dence axiom complete listing of the violations can be obtained from the program see Appendix F for more information In this manner it is possible to assess whether the violations were simply inadvertent or the subject actually was giving contradictory infor mation If the subject follows a certain pattern he she may be placed in the appropriate prototype group If not additional instructions and possibly an entire new sort may be necessary 2 5 CARD SORT ANALYSIS 2 5 1 Introduction The card sort analysis is performed to accomplish two objectives First of all the conjoint measure ment algorithm performs the axiom tests to assess the validity of an additive model for the data Secondly the scaling algorithms produce interval level rescaled values for each of the levels of the three dimensions This is accomplished by using the microcomputer based software The next two sections describe information concerning prototyping and a sample data analysis section for instructional purposes Appendix F contains detailed instructions for program operation and is designed to be a reference guide for using the program 2 5 2 Prototyping One of the unique aspects of SWAT is what is known as prototyping Prototyping refers to a procedure of stratifying the subjects into homogeneous groups based on their perceptions of the relative import
109. ns on the Program Setup screen the program will display as shown in Figure 28 the following information on the Scaling Information screen 1 The last five iterations of the scaling algorithm For a more detailed explanation of these values refer to Section 2 5 3 Sample Data Analysis 2 The rescaled values for each level of the subscales These are the additive values which through all possible combinations form the 27 values of the scaling solution 3 The approximate relative importance of each factor This indicates the amount of change from level of a dimension to level 3 of the same dimension This information will be from the program selected algorithm Refer to Section 2 3 5 for more information on the selection of the appropriate scaling algorithm The following options exist on this screen Plot of Rescaled This plot which is depicted in Figure 29 gives an indication of the Versus Raw Data goodness of fit of the rescaled values The appropriate plot is linearly decreasing from left to right und data points not lying on the line indicute cards which were displaced from the pattern While viewing the plot you may either print it out or return to this screen 100 UAF gz 25 1135 NI N3SOHO 140 OL 09 GJ 0111105 9HI TJVOS MAIA 54 SNOIIVH3II TIV 1814 E ONIIVIS 1 23 SA G3TV2SS8 4
110. ntation of SWAT in both laboratory and operational settings lt was per formed under Work Unit 7184 14 07 The effort is supported by Systems Rescarch Laboratories Inc SRL Dayton Ohio under Contract Number F33615 85 C 0541 Mr Robert Linhart is the contract monitor We wish to acknowledge Dr Thomas E Nygren of The Ohio State University who created the original analysis software for Version 3 1 and Brian Porter of SRL for his time effort and patience in producing the user interface for this version The disk which this report refers to is Version 3 1 of SWAT and contains four files They are 1 SWATPRGM BAT 2 MAIN EXE 3 SWAT DAT and 4 TEST DAT To run the program type SWATPRGM and RETURN The first three files which comprise the executable program must be kept together while the fourth file is a sample data set included for instructional and testing purposes Since the program needs to write a file to the program disk it is not write protected it is advisable to make a backup copy of the disk You are asked not to distribute the program disk We would like to maintain an accurate list of recipients to be able to supply program updates or advise users of problems should they be dis covered Potential users can obtain the program by writing to Mr Gary B Reid AAMRL HEG Wright Patterson Air Force Base Ohio 45433 6573 TABLE OF CONTENTS INTRODUCTION SCALE DEVELOPMENT
111. ntire history printed before selecting F3 ESC Escape As before this retains the entered data and returns you to the Main Menu in case of problems or for a fast way to exit the program If the Axiom Test Summary is for a prototype group the additional option Go To Next Proto type exists which will display summary information of axiom violations for the next prototype group Also selection of either of the print options will print axiom test information for all three e prototype groups If prototype scale was selected on the Program Setup screen sclection of the 99 Go to Next Option Chosen in Program Setup option in this screen will calculate and display scaling information for the first prototype group If the summary is for an individual subject tlie additional option Go To Next Individual exists which will display this summary information for the next individual selected Also either of the print options will print information for each subject chosen Therefore if numerous subjects have been included printing the complete axiom history will result in quite a large amount of informa tion being printed All other options will operate as previously described SCALING INFORMATION In all three cases when Go to Next Option Chosen in Program Setup is selected on the Sum mary of Axiom Violations screen and a scaling solution option had been chosen on the Program Setup screen or upon selection of any of the Scaling optio
112. nty unpredictability or unfamiliarity Considerable attention required 3 Extensive mental effort and concentration are necessary Very complex activity requiring total attention Psychological Stress Load Psychological Stress Load refers to the contribution to total workload of any conditions that pro duce anxiety frustration or confusion while performing a task or tasks At low levels of stress one feels relatively relaxed As stress increases confusion anxiety or frustration increases and greater concentration and determination are required to maintain control of the situation Psychological Stress Load may be rated on the three point scale below 1 Little confusion risk frustration or anxiety exists and can be easily accommodated 2 Moderate stress due to confusion frustration or anxiety noticeably adds to workload Significant compensation is required to maintain adequate p 4 formance 79 3 High to very intense stress due to confusion frustration or anxiety High to extreme deter mination and self control required Each of the three dimensions just described contribute to workload during performance of a task or group of tasks Note that although all three factors may be correlated they need not For example one can have many tasks to perform in the time available high Time Load but the tasks may require little concentration low Mental Effort Load Likewise one can be anxious and frus trated hi
113. oderate conscious mental effort or concentration required Complexity of activity is moderately high due to uncertainty unpredictability or unfamiliarity Considerable attention required Little confusion risk frustration or anxiety exists and can be easily accommodated Occasionally have spare time Interruptions or overlap among activities occur frequently Very little conscious mental effort or concentration required Activity is almost automatic requiring little or no attention High to very intense stress due to confusion frustration or anxiety High to extreme determina tion and self control required 18 58 98 58 24 86 88 80 00 1 88 20 58 OS 00 08 48 hn 05 08 Occasionally have spare time Interruptions or overlap among activities occur frequently Moderate conscious mental effort or concentration required Complexity of activity is moderately high due to uncertainty unpredictability or unfamiliarity Considerable attention required High to very intense stress due to confusion frustration or anxiety High to extreme determination and self control required Appendix B OUTLINE FOR VERBAL CARD SORT INSTR
114. of interest What is generally known about the fallibility of the human mem ory would seem to suggest that ratings should be taken as close to the event being ruted as possi ble If for example the investigator desires to obtain data on the aircraft approach task he might want to obtain ratings as the aircraft passes the outer marker the middle marker and at touch down However operational constraints or safety of flight considerations might preclude obtaining ratings at each of these points and may dictate that this recommendation be altered One way that might be considered in attempting to get data in these kinds of demanding situations is to give the operator the option of not responding The operator could be instructed to omit the response any time that workload is so high that to attempt a response would interfere with task performance Then for every task segment that the operator failed to rate the investigator would enter a 3 3 3 or 100 in the data matrix since the operator was at the defined maximum workload level An alternative approach to this problem would be to delay data collection until a conveniciit break In the example the three ratings for the landing might all be taken after the airplane has finished the landing roll and the pilot has taxied off the active runway This approach suggests the question how long can the investigator wait to obtain the ratings 52 Several investigations have been directed toward answeri
115. or remem bers to obtain the baseline tasks in the same manner then whatever effect on the ratings that exist that might be attributable to the post hoc rating approach is a constant across all tasks and thus will not affect the relative standing However the investigator should remember how the data were obtained when interpreting the data 3 3 DATA ANALYSIS After the event scoring phase has been completed and all ratings have been obtained rescaled values for each of the ratings must be assigned These rescaled values were generated by the scale development phase as described earlier When assigning rescaled values it is imperative that the appropriate scale be used for each individual If a single group solution was deemed appropriate it may be used for all of the subjects However if subgroup or individual solutions were necessary the proper scaling solution must be used for each subject e g if subject No 1 is a time prototype subject and ptototype group solu tions are being used then use the scaling solution for the time group in assigning subject No 1 s rescaled values As an example of this procedure suppose that a given task contained ten events for which SWAT ratings were obtained For simplicity only a single subject will be described the first event the subject was under moderate time pressure had a moderate amount of memory demands and 53 calculations and very little psychological stress load Thi
116. out this manual references to mental workload will be shortened to workload This is done solely for convenience and should not contribute to over looking other important components of workload This manual is organized around an explanation of the two phases of SWAT the Scale Develop ment phase and the Event Scoring phase The how to portions have been placed in appendices 9 to facilitate easy access after you have become reasonably familiar with the SWAT analysis pro cess You may wish to separate certain appendices from the main document so they may be used e as a master for copies or as a quick reference guide for running the SWAT software 10 Section 2 SCALE DEVELOPMENT 2 1 INTRODUCTION The Scale Development phase is the principal aspect which differentiates S from other subjec tive workload approaches Usually descriptors are provided in order to define some number of workload levels seven for example and subjects are carefully trained to know what is repre sented by each level of the scale In SWAT descriptors of components of workload are provided but the task of the subject is not to learn what the various levels mean but rather to make judgments that allow the investigator to determine how the factors combine for the particular subjects involved in the investigation The first requirement in the development of this scaling approach is to establish an operational defi nition of mental wo
117. present the card sort data The definition of best fitting is what differentiates the two procedures as will be described below Being nonmetric scaling pro cedures both algorithms attempt to determine the best fitting set of interval scaled values for the levels of the dimensions and their combined effect based only on rank ordering of the combinations of the dimensions Therefore the SWAT procedure begins by rank ordering the data from the smallest to the largest if they are not already in that form From this point on only this rank order of the data and not the data values themselves are used in the analysis 2 3 5 1 MONANOVA e The first approach MONANOVA finds and applies a monotonic transformation to the original card sort data such that a set of distances can be estimated for the combinations of levels with the constraint that the distances fit the transformation in a least squares analysis To begin an arbitrary set of initial scale values for the levels of the factors is formed to produce initial estimates of the 27 stimulus combinations From these initial scale values a matrix of what is called disparities is formed Disparities are transformed values that are monotonic with the original data and as close as possible to the initial set of workload scale values Next a badness of fit measure STRESS is computed to determine how closely the monotonically transformed disparity values match the esti mated scale values from the addi
118. previously specified on the Main Menu screen or a new file which you able to specify Specifying a new name will create a new file while not changing the original file Be sure to always save the data set to prevent accidental loss of data This allows you to obtain a hard copy of the input data set for easy refer ence Since the entire data table when filled cannot fit on the screen at one time this may be a more desirable way to proof a data set This option directs the program to proceed to the Program Setup screen in order to continue the normal analysis process Use this option after all data has been entered and checked for accuracy This option takes you back to the Main Menu and easily allows you to leave the program help you get reoriented if you were lost or confused or make changes in earlier screens REMEMBER though that data are saved to disk only through the Save Data option on this screen e PROGRAM SETUP This screen is presented in Figure 25 and gives you options for the types of analyses to be per formed Enter the number s of the analyses which you want performed There are three main analyses 1 Prototype correlations and Kendall s Coefficient of Concordance 2 Axiom Testing 3 Scaling Solution eee eee PROGRAM SETUP eee eee TO RUN ANY OF THESE PROGRAMS OR COMBINATIONS OF PROGRAMS CHOOSE THE CORRESPONDING NUMBER S AND PRESS RETURN 1 PRhTOTYPE CORRELATIONS AND KENDALL S e 2 GROUP AXIOMS
119. r 9 nN3H 253 lt 5 NVN9OHd VIVA LNIHd VIVA HZJLNJ LIQGJ viva 3AYS 3 d Td Qo 21 2079 Qo I Qo Qo Zt 20 IT aT 0036 0078 00 7 0079 007 BO 00 I g 092125 nug OO LT 00 9 00 T 5 DOET 90 20 9 eo zi 1 00 12 00 6 0 02 09 89 90361 90 0 00 79 D IT S 0 0 207 20 90 00 6 00 1 L 9 SIHL yc omar 00 Z eo v 702 08 9 007 0076 OO t 20 0076 00 79 0 R 30 E 2 90 99 9 61 0 009791 20 OD II 00 909 0078 008 02072 09075 T Q T s 5193 5 oo 0072 0079 eo TZ 20 20 r oo Oz 9076 90 Sc eT 9 00 T eo 2971 NI 193 8 5 Qo FZ Qo 9 2077 ee 00 227 2017 OO TT oo 00 Qo PI 20976 oz 9o 21 202 Qe 99 OT ge T c en Qo EZ oe 22 20772 Oz go oo 2 9 Qo c 00 rT 99 00791 9o eo 007 20 2972 1 N m x L gt x n 5 N
120. rd and that letter has a specific workload com bination assigned to it For example card N has a 1 1 1 workload combination and card K has a 2 3 1 combination The 1 1 1 card is composed of the lowest level of Time Load Mental Effort Load and Psychological Stress Load while the 2 3 1 card is made up of the middle level of Time Load the highest level of Mental Effort Load and the lowest level of Psychological Stress Load dimension Examples are Often have spare time Interruptions or overlap among activities occur infrequently or not at all e CARD Very little conscious mental effort or concentration required Activity is N almost automatic requiring little or no attention Little confusion risk frustration or anxiety exists and can be easil y accommodated Occasionally have spare time Interruptions or overlap among activities oceur frequently CARD Extensive mental effort and concentration are necessary Very complex K activity requiring total attention Little confusion risk frustration or anxiety exists and cun be easily accommodated e 18 NOTE It is important to keep each letter with the combinations as they appear in Appendix A The SWAT analysis program is designed to accept a specific order of those combinations when data are input Changing the letters will affect the input order and therefore the outcome of the SWAT analysis 2 3 4 Axiom Testing Programs As discussed earlier in most application
121. rd sort as being meaning ful information Therefore inconsistencies will cause the scaling to incorporate ties in the levels As a result this is the preferred algorithm when a subject s conceptualization of workload is defined by a model with fewer than three levels of each of the dimensions For example a subject may feel t t Time Load does not really make any difference in how high his workload is up to the point of not being able to keep up with the task From that point on this subject might think that Time Load is an extremely important factor In this case the subject would probably treat the level 1 and level 2 descriptors as equivalent while level 3 would be distinctly different In other words the subject would have only two levels of Time Load and perhaps three levels of Mental Effort Load and three levels of Psychological Stress Load Therefore the SWAT program would auto matically select and present the MONANOVA scaling solution as the best fitting solution 2 3 5 2 Johnson s Algorithm The second scaling procedure in SWAT NONMETRG is used to provide another scaling of the data this time based on a badness of fit measure different from STRESS This measure THETA differs from STRESS in that it is based on a pairwise method in which the differences in scale values for all possible pairs of stimuli 351 pairs for the 27 stimuli in SWAT are compared with the differences in the original ranks As in the previous scaling algorit
122. ree dimensions and their levels Consequently only slight additional training will be required see Section 3 for more information on Event Scoring Secondly the card sort provides motivation for subjects to take the rating scale seriously One problem inherent in traditional subjective measures is that of gaining subject acceptance of the 23 rating scale being used If subjects reject the technique or take the rating task too lightly then the chances of obtaining accurate ratings are greatly reduced Our experience has indicated that per e forming the card sort provides subjects with a feeling of greater involvement and thus facilitates seriousness and greater reliability of ratings Appendices B C and D provide information pertinent to the card sort procedure Appendix B provides an outline which is to serve as a guide for verbal instructions for a typical sample of college students Similarly Appendix C is an outline of instructions geared toward a pilot popu lation You will notice that these two sets of instructions have some similarities and yet they are not identical This illustrates the point that you should tailor the instructions to fit the specific application and subject population These samples are included to indicate some of the points which should be included in a briefing but not necessarily read verbatim to the subjects In writing a set of instructions points and approaches from both of these sets as well as ot
123. rkload While researchers have not arrived at a consensus about a technical definition of workload there is considerable agreement that mental workload is a combination of several factors related to task demands operator state and time factors Therefore workload has been defined for SWAT to be composed primarily of Time Load Mental Effort Load and Psycho logical Stress Load Reid and Nygren 1988 Time Load refers to the total amount of time availa ble to an operator to accomplish a task as weil as overlap of tasks or parts of tasks Mental Effort Load is the amount of attention or concentration that is required to perform a task and Psychologi cal Stress Load is the presence of confusion frustration and or anxiety associated with task per formance This definition is not intended to represent a sufficient technical definition of mental workload rather it provides a useful operational definition It does appear to reflect most of what a majority of people are talking about when they refer to mental workload NM DESCRIPTION OF THE DIMENSIONS The three factors or dimensions used to operationally define workload have each been further defined by a set of descriptors that specify three levels of each of the dimensions These dimen sions are based largely on the theoretical work of Sheridan and Simpson 1979 in defining pilot workload We have attempted to generalize the wording of the descriptors in order to create a scale that is
124. s not imply that joint indepen dence will be satisfied for all pairs of factors We can of course state two other forms of the joint independence property for and C of B and B and C of A For SWAT this asserts that the ordering of any combination of two dimensions Time Load and Mental Effort Load for example will hold for all levels of the third dimension Psychological Stress Load 2 3 2 3 Double Cancellation The third property examined by Krantz and Tversky 1971 is usually referred to as double cancel lation stated for factors A and B as If a2 b3 c1 gt 1 52 1 and a3 b2 1 gt a2 b1 c1 then a3 b3 c1 gt al bl cl 8 Note that double cancellation requires at least three levels of each of the factors A and B and deals with only two such factors at a time Hence it must be satisfied for all pairs of factors If factors A and B each have three levels then there will be one possible test of double cancellation for these two factors For the purposes of SWAT this axiom tests the consistency in the row column and diagonal relations for two factors at a time holding the third factor constant Therefore there are three possible tests of double cancellation for each factor Note that in this test there are two antece dent conditions both of which must be met for the test to be performed 2 3 3 Scale Development Data Collection Note that these properties are stated in terms of order relations and only these
125. s of conjoint measurement methodology it is the additive representation that is of interest However even for an additive model as small as the 3x 3x 3 design used in the SWAT methodology both the testing procedures for the properties mentioned above and the actual scaling procedure for obtaining the numerical scale values become extremely impractical without the aid of a computer based algorithm Indeed the actual counting of the num ber of tests the number of successful orderings and the number of violations of the three axioms described above can become computationally overwhelming very quickly The SWAT program was designed to meet these computational axiom testing needs practically and efficiently There are several programs which have been designed to do the testing of the preceding axioms The SWAT program is a combination of several of these programs One attempt to develop a general diagnostic program for testing the conjoint measurement axioms was made by Holt and Wallsten 1974 Their program CONJOINT was designed to test each of the axioms mentioned above CONJOINT was written in PL 1 and has been modified to run on an IBM 370 or Amdahl 470 operating system Ullrich and Cummins 1973 developed two other pro grams PCJM and PCJM2 written in FORTRAN to do essentially the same thing as CONJOINT There are however several differences between the programs which make both useful as diagnos tic tools For a more detailed description of
126. s subject most likely would give the three dimensions ratings of 2 2 and 1 respectively During scale development this combination of levels could have been assigned a scale value of for example 14 1 based on position of the 2 2 1 combination relative to all of the other 26 combinations represented in the card deck Now that this combination has been selected as being descriptive of the experienced event the 14 1 is entered into the investigator s data matrix as the workload scale value associated with performance of that event by that subject under that set of conditions Table 3 graphically demonstrates how rescaled values are assigned to ratings Based on this example event number one for this subject would be assigned a value of 14 1 Similarly event number two would be assigned a rescaled value of 57 9 event number three would receive a rescaled value of 0 and so on until all ratings have been assigned rescaled values From this point on data analysis is situation specific and will conform to the type of experiment being conducted TABLE 3 EXAMPLE OF HOW TO ASSIGN A RESCALED VALUE TO A SUBJECT S GIVEN RATING _ SWAT Scale _ U Card Combination Rescaled Value 1 2 2 1 111 0 0 2 2 1 3 112 24 4 3 1 1 1 113 51 4 4 3 1 2 121 7 6 5 1 3 3 122 32 0 6 2 3 1 123 59 0 7 2 1 2 131 27 7 8 1 2 2 132 52 1 9 3 3 3 133 79 1 10 3 2 1 211 6 5 212 30 9 213 57 9 221 14 1 222 38 5 223 65 5 231 34 2 233 85 6
127. scriminability in Proceedings of the Human Factors Society 30th Annual Meeting p 63 Santa Monica California Human Factors Society 108 Eggemeier F T Crabtree M S and LaPointe P 1983 The effect of delayed report on subjective ratings of mental workload in Proceedings of the Human Factors Society 27th Annual Meeting pp 139 143 Santa Monica California Human Factors Society Eggemeier T Crabtree 5 Zingg J J Reid B and Shingledecker 1982 Subjective workload assessment in a memory update task in Proceedings of the Human Factors Society 26th Annual Meeting pp 643 647 Santa Monica California Human Factors Society Eggemeier T McGhee 7 Z and Reid B 1983 The effects of variations in task loading on subjective workload rating scales in Proceedings of the IEEE 1983 National Aerospace and Electronics Conference pp 1099 1105 Dayton Ohio IEEE Eggemeier F T Melville B E and Crabtree M S 1984 The effect of intervening task performance on subjective workload ratings in Proceedings of the Human Factors Society 28th Annual Meeting pp 954 958 Santa Monica California Human Factors Society Eggemeier F T and Stadler M A 1984 Subjective workload assessment in a spatial memory task in Proceedings of the Human Factors Society 28th Annual Meeting 680 684 Santa Monica California Human Factors Society Eggleston R G
128. scriptor combinations in the card sort procedure To determine which method is appropriate for developing the scaling solution the data must be analyzed using the SWAT program If the Kendall s Coefficient is 75 or higher then the scale from that group run may be used with all subjects If the Kendall s Coefficient is below 75 the subjects should be prototyped as either time effort or stress The appropriate options must be selected on the Program Setup screen in order to produce the required scaling solutions This pro cedure will be described with an example in the next section 2 5 3 Sample Data Analysis This section will describe the analysis for the sample data set which is included on the SWAT disk The name of this file is TEST DAT To use this file it should be copied over to disk drive B before beginning operation The program disk should be in disk drive A or copied over to a hard disk This sample data file includes data for 12 subjects that can be used to test the program and as e a tutorial on program Throughout this section emphasis will be on interpreting the analysis Refer to Appendix F for procedural guidance if necessary Appendix F provides detailed information on each of the menus and screens encountered in an analysis For the novice user these two sections should probably be used in tandem It is suggested that these data be entered in another data file for practice in using the program and the resu
129. se the numbers associated with the levels or in some other way attempt to mechanize the ordering The ordering information is very important in helping to define the scale but equally important to us is the training value associated with carefully considering the relationships of the meaning of the levels for each of the three dimensions Several points need to be made at this stage Remember that there is not a correct answer You are making judgments about conditions in terms of the degree of workload associated with an event like the one under consideration This is a communication process that we use for you to try to explain to us the way you view workload in terms that allow us to put numbers on your judgments In this process it may become necessary for us to later come back to you to confirm our opinion of what you are trying to tell us or resolve some area that is not clear to us If this does become necessary do not change your judgments because you think you got the wrong answer There is not right and wrong answer Just try to be consistent in giving your judgment about events Because people do differ it is best that you not compare notes with anyone Do not dis cuss things in an attempt to come up with a consensus Asyou do the card sort try to think of an experience that you have had that each card set of descriptors would describe Then put the cards in order by deciding which of the experi ences had
130. second highest correlation is with ETS Stress is clearly the least important of the three dimensions but time is not clearly the most impor tant In this situation the subject would be considered to belong to a time effort prototype group For this occurrence the experimenter may wish to create a fourth prototype group and generate a separate scale for this group or the experimenter may choose to place the T E subject into which ever one of the three main groups seems to best fit the data These choices ure subjective and the value of the correlation coefficients and the number of T E subjects should both be considered For simplicity the remaining discussion will only consider the three main prototypes At this point it is important to note the three methods for handling the data and creating the final SWAT scale 1 Group Sealing Solution The data from all subjects will be averaged together and the con joint scaling algorithm will derive the scale from this average 2 Prototyped Scaling Solution Subjects are prototyped as either time effort or stress sub jects and each of these three homogeneous groups then has its own SWAT scale The investigator may override the automatic selection of prototypes and create as many proto type groups or reassign a subject to a prototype as is considered necessary 3 Individual Scaling Solution Each subject s data are analyzed separately and a SWAT scale is derived for each subject individually
131. stigate the differential sensitivity of the three subscales Eggemeier McGhee and Reid 1983 Potter and Acton 1985 Both of these studies indicated potential advantages to analyzing the subscales as well as the overall combined values 2 4 CARD SORT PROCEDURE 2 4 1 Introduction The principal reason for completing the card sort is to generate data that are used to produce a scaling solution tailored to the group s or individual s perception of workload This is one aspect of SWAT which is different from most other subjective workload assessment approaches The results of the card sort are analyzed by the conjoint scaling program to produce an interval level workload scale Since the results from card sorts are used to generate the workload scale the card sort session is the key to a successful application of SWAT The subject must be convinced of the importance of providing the best possible information regarding how he or she percvives trades off the three dimensions which we have defined as being the primary contributors to workload Inaccurate or invalid card sort information can have a considerable effect on the results of the experiment Aside from the scale generation there are several other very important aspects of the card sort which need to be emphasized Primarily the card sort procedure serves as training for the subse quent Event Scoring phase of SWAT After sorting the cards subjects ure very familiar with the use of the th
132. tely high due to uncertainty unpredictability or unfamiliarity Considerable atten tion required 3 Extensive mental effort and concentration are necessary Very complex activity requiring total attention III PSYCHOLOGICAL STRESS LOAD 1 Little confusion risk frustration or anxiety exists and can be easily accommodated 2 Moderate stress due to confusion frustration or anxiety noticeably adds to workload Significant compensation is required to maintain adequate performance 3 High to very intense stress due to confusion frustration or anxiety High to extreme deter mination and self control required 106 BIBLIOGRAPHY Acton W H and Colle H A 1984 The effect of task type and stimulus pacing rate on subjec tive mental workload ratings in Proceedings of the 1984 National Aerospace and Electronics Conference pp 818 823 Dayton Ohio IEEE Acton W and Crabtree M S 1985 Work assessment techniques in system redesign in Proceedings of the IEEE National Aerospace wlectronics Conference Dayton Ohio IEEE Albery W B Ward S L and Gill T 1985 The effect of acceleration stress on human workload Technical Report AAMRL TR 85 039 Wright Patterson Air Force Base Ohio Arm strong Aerospace Medical Research Laboratory Albery W B Repperger D W Reid B Goodyear C Ramirez L and Roe M 1987 Effect of noise on a dual task Subjective a
133. that one task has lower workload associated with it than another In fact the first task could be so low in demand that in your judgment it would be boring Someone who has a low tolerance for boredom might be tempted to think I know this task is low in workload but I really do not like to be bored so I will rank it higher than this other task Remember it is not your prefer ence we are asking for but the amount of workioad you think exists in a situation You may use whatever strategy that seems best to you to do the card sort The procedure of dividing the 27 card deck into three smaller decks of low medium and high and ordering these smaller decks and then recombining into the entire 27 is a strategy that has proven use ful to many people However this strategy is not mandatory This is not an easy task It will probably take you from 30 minutes to an hour to finish and some of the discriminations are going to be difficult Please concentrate and give us the best sort possible Even though this is a laborious process we think that it pays off in the long run When you get to the simulator the rating task will be easier because of the effort you are putting in now Are there any questions Then go ahead and start If you have any questions at any time please ask them The thing that is bothering you may also be bothering someone else If you are not certain about how to do the ordering it will affect your data so please ask if you have any
134. the needed data This may include some standard events as well as the events where task overload is suspected It should be remembered that at least at this time the measurement of workload can be attained only in a relative sense In other words if we take measurements we can say that Task A has a workload of 88 while Task B has a workload of 37 for instance Con sequently we are able to state which task has the highest workload and with certain measurement tools SWAT for example we can say how much more workload one task has than the other This relative nature of the current measurement tools requires the use of a comparison or baseline task For example if the investigator is concerned that the use of a particular automatic landing sys tem has an excessive workload associated with its use then data might also be collected employing system that is already in service It is assumed that the new system would have increased capa bilities associated with it but the increased capability could also place increased demands upon the operator Data that are descriptive of the current system would give system designers and evalua tors the information necessary to trade off cost against increased capability Clearly the investiga tor must be careful in these kinds of evaluations to equate training to the maximum degree possible Another aspect of attempting to remain unintrusive pertains to when the ratings must be obtained relative to the segment
135. these programs refer to Nygren 1982 SWAT is a combination of what is believed to be the most useful parts of these programs First SWAT provides some of the same information as CONJOINT for testing conjoint measurement axioms However SWAT also provides a more detailed analysis of violations of these axioms especially for the critical axioms of simple independence and joint independence In addition SWAT is written in FORTRAN whereas CONJOINT is written in SWAT employs some of the same algorithms used in PCJM for examining the axioms SWAT however makes some very important corrections to logical and theoretical errors made by the PCJM analysis of the con joint measurement axioms 19 2 3 5 Scaling After the axiom tests have been completed and an additive model has been found to be an adequate representation of the data the SWAT program calculates a scaling solution for the data By a scaling solution we mean that numerical values can be found for each of the levels of the three factors and their additive combinations that will maintain the order of the subjects card sort and conform to an additive model A number of algorithms are now available for obtaining scaling solutions for multiplicative dis tributive and dual distributive models as well as an additive model Two of these nonmetric algo rithms MONANOVA Kruskal 1965 and NONMETRG Johnson 1973 are used for SWAT to provide a best fitting scaling solution to re
136. till acceptable 39 SUMMARY OF AXIOM VIOLATIONS INDIVIDUAL ANALYSIS SUBJECT 8 INDEPENDENCE T INDEPENDENT OF E AND S 16 FAILURES OUT E INDEPENDENT OF T AND 5 9 FAILURES OUT S INDEPENDENT OF T AND E 16 FAILURES QUT DOUBLE CANCELLATION DOUBLE CANCELLATION IN T x E s 1 FAILURES DOUBLE CANCELLATION IN E x 5 2 FAILURES DOUBLE CANCELLATION IN S x T 0 FAILURES JOINT INDEPENDENCE T OF 108 OF 108 OF 108 QUT OF OUT OF OUT OF He eH TESTS TESTS TESTS 2 TESTS 2 TESTS i TESTS E INDEPENDENT OF 8 14 FAILURES OUT OF 108 TESTS E 8 INDEPENDENT OF 12 FAILURES OUT OF 108 TESTS 5 x T INDEPENDENT OF E 14 FAILURES QUT OF 108 TESTS OPTIONS INDIVIDUAL Fi GO TO NEXT INDIVIDUAL F2 GO TO NEXT OPTION CHOSEN IN PROGRAM SETUP PRINT SUMMARY OF AXIOM VIOLATIONS F4 PRINT COMPLETE AXIOM HISTORY ESC MAIN MENU Figure 9 Summary of Axiom Violations for Su ject No 8 Individually 40 44 8 8 SUMMARY OF AXIOM VIOLATIONS 646 INDIVIDUAL ANALYSIS SUBJECT 12 INDEPENDENCE T INDEPENDENT OF E AND 5 34 FAILURES OUT OF 108 TESTS E INDEPENDENT OF T 5 34 FAILURES OUT OF 108 TESTS S INDEPENDENT OF T AND 36 FAILURES OUT OF 108 TESTS DOUBLE CANCELLATION DOUBLE CANCELLATION IN Tx E 0 FAILURES OUT OF 1 TESTS DOUBLE CANCELLATION IN E x S 2 FAILURES OUT OF 2 TESTS DOUBLE CANCELLATION IN 8 x 0 FAILURES QUT OF 9 TESTS JOINT INDEPENDENCE T
137. tion Moderate stress due to confusion frustration or anxiety noticeably adds to workload Significant compensation is required to maintain adequate performance Almost never have spare time Interruptions or overlap among activities are very frequent or occur all the time Extensive mental effort and concentration are necessary Very complex activity requiring total attention Moderate stress due to confusion frustration or anxiety noticeably adds to workload Significant compensation is required to maintain adequate performance OIL Occasionally have spare time Interruptions or overlap among activities occur frequently Very little conscious mental or concentration required Activity is almost automatic requiring little or no attention Little confusion risk frustration or anxiety exists and can be easily accommodated wanana i 08 06 09 58 26 08 00 58 08 30 06 8 26 28 84 88 DDR Occasionally have spare time Interruptions or overlap among activities occur frequently Moderate conscious mental effort or concentration required Complexity of activity is moderately high due to uncertainty unpredictability or unfamiliarity Considerable attention required Little confusion risk frustration or anxiety exists and can be easily accommodated am kao
138. tive model STRESS is formed by finding the square root of the sum of the squared deviations between the disparity values and the estimated stimulus values If the original rank data are in perfect agreement with an additive representation then monotoni cally transformed disparities will be found that when suitably normalized are identical to the estimated stimulus scale values producing a STRESS value of zero Subjects data are however generally not without some random error Typically then the algorithm will not find a STRESS value of zero In these cases the algorithm works iteratively Following the computation of STRESS the estimated stimulus scale values are recalculated via a least squares estimation pro cedure similar to that employed in standard regression analysis The partiai derivative of STRESS with respect to each scale value is found and a numerical analysis procedure known as the method of gradients is used to find a new set of best fitting in the least squares sense stimulus scale values New disparities are formed a new STRESS value is computed and the iterative process is continued until no improvement in STRESS can be found Following the last iteration the esti mated scale values for the 27 stimulus combinations are found and are normalized so that combina tion 1 1 1 has a scale value of 0 and 3 3 3 has a scale value of 100 This approach treats inconsistencies in the arrangement of levels in the ca
139. uires total attention or concentration due to task complexity or the amount of information that must be processed Activities such as performing calculations making decisions remembering or storing information and problem solving are all examples of mental effort The exact descriptors used are 1 Very little conscious mental effort or concentration required Activity is almost automatic requiring little or no attention 2 Moderate conscious mental effort or concentration required Complexity of activity is moderately high due to uncertainty unpredictability or unfamiliarity Considerable atten tion required 12 3 Extensive mental effort and concentration are necessary Very complex activity requiring total attention 2 2 3 Psychological Stress Load Psychological Stress Load refers to conditions that produce confusion frustration and or anxiety during task performance and therefore make task accomplishment seem more difficult At low levels of stress one feels relatively relaxed As stress increases distraction from relevant aspects of the task is caused by factors within the environment or the individual These factors include such things as motivation fatigue fear skill level or temperature noise vibration and comfort Many of these factors can directly affect task performance when they reach high levels However for the purposes of SWAT and the measurement of mental workload we are talking about these
140. uld be used for the appropriate subjects as the workload levels for the corresponding ratings from the Event Scoring phase Event Scoring will be described in more detail in the next section A similar pro cedure should be followed for every group of card sort data While the exact approach will vary for each group this general framework should prove successful 47 si ofqns oum ay 3ureos 79 AddALOLlO 253 0109 94 40135 NI NASOHD NOILdO OI 09 S S 010 4 HOH X FETS 42 553815 6 3 5012 4 304 X 708 92391 553815 1 401993 404 5 00205 67 S SZ T 553515 Z 98 t 1 0443 9 4041094 HOV3 iO 44 TO LHOdld3 32NVIHOdHI dAILV 3H 31VRIVONGdV Z c CE 140443 5 LL TY es 3HIL 99 TO c9ito 8558510 8 S 1 1956 OSPIO 56 8691607 LT 1 3AILIGGV cLgv6 85 9 9t cOtbvb6 O9GPIO SI et ON NOILVHSLI CALNIYd nv NOIIVH3lI AO13H SNOILVHYALI AHL 3745 AHL SNOTLYUSLI 1591 NOILNTOS SHNOIIVH3 NOIIVHHOJNI 9 SA AdALOLOUd SN SNITVOS ASIA 74 LI d 5 4 Zd 2S3H 40 1014 14 01140 1 05 JHIL NOTLVHHOANI 9417425
141. upon receipt of the program We do ask however that you do not distribute the software to others Since the SWAT program is government property any individual can obtain a copy simply by writing the Harry G Armstrong Aerospace Medical Research Laboratory If copies are distributed without our knowl edge the recipient will not be advised of program updates or problems which may be discovered as the program is used The remainder of this appendix is organized for a complete description of each of the screens with which the user will interact The program is interactive and the user menus and screens are designed to help guide you through the analysis process The interface has been designed for ease of use and should facilitate data analysis First a sample screen will be presented for reference and then the screen functions will be described To the degree possible the order of execution for a routine analysis will be followed We will deviate from this practice only to completely describe all of the options presented on a given screen Output information resulting from this analysis includes prototype correlations axiom test results and the conjoint scaling solution As an additional aid the ESC key has been programmed to be the user s panic button At anytime while you are working with the SWAT program that you become confused or lost in the analysis you can press the ESC key and return to the Main Menu screen and all data will be retai
142. urn to the Program Setup screen and choose No 4 Prototype Axioms to assess the appropriateness of an additive model on the three groups separately The results of the axiom tests are presented in Figures 6 7 and 8 The summary of axiom violations for the time group as depicted in Figure 6 indicates very good agreement with only a maximum of seven viola tions for joint independence In general the criterion for either independence or joint independence is 20 failures Enter F1 to go to the next prototype group and display the results for the effort group As indicated in Figure 7 the correlations for these two subjects do not reach the criterion for acceptance of the analysis Based on this information it is advisable to investigate each of these two subjects individually Before doing that however examine the axiom test results for the stress group As presented in Figure 8 there are no axiom violations for this group NIVN 053 40135 HVHOOHd LNIUd VLVG SAYS d 23 T3 125 aj dures 22125 eed 20 go oo 9936 T 00 c 00 99 tc 90 2079 09 22 GOTT 9 9 t eot 8 eo vc 90 2 00 22 T 0 S eo 90 090 gt 20 ETS eo eo 02 00 TT eec 61 20 OT 90 1 d idVL SIHL 0
143. ver have spare time Interruptions or overlap among activities are very frequent or occur all the time Extensive mental effort and concentration are necessary Very complex activity requiring total attention High to very intense stress due to confusion frustration or anxiety High to extreme determina e tion and self control required NSS SES SCE EY SESE P namu 08 OD O GR AD AA GA D G a 80 88 13 Often have spare time Interruptions or overlap among activities occur infrequently or not at all Moderate conscious mental effort or concentration required Complexity of activity is moderately high due to uncertainty unpredictability or unfamiliarity Considerable attention required Moderate stress due to confusion frustration or anxiety noticeably adds to workload Significant compensation is required to maintain adequate performance
144. x E INDEPENDENT OF S 36 FAILURES OUT OF 108 TESTS E x S INDEPENDENT OF T 18 FAILURES QUT OF 108 TESTS S x T INDEPENDENT OF E 40 FAILURES OUT OF 108 TESTS OPTIONS INDIVIDUAL Fi GO TO NEXT INDIVIDUAL F2 GO TO NEXT OPTION CHOSEN IN PROGRAM SETUP F3 PRINT SUMMARY OF AXIOM VIOLATIONS F4 PRINT COMPLETE AXIOM HISTORY ESC MAIN MENU Figure 10 Summary of Axiom Violations for Subject No 12 Individually 41 ZI ON 1991915 Suneorpu 029125 5 711 LINO OL 95 49 6 7 047 2 SR 5 0071 95 BE 03 ER UG 9895 95 99 6 16 l GE 95 6 66 S 947 BL 62 97 97 5 0031 46 9 6S 2 1 09 4945 09 PB 64 5 O T 967 tb 09 ER ZZ 99 5 96 0931 99 EF L 027 897 127 6 7 BB l 9 65 19 99 68 09 3dA ILO0 LOHd 315 135 153 513 351 531 564 4 931539905 SIO3funs 153 15 H VA HOd SH OVEZ A SVM SDNVGEOONOD 40 2144302 5 JHL Vivd 5 123 5 H VA 40 515 34 101084 4 QAN 42 ZI 3421244209 S Iepu23 Jo 515 4 101054 I 85 19 647 047 5 901 9
145. y of interval scale psychometric techniques in the assessment of subjective mental workload unpublished masters thesis the University of Dayton Dayton Ohio Masline P 7 and Biers D W 1987 An examination of projective versus post task subjective workload ratings for three psychometric scaling techniques in Proceedings of the Human Factors Society 31st Annual Meeting pp 77 80 Santa Monica California Human Factors Society Nataupsky M and Abbott T S 1987 Comparison of workload measures on computer generated primary flight displays in Proceedings of the Human Factors Society 31st Annual Meeting pp 548 552 Santa Monica California Human Factors Society Notestine 7 C 1984 Subjective workload assessment in a probability monitoring task and the effect of delayed ratings in Proceedings of the Human Factors Society 28th Annual Meeting pp 685 689 Santa Monica California Human Factors Society Nygren T E 1985 Axiomatic and numeric conjoint measurement A comparison of three methods of obtaining subjective workload SWAT rankings in Proceedings of the IEEE 1985 National Aerospace and Electronics Conference pp 878 883 Dayton Ohio IEEE O Donnell R D and Eggemeier T 1986 Workload assessment methodology in Boff L Kaufman and J P Thomas Eds Handbook of Perception and Human Performance Volume 2 cognitive Processes and Performance New York Wiley Interscience Ossard G
146. y which hinders completion of your task Example in Everyday Life Psychological Stress Load includes such things as pressure to excel anxiety over physical dangers tension fatigue general state of health or feelings and comfort factors such as temperature or noise In the test situation if your course grade was to be determined by your performance on a certain test there would probably be quite a high level of stress However in a situation where your grade was fairly well determined the stress level would undoubtedly be less regardless of the time pressure or the amount of concentration required Also if construction was going on near the test room noise and distructions could affect your ability to concentrate and therefore impose psychological stress In a driving situation stress could be produced by obscure road signs heavy traffic or inclement weather which could cause you to become lost frustrated or concerned for your safety Description of the Levels Within the Dimeusions Figure 18 presents the three dimensions which have just been described as the main contributors to workload and the levels of each dimension Notice that there are three levels of each dimension which can be used to give a rating One is associated with the lowest degree of each of the dimensions three is associated with the highest degree and two is a middle degree Verbal descriptors are provided to define how you should eval uate the levels o

Reid, G.B., Potter, S.S. & Bressler, J.R.

Contents

Download Pdf Manuals

Related Search

Related Contents

Reid, G.B., Potter, S.S. &amp; Bressler, J.R.

Contents

Download Pdf Manuals

Related Search

Related Contents

Reid, G.B., Potter, S.S. & Bressler, J.R.