Home

Voice Recognition Evaluation Report

1. 22 Idcm ES 22 nbus et MCA RR QOIS ER 22 Appendix A Out of the Box Test Results for Phraselator u ceret rete traer eh ea eee 23 Appendix B Response Time DIBlel e oneri herb ob eti banda eee eee eee 46 borgo ooo 46 Phrase 46 Loa a EE 50 LIST OF TABLES Table 1 Language Features and Operational Parameters teret erat entra rennes v ausa aree 5 Table 2 General Unit Characteristics 5 Table 3 VRT Ambient Noise Test emen em ene nnne 14 Table 4 VRT Ambient Noise Test 96 ene e enne 14 Table 5 VRT Pink Noise eee Table 6 VRT Pink Noise Test 96 Correct 1 0 0 0 c cc ccc cece cece cece I 15 Table 7 VRT White Noise Test Results Hee Hee eee 15 Table 8 VRT White Noise Test Correct 00 0 cece cece cece eene 1 Table 9 Phraselator Ambient Noise Level 16 Table 10 Phraselator Ambient Noise 96 Co
2. Table 37 UT 103 80 Db White Noise Response Times 80 DB Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Test 1 2 3 4 5 6 7 8 9 10 Resp Resp Resp Resp Resp Resp Resp Resp Resp Resp Total Try Time Time Time Time Time Time Time Time Time Time Avg 1 4 3 4 3 3 3 3 4 3 3 2 3 3 3 3 4 4 3 3 3 4 3 3 4 3 3 3 3 3 3 3 3 4 4 3 3 3 4 4 4 3 4 3 5 3 3 3 4 3 3 4 3 3 4 6 3 3 3 3 3 3 3 4 3 4 7 4 4 3 3 4 3 3 3 4 3 8 3 3 3 3 3 4 3 3 4 4 9 3 3 4 3 3 3 3 4 3 3 10 3 3 3 4 3 3 3 3 3 3 Avg 3 3 3 2 3 2 3 2 3 3 3 3 3 2 3 3 3 3 3 4 3 27 Table 38 UT 103 90 Db White Noise Response Times 90 DB Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Test 1 2 3 4 5 6 7 8 9 10 Resp Resp Resp Resp Resp Resp Resp Resp Resp Resp Total Try Time Time Time Time Time Time Time Time Time Time Avg 1 3 3 4 3 3 3 4 3 3 3 2 3 3 3 4 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 3 4 4 4 3 3 4 4 3 3 3 3 5 4 3 4 4 4 3 3 4 3 3 6 3 4 3 3 3 3 4 4 3 4 7 3 4 3 3 3 3 3 3 4 3 8 3 3 3 3 3 4 3 4 3 3 9 3 3 3 4 3 3 3 3 3 4 10 3 3 4 3 5 3 3 4 4 3 Avg 3 2 3 3 3 3 3 3 3 4 3 2 3 2 3 4 3 3 3 2 3 28 This document is a research report submitted to the U S Department of Justice 54 This report has not been published by the Department Opinions or points of view expressed are those of the author s and do not necessarily reflect the official position or policies of t
3. This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view 46 expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 Table 23 Phraselator 70 Db Pink Noise Response Times 70 DB Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Test 1 2 3 4 5 6 7 8 9 10 Resp Resp Resp Resp Resp Resp Resp Resp Resp Resp Total Try Time Time Time Time Time Time Time Time Time Time Avg 1 4 5 4 4 4 4 4 4 4 4 2 4 4 5 5 4 5 4 4 4 4 3 4 5 4 4 4 4 4 5 5 5 4 4 4 4 5 4 5 5 5 5 4 5 4 4 4 5 4 4 5 4 4 4 6 4 4 5 4 4 4 5 4 4 4 7 4 4 4 4 4 4 5 4 4 4 8 4 4 4 4 4 4 4 5 5 4 9 4 5 4 4 4 4 5 5 5 5 10 5 5 5 4 4 4 4 5 5 5 AVG 4 1 4 4 4 3 4 3 4 4 2 4 5 4 5 4 5 4 3 4 81 Table 24 Phraselator 80 Db Pink Noise Response Times 80 DB Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Test 1 2 3 4 5 6 7 8 9 10 Resp Resp Resp Resp Resp Resp Resp Resp Resp Resp Total Try Time Time Time Time Time Time Time Time Time Time Avg 1 4 4 4 4 5 4 5 4 5 4 2 4 3 5 5 5 5 4 5 6 4 3 4 4 5 5 6 5 5 4 5 5 4 4 4 5 5 6 5 5 5 5 5 5 5 6 6 5
4. 5 Do you speak any languages other than English circle one Yes No If yes to the question above please indicate which languages and how proficient speaking and understanding the language you feel you are Language Proficiency Level Proficiency Level 1 poor 2 moderate 3 fluent 1 Have you ever used a speech recognizer recognizes input sounds and translates maps them into words circle one Yes 2 Have you ever used a speech translator converts one language into another circle one Yes 8 What type of microphones do you have experience feel comfortable using circle all that apply Live mic Push to talk mic Omer 80 This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view 24 expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 Participant Initial Phraselator Contact Instructions This section of the Evaluation is intended to determine your initial familiarity with the device s hardware components and the supporting contents of the device As part of this exercise you will label a block diagram device template Item 5 of key component features and will be able to use this as a reference for the following section of the Evaluation While you are completing this section y
5. Phrase Phrase Phrase Phrase Phrase Phrase Test 1 2 3 4 5 6 7 8 9 10 Resp Resp Resp Resp Resp Resp Resp Resp Resp Resp Total Try Time Time Time Time Time Time Time Time Time Time Avg 1 3 3 3 3 3 3 3 4 3 3 2 4 3 3 3 3 3 4 3 3 3 3 3 3 4 3 3 3 3 3 3 4 4 3 4 4 4 4 3 4 4 3 3 5 3 3 3 3 3 3 3 3 4 3 6 4 4 4 3 4 4 3 3 3 4 7 3 3 3 3 3 3 4 3 3 3 8 3 3 3 3 3 4 3 4 3 3 9 4 4 3 3 3 3 3 3 4 3 10 3 3 4 3 3 3 3 3 4 3 Avg 3 8 3 8 3 4 3 1 3 2 3 2 3 3 3 3 3 3 3 2 3 26 Table 36 UT 103 70 Db White Noise Response Times 70 DB Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Test 1 2 3 4 5 6 7 8 9 10 Resp Resp Resp Resp Resp Resp Resp Resp Resp Resp Total Try Time Time Time Time Time Time Time Time Time Time Avg 1 3 3 3 3 4 3 3 4 3 3 2 3 3 3 4 3 3 3 3 3 3 3 3 4 3 3 3 4 3 3 3 4 4 4 3 3 3 3 3 3 3 3 4 5 3 3 4 4 3 3 3 3 3 3 6 4 4 4 3 3 3 3 4 3 3 7 3 3 4 3 4 3 4 3 3 4 8 3 3 3 3 3 3 3 4 4 3 9 3 3 3 3 3 4 3 3 3 3 10 4 3 3 3 3 3 3 3 3 3 Avg 3 8 3 2 3 8 3 2 3 2 3 2 3 1 3 8 3 1 3 8 3 22 This document is a research report submitted to the U S Department of Justice 53 This report has not been published by the Department Opinions or points of view expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003
6. 80 amp 90 noise level testing Gun range hearing plugs protection allows you to filter out the noise and still hear what the translator is saying This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view 13 expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 VRT Tables 3 8 list the test results for the VRT The tables list the command phrase each spoken 10 times and the number the unit correctly translated The tables cover the ambient room noise 60 90 Db pink noise and 60 90 Db white noise test results Table 3 VRT Ambient Noise Test Results Ambient Room Noise lt 50 dB Phrase Tries Correct 1 Go Aft 10 10 2 Arrival Date 10 10 3 What s Your Cargo 10 9 4 Registration Number 10 9 5 Vessel Owner 10 3 6 Steady Course 10 7 7 Maintain Speed 10 10 8 Slow Down 10 10 9 Stop Vessel 10 10 10 Port of Call 10 3 Table 4 Ambient Noise Test Correct Phrase Correct 1 100 0 2 100 0 3 90 0 4 90 0 5 30 0 6 70 0 7 100 0 8 100 0 9 100 0 10 30 0 Table 5 VRT Pink Noise Test Pink Noise 60dB 70 dB 80 dB 90 dB Phrase Tries Correct Tries Correct Tries Correct Trie
7. Insert load battery pack not the 4 AA batteries but the actual battery pack 1 Turn device on 2 Start the Phraselator application by double clicking the Phraselator icon with the stylus 3 Select ArabicFoceProtection SC module by one clicking the ArabicForceProtection SCY menu with the stylus EE 3 2 and Do you speak English and acknowledge correct response with observer or repeat until correct response is obtained and confirmed by observer 4 Ensure Settings Options and Translated are selected No 5 Ensure category All Phrases is selected No 6 Ensure language Arabic is selected No 7 Translate say via PTT Pick Different Phrases Do you speak Arabic No 9 View input and output volume by first selecting Settings from the menu amp then Controls from the menu Report output volume level reading to observer i e 200 in 243 out Z 9 Lower output volume to between 80 100 via external volume button while monitoring output volume reading 10 Change the category from the menu to Restricted Area 11 Change language to Urdu 12 Translate say via Phrases menu Halt or release my dog Success 13 Repeat recognized phrase outputted above via microphone playback 14 Search for phrases that contain the word shoot use directions below and write down how many yo
8. 80 Db White Noise Response Times 54 Table 38 UT 103 90 Db White Noise Response Times essere eene enne 54 LIST OF FIGURES Figure T Noise MU TERN MENS 13 This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view 3 expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 Introduction This study investigated and compared several voice translator units Three units were tested which were the Phraselator Voice Response Translator and the Universal Translator UT 103 The intent of the study was to provide a comparison report that outlines the units performance capabilities including operation within noisy environments ease of use learning and other operational characteristics such as battery life In 1997 the National Institute of Justice NIJ and NAVAIR ORL Training Systems Division formerly the Naval Air Warfare Center Training Systems Division signed an Interagency Agreement IAA to establish a collaborative relationship to share modeling simulation and training technology to address law enforcement and military requirements Under this IAA NIJ requested that NAVAIR ORL TSD develop training support and conduct field evaluations for the Voice Response Tra
9. Settings Options and Translated are selected 5 Ensure category All Phrases is selected 6 Ensure language Arabic is selected 7 Translate say via PTT Pick Different Phrases Do you speak Arabic and Do you speak English and acknowledge correct response with observer or repeat until correct response is obtained and confirmed by observer 8 View input and output volume by first selecting Settings from the menu amp then Controls from the menu Report output volume level reading to observer i e 200 in 243 out 9 Lower output volume to between 80 100 via external volume button while monitoring output volume reading 10 Change the category from the menu to Restricted Area 11 Change language to Urdu 12 Translate say via Phrases menu Halt or release my dog 13 Repeat recognized phrase outputted above via microphone playback 14 Search for phrases that contain the word shoot use directions below and write down how many you find Select Search from the menu Select Keyboard from the menu Activate Search for Text Box by clicking on the text box via stylus on the keyboard Shoot e De activate keyboard remove the keyboard from the screen 15 Exit Phraselator under Settings 16 Turn unit off suspend under Start Suspend This document is a research report submitted to the U S Department of Justice
10. This report has not been published by the Department Opinions or points of view 28 expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 1 3 7 October 2003 Participant Exit Questionnaire Do you feel this device could be a valuable tool for communicating with individuals who speak a language different than the user Yes No Explain choice fMSSWBISSSSETySNGNUGDISWORSHRDIERIANES _ ___ _ Do you feel the documentation provided allowed you to successfully identify the different components of the device during the template exercise No Explain choice GNWASNSBNENSNONdenUSNEpa 0 0 0 Did you ask the observer for help at any time Yes No If yes what specifically did you ask for help with MASM OOA wit computer 4 Do you feel you had enough time to become familiar with the device No 5 Would you feel proficient enough with the device at this time to take it out in public and use it to communicate with someone who speaks a language that you do not understand Yes If no how much more time do you feel you would need with the device to become proficient in using it circle one 0 1 hours 2 5 hours Over 5 hours 6 Were the written instructions provided to you circle one Easy to Perform imewhallDiticulltolPerorm Difficult to Perform Explain choice __someone with computer skill would have been very easy 7 Did you fi
11. U S Department of Justice This report has not been published by the Department Opinions or points of view 34 expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 Participant Entry Questionnaire 3 Job title BMINGNSTERNSERORGEmNERUSpeOHMS 4 How would you rate your level of computer experience l 3 q 5 none minimal somewhat experienced experienced advanced 3 Which of the following computers do you personally use circle all that apply Laptop Desktop PDA Pen Tablet Other 4 Which of the following computer interfaces have you had experience using circle all that apply Stylus KeybOWH Microphone Touch screen Mouse Other 5 Do you speak any languages other than English circle one Yes No If yes to the question above please indicate which languages and how proficient speaking and understanding the language you feel you are Language Proficiency Level Proficiency Level 1 2 moderate 3 fluent 3 Have you ever used a speech recognizer recognizes input sounds and translates maps them into words circle one Yes 4 Have you ever used a speech translator converts one language into another circle one Yes 8 What type of microphones do you have experience feel com
12. U S Department of Justice This report has not been published by the Department Opinions or points of view 36 expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 2 0 Phraselator Features and Controls 2 Front or Top This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 LE 37 2 2 Back or Bottom NER 80 530 This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice HOL1V13SVHHd 7 October 2003 38 7 October 2003 Participant Application Instructions This section of the Evaluation involves a set of instructions that you are to read and complete one at a time sequentially It is intended for you to exercise the operational software features of the unit During this section of the Evaluation you will be able to utilize your block diagram device template constructed in the previous sec
13. intended to exercise the operational software features of the device During this section of the Evaluation the participant will be able to utilize the block diagram device template constructed in the previous section as a reference Specific evaluator instructions are to 1 Record the start time of this section 2 Observe the participant complete each step and indicate whether each task was completed or not 3 Note any significant events during this section e g if the participant asks for help or appears to need assistance by completing the questions at the end and 4 Record the end time for this section Note that items in single quotes represent a menu name from that device and that PTT stands for the Push To Talk microphone on the device START TIME ___ Did the Participant YES NO circle one la Insert load battery pack not the 4 AA batteries but the actual battery pack Z 1 Turn device on Z 2 Start the Phraselator application by double clicking the Phraselator icon with the stylus Z 3 Select ArabicFoceProtection SC module by one clicking the ArabicForceProtection SCY menu with the stylus Z 4 Ensure Settings Options and Translated are selected Z 5 Ensure category All Phrases is selected Z 6 Ensure language Arabic is selected Z 8 Translate say via PTT Pick Different Phrases Do you spea
14. on the keyboard Shoot found 2 phrases 2 2 De activate keyboard remove the keyboard from the screen Z 15 Exit Phraselator under Settings Z 16 Turn unit off suspend under Start Suspend END TIME 0043 Did the participant complete each of the sixteen items Yes No Z BE Baa eae eee eT Z If no which item s did they not identify Did the participant ask for help at any time Yes No If yes what specifically did the participant ask for help with Did the participant ever cause you to step in and offer assistance because they were having difficulty with an item Yes No If yes what specifically did you have to offer help with _The keyboard did not activate Additional Comments This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view 44 expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view 45 expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 Appendix B Response Time Tabl
15. questioning and commands The target audience is military and law enforcement The Phraselator was a part of the DARPA one way translation effort Marine Acoustics Inc was awarded a DARPA SBIR grant in January 2001 to develop the Phraselator Phraselators were built and delivered to military units in support of operation Enduring Freedom The Phraselator covers medical and military phrases search checkpoint and interrogation The target audience is medical personnel and military personnel The Universal Translator UT 103 is a commercial system designed to provide translation capability centered on transportation and other tourist type interactions It translates English to common European languages Spanish German and French The areas covered are plane luggage rail taxi and restaurant ordering type phrases The target audience is English speaking tourists on a European vacation The physical testing was performed in 3 parts The first was an of the box test in which just the Phraselator device was used It consisted of 2 different subjects opening the device and trying to learn to use it to perform simple translation task without any prior contact or familiarization with the unit The second was to test the units with 5 different phrases picked from the units phrase list first with ambient room noise 50Db then with different levels of pink and white background noise The sound level meter This document is a research report s
16. s initial familiarity with the device s hardware components and the supporting contents of the device As part of this exercise the participant will label a block diagram device template of key component features and will use this as a reference for the following section of the Evaluation An evaluator will be present If the participant has tried to complete an instruction and feels that they cannot and would like to ask questions they can ask the evaluator The evaluator will also be evaluating the participant during and after this section is complete The Participant Application Instructions are a set of operational instructions intended to determine if the participant can navigate through key software features During this session of the evaluation the participant will be able to utilize the block diagram device template constructed in the previous section asareference An evaluator will be present If the participant has tried to complete an instruction and feels that they cannot and would like to ask questions they can ask the evaluator The evaluator will also be evaluating the participant during and after this section is complete The Participant Exit Questionnaire is predominantly composed of multiple choice questions developed to gather information on the participant s opinions of the device This also includes questions pertaining to the unit s usability reliability and accuracy This document is a research report submitted to the
17. you need help 10 9 10 6 10 9 10 7 6 May search your vehicle 10 7 10 6 10 6 10 4 7 Where are you going 10 8 10 8 10 8 10 4 8 Do you have a place to live 10 9 10 7 10 9 10 7 9 Do not enter 10 9 10 7 10 9 10 4 10 will get an interpreter 10 9 10 7 10 6 10 6 Table 12 Phraselator White Noise Test Correct Phrase 60 Db 70 Db 80 Db 90 Db Correct Correct Correct Correct 1 70 0 40 0 50 0 40 0 2 70 0 50 0 60 0 90 0 3 60 0 50 0 70 0 30 0 4 80 0 90 0 70 0 70 0 5 90 0 60 0 90 0 70 0 6 70 0 60 0 60 0 40 0 7 80 0 80 0 80 0 40 0 8 90 0 70 0 90 0 70 0 9 90 0 70 0 90 0 40 0 10 90 0 70 0 60 0 60 0 Table 13 Phraselator White Noise Test Results White Noise 60dB 70 dB 80 dB 90 dB phrase Tries Correct Tries Correct Tries Correct Tries Correct 1 Did anyone see what happened 10 6 10 5 10 4 10 4 2 Halt or I ll release my dog 10 7 10 6 10 7 10 8 Visitors must check in at the main 3 entrance 10 5 10 5 10 7 10 3 4 Step away from the vehicle 10 8 10 9 10 8 10 8 5 Do you need help 10 9 10 5 10 9 10 6 6 May I search your vehicle 10 8 10 7 10 6 10 5 7 Where are you going 10 7 10 8 10 8 10 4 8 Do you have a place to live 10 9 10 9 10 8 10 6 9 Do not enter 10 9 10 9 10 9 10 4 10 will get an interpreter 10 8 10 7 10 6 10 5 This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view expresse
18. 0 0 Driving gt Gas station gt Would you 8 check the tires please 10 9 10 9 10 5 10 0 Telephone gt Inquiries gt Can 9 access the internet here 10 10 10 9 10 4 10 1 Telephone Inquiries What are 10 the charges per hour 10 10 10 10 10 4 10 0 This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view 19 expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 Table 18 UT 103 Pink Noise Test o Correct Phrase 60 Db 70 Db 80 Db 90 Db Correct Correct Correct Correct 1 80 0 60 0 80 0 20 0 2 80 0 80 0 80 0 0 0 3 90 0 90 0 60 0 40 0 4 0 0 0 0 0 0 0 0 5 100 0 80 0 40 0 20 0 6 90 0 80 0 30 0 10 0 7 90 0 100 0 40 0 0 0 8 90 0 90 0 50 0 0 0 9 100 0 90 0 40 0 10 0 10 100 0 100 0 40 0 0 0 Table 19 UT 103 White Noise Test Correct White Noise 60dB 70 dB 80 dB 90 dB phrase Tries Correct Tries Correct Tries Correct Tries Correct Traveling gt Tickets gt A seat next 1 to the window please 10 8 10 8 10 8 10 4 Traveling Tickets want to 2 change my reservation 10 9 10 9 10 7 10 2 Traveling gt Customs gt Please show me how to fill in this for
19. 0 100 via external volume button while monitoring output volume reading 15 Change the category from the menu to Restricted Area 16 Change language to Urdu 17 Translate say via Phrases menu Halt or release my dog 18 Repeat recognized phrase outputted above via microphone playback 19 Search for phrases that contain the word shoot use directions below and write down how many you find Select Search from the menu Select Keyboard from the menu Activate Search for Text Box by clicking on the text box Type via stylus on the keyboard Shoot e De activate keyboard remove the keyboard from the screen 15 Exit Phraselator under Settings 16 Turn unit off suspend under Start Suspend This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view 39 expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 Participant Exit Questionnaire 5 Do you feel this device could be a valuable tool for communicating with individuals who speak a language different than the user Yes Explain choice It did not translate any of my phrases after several attempts 4 Do you feel the documentation provided allowed you to successfully identify the different components of t
20. 5 3 4 4 3 4 3 3 3 4 3 6 4 3 4 3 3 4 3 4 3 3 7 3 3 3 4 4 3 4 3 3 3 8 3 3 3 3 3 3 3 3 3 3 9 3 3 3 3 3 3 3 3 3 3 10 3 3 3 3 3 3 3 3 4 3 Avg 3 2 3 1 3 2 3 2 3 2 3 1 3 2 3 2 3 2 3 1 3 17 Table 32 UT 103 70 Db Pink Noise Response Times 70 DB Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Test 1 2 3 4 5 6 7 8 9 10 Resp Resp Resp Resp Resp Resp Resp Resp Resp Resp Total Try Time Time Time Time Time Time Time Time Time Time Avg 1 4 3 3 3 3 3 3 3 3 3 2 3 3 3 3 3 3 3 4 3 3 3 3 3 3 3 3 3 4 3 3 3 4 3 3 3 3 3 3 3 4 3 4 5 3 3 3 3 3 3 3 3 3 3 6 3 3 3 3 4 3 3 3 4 3 7 3 3 3 3 3 3 3 3 4 3 8 3 3 3 3 3 3 3 3 3 3 9 4 3 4 3 3 3 3 3 3 3 10 3 4 4 4 3 3 3 3 3 4 Avg 3 2 3 1 3 2 3 1 3 1 3 3 1 3 2 3 2 3 2 3 14 This document is a research report submitted to the U S Department of Justice 51 This report has not been published by the Department Opinions or points of view expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 Table 33 UT 103 80 Db Pink Noise Response Times 80 DB Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Test 1 2 3 4 5 6 7 8 9 10 Resp Resp Resp Resp Resp Resp Resp Resp Resp Resp Total
21. 5 5 5 5 5 4 6 5 4 6 4 4 5 4 5 7 5 7 4 4 4 5 5 5 4 5 5 5 8 5 4 4 6 6 6 4 5 4 6 9 4 4 4 5 6 7 5 5 5 6 10 5 4 4 5 5 7 5 5 5 6 AVG 4 4 4 1 4 7 4 9 5 8 5 4 4 6 4 8 5 2 5 4 84 This document is a research report submitted to the U S Department of Justice 47 This report has not been published by the Department Opinions or points of view expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 Table 25 Phraselator 90 Db Pink Noise Response Times 90 DB Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Test 1 2 3 4 5 6 7 8 9 10 Resp Resp Resp Resp Resp Resp Resp Resp Resp Resp Total Try Time Time Time Time Time Time Time Time Time Time Avg 1 7 2 3 3 4 3 4 3 3 3 2 3 3 10 3 4 3 3 3 5 4 3 7 3 3 3 3 4 3 4 3 3 4 5 5 3 4 3 3 3 3 3 3 5 3 3 3 5 3 3 4 4 5 4 6 4 4 3 3 4 4 5 4 3 3 7 3 3 3 3 3 4 3 4 3 3 8 3 2 3 4 4 3 4 3 3 3 9 3 3 4 4 3 3 3 3 4 4 10 3 4 4 3 3 3 3 3 3 3 AVG 4 1 3 2 3 9 3 5 3 4 3 8 3 5 3 4 3 5 3 8 3 51 Table 26 Phraselator 60 Db White Noise Response Times 60 DB Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Test 1 2 B 4 5 6 n 8 9 10 Resp Resp Resp Resp Resp Resp Resp Resp Resp Resp Try Time Mime Time Mime Time Time Time
22. Db Pink Noise Response 46 Table 23 Phraselator 70 Db Pink Noise Response Times eeeeeeeeeeeneeeneneee nennen 47 Table 24 Phraselator 80 Db Pink Noise Response Times cescccsescesseececeeeeeceeeeecnseeeeseeeenaeeeenaeeeaes 47 Table 25 Phraselator 90 Db Pink Noise Response Times eese nennen 48 Table 26 Phraselator 60 Db White Noise Response Times ccccececcecssececsseeeceeeeecseeeeseeesteeecsteeeenaes 48 Table 27 Phraselator 70 Db White Noise Response Times eeeeeeeeeeeeeeneee nennen 49 Table 28 Phraselator 80 Db White Noise Response Times eene nennen 49 Table 29 Phraselator 90 Db White Noise Response Times 50 Table 30 UT 103 Ambient Room Noise Response Times eese 50 Table 31 UT 103 60 Db Pink Noise Response Times eeeeseeeeseeeeeeeeeeenee 51 Table 32 UT 103 70 Db Pink Noise Response 51 Table 33 UT 103 80 Db Pink Noise Response Times iuo eres rear aca nette 52 Table 34 UT 103 90 Db Pink Noise Response 52 Table 35 UT 103 60 Db White Noise Response Times eeeeeeeeeeeeeeneeeennene nennen nennen 53 Table 36 UT 103 70 Db White Noise 53 Table 37 UT 103
23. Gas station Would you check the tires please 10 8 9 Telephone gt Inquiries gt Can access the internet here 10 9 10 Telephone gt Inquiries gt What are the charges per hour 10 10 This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view 18 expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 Table 16 UT 103 Ambient Correct Phrase Correct 1 100 0 2 100 0 3 90 0 4 0 0 5 100 0 6 90 0 7 80 0 8 80 0 9 90 0 10 100 0 Table 17 UT 103 Pink Noise Level Test Results Pink Noise 60dB 70dB 80 dB 90 dB Phrase Tries Correct Tries Correct Tries Correct Tries Correct Traveling gt Tickets gt A seat next 1 to the window please 10 8 10 6 10 8 10 2 Traveling Tickets want to 2 change my reservation 10 8 10 8 10 8 10 0 Traveling gt Customs gt Please show me how to fill in this form 10 9 10 9 10 6 10 4 Traveling gt Customs gt lI m a 4 tourist 10 0 10 0 10 0 10 0 Traveling gt Railroad gt When does 5 the next train depart 10 10 10 8 10 4 10 2 Driving gt Road Emergency gt Can 6 you repair it today 10 9 10 8 10 3 10 1 Driving gt Road Emergency gt The 7 engine won t start 10 9 10 10 10 4 1
24. Mime Time Mime Total Avg 1 4 4 4 5 5 4 4 4 4 4 2 4 4 4 4 4 4 4 5 4 5 3 5 4 4 5 4 4 4 4 4 4 4 4 4 5 4 4 4 4 5 4 4 5 5 4 4 4 4 5 4 4 5 5 6 4 4 5 4 5 5 4 4 4 4 7 4 5 4 4 4 4 4 4 4 5 8 4 5 4 4 5 4 4 4 5 5 9 4 4 4 4 4 5 4 4 4 4 10 4 4 4 5 4 4 4 5 4 4 AVG 4 2 4 2 4 2 4 3 4 3 4 3 4 4 3 4 2 4 4 4 24 This document is a research report submitted to the U S Department of Justice 48 This report has not been published by the Department Opinions or points of view expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 Table 27 Phraselator 70 Db White Noise Response Times 70 DB Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Test 1 2 3 4 5 6 n 8 9 10 Resp Resp Resp Resp Resp Resp Resp Resp Resp Total Try Time Mime Time Time Time Time Time Time Time Time Avg 1 4 4 4 4 4 4 5 4 4 5 2 5 4 4 4 5 4 4 5 4 4 3 5 4 4 5 4 4 4 5 4 4 4 4 4 4 4 5 4 4 4 4 5 5 4 4 4 5 4 5 5 4 4 4 6 4 5 4 4 4 4 5 4 4 4 7 4 4 4 4 5 5 5 4 4 5 8 4 4 4 5 4 4 4 4 5 4 9 4 4 4 4 4 4 4 4 4 4 10 4 4 4 4 5 4 4 5 5 4 AVG 4 2 4 1 4 4 3 4 4 4 2 4 4 4 3 4 2 4 3 4 24 Table 28 Phraselator 80 Db White Noise Response Times 80 DB Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase P
25. T 103 Table 30 UT 103 Ambient Room Noise Response Times 50 DB Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Test 1 2 3 4 5 6 7 8 9 10 Resp Resp Resp Resp Resp Resp Resp Resp Resp Resp Total Try Time Time Time Time Time Time Time Time Time Time Avg 1 3 3 3 3 3 3 3 3 3 3 2 3 4 3 3 3 4 3 3 3 3 3 3 3 3 3 3 4 3 3 3 3 4 3 3 3 3 3 3 3 3 3 3 5 3 3 3 3 3 3 3 3 4 4 6 3 3 3 3 4 3 3 3 3 3 7 3 3 3 3 3 3 3 4 3 3 8 4 3 3 3 4 3 5 4 4 3 9 4 3 3 3 3 3 3 3 3 3 10 3 3 4 3 3 3 4 3 3 4 Avg 3 2 3 1 3 1 3 3 2 3 2 3 3 3 2 3 2 3 2 3 17 This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 50 7 October 2003 Table 31 UT 103 60 Db Pink Noise Response Times 60 DB Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Test 1 2 3 4 5 6 7 8 9 10 Resp Resp Resp Resp Resp Resp Resp Resp Resp Resp Total Try Time Time Time Time Time Time Time Time Time Time Avg 1 3 3 3 4 3 3 4 3 3 4 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 3 3 4 4 3 3 3 3 3 3 3 3 3
26. The author s shown below used Federal funds provided by the U S Department of Justice and prepared the following final report Document Title Voice Recognition Evaluation Report Author s Kevin Geib Laurie Marshall Document No 202733 Date Received October 2003 Award Number 2002 LB R 045 This report has not been published by the U S Department of Justice To provide better customer service NCJRS has made this Federally funded grant final report available electronically in addition to traditional paper copies Opinions or points of view expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 Voice Recognition Evaluation Report Prepared For OFFICE OF SCIENCE AND TECHNOLOGY NATIONAL INSTITUTE OF JUSTICE Washington D C 7 OCTOBER 2003 Prepared By Kevin Geib Laurie Marshall NAVAIR ORL Training Systems Division 12350 Research Parkway Orlando FL 32826 3224 This research was supported under Interagency Agreement 2002 LB R 045 with the U S Department of Justice Office of Justice Programs National Institute of Justice The products manufacturers and organizations discussed in this publication are presented for informational purposes only and do not constitute product approval or endorsement by the National Institute of Justice or the U S Department of Justice This document is a research report submitte
27. Try Time Time Time Time Time Time Time Time Time Time Avg 1 3 3 3 3 3 3 4 3 3 3 2 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 3 3 4 3 3 4 3 3 3 3 3 3 3 3 3 3 5 3 4 3 3 3 4 3 3 4 3 6 3 3 4 3 3 3 4 3 3 3 7 3 3 3 3 4 4 3 3 3 5 8 3 3 3 4 3 3 3 3 3 4 9 3 4 3 3 3 3 3 3 3 3 10 4 3 3 3 3 3 3 4 4 4 Avg 3 1 3 2 3 2 3 2 3 2 3 2 3 2 3 2 3 2 3 4 3 21 Table 34 UT 103 90 Db Pink Noise Response Times 90 DB Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Test 1 2 3 4 5 6 7 8 9 10 Resp Resp Resp Resp Resp Resp Resp Resp Resp Resp Total Try Time Time Time Time Time Time Time Time Time Time Avg 1 3 3 4 3 3 3 4 3 3 3 2 3 3 3 4 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 3 4 4 3 3 3 4 4 3 3 3 3 5 4 3 4 3 3 3 3 3 3 3 6 3 4 3 3 3 3 4 3 3 4 7 3 4 3 3 3 3 3 3 4 3 8 3 3 3 3 3 4 3 4 3 3 9 3 3 3 4 3 3 3 3 3 4 10 3 3 4 3 5 3 3 4 4 3 Avg 3 2 3 2 3 3 3 2 3 3 3 2 3 2 3 2 3 3 3 2 3 23 This document is a research report submitted to the U S Department of Justice 52 This report has not been published by the Department Opinions or points of view expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 Table 35 UT 103 60 Db White Noise Response Times 60 DB Phrase Phrase Phrase Phrase
28. al Date 10 10 10 9 10 8 10 6 3 What s Your Cargo 10 9 10 10 10 6 10 4 4 Registration Number 10 9 10 10 10 9 10 9 5 Vessel Owner 10 2 10 3 10 1 10 1 6 Steady Course 10 7 10 7 10 7 10 7 7 Maintain Speed 10 9 10 10 10 9 10 9 8 Slow Down 10 10 10 9 10 10 10 10 9 Stop Vessel 10 10 10 10 10 9 10 10 10 Port of Call 10 3 10 2 10 2 10 3 Table 8 White Noise Test Correct Phrase 60 Db 70 Db 80 Db 90 Db Correct 96 Correct 9o Correct 96 Correct 1 100 0 90 0 80 0 60 0 2 100 0 90 0 80 0 60 0 3 90 0 100 0 60 0 40 0 4 90 0 100 0 90 0 90 0 5 20 0 30 0 10 0 10 0 6 70 0 70 0 70 0 70 0 7 90 0 100 0 90 0 90 0 8 100 0 90 0 100 0 100 0 9 100 0 100 0 90 0 100 0 10 30 0 20 0 20 0 30 0 This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 15 Phraselator 7 October 2003 Tables 9 14 list the test results for the Phraselator The tables list the command phrase each spoken 10 times and the number the unit correctly translated The tables cover the ambient room noise 60 90 Db pink noise and 60 90 Db white noise test results Table 9 Phraselator Ambient Noise Level Test Ambie
29. being used For repairs on the back of the manual the manufacturer states If your Phraselator fails notify us and we will send you a replacement and return mailer for your failed unit Contact information is also provided on the user guide Universal Translator Number of languages The languages supported by this unit are Spanish German and French Adding languages would have to be done by the manufacturer Visiting the Ectaco web site revealed claims of supporting up to 17 languages Number of phrases The manufacturer claims that the unit contains about 3000 unique phrases The domains that it is subdivided into however are fairly restrictive There are only about 10 phrases per domain The user is forced to select down to the domain that contains the desire phrase This technique boosts accuracy at the expense of ease of use Technology The unit uses wave playback for the translated phrase and the input is speaker independent Recognition technology is the Lingvobit speech recognition system This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view 10 expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 System Recognition Training The unit is speaker independent no training is required Noise Degradation No claims were
30. d are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 17 7 October 2003 Table 14 Phraselator White Noise Test Correct Phrase 60 Db 70 Db 80 Db 90 Db Correct Correct Correct Correct 1 60 0 50 0 40 0 40 0 2 70 0 60 0 70 0 80 0 3 50 0 50 0 70 0 30 0 4 80 0 90 0 80 0 80 0 5 90 0 50 0 90 0 60 0 6 80 0 70 0 60 0 50 0 7 70 0 80 0 80 0 40 0 8 90 0 90 0 80 0 60 0 9 90 0 90 0 90 0 40 0 10 80 0 70 0 60 0 50 0 Universal Translator Tables 15 20 list the test results for the Universal Translator The tables list the command phrase each spoken 10 times and the number the unit correctly translated The tables cover the ambient room noise 60 90 Db pink noise and 60 90 Db white noise test results Table 15 UT 103 Ambient Noise Level Test Results Room ambient noise 50 dB Phrase Tries Correct 1 Traveling gt Tickets gt A seat next to the window please 10 10 2 Traveling Tickets want to change my reservation 10 10 Traveling gt Customs gt Please show me how to fill in this 3 form 10 9 4 Traveling gt Customs gt l m a tourist 10 0 5 Traveling gt Railroad gt When does the next train depart 10 10 6 Driving gt Road Emergency gt Can you repair it today 10 9 7 Driving gt Road Emergency gt The engine won t start 10 8 8 Driving gt
31. d to the U S Department of Justice This report has not been published by the Department Opinions or points of view 1 expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 TABLE OF CONTENTS Ti 4 LES 5 Voice Response Translator Components NERO 5 irm HT 5 by dui P 6 FPhraselator ODDO MUN UEM DOE Dora Cun RM CDs De MEORUM aaaea NS al eik 6 Universal 6 Unit Capa IS T ia 7 7 PEAS C METELLI 9 DET TS TU ROT 10 HCM TAE NUT 12 Dutor fe BOX 12 8 770 13 14 PUEDE ROI esos TT 16 UuciurtiBbriudb M 18 I coa MI 1 NEM T 21 pg 21 PUNT AG m 21 Universal Tns lAr sainia i 21 EDD 21 dcl
32. ed instructions This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 30 7 October 2003 6 Review included written documentation if necessary to help accurately identify items on paper template Participant should realize that they need to review the included written documentation in order to correctly identify the items on the paper template TOTAL TIME How much time it took in total for the participant to complete all tasks 45 Did the participant complete each of the five items within the excessive time limit Yes No If no which item s did they not complete within the excessive time limit Did the participant correctly identify each of the nine items listed on the paper template Yes No If no what items were not correctly identified Did the participant ask for help at any time Yes No If yes what specifically did the participant ask for help with Did the participant ever exceed the excessive time limit and cause you to step in and offer assistance Yes WO If yes what specifically did you have to offer help with Additional Comments This document is a research report submitted t
33. elease my dog worked the first time for the first tester and was never recognized for the second one 7 The microphone level though monitoring during the spoken test always seemed to reset itself to 0 when the microphone volume was checked according to the test procedure This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view 12 expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 Refer to Appendix A for details concerning this test Noise testing Noise testing was performed using white noise pink noise and best case ambient It should be noted that since the 50 Db noise level would not occur in any normal operational environment it is the best case in which the units may be tested White noise is defined as each frequency in the sound having equal energy throughout the spectrum An octave is the doubling of the frequency 1 octave from 100 Hz is 200 Hz As the frequencies are increased the higher frequency octave have greater number of frequency components than the lower frequency octaves 20 Hz to 40 Hz octave at a 1 Hz resolution have fewer frequency components 20 than the 1000 Hz to 2000 Hz octave 1000 frequency components at a 1 Hz resolution Therefore white noise has an emphasis on the higher frequencies and gives a stro
34. ent Opinions or points of view expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice HOL1V13SVHHd 7 October 2003 27 7 October 2003 Participant Application Instructions This section of the Evaluation involves a set of instructions that you are to read and complete one at a time sequentially It is intended for you to exercise the operational software features of the unit During this section of the Evaluation you will be able to utilize your block diagram device template constructed in the previous section as a reference While you are completing this section you will be observed by an evaluator We are not interested in how fast you complete each step but rather want to observe how an individual with little experience with the device begins using it If you feel you cannot complete a step you may ask for assistance from the evaluator who will otherwise be seated nearby Note that items in single quotes represent a menu name from the device and that PTT stands for the Push To Talk microphone on the device Please begin 1 Insert load battery pack not the 4 AA batteries but the actual battery pack 1 Turn device on 2 Start the Phraselator application by double clicking the Phraselator icon with the stylus 3 Select ArabicFoceProtection SC module by one clicking the ArabicForceProtection SCY menu with the stylus 4 Ensure
35. es Response times were measured with a wrist watch Therefore time measurements are accurate to the second VRT Response responses occurred in about 1 second or less Phraselator Table 21 Phraselator 50 Db Ambient Room Noise Response Times 50 DB Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Test 1 2 3 4 5 6 7 8 9 10 Resp Resp Resp Resp Resp Resp Resp Resp Resp Resp Total Try Time Time Time Time Time Time Time Time Time Time Avg 1 4 4 4 5 4 4 4 4 4 4 2 5 4 4 4 4 4 4 4 4 5 3 4 4 5 4 4 5 4 5 5 4 4 4 4 4 4 4 4 5 4 4 4 5 3 4 3 4 4 4 4 5 4 5 6 4 4 4 4 5 3 5 4 4 4 7 4 5 4 4 4 4 4 5 5 4 8 4 4 5 5 4 4 4 4 4 5 9 4 5 4 4 5 4 5 4 4 4 10 4 4 4 4 4 4 4 4 4 4 AVG 4 4 2 4 1 4 2 4 2 4 4 3 4 3 4 2 4 3 4 18 Table 22 Phraselator 60 Db Pink Noise Response Times 60 DB Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Test 1 2 3 4 5 6 7 8 9 10 Resp Resp Resp Resp Resp Resp Resp Resp Resp Resp Total Try Time Time Time Time Time Time Time Time Time Time Avg 1 4 5 4 4 5 4 5 5 4 4 2 5 4 5 4 4 4 4 4 4 5 3 4 4 4 4 4 4 5 4 5 4 4 4 4 4 5 5 4 4 4 4 5 5 5 5 4 4 5 4 4 5 4 4 6 5 4 5 4 4 4 4 4 5 4 7 4 4 4 4 4 4 4 4 4 4 8 4 5 4 4 4 5 4 5 4 4 9 4 4 4 5 4 5 5 4 5 4 10 5 4 4 4 4 4 4 4 4 5 AVG 4 4 4 3 4 2 4 2 4 3 4 2 4 3 4 3 4 3 4 3 4 28
36. fortable using circle all that apply Live mic Push to talk mic Other This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view 35 expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 Participant Initial Phraselator Contact Instructions This section of the Evaluation is intended to determine your initial familiarity with the device s hardware components and the supporting contents of the device As part of this exercise you will label a block diagram device template Item 5 of key component features and will be able to use this as a reference for the following section of the Evaluation While you are completing this section you will be observed by an evaluator If you feel you cannot complete a step you may ask for assistance from the evaluator who will otherwise be seated nearby Please begin Open the box Take out and identify included written documentation Take out and identify battery pack Take out and identify Phraselator unit Identify physical features of Phraselator using paper template provided see further instructions on paper template provided Review included written documentation if necessary to help accurately identify items on paper template EPA e This document is a research report submitted to the
37. h sensitive hand held computer screen It has buttons to control output volume and microphone sensitivity There is a push to talk button and a button that causes the currently selected phrase to play the translation There is no switch other than to turn the unit off to prevent the unit translating a phrase that was incorrectly recognized from playing Scalability phrases domains ability required to do so ease and time Phrases and domain changes would have to be done by the manufacturer The capability exists for adding new domains Coast Guard etc and languages by request Operational modes i e hands free This unit must be hand held for operation It requires user to keep the unit at a distance of about 4 to 6 inches from the mouth while speaking While the push to talk microphone does not allow this unit to be used hands free it is possible to operate with one hand after the recognition program has been started and the phrase set selected Response time The unit usually responded within 4 to 5 seconds after the user finished a phrase See Appendix B Phraselator for details in response times Response time slows significantly as the batteries wear down Dimensions Width 5 0 inches This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view 9 expressed are those of the author s and do not necessarily reflect the official positi
38. he U S Department of Justice
39. he ability to bypass voice recognition if needed and The Universal Translator is better suited for tourist travel Below are some specific comments for each device This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view 2l expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 VRT The main advantage of this unit is the fact that no user intervention is required for operation After turning the unit on setting the phrase group Coast Guard in this case the user simply talked to the unit The other translator devices with push to talk and the GUI Phraselator required operator intervention The VRT was also the fastest unit with response times of less than a second Testing for the VRT proceeded faster than with the other two units as it was easier to determine if the phrase was recognized 1 the unit responded immediately The battery life for the VRT was greater than the other two units Battery testing was performed with a computer playing a recorded phrase set repeatedly for 8 consecutive hours The VRT limit was not found during this period and it is estimated that battery life would be several days The battery life of the other two units would be at most a day with constant use Additionally the fact that the headset microphone
40. he device during the template exercise No Explain choice The manual amp pamphlet had pictures which labeled the parts __ 3 Did you ask the observer for help at any time Yes No If yes what specifically did you ask for help with During the voice translation steps_ 4 Do you feel you had enough time to become familiar with the device No 6 Would you feel proficient enough with the device at this time to take it out in public and use it to communicate with someone who speaks a language that you do not understand Yes If no how much more time do you feel you would need with the device to become proficient in using it circle one 0 2 hours 2 5 hours Over 5 hours 6 Were the written instructions provided to you circle one EasytolPetfomm Somewhat Difficult to Perform Difficult to Perform Explain choice 8 Did you find the device itself easy to use Yes No Explain choice 8 Did the device follow the commands that you told instructed it to Yes M6 9 Any additional comments you would like to add This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view 40 expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 Observer Evaluator Initial Phraselator Contact Evaluation During this section of the Evaluation the
41. he entire phrase to be spoken just a few key words These key words are given on a laminated card Adding a large number of phrases then would increase the number of key words that a user would have to remember and may result in the user consulting the phrase key word card constantly while using the unit This would defeat the hand free operation of the unit so care must be taken when adding phrases to this unit Technology The VRT uses 8 bit wave playback for the foreign language phrase the input of which is matched to a user trained input The user speaks a shortened key word phrase which the match is repeated and then the foreign language phrase is spoken System Recognition Training This unit must be trained in order for it to recognize a user The unit guides you automatically through the training Training involves the potential user saying each phrase from the reference cards that would trigger the foreign language phrase Two units were trained and each took 45 minutes to train The unit tests the recorded wave quality during the training session and when a phrase is not recognized repeats the training until it is This caused the training to take longer than expected and many phrases required repeated training attempts but the payoff is better recognition Noise Degradation The manufacturer stated that unit would work in about 100 Db of background noise and claims that higher noise levels could be achieved by training in a n
42. he next section of the Evaluation Specific evaluator instructions are to 1 Record the actual time it takes the participant to complete each task 2 Provide the participant with the block diagram device template when they reach that item 3 Note any significant events during this section e g if the participant asks for help or appears to need assistance by completing the questions at the end and 4 Add the total time overall it took the participant to complete all of the items Item Participant Instruction Description of Exces Actual Comments Observations Task sive Time Time min min Openbox Participant should 5 0 5 be able to open the box the factory packed box 2 Take out and identify Participant should 5 0 25 included written be able to remove documentation the written documentation from the box and should be aware that it includes instructions for using the device 3 Take out and identify Participant should 5 0 5 ee battery pack be able to identify and remove the battery pack from the box 4 Take out and identify Participant should 3 0 5 Phraselator unit be able to identify and remove the Phraselator device itself from the box 5 Identify physical features See block diagram 25 3 25 of Phraselator using device template of for paper template provided Phraselator with items see further instructions items to be 85 amp on paper template identified for 6 provid
43. he paper template TOTAL TIME How much time it took in total for the participant to complete all tasks 45 Did the participant complete each of the five items within the excessive time limit Yes No If no which item s did they not complete within the excessive time limit Did the participant correctly identify each of the nine items listed on the paper template Yes No If no what items were not correctly identified Did the participant ask for help at any time Yes No If yes what specifically did the participant ask for help with Did the participant ever exceed the excessive time limit and cause you to step in and offer assistance Yes 8 If yes what specifically did you have to offer help with Additional Comments This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 42 7 October 2003 Observer Evaluator Participant Application Evaluation During this section of the Evaluation the evaluator will monitor the participant and determine if the participant completes each of the items This section of the Evaluation involves a set of instructions that the participant should complete one at a time sequentially It is
44. his session of the evaluation the participant will be able to utilize the block diagram device template constructed in the previous section asareference An evaluator will be present If the participant has tried to complete an instruction and feels that they cannot and would like to ask questions they can ask the evaluator The evaluator will also be evaluating the participant during and after this section is complete The Participant Exit Questionnaire is predominantly composed of multiple choice questions developed to gather information on the participant s opinions of the device This also includes questions pertaining to the unit s usability reliability and accuracy This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view 23 expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 Participant Entry Questionnaire 1 Job title 2 How would you rate your level of computer experience none minimal somewhat experienced experienced advanced 3 Which of the following computers do you personally use circle all that apply Laptop Desktop PDA Pen Tablet Other 4 Which of the following computer interfaces have you had experience using circle all that apply Stylus Keyboard Microphone Touch screen Mouse Other
45. hrase Test 1 2 3 4 5 6 n 8 9 10 Resp Resp Resp Resp Resp Resp Resp Resp Resp Resp Try Time lime lime lime lime lime lime lime lime Total Avg 1 4 4 4 5 4 5 5 4 4 4 2 4 4 4 4 4 4 4 4 4 4 3 4 4 5 4 4 4 4 5 4 4 4 5 5 4 4 5 5 4 4 5 4 5 6 5 5 4 4 5 4 4 5 5 6 5 5 4 4 4 4 4 4 4 4 7 4 4 5 5 5 4 5 4 4 4 8 4 4 4 5 5 4 5 6 4 5 9 5 5 4 4 4 4 4 4 4 4 10 4 4 4 4 4 4 4 4 4 4 AVG 4 5 4 4 4 3 4 3 4 3 4 3 4 3 4 3 4 2 4 2 4 31 This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 49 7 October 2003 Table 29 Phraselator 90 Db White Noise Response Times 90 DB Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Phrase Test 1 2 3 4 5 6 n 8 9 10 Resp Resp Resp Resp Resp Resp Resp Resp Resp Resp Try Time Time lime lime Time lime lime Time Time Total Avg 1 4 4 4 5 4 5 5 4 4 5 2 4 4 4 4 4 4 4 5 4 4 3 5 5 4 4 4 5 5 5 5 4 4 4 6 4 5 4 4 5 4 4 4 5 5 4 5 6 4 5 4 4 5 4 6 4 5 5 5 5 5 4 4 5 4 7 5 4 5 5 5 4 5 4 5 5 8 4 4 5 4 5 4 5 5 4 5 9 4 5 4 4 4 4 5 5 4 5 10 4 5 4 4 4 4 4 4 4 4 AVG 4 3 4 6 4 4 4 6 4 3 4 4 4 6 4 4 4 4 4 4 4 44 U
46. ipant Instruction Set Today you will be participating in an Out of The Box evaluation of a speech translator unit the Phraselator which is capable of translating commands from the English language into multiple foreign languages The purpose of this evaluation is to provide information on a participants initial contact with the device The evaluation is divided into four sections Participant Entry Questionnaire Participant Initial Phraselator Contact Instructions Observer evaluating Participant Application Instructions Observer evaluating Participant Exit Questionnaire Bor This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view 39 expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 The Participant Entry Questionnaire is predominantly composed of multiple choice questions developed to gather information on the participant s background and experience with computers This information will be used to assist the Principal Investigator in interpreting results throughout the remaining sections of the Evaluation The Initial Phraselator Contact Instructions are a set of six basic instructions consisting of opening the box the unit came in to reviewing items contained within it This section is intended to test the participant
47. k USB cable AA Batteries Hardware User Manual Laminated Phrase card Phraselator Brochure Nylon Carrying Case with shoulder strap and key ring hook holds all above materials NOR BR Re There were total of 2 units used in testing each containing the same items Universal Translator Universal Translator UT 103 1 Neck strap Ear Headphone combo AA batteries User Manual _ NOS This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view 6 expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 Unit Capabilities VRT Number of languages The languages supported by this unit are Spanish Creole Portuguese Arabic and Dari Farsi Since this unit like all the others plays back fixed stored responses the potential exists to support any spoken language Number of phrases The VRT contains about 204 unique phrases Phrases are not spoken completely they are flagged by short key word phrases that trigger the complete spoken phrase in the recorded language The key word phrases are given on a small laminated key ring bound set Manufacturer states that the unit has the capacity to hold a virtually unlimited number of phrases and languages therefore the phrase set could be expanded The unit does not require t
48. k Arabic and Do you speak English and acknowledge correct response with observer or repeat until correct response is obtained and confirmed by observer BEBE Z 12 View input and output volume by first selecting Settings from the menu amp then Controls from the menu Report output volume level reading to observer i e 200 in 243 out 9 Lower output volume to between 80 100 via external volume button Z E This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 43 7 October 2003 Z while monitoring output volume reading 87 used button not the 15 Change the category from the menu to Restricted Area Z 16 Change language to Urdu Z 17 Translate say via Phrases menu Halt or release my dog Never 2 18 Repeat recognized phrase outputted above via microphone playback Z 19 Search for phrases that contain the word shoot use directions below and write down how many you find e Select Search from the menu Z e Select Keyboard from the menu 2 e Activate Search for Text Box by clicking on the text box Type via stylus
49. m 10 10 10 8 10 7 10 0 Traveling gt Customs gt l m a 4 tourist 10 2 10 0 10 0 10 0 Traveling gt Railroad gt When does 5 the next train depart 10 8 10 8 10 2 10 0 Driving gt Road Emergency gt Can 6 you repair it today 10 7 10 8 10 2 10 0 Driving gt Road Emergency gt The 7 engine won t start 10 10 10 9 10 6 10 0 Driving gt Gas station gt Would you 8 check the tires please 10 9 10 9 10 3 10 0 Telephone gt Inquiries gt Can 9 access the internet here 10 9 10 8 10 4 10 0 Telephone gt Inquiries gt What are 10 the charges per hour 10 9 10 10 10 0 10 0 This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 20 7 October 2003 Table 20 UT 103 White Noise Test Correct Phrase 60 Db 70 Db 80 Db 90 Db Correct Correct Correct Correct 1 80 0 80 0 80 0 40 0 2 90 0 90 0 70 0 20 0 3 100 0 80 0 70 0 0 0 4 20 0 0 0 0 0 0 0 5 80 0 80 0 20 0 0 0 6 70 0 80 0 20 0 0 0 7 100 0 90 0 60 0 0 0 8 90 0 90 0 30 0 0 0 9 90 0 80 0 40 0 0 0 10 90 0 100 0 0 0 0 0 Battery Life Testing This test was to see how long a unit s battery life would last before requiring recharging If a unit went over 8 hours withou
50. made in the manufacturer s literature on the unit s ability to operate in a noisy environment Output Volume Options The unit is designed to operate in a one on one environment with a built in speaker The system has a headphone jack but that consists of a microphone and speaker output Given the language training documentation that comes with the unit the headset is apparently only intended to be used for language learning Feedback indicators for users The unit has a small LCD display Scalability phrases domains ability required to do so ease and time Phrases and domain changes would have to be done by the manufacturer It appears from the distributor s web site that it might be possible to support a specific language Operational modes i e hands free This is a hand held unit The microphone automatically turns on when input is detected Response time The unit usually responds within 3 to 4 seconds after the user finishes speaking the phrase See Appendix B Universal Translator for detailed response time data Response time slows significantly as batteries wear down Dimensions Width 2 5 inches Height 4 375 inches Thickness 1 0 inch Weight Unit measured 4 7 oz Platform type OS The unit is an embedded system based on proprietary hardware Microphone The unit has an embedded microphone and a noise canceling headset mike speaker However using this combination does not allow anyone but the user to here the
51. nd the device itself easy to use Explain choice 8 Did the device follow the commands that you told instructed it to W No 8 Any additional comments you would like to add This document is a research report submitted to the U S Department of Justice 29 This report has not been published by the Department Opinions or points of view expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 Observer Evaluator Initial Phraselator Contact Evaluation During this section of the Evaluation the observer will monitor the participant and record the time it takes for each item to be completed This section is intended to test the participants initial familiarity with the device s hardware components and the supporting contents of the device The participant should determine when they have completed all of the items If for some reason it seems to take an excessive amount of time exceeds excessive time as documented in the table below the evaluator will provide some guidance and record the specific guidance that was needed and then provided As part of this exercise the participant will label a block diagram device template of key component features Item 5 This diagram will need to be reviewed by the evaluator and corrected before moving on to the next section of the Evaluation The participant may use this block diagram device template during t
52. ng sssssssssss sound Pink noise has a spectral component defined as 1 f which is equal energy per octave Therefore the 20 Hz to 40 Hz example would have the same energy as the 1000 Hz to 2000 Hz range Pink noise sounds more natural to the human ear and is used to generate rainfall waterfall wind rushing river and other sounds that occur in nature Both noise files were generated by Cool Edit Pro s noise generation capability Cool Edit Pro uses a random frequency generator for the white and pink noise generator that adheres to the spectral component requirement for each type of noise Five phrases were selected from each unit s supported phrase list and the microphone placement was measured to make sure the distance and positioning was according to manufacturers specifications Figure 1 shows the test setup Tester position Speakers Microphone position Position behind tester but so that the head does not block the sound Distance to mouth must be according to manufacturers specification Unit under test Db meter Measurement must be at point of microphone Figure 1 Noise Test Setup The sound generator used was a standard laptop computer driving Dell Harman Kardon PC speakers The Db meter used was a Radio Shack Sound Level Meter catalog number 33 2055 Noise level measurements were made at the microphone to ensure a consistent level for the test Gun range hearing protection was also used with the 70
53. nslator VRT a language translation device developed by Integrated Wave Technologies specifically for law enforcement officers As NAVAIR ORL TSD and NIJ worked to expand and improve the utility of the VRT for law enforcement potential military applications were identified Simultaneously information regarding other translation devices such as the Phraselator developed by the Defense Advanced Research Projects Agency DARPA and the Universal Translator UT 103 a commercial product from Ectaco Inc emerged While there were testimonials regarding the utility and potential applications of all these devices there was no known existing laboratory test to evaluate the devices In 2002 NIJ requested that NAVAIR ORL conduct these tests in order to determine the speech accuracy of the units and include degradation in noisy environments As plans for the testing evolved representatives from the Special Operations Command SOCOM contacted NIJ regarding potential use of the VRT There was already substantial military interest in translation devices the Phraselator had already been involved in several field tests SOCOM agreed to supply Phraselator units for the NAVAIR ORL laboratory tests and requested specifically that an out of the box test be conducted for the Phraselator because this could have direct bearing on SOCOM s plans for fielding units The VRT is a system currently used by law enforcement and the Coast Guard The phraseology covers search
54. nt Noise lt 50 dB phrase Tries Correct 1 Did anyone see what happened 10 6 2 Halt or I ll release my dog 10 8 3 Visitors must check in at the main entrance 10 7 4 Step away from the vehicle 10 8 5 Do you need help 10 10 6 May I search your vehicle 10 9 7 Where are you going 10 8 8 Do you have a place to live 10 9 9 Do not enter 10 10 10 will get an interpreter 10 9 Table 10 Phraselator Ambient Noise Correct Phrase 96 Correct 1 60 0 2 80 0 3 70 0 4 80 0 5 100 0 6 90 0 7 80 0 8 90 0 9 100 0 10 90 0 This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 16 7 October 2003 Table 11 Phraselator Pink Noise Test Results Pink Noise 60dB 70 dB 80 dB 90 dB phrase Tries Correct Tries Correct Tries Correct Tries Correct 1 Did anyone see what happened 10 7 10 4 10 5 10 4 2 Halt or I ll release my dog 10 7 10 5 10 6 10 9 Visitors must check in at the main 3 entrance 10 6 10 5 10 7 10 3 4 Step away from the vehicle 10 8 10 9 10 7 10 7 5 Do
55. o the U S Department of Justice This report has not been published by the Department Opinions or points of view expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 3l 7 October 2003 Observer Evaluator Participant Application Evaluation During this section of the Evaluation the evaluator will monitor the participant and determine if the participant completes each of the items This section of the Evaluation involves a set of instructions that the participant should complete one at a time sequentially It is intended to exercise the operational software features of the device During this section of the Evaluation the participant will be able to utilize the block diagram device template constructed in the previous section as a reference Specific evaluator instructions are to 1 Record the start time of this section 2 Observe the participant complete each step and indicate whether each task was completed or not 3 Note any significant events during this section e g if the participant asks for help or appears to need assistance by completing the questions at the end and 4 Record the end time for this section Note that items in single quotes represent a menu name from that device and that PTT stands for the Push To Talk microphone on the device START TIME _ 0928 Did the Participant YES NO circle one 1
56. observer will monitor the participant and record the time it takes for each item to be completed This section is intended to test the participants initial familiarity with the device s hardware components and the supporting contents of the device The participant should determine when they have completed all of the items If for some reason it seems to take an excessive amount of time exceeds excessive time as documented in the table below the evaluator will provide some guidance and record the specific guidance that was needed and then provided As part of this exercise the participant will label a block diagram device template of key component features Item 5 This diagram will need to be reviewed by the evaluator and corrected before moving on to the next section of the Evaluation The participant may use this block diagram device template during the next section of the Evaluation Specific evaluator instructions are to 1 Record the actual time it takes the participant to complete each task 2 Provide the participant with the block diagram device template when they reach that item 3 Note any significant events during this section e g if the participant asks for help or appears to need assistance by completing the questions at the end and 4 Add the total time overall it took the participant to complete all of the items Item Participant Instruction Description of Exces Actual Comments Obse
57. oise environment where there is a fairly steady loud background noise Output Volume Options The unit is designed to operate in a one on one environment with a built in speaker but also contains attachment plugs and units to connect to a megaphone or voice instrument amplifier This should give sufficient volume for any noise situation The built in speaker measured about 85 Db at the speaker during playback This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view 7 expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 Feedback indicators for users The unit has a switch to turn it on a button for reset or user number selection and two LED s one that indicates that your voice input is loud enough turns green and one that turns red when the phrase was recognized and the translation playback is about to begin There is no switch other than to turn the unit off to prevent the unit translating a phrase that was incorrectly recognized from playing Scalability phrases domains ability required to do so ease and time Phrases languages and domain changes such as adding specific medical or travel environment phrases sets would have to be done by the manufacturer The supplied documentation supplies a Point of Contact POC for thi
58. on or policies of the U S Department of Justice 7 October 2003 Height 7 0 inches Thickness 1 75 inches Weight Unit measured 1 16 3 3 oz Platform type OS Windows CE handheld computer device Microphone The microphone is attached to the unit and is a noise canceling mike It requires holding down the push to talk button while using The unit allows for connecting an external headset mike The microphone level is adjustable and should be adjusted before using the unit Number of users Enrolled Users per unit This unit is user independent any user may use the system Ruggedness The unit is ruggedized with heavy plastic cover and grips A plastic flip cover protects the touch screen on the unit The manufacturer claims the unit is weather resistant The unit does not appear to be waterproof and would not survive a submersion especially a salt water submersion Maintenance Maintenance consists of recharging the battery pack Battery charging is done using the AC DC converter supplied with the unit The charger uses a standard 120V wall outlet The unit also accepts standard batteries 4 AA and works using the charger as a power source Unit recognition and response time is affected by weak batteries so recharging or battery replacement should be done when the unit begins failing to recognize the majority of the phrases The manufacturer states that recharging takes 4 hours when not in use and 5 6 hours if charging while unit is
59. ou will be observed by an evaluator If you feel you cannot complete a step you may ask for assistance from the evaluator who will otherwise be seated nearby Please begin 1 Open the box 2 Take out and identify included written documentation 3 Take out and identify battery pack 4 Take out and identify Phraselator unit 5 Identify physical features of Phraselator using paper template provided see further instructions on paper template provided 6 Review included written documentation if necessary to help accurately identify items on paper template This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view 25 expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 2 0 Phraselator Features and Controls 2 Front or Top This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view 26 expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 2 2 Back or Bottom CUES EE This document is a research report submitted to the U S Department of Justice This report has not been published by the Departm
60. output This headset was apparently supplied for private language training There is no level adjustment required for the microphone Number of users Enrolled Users per unit Unit is speaker independent This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view 11 expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 Ruggedness This is a commercial off the shelf COTS unit There is no special ruggedization The small size and weight may give it some resistance to light abuse Maintenance The unit is battery operated only Rechargeable batteries could be used but would have to be purchased separate as well as a charger Weak batteries affect recognition accuracy and response time so it is important to replace the batteries when the recognition begins to suffer Unit Testing Results The following sections cover the tests performed on the individual units and reports the results obtained Out of the Box SOCOM requested NAVAIR ORL to perform an out of the box test on the Phraselator This was done because SOCOM felt that this could have direct bearing on SOCOM s plans for fielding these units This test was conducted by having an individual with no previous experience walk in remove the unit from its packaging and begin to learn to
61. rgeted audience as detailed below Voice Response Translator Components The VRT was delivered in 2 types of packages Below is listed the contents for each unit VRT Unit 1 VRT Unit with Headset 1 Megaphone with Connection cable attachment 1 This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view 5 expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 US Coast Guard Training Video Megaphone instructional manual Quick laminated operational guide and Phrase list Voice Response Translator Operating Instructions and Phrase List US Coast Guard Version Handheld microphone unit VRT Power adapter AC DC VRT Power adapter car DC plug in Extra VRT Connection Cable Large Nylon carrying case fits Megaphone and all above items M VRT Unit 2 VRT Unit with Headset Fender Portable Guitar Voice Amplifier Power Adapter AC DC VRT Connection cable unit to amplifier US Coast Guard Training Video Medium Nylon carrying case fits Megaphone and all above items See NOK eS Also were included were 16 extra Operating and Phrase List manuals with key ring laminated quick reference phrase cards Phraselator components Phraselator Computer Unit Power Converter AC DC Battery Pac
62. rrect csse menn 16 Table 11 Phraselator Pink Noise Test Results 17 Table 12 Phraselator White Noise Test Correct 1 7 This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view 2 expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 Table 13 Phraselator White Noise Test Results 17 LIST OF TABLES continued Table 14 Phraselator White Noise Test Correct eee S ere ve aeos pide rines ids 18 Table 15 UT 103 Ambient Noise Level Test Results esee enne nnne eene nne 18 Table 46 LU T 103 Ambient 96 COSC toca 19 UT 103 Pink Noise Tse vel Test cose eq eoa eus Facies traten nde oes aon 19 Table 18 UT 103 Pink Noise Test aod oaa Pee eB Poo Ode Geta sas 20 Table 19 UT 103 White Noise Test ento yere a ehe der MORI 20 Table 20 UT 103 White Noise Test 96 Correct iens tete apium toa Rea eh 21 Table 21 Phraselator 50 Db Ambient Room Noise Response Times eee 46 Table 22 Phraselator 60
63. rvation Task sive Time S Time min min Openbox Participant should 5 0 25 be able to open the box the factory packed box 2 Take out and identify Participant should 5 0 25 included written be able to remove documentation the written documentation from the box and should be aware that it includes instructions for using the device 3 Take out and identify Participant should 5 0 25 battery pack be able to identify and remove the battery pack from the box 4 Take out and identify Participant should 5 0 25 Phraselator unit be able to identify and remove the Phraselator device itself from the box 5 Identify physical features See block diagram 25 4 5 Amm of Phraselator using device template of for paper template provided Phraselator with items see further instructions items to be 5 amp on paper template identified for 6 provided instructions This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 41 7 October 2003 6 Review included written documentation if necessary to help accurately identify items on paper template Participant should realize that they need to review the included written documentation in order to correctly identify the items on t
64. s Correct 1 Go Aft 10 10 10 10 10 8 10 7 2 Arrival Date 10 9 10 9 10 9 10 6 3 What s Your Cargo 10 9 10 9 10 6 10 3 4 Registration Number 10 9 10 10 10 10 10 10 5 Vessel Owner 10 1 10 5 10 1 10 0 6 Steady Course 10 7 10 7 10 7 10 7 7 Maintain Speed 10 9 10 10 10 9 10 10 8 Slow Down 10 10 10 10 10 10 10 10 9 Stop Vessel 10 10 10 9 10 8 10 10 10 Port of Call 10 4 10 2 10 3 10 4 This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view 14 expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 Table 6 Pink Noise Test Correct Phrase 60 Db 70 Db 80 Db 90 Db Correct Correct Correct Correct 1 100 0 100 0 80 0 70 0 2 90 0 90 0 90 0 60 0 3 90 0 90 0 60 0 30 0 4 90 0 100 0 100 0 100 0 5 10 0 50 0 10 0 0 0 6 70 0 70 0 70 0 70 0 7 90 0 100 0 90 0 100 0 8 100 0 100 0 100 0 100 0 9 100 0 90 0 80 0 100 0 10 40 0 20 0 30 0 40 0 Table 7 VRT White Noise Test Results White Noise 60dB 70 dB 80 dB 90 dB Phrase Tries Correct Tries Correct Tries Correct Tries Correct 1 Go Aft 10 10 10 9 10 8 10 6 2 Arriv
65. s purpose Operational modes i e hands free The VRT is hands free capable or the user may hold the unit The microphone automatically turns on when input is detected No adjustment is required by the user Response time During testing if the phrase was recognized the response was always in less than 1 second i e it does not take long to determine that the phrase was not recognized The battery charge level does not appear to affect the response time of the unit Dimensions Width 3 25 inches Height 5 125 inches Thickness 1 inch Weight Unit measured 10 5 oz Platform type OS Embedded proprietary hardware based system Microphone Headset noise canceling mike Microphone is voice activated and no level adjustments are required Number of users Enrolled Users per unit This unit allows up to 7 different users per device Ruggedness Unit is ruggedized with a metal case that could withstand moderate abuse The unit is stated to be weather proof by the manufacturer but it is not waterproof and would not survive submersion especially in salt water Maintenance Maintenance consists of recharging the battery The battery charger is a supplied AC DC converter that plugs into standard wall outlets If the battery s functioning lifespan has expired then the manufacturer recommends returning unit to factory for battery replacement Repairs would have to done by the manufacturer This document is a research report submitted
66. s the phrase set as it appeared in the computer This unit would be most beneficial where power is available for recharging and where a more complete set of phrases would be required for instance in a medical environment or prisoner interrogation in a designated room Universal Translator The UT 103 was clearly intended for the commercial travel environment The VRT and Phraselator had phrases languages meant for law enforcement or military use while the UT s phrase language set was mainly designed to assist users with airports taxis rail commutes and hotel rooms in Europe This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view 22 expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 Appendix A Out of the Box Test Results for Phraselator Tester 1 Responses Note Tester s Observers selection inputs highlighted in green Participant Instruction Set Today you will be participating in an Out of The Box evaluation of a speech translator unit the Phraselator which is capable of translating commands from the English language into multiple foreign languages The purpose of this evaluation is to provide information on a participant s initial contact with the device The evaluation is divided into four sections 1 Participant Entr
67. t requiring a charge then no further testing was done to drain the battery VRT The unit did not require recharging for 8 hours of operation This unit requires the least amount of recharging and uses the least amount of power When unit was recharged it appeared to recharge in about 30 45 minutes Phraselator The unit lasted through about 6 hours of testing using the battery pack and there were problems with trying to get the battery recharged The charger did not appear to work and the low battery popup kept appearing The testing was finished using supplied AA batteries and purchased batteries With a working charging unit it may be best to use the rechargeable batteries or the wall power unit This unit seems to consume the most power of the 3 units tested and using rechargeable batteries or wall power would be the most cost effective way to use it Universal Translator Unit lasted for eight hours of operation Batteries left in the unit overnight went dead Apparently the unit is somehow draining the batteries constantly The unit contains a memo recording feature therefore it is most likely keeping the recorded memo memory refreshed To save battery life with this unit the batteries should be removed when not in use Conclusion After considering the results of the test and unit behavior during testing the seems to be the easiest least intrusive to use device The Phraselator ranks second with a large phrase set and t
68. tion as a reference While you are completing this section you will be observed by an evaluator We are not interested in how fast you complete each step but rather want to observe how an individual with little experience with the device begins using it If you feel you cannot complete a step you may ask for assistance from the evaluator who will otherwise be seated nearby Note that items in single quotes represent a menu name from the device and that PTT stands for the Push To Talk microphone on the device Please begin 1 Insert load battery pack not the 4 AA batteries but the actual battery pack 1 Turn device on 2 Start the Phraselator application by double clicking the Phraselator icon with the stylus 3 Select ArabicFoceProtection SC module by one clicking the ArabicForceProtection SCY menu with the stylus 4 Ensure Settings Options and Translated are selected 5 Ensure category All Phrases is selected 6 Ensure language Arabic is selected 10 Translate say via PTT Pick Different Phrases Do you speak Arabic and Do you speak English and acknowledge correct response with observer or repeat until correct response is obtained and confirmed by observer 11 View input and output volume by first selecting Settings from the menu amp then Controls from the menu Report output volume level reading to observer i e 200 in 243 out 9 Lower output volume to between 8
69. to the U S Department of Justice This report has not been published by the Department Opinions or points of view 8 expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 Phraselator Number of languages The languages that this unit supported are Arabic Dari Pashto and Urdu Since this unit plays back fixed stored responses the potential exists to support any spoken language only limited by the amount of disk space required to hold the playback recordings Number of phrases Unit contains about 326 phrases in the Force Protection domain Phrases are spoken completely and appear on the display in a list box Technology The unit uses the Dynaspeak speech recognizer which is a Hidden Markov Model HMM based system that is speaker independent System Recognition Training No training is required Noise Degradation The system is supplied with a noise canceling microphone that enables the unit to have noise immunity Output Volume Options The unit is designed to operate in a one on one environment with a built in speaker but also contains a line out jack that would allow you to connect it to a voice instrument amplifier This should give sufficient volume for any noise situation The built in speaker measured about 95 Db at the speaker during playback Feedback indicators for users The unit has a switch to turn it on and a touc
70. u find e Select Search from the menu B BS HS S B e This document is a research report submitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 32 7 October 2003 e Select Keyboard from the menu Yes No e Activate Search for Text Box by clicking on the text box asked Yes No ___what is a text box e via stylus on the keyboard Shoot found 2 phrases Yes No e De activate keyboard remove the keyboard from the screen Yes No 15 Exit Phraselator under Settings A No 16 Turn unit off suspend under Start Suspend No END TIME __ 0951 Did the participant complete each of the sixteen items Yes No If no which item s did they not identify Did the participant ask for help at any time Yes No If yes what specifically did the participant ask for help with _ TAGWES Specified selecting altext box Did the participant ever cause you to step in and offer assistance because they were having difficulty with an item MES No If yes what specifically did you have to offer help with __ Additional Comments Tester 2 Responses Tester s Observer s selection inputs highlighted in green Partic
71. ubmitted to the U S Department of Justice This report has not been published by the Department Opinions or points of view 4 expressed are those of the author s and do not necessarily reflect the official position or policies of the U S Department of Justice 7 October 2003 used is only sensitive to 50 Db so measurements referred to as lt 50 Db mean that it s below the meter s ability to make a measurement The last test was to determine the battery life of the unit and recharge times The following two tables summarize the specific items that were analyzed Table 1 Language Features and Operational Parameters Number of languages Number of phrases Types of translations Spoken Word Accuracy Technology Noise degradation Output Volume Vocabulary Feedback indicators for users Scalability phrases domains ability required to do so ease and time Operational modes i e hands free Response time mtr CO ON Cc WIN Table 2 General Unit Characteristics Dimensions Weight Battery required life amp recharge time Screen intensity out in door Front end interface Platform type OS Microphone Number of users Enrolled Users per unit Ruggedness 0 Maintenance Unit Components Each translation unit was delivered to NAVAIR ORL complete with a complement of items for each ta
72. use it The test consisted by having the individual label the important pieces of the unit turning it on and starting up the program navigate some of the menus setting some of the devices parameters and then speaking the phrases and verifying the response The out of the box test procedure and results are transcribed from the originals in Appendix A This test was only performed on the Phraselator Unit Some observations made during the test that are not in Appendix A are outlined below Test room had less than 50 Db noise level 2 The testers were quickly able to understand the unit s operation One tester had a very good computer background the other did not Neither had experience in speech recognition software This experience did not factor in their ability to use the unit Neither tester had a strong accent The first had a slight southern accent and the second had almost no accent at all 3 The most difficult part of the assembly was inserting the rechargeable battery pack It did not seem to fit very well and required a few minutes of manipulation to get it to set in 4 It did not recognize Do you speak Arabic for either tester For the second tester phrases that were not related were translated Follow me Can you walk for example 5 Using the buttons at the top of the unit worked much faster than trying to use the stylus The touch screen did not seem to work well with the stylus 6 The phrase Halt or r
73. was voice activated offered the advantage of hands free operation One disadvantage noted regarding the VRT was the absence of a panic button or mechanism to prevent the unit from broadcasting a phrase that had been incorrectly read Twice during testing Stop Vessel was translated into Vessel Owner It would be advantageous to have a button to immediately stop the unit from translating when situations like this occur One additional disadvantage results from the need to re train the unit when a phrase is not recognized Training does take a significant amount of time and it became apparent during this testing that a single training session of 45 minutes was not sufficient The benefit however of spending the additional time is greater recognition accuracy The shortened key word activation is considered a significant benefit For testing it was easy to memorize the Coast Guard set of trigger words and phrases However this could become burdensome if the phrase set became too large and required the user to constantly refer to the phrase card and eliminating the hands free operation this device affords Phraselator The Phraselator seems better suited to a more relaxed environment in which more attention may be given to operation of the unit The user has the ability to bypass the voice recognition by selecting the phrase using the stylus Additionally the user does not have to learn shorten phrase sets as required for the VRT The user say
74. y Questionnaire 2 Participant Initial Phraselator Contact Instructions Observer evaluating 3 Participant Application Instructions Observer evaluating 4 Participant Exit Questionnaire The Participant Entry Questionnaire is predominantly composed of multiple choice questions developed to gather information on the participant s background and experience with computers This information will be used to assist the Principal Investigator in interpreting results throughout the remaining sections of the Evaluation The Initial Phraselator Contact Instructions are a set of six basic instructions consisting of opening the box the unit came in to reviewing items contained within it This section is intended to test the participant s initial familiarity with the device s hardware components and the supporting contents of the device As part of this exercise the participant will label a block diagram device template of key component features and will use this as a reference for the following section of the Evaluation An evaluator will be present If the participant has tried to complete an instruction and feels that they cannot and would like to ask questions they can ask the evaluator The evaluator will also be evaluating the participant during and after this section is complete The Participant Application Instructions are a set of operational instructions intended to determine if the participant can navigate through key software features During t

Voice Recognition Evaluation Report

Contents

Download Pdf Manuals

Related Search

Related Contents