Automatic speech recognition system and method for aircraft
Contents
1. may be at a maximum volume whereas the call out Change Entertain ment Setting may be at a minimum volume Although the invention has been described with reference to the preferred embodiment illustrated in the attached draw ing figures it is noted that equivalents may be employed and substitutions made herein without departing from the scope of the invention as recited in the claims For example the outbound and inbound communications which are inter cepted and recognized by the avionics system 10 of the present invention are not limited to conversations between the aircraft and ATC but may also include communications between the aircraft and a dispatch or even communications between the pilots or co pilots of other aircraft Tn this manner the system 10 may be used to reduce pilot workload using normal outbound transmissions For example the system 10 may change a frequency setting on a COM or NAV radio set a heading bug or autopilot setting set a altitude bug or autopilot setting set a altimeter bug or 20 25 30 35 40 45 50 55 60 65 14 autopilot setting or set a airspeed bug or autopilot engine setting The system 10 may also configure the FMS with holding pattern confirmed in an outbound transmission and provide an alert to the pilot if the aircraft is still in the holding pattern after the expect further clearance time has elapsed The system 10 may also setup and or modify a VFR or IFR2. Confirm or another similar word which is recog nized by the ASR Once the pilot or co pilot has acknowl edged the change of frequency the system automatically changes the frequency to the appropriate tower frequency The system may be used in a similar manner to automatically navigate the aircraft to the nearest airport find the nearest fuel stop find the closest airport with a rental car determine the best altitude based on wind speed or determine the closest obstacle based on information sensed by a sensor or provided by a database In another embodiment the system of the present invention may be used by a pilot or co pilot to more quickly and easily check the status of an aircraft by combining ASR commands and TTS readbacks For example the system may be used to automatically retrieve a checklist read off items from the checklist and then receive confirmation that functions have been performed in accordance with the checklist A pilot or co pilot will often read offitems on a checklist while the other performs certain operations in accordance with the checklist and then confirms that the items have been performed The system of the present invention may be used to partially automate this process The pilot or co pilot may push the PTC switch or other switch and say something like Pre flight checklist Pressing and then releasing the PTC triggers the ASR to attempt to recognize the pilot s or co pilot s request for a pre
3. istics Rather passengers may be only allowed to control in flight entertainment systems but still be able to request status information such as airspeed altitude maps ETA ETE etc In this regard the system 10 is able to process and respond to simultaneous and differing commands from mul tiple people and respond differently to those requests The system 10 can also access database information based on recognized airports or other locations In this manner the pilot can extract and use information such as frequencies pattern altitude and any other information in the databases 66 Not only can the system 10 respond to a generic command like tune tower and determine which tower and frequency based on location but the system 10 can also respond appro priately to other generic commands such as tune center or tune FSS and use the aircraft s location and information in the database to select the appropriate tower ground clear ance airport center or Flight Service Station frequencies based on location and or destination The system 10 may also be used to synthesize all pilot communications to the passengers such as the pre flight briefing status updates such as ETA ETE airspeed altitude etc provide a guided tour functionality such as pointing out the Grand Canyon on the right warn of turbulent weather on pilot command or as recognized through background moni toring of ATIS AWOS ASOS etc The sy
4. 70 to convert this confirmation to speech and deliver to the pilot via the DAC 108 and speaker 30 The pilot then acknowl edges the change of frequency by operating the push to control PTC button 36 or other device or by saying Con firm or another similar word which would then be recognized by the ASR 68 Once the change of frequency has been acknowledged the command interpreter changes the frequency of the radio without requiring further action from the pilot or co pilot The system may be used in a similar manner to automatically change the heading altitude air speed or any other controllable aspect of the aircraft Importantly the system and method of the present inven tion permit automatic control of certain aspects of an aircraft without requiring extra steps by the pilot or co pilot In the frequency change example above the radio 18 is controlled as a result of the pilot s normal communications with the ATC The pilot is not required to perform any additional steps voice additional commands operate additional switches etc other than the steps he or she would normally perform to confirm a frequency change with the ATC As another example the avionics system 10 may recognize a pilot s or co pilot s confirmation of a Clear to Land com munication and then automatically prompt the pilot or co pilot to begin performing tasks on a pre landing checklist retrieved from one of the databases 66 Similar examples i
5. The computer readable medium can be for example but not limited to an electronic magnetic optical electro magnetic infrared or semi conductor system appa ratus device or propagation medium More specific although not inclusive examples of the computer readable medium would include the following an electrical connec tion having one or more wires a portable computer diskette a random access memory RAM a read only memory ROM an erasable programmable read only memory EPROM or Flash memory an optical fiber and a portable compact disk read only memory CDROM The computer readable medium can even be paper or another suitable medium upon which the program is printed as the program can be electronically captured via for instance optical scan ning of the paper or other medium then compiled inter preted or otherwise processed in a suitable manner if neces sary and then stored in a computer memory The ASR program 68 is preferably coupled with a grammar definition 72 which stores known commands The ASR pro gram accesses the grammar definition in order to recognize spoken words or phrases The ASR program is also coupled with a voice command interpreter 74 The command inter preter is coupled with the displays 14 16 by lines 76 78 with the radios 18 20 by lines 80 82 with the controlled compo US 7 912 592 B2 7 nents 62 by the line 84 and with the sensors 64 and databases 66 by the line 86 The voice
6. also be used by multiple people on the aircraft For example the system may monitor selected communications from pilots co pilots attendants and pas sengers and comply with certain requests from a pilot or co pilot and different or more limited requests from passen gers and or attendants The system of the present invention also prioritizes communications so that for example com US 7 912 592 B2 9 mands from pilots take precedence over simultaneous and or conflicting commands from passengers The system may also prioritize the level of confirmation required before a requested function is performed so that highly critical func tions lower landing gear require at least one confirmation whereas non critical functions change an entertainment option require no confirmation In a related embodiment the avionics system 10 may monitor inbound communications from ATC or other source in an attempt to recognize instructions If an instruction is recognized the command interpreter 74 may control related functions in response thereto For example the ASR program 68 and command interpreter 74 may monitor one of the radios 18 20 recognize an ATC request to change frequency or instructions such as Change COM 2 to XYZ and then automatically change the radio frequency As with the previ ous embodiment the command interpreter 74 may first request confirmation before actually performing the recog nized function The avionics syste
7. the speakers The pilot or US 7 912 592 B2 13 co pilot may then perform some function in accordance with the tutorial and say Next or Check to cause the system to display or speak another item from the tutorial In another embodiment the avionics system 10 can be used as a virtual examiner to monitor a pilot s operation of the aircraft To implement this feature a pilot or co pilot can push one of the PTC switches and then say something like Virtual Examiner The ASR program 68 and command interpreter 74 recognize this command and retrieve a custom program from one of the databases 66 that provides standards for a check ride such as would be required to acquire a pilot s license or endorsements or advancements thereto The com mand interpreter 74 may display items from the check ride on one of the displays 14 16 or may convert the items to speech with the TTS program 70 and one or more of the speakers The pilot may then perform some maneuver in accordance with the check ride while the system 10 monitors his or her per formance as well as the aircraft s performance The system 10 can then provide the pilot with feedback as to whether they met the standards and or constructive criticism to enable to pilot to perform better next time The pilot or co pilot can then say Next or Check to cause the system to display or speak another item from the check ride For all of the above described embodim
8. without forcing the pilot or co pilot to listen to the inbound radio transmission The system 10 may then display that information in text and or graphical form on one of the displays use the TTS 70 and speakers 30 38 46 to present the information to the pilot or use that information to configure the FMS In another embodiment the avionics system 10 may be used to alert a pilot or co pilot of relevant communications Aircraft radios are often tuned to frequencies which are simultaneously used by multiple aircraft at an airport The pilot or co pilot of an aircraft must listen to all of the com munications and then respond or comply with instructions that are relevant to his or her particular aircraft With the present invention the ASR 68 monitors inbound communi cations on the radios 18 20 and then attempts to recognize portions of the inbound communications that identify the aircraft For example the ASR may monitor communications from a tower and then recognize the aircraft s tail number in a communication Once the tail number is recognized the command interpreter 74 may alert the pilot or co pilot by directing the TTS 70 and speakers 30 38 or 46 to provide a speech message such as Relevant Communication The command interpreter 74 may also direct one of the displays 14 16 to provide an alert In another embodiment a pilot or co pilot may use the ASR to control informational messages which are not typically generated by t
9. 3 change to Kansas City Center on 123 45 The pilot responds by push ing the push to talk PTT switch 34 and then saying in an outbound radio communication over the radio 18 Changing to 123 45 N123 Pressing the PTT 34 or any other button or triggering device triggers the ASR 68 to begin monitoring the outbound communication Once the PTT 34 is released the ASR 68 analyzes the pilot s response to recognize words or phrases which may correspond to a control function of the aircraft such as changing the radio frequency In the present example the ASR 68 recognizes the pilot s confirmation as a request to change radio 18 to a frequency of 123 45 The command recognized by the ASR 68 is then passed to the command interpreter 74 After the ASR 68 and command interpreter 74 recognize a frequency change request the com an 5 40 45 55 8 mand interpreter may automatically set the recognized fre quency as the communication radio standby frequency auto matically set the active communication radio frequency to the read back frequency when the transmission is completed automatically set the communication radio frequency to the read back frequency if the frequency is consistent with the flight plan or local operating environment and or ask for a confirmation The command interpreter may require confir mation by displaying something like Change COM 1 to 123 45 on the display 14 or by directing the TTS program
10. 68 to begin monitoring the pilot s commands Once the PTC 36 is released the ASR 68 attempts to recognize words or phrases of the pilot s command which may correspond to a control function of the aircraft such as changing the radio frequency In this example the ASR 68 recognizes the pilot s request to tune one of the radios 18 20 to the airport tower The command interpreter 74 then automatically looks up the frequency of the tower for the current airport from one of the databases 66 based on the current position of the aircraft as determined by a Global Positioning Satellite GPS receiver Once the command interpreter 74 has retrieved the tower frequency for the airport it automatically tunes one of the radios 18 20 to the relevant tower frequency The command interpreter 74 may tune either the active or stand by fre quency of either radio As with the first exemplary embodiment described above the command interpreter 74 may also require confirmation before changing the frequency by displaying something like Tune to tower at XYZ airport on the display 14 or using the TTS program 70 to convert this confirmation to speech for delivery to the pilot The pilot can then acknowledge the change of frequency by operating the PTC button 36 or other device or by saying Confirm or another similar word which would then be recognized by the ASR Once the pilot has acknowledged the change of frequency the command inter preter 74 ch
11. US007912592B2 az United States Patent 10 Patent No US 7 912 592 B2 Komer et al 45 Date of Patent Mar 22 2011 54 AUTOMATIC SPEECH RECOGNITION 56 References Cited SYSTEM AND METHOD FOR AIRCRAFT U S PATENT DOCUMENTS 75 Inventors Joseph L Komer Lenexa KS US 6 163 768 A 12 2000 Sherwood etal 704 235 Joseph E Gepner Olathe KS US 6 720 890 B1 4 2004 Ezroni et al Charles Gregory Sherwood Olathe KS 6 895 380 B2 5 2005 Sepe EEE 704 275 US 7 006 013 B2 2 2006 Mitchell et al 7 089 108 B2 8 2006 Merritt neccsser 701 206 7 415 326 B2 8 2008 Komeretal wu 701 3 73 Assignee Garmin International Inc Olathe KS 2001 0030611 Al 10 2001 O Rourke US 2002 0107694 Al 8 2002 Lerg 2003 0025682 Al 2 2003 Dame x a P 2004 0006412 Al 1 2004 Doose et al Notice Subject to any disclaimer the term of this 2004 0124998 Al 7 2004 Dame patent is extended or adjusted under 35 2005 0203700 Al 9 2005 Merritt U S C 154 b by 941 days 2006 0015338 Al 1 2006 Poussin 2010 0030400 Al 2 2010 Komer etal we 701 3 This patent is subject to a terminal dis claimer OTHER PUBLICATIONS Scansoft User Manual published 2004 p 24 21 Appl No 11 423 316 ras 22 Filed Jun 9 2006 Primary Examiner Richard M Camby 65 A a i 74 Attorney Agent or Firm Samuel M Korte rior Publication Data US 2007 0288128 Al Dec 13 2007 67 ABSTRACT A system and method which implement automatic sp
12. ading of an aircraft a pilot or co pilot can simply push the PTC switch or other control device and say something like Find nearest airport Pressing and releasing the PTC trig gers the ASR to recognize the command and trigger the sys tem to find the closest airport based on sensed information such as the current position heading and altitude of the aircraft The system then displays the name of coordinates of and or the heading to the closest airport and or provides this information audibly with the TTS The system may be used in a similar manner to quickly and easily provide other infor an 5 40 45 65 4 mation to the pilot or co pilot such as the position heading altitude and speed of the aircraft as well as weather condi tions currently experienced by the aircraft or weather condi tions at a selected destination of the aircraft In another embodiment the system of the present invention may be used to alert a pilot or co pilot of relevant communi cations Aircraft radios are often tuned to frequencies which are simultaneously used by multiple aircraft at an airport The pilot or co pilot of an aircraft must listen to all of the com munications and then respond or comply with instructions that are relevant to his or her particular aircraft The system of the present invention monitors inbound communications to the aircraft and then attempts to recognize portions of the inbound communications that identify the
13. aircraft For example the system may monitor communications from a tower and then recognize the aircraft s tail number in a com munication Once the tail number is recognized the system alerts the pilot or co pilot with an audible textual or speech prompt These and other important aspects of the present invention are described more fully in the detailed description below BRIEF DESCRIPTION OF THE DRAWING FIGURES A preferred embodiment of the present invention is described in detail below with reference to the attached draw ing figures wherein FIG 1 is a schematic diagram of selected components of an avionics system in which the system of the present invention may be incorporated FIG 2 is a schematic diagram of the avionics system show ing selected components in more detail The drawing figures do not limit the present invention to the specific embodiments disclosed and described herein The drawings are not necessarily to scale emphasis instead being placed upon clearly illustrating the principles of the inven tion DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The following detailed description of the invention refer ences the accompanying drawings that illustrate specific embodiments in which the invention can be practiced The embodiments are intended to describe aspects of the inven tion in sufficient detail to enable those skilled in the art to practice the invention Other embodiments can be utilized and c
14. anges the frequency to the appropriate tower frequency For example the command interpreter 74 may then change the stand by frequency to the active frequency and or may switch to the tuned radio The system may be used in a similar manner to automatically navigate the aircraft to the nearest airport find the nearest fuel stop find the closest airport with a rental car determine the best altitude based on wind speed or determine the closest obstacle based on infor mation sensed by a sensor provided by a database or other wise available to the system In another embodiment the avionics system 10 may be used by a pilot co pilot or another person to more quickly and easily check the status of an aircraft by combining ASR commands and TTS readbacks For example the system 10 US 7 912 592 B2 11 may be used to automatically read off status information retrieve a checklist read off items from the checklist and then receive confirmation that functions have been performed in accordance with the checklist A pilot or co pilot will often read off items on a checklist while the other performs certain operations in accordance with the checklist and then confirms that the items have been performed The system 10 may be used to partially automate this process and serve as a virtual co pilot The pilot may push the PTC switch 36 or other switch and say something like Pre flight checklist Press ing and then releasing the PTC 36 tr
15. approach route or flight plan based on the pilot s read back to ATC The system 10 may also automatically pull up and go through a checklist upon recognizing data in an outbound transmission For example the system 10 may automatically bring up and or go through a pre landing check list or pre takeoff check list upon recognizing clear to land or clear to take off respectively The system 10 may also respond in this manner to basic position reporting such as downwind turning base and or turning final The system 10 may also provide a reminder to report according to read back of an ATC request or even make the report for the pilot The system may also be useful on the ground For example upon recog nizing taxi instructions the system 10 can display a taxi route for the pilot This taxi route can incorporate hold short instructions such that the pilot is warned if they are approach ing a hold short point too quickly or approaching a runway after recognizing hold short and before recognizing a release or clear to take off While the above functionality is expected to work with transmissions to ATC it can also be utilized with transmissions to a dispatching center and or other aircraft In providing the checklist functionality the system 10 may simply read them using the TTS 70 and listen for the pilot s confirmation using the ASR program 68 Alternatively the system 10 may take a more a
16. command interpreter 74 deter mines the appropriate action to take based on the words and phrases detected by the ASR program 68 the state of the aircraft as sensed by the sensors 64 and or information in the aviation databases 66 The voice command interpreter 74 also causes the TTS program 70 to speak words or phrases and controls the display of information on displays 14 16 The voice command interpreter can also dynamically change the known commands in the grammar definition 72 As illustrated the audio panel 12 includes a number of analog to digital converters ADCs 88 90 92 94 96 coupled between the ASR program 68 and the microphones 32 40 48 52 58 by lines 98 100 102 104 and 106 The microphones convert communications from the pilot co pilot attendant and passenger into analog electrical signals The ADCs con vert the analog voice signals to a digital representation of the analog signal The digital representation of the analog signal is then passed to the ASR program The audio panel 12 also includes a number of digital to analog converters DACs 108 110 112 114 116 coupled between the TTS program 70 and the speakers 30 38 46 50 56 by lines 118 120 122 124 and 126 When commanded by the voice command interpreter 74 the TTS 70 generates a digital representation of speech and delivers the digital sig nals to the DACs The DACs convert the digital signals to corresponding analog voice signals for driving the
17. ctive role by checking one or more of the items itself either completely in the background or checking and confirming each item to the pilot For example when it gets to a landing gear item the system 10 may confirm either to itself or to the pilot that the landing gear it down and locked thereby essentially removing an item from the checklist The system 10 may also add items to the checklist such as asking the pilot to slow the aircraft before deploying the landing gear The system 10 may also pull up the appropriate checklist with little or no pilot intervention Furthermore the system 10 may be able to select which one of a plurality of checklists should be used for any given situation For example the system 10 may be able to detect a nature of an emergency and pull up the appropriate emer gency checklist to deal with emergencies of that nature The system 10 is preferably able to incorporate database and other information such as aircraft status For example the system 10 is able to recognize tune tower and then tune the radio to the tower frequency for the airport at the aircraft s current location or destination if on a flight plan or route The system 10 can also respond to turn on runway lights by looking up the UNICOM frequency and keying the PTT according to the control information stored in the database The system 10 can be eve smarter and recognize that the aircraft is approaching the airport and automaticall
18. dio frequency In the example above the ASR recognizes the pilot s or co pilot s confirmation as a request to change a COM 1 radio to a frequency of 123 45 A command inter preter may then automatically change the frequency of the appropriate radio without requiring the pilot or co pilot to take any further action This allows the system to control a function of the aircraft without requiring any further action beyond responding to ATC as the pilot or co pilot normally would This reduces pilot workload Alternatively the system of the present invention may also provide confirmation by displaying something like Change COM 1 to 123 45 on a display or using the TTS program to convert this confirmation to speech The pilot or co pilot then acknowledges the change of frequency by operating a push to control PTC button such as the PTT switch or other device and or by saying Confirm or another similar word which is then recognized by the ASR Once the pilot or co pilot has acknowledged the change of frequency the sys tem of the present invention automatically changes the fre quency of COM 1 without requiring further action from the pilot or co pilot The system may be used in a similar manner to automatically change the heading altitude air speed or any other controllable aspect of the aircraft The system of the present invention can also be used by multiple people on the same aircraft For example the system may monito
19. ed avionics system or any other avionics system Alternatively the ASR and TTS pro grams may be implemented in a stand alone piece of avionics equipment which interfaces with an avionics system such as the Garmin G1000 In one embodiment the system of the present invention may be used to automatically control a function of an aircraft in response to a pilot s or co pilot s confirmation of instruc tions from an external source such as air traffic control ATC For example the system may be used to automatically change the frequency of an aircraft radio when a pilot or co pilot is instructed to do so by ATC When ATC assigns communica tion frequencies pilots or co pilots commonly read back the frequency assignments to ensure accurate communication In an inbound radio communication to the aircraft ATC may say something like N123 change to Kansas City Center on 123 45 The pilot or co pilot responds by pushing a push to talk PTT switch and then saying in an outbound radio communication from the aircraft Changing to 123 45 N123 With the present invention pressing the PTT or any other button or triggering device triggers the ASR to begin monitoring the outbound communication Once the PTT is released the ASR analyzes the pilot s or co pilot s response to recognize words or phrases which may correspond to a jai 5 30 35 40 45 55 65 2 control function of the aircraft such as changing the ra
20. eech 51 Int CI recognition ASR and text to speech TTS programs to per mit pilots co pilots and other persons to more quickly and GOIC 23 00 2006 01 pa 701 easily perform control and monitoring tasks on aircraft The 52 US Claas E nae tai 701 3 701 36 system may be used to automatically change the frequency of 58 Field of Classification Search 701 3 18 an aircraft radio when a pilot or co pilot is instructed to do so 701 23 36 27 704 235 275 270 256 by ATC 704 240 See application file for complete search history 8 Claims 1 Drawing Sheet qs 3 12 J 70 TTS COMMAND A INTERPRETER FY So 72 US 7 912 592 B2 Mar 22 2011 U S Patent YVANYY9 SISVEVIWO el 99 ie SUOSNIS ne tS Ey hns ae SiNaNodWwad 292 OITIOYINOO cn TANVd olany amp ht j th ias ae e th po AJ e o 97 d T th ge ve Al PS pe bola j A al US 7 912 592 B2 1 AUTOMATIC SPEECH RECOGNITION SYSTEM AND METHOD FOR AIRCRAFT FIELD OF THE INVENTION The present invention relates to avionics systems More particularly the invention relates to a system and method which implement automatic speech recognition and text to speech programs to assist pilots and co pilots in monitoring the status of and controlling the operation of aircraft BACKGROUND Pilots and co pilots are required to monitor a myriad of communication channels sensors and gauges and perform n
21. ent position heading and altitude of the aircraft The com mand interpreter 74 then displays the name and coordinates of the closest airport on the display 14 and or provide this information audibly with the TTS 70 and one of the speakers 30 38 46 The system 10 may be used in a similar manner to quickly and easily provide other information such as the position heading altitude and speed of the aircraft as well as weather conditions currently experienced by the aircraft or an 5 20 25 30 35 40 45 50 55 60 65 12 weather conditions at a selected destination of the aircraft The system may also be used to automatically provide a call out when a sensed condition changes For example the command interpreter 74 and TTS 70 may call out an alert when the aircraft descends below a selected threshold altitude or when weather conditions change as sensed by one of the sensors 64 Other TTS alerts and warnings that may be pro vided by the avionics system include engine fire or other failure warnings and restricted or prohibited airspace warn ings In another embodiment the avionics system 10 may be used to monitor ATIS AWOS and or ASOS along a route or near a destination For example the system 10 may automati cally tune an unused COM or NAV radio to an appropriate frequency and use ASR to recognize information presented by the ATIS AWOS or ASOS The system 10 may capture that information in the background
22. ents the ASR 68 grammar definition 72 and command interpreter 74 may be programmed to quickly recognize and act upon certain com mands frequently used by pilots and co pilots For example the system 10 may quickly recognize the commands Mute Terrain and in response mute a Terrain Awareness and Warn ing System TAWS warning Mute Traffic and in response mute a Traffic Alerting System TAS warning set up ILS and in response take all the necessary steps to set up an instrument landing of an aircraft Tune Tower and in response tune one of the radios to a tower frequency Emer gency Checklist and then automatically retrieve read off and or display such a checklist Request Weather and then tune one of the radios to the nearest AWOS ASOS or ATIS frequency based on the location of the aircraft as sensed by a GPS device The ASR 68 grammar definition 72 and com mand interpreter 74 may also be programmed to recognize a voice command such as say Again and in response call out or display information a second time Importantly the system 10 is set up to recognize voice commands worded similarly to how pilots or co pilots normally speak to one another and or to ATCs and others In all of the embodiments the command interpreter 74 and TTS program 70 may adjust the volume of speech call outs based on the criticality of the information being called out For example the call out Danger Stall Speed
23. f flight of the aircraft a track of the aircraft a desired track of the aircraft an altitude pre select a height above terrain or obstacles a position of other aircraft an auto pilot setting a trim setting a flaps setting a landing gear state an engine setting a manifold pressure an oil tempera ture an oil pressure an exhaust gas temperature a cylinder head temperature a state of an electrical system a vacuum system state an environmental control system state or setting an entertainment system setting an anti icing setting and a V speed setting For example the system 10 may be used to automatically tune one of the radios 18 20 based on a pilot s or co pilot s command and the current position of the aircraft While an aircraft is on the ground pilots often monitor weather and airport conditions on an ATIS frequency then attempt to gain IFR clearance on a clearance frequency then obtain taxi instructions from a ground frequency and then finally obtain takeoff permission on a tower frequency Instead of looking up the numeric frequencies for each of these channels for the current airport and then manually switching between the channels the present invention permits a pilot or co pilot to merely issue a voice command to automatically tune to a particular frequency The pilot may push the PTC switch 36 and say something like Tune tower Pressing the PTC 36 or any other button or triggering device triggers the ASR
24. flight checklist Once recognized the system auto matically retrieves the pre flight checklist and then reads off items on the checklist with the TTS and or displays the items on a display After each item in the checklist is read off or displayed the pilot or co pilot responds by saying something like Check The ASR recognizes the confirmation and then displays or reads off the next item in the checklist Before reading off or displaying the requested checklist the system may also ask for confirmation by displaying or reading off something like Pre flight checklist The pilot or co pilot acknowledges that the proper checklist was retrieved by operating the PTC button or by saying Con firm The system of the present invention may also dynami cally alter the contents ofa checklist based on the current state of the aircraft or another sensed parameter For example if a pre landing checklist is requested and retrieved and the checklist requires the pilot or co pilot to lower an aircraft s landing gear the system may first sense the current air speed of the aircraft and instruct the pilot or co pilot to reduce the air speed to an acceptable level before lowering the landing gear Tn another embodiment the system of the present invention may be used to call out requested data or other information to a pilot or co pilot For example rather than manually deter mining the closest airport based on the current position and he
25. hanges can be made without departing from the scope of the present invention The following detailed description is therefore not to be taken in a limiting sense The scope of the present invention is defined only by the appended claims along with the full scope of equivalents to which such claims are entitled The present invention can be implemented in hardware software firmware or a combination thereof In a preferred embodiment however the invention is implemented with an automatic speech recognition ASR program and a text to speech TTS program integrated into an avionics system such as the G1000 integrated avionics system provided by Garmin International Inc Certain components of an exem plary control avionics system are broadly referred to by the numeral 10 in FIG 1 The avionics system equipment and computer programs illustrated and described herein are merely examples of computer equipment and programs that may be used to implement the present invention and may be US 7 912 592 B2 5 replaced with other avionics equipment and computer pro grams without departing from the scope of the present inven tion Referring again to FIG 1 the avionics system 10 in which the ASR and TTS programs are incorporated may include among other components an audio panel 12 and one or more displays 14 16 and radios 18 20 coupled with the audio panel 12 The audio panel 12 may serve several stations within an aircraft For example
26. he avionics system The TTS is used to generate the messages The pilot or co pilot commands or programs the avionics system to generate the messages For example the pilot may want to know when a certain altitude airspeed or position is reached Using ASR the pilot may command Advise altitude 1 000 feet or Advise altitude flight level 120 descending ascending and departing would also be options Other information would include airspeed position heading fuel remaining in time Ibs Kg gallons liters etc time fuel burn above or below a rate and other aircraft parameters Similarly a pilot or co pilot may use the ASR and TTS in a Callout mode to callout airspeed altitude fuel flow etc The command interpreter would cause a Callout to occur when the callout parameter changed significantly In another embodiment the avionics system 10 can be used as a virtual instructor to teach a pilot or co pilot how to use all functions of the system 10 and or aircraft To implement this feature a pilot or co pilot can push one of the PTC switches and then say something like Virtual Instructor The ASR program 68 and command interpreter 74 recognize this command and retrieve a custom program from one of the databases 66 that provides a tutorial The command inter preter 74 may display items from the tutorial on one of the displays 14 16 or may convert the items to speech with the TTS program 70 and one or more of
27. iggers the ASR 68 to attempt to recognize the pilot s request for a pre flight check list Once recognized the command interpreter 74 retrieves the pre flight checklist and then instructs the TTS 70 to read offitems on the checklist via the speaker 30 or 46 Alternately or additionally the command interpreter 74 may display items from the checklist on one of the displays 14 16 After each item in the checklist is read off or displayed the pilot responds by saying something like Check The ASR pro gram 68 recognizes the confirmation and the command inter preter 74 then displays and or reads off the next item in the checklist The pilot may also confirm using the PTC switch such as by pressing the PTC 36 twice in rapid succession Before reading off or displaying the requested checklist the command interpreter 74 may require confirmation by displaying or reading off something like Pre flight check list The pilot acknowledges that the proper checklist was retrieved by operating the PTC switch 36 or by saying Con firm The command interpreter 74 may also dynamically alter the contents of the checklist based on the current state of the aircraft For example if a pre landing checklist is requested and retrieved and the checklist requires the pilot to lower an aircraft s landing gear the command interpreter 74 may first acquire the current air speed of the aircraft through one of the sensors 64 and instruct the pilot t
28. m 10 may also monitor and record a selected duration of all inbound communications and then attempt to recognize data and or commands in the communi cations The duration may be fixed adjustable and or sliding For example particularly when expecting the receive ATC clearance instructions the pilot may instruct the system to record and recognize all audio for the next two minutes All recognized data and commands are then stored until some future time in a scratchpad memory for possible later use by the pilot or co pilot Alternatively the system 10 may monitor all audio and store recognized data and commands in the scratch pad for some duration such as two minutes or until a buffer is full at which point older data and commands drop out of the scratch pad The avionics system 10 may also recognize ATC delivered clearance instructions and then automatically program the recognized instructions into a flight management system FMS such as an auto pilot system The system 10 may also recognize automated terminal information system ATIS messages or AWOS or ASOS messages and then display recognized information on one of the displays 14 16 convert it to speech with the TTS 70 and associated speakers and or use it to configure the FMS The avionics system 10 may also monitor and recognize taxi and airport clearance instructions that are combined with the airport database and GPS position to warn the pilot of unintentional runway incursi
29. nclude the recognition of Clear to Take off Turning Final and Turning Base transmissions or commands or other normal radio traffic to retrieve pre take off pre land ing or other appropriate checklists The system of the present invention may store and use several different checklists for the aircraft For example the system may manage emergency checklists pre flight check lists in flight checklists landing checklists post flight checklists and any other checklist for the aircraft The system may run through the checklists with the pilot and or co pilot or may substantially automatically run through the checklists on its own For example when the system detects that the pilot is preparing to land such as by intercepting an outbound radio transmission detecting that the aircraft has entered controlled airspace or a landing pat tern or detecting that the aircraft is approaching a destination airport the system may automatically pull up the pre landing checklist and begin reading off items and listening for the pilot s Check Alternatively for example upon recogniz ing Turning Final the system may automatically run through the pre landing checklist in the background without interrupting the pilot to autonomously confirm that every thing is properly configured for landing and alert the pilot if anything is amiss such as landing gear not showing fully down and locked The avionics system 10 can
30. nd then manually switching between the channels the present invention permits a pilot or co pilot to merely issue a voice command to automatically tune to a particular frequency The pilot or co pilot may push the PTC switch and say something like Tune tower Pressing the PTC or any other button or triggering device triggers the ASR to begin monitoring the pilot s or co pilot s commands Once the PTC is released the ASR attempts to recognize words or phrases of the pilot s or co pilot s command which may correspond to a control function of the aircraft such as changing the radio frequency In the example above the ASR recognizes the pilot s or co pilot s request to tune a communication radio to the airport tower The system then automatically looks up the tower frequency for the current airport based on the current position of the aircraft as determined by a Global Positioning Satellite GPS receiver or other source of position informa US 7 912 592 B2 3 tion such as inertial navigation coupled with or incorporated in the avionics system and automatically tunes a communi cation radio to the relevant tower frequency The system may also provide confirmation by displaying something like Tune to tower at XYZ airport on a display or using the TTS program to convert this confirmation to speech The pilot or co pilot then acknowledges the change of frequency by operating the PTC button or other device or by saying
31. o reduce the air speed to an acceptable level before lowering the landing gear The system 10 may perform and or confirm checklist items automatically or semi automatically For example upon reaching an item for lowering the landing gear on a pre landing check list the system 10 may confirm that the air speed is low enough and use the TTS 70 to say something like Airspeed is acceptable shall I lower the landing gear and then autonomously lower the landing gear upon recognizing the pilot s Check Alternatively upon reaching an item for lowering the landing gear on the pre landing check list the system 10 may confirm that the airspeed is low enough and use the TTS 70 to say something like Airspeed is acceptable lowering the landing gear and then autonomously lower the landing gear without requiring further the pilot intervention In another embodiment the avionics system 10 may be used to call out or display requested data sensed data or other information For example rather than manually deter mining the closest airport based on the current position and heading of an aircraft a pilot or co pilot can simply push one of the PTC switches 36 44 or other control device and say something like Find nearest airport Pressing and releasing the PTC triggers the ASR program 68 to recognize the com mand and triggers the command interpreter 74 to find the closest airport based on sensed information such as the cur r
32. on to confirm by operating a switch two times in succession 6 The method as set forth in claim 5 wherein the switch is selected from the group consisting of a push to talk switch and a push to control switch 7 The method as set forth in claim 1 wherein the moni toring step is triggered by a push to talk switch on the air craft 8 The method as set forth in claim 1 wherein the speech recognition program compares the portion of the outbound communication to a database of known commands corre sponding to the function in order to match the recognized word or phrase to one of the known commands in the data base
33. ons or visually confirm to the pilot that he is cleared to take cross a runway The system 10 may also monitor and recognize taxi instructions and plot a map on the airport diagram and then give turn by turn direc tions The avionics system 10 may also recognize an incoming approach instruction or a pilot s or co pilot s outbound con firmation of such an instruction and then display a list of possible approaches retrieved from the databases 66 The command interpreter 74 may determine and display an optional approach based on information from the sensors 64 such as wind speed airport elevation aircraft altitude aircraft position etc In another embodiment the avionics system 10 may be used to automatically control a function of an aircraft by combining ASR commands and information about the state of the aircraft from the sensors 64 and or information stored in the databases 66 The information about the state of the air craft may be any of the following a position of the aircraft a heading of the aircraft an altitude of the aircraft a speed of the aircraft a flight plan for the aircraft a fuel level a wind speed experienced by the aircraft a wind direction experi 20 25 30 35 40 45 50 55 60 65 10 enced by the aircraft a temperature experienced by the air craft a pressure experienced by the aircraft a weather con dition currently experienced or to be experienced by the aircraft a phase o
34. ormat LCD dis plays The radios 18 20 are preferably dual integrated radio modules that provide IFR oceanic approved GPS VHF navi gation with ILS and VHF communication with transceivers and 8 33 kHz channel spacing The audio panel 12 may be coupled with relays control lers and or other equipment to control certain aircraft com ponents broadly referred to by the numeral 62 in FIG 1 For example and as explained in more detail below the avionics system 10 may be used to change a frequency of one of the radios 18 20 set a heading of the aircraft set an altitude of the aircraft set a holding pattern of the aircraft alert an Expect Further Clearance instruction configure an auto pilot of the aircraft set an approach of the aircraft set an air speed of the aircraft set or change a flight plan for the aircraft draw a VFR approach for the aircraft set a barometric pressure correction setting of the aircraft change a navigation setting of the aircraft control the aircraft s engine and or propeller and control the lowering and raising of the aircraft s landing gear and or flaps These are only examples of aircraft components which may be controlled by the system of the present inven tion The audio panel 12 may also be directly or indirectly coupled with sensors 64 or other devices which sense the state of certain aspects of the aircraft For example and as described in more detail below the avionics system 10 may receive in
35. puts indicative ofa position of the aircraft a heading of the aircraft an altitude of the aircraft an attitude of the aircraft a speed of the aircraft a flight plan for the aircraft fuel level a wind speed experienced by the aircraft a wind direction and or angle of attack experienced by the aircraft a temperature experienced by the aircraft and a weather con 20 25 30 35 40 45 50 55 60 65 6 dition currently experienced or to be experienced by the air craft These are only examples of inputs related to the state of the aircraft The audio panel 12 may also be coupled with various databases broadly referred to by the numeral 66 in FIG 1 The databases may include for example the ATIS clearance ground and tower frequencies for all known airports stored waypoints and other navigation information pre flight pre landing and pre taxi checklists and other checklists stored flight plans and general information about the aircraft air ports used by the aircraft hotels and restaurants near the airports topographic data obstacle locations and heights terrain elevation data airplane configuration settings pilot profiles arrival procedures departure procedures approach procedures airport diagrams runway and taxiway data weather frequencies fuel availability and pricing user defined waypoints VORs NDBs intersections airways and airspace boundaries For example one or more of the data ba
36. r selected communications from pilots co pilots attendants and passengers and comply with certain requests from a pilot or co pilot and different or more limited requests from passengers and or attendants The system also priori tizes communications so that for example commands from pilots take precedence over simultaneous and or conflicting commands from passengers The system may also prioritize the level of confirmation required before a requested function is performed so that for example highly critical functions lower landing gear require at least one confirmation whereas non critical functions change an entertainment option require no confirmation In another embodiment the system of the present invention may be used to automatically control a function of an aircraft by combining ASR commands and information about the state of the aircraft and or other information stored in a data base For example the system may be used to automatically tune an aircraft radio based ona pilot s or co pilot s command and the current position of the aircraft While an aircraft is on the ground pilots often monitor weather and airport condi tions on an ATIS frequency then attempt to gain IFR clear ance on a clearance frequency then obtain taxi instructions from a ground frequency and then finally obtain takeoff permission on a tower frequency Instead of looking up the numeric frequencies for each of these channels for the current airport a
37. respond ing to the ATC clearance Then the system 10 can use the pilot s outbound read back as a confirmation of data recog nized from the inbound ATC instructions In this manner the 20 30 40 45 50 55 16 ASR program s 68 speech recognition can be supported supplemented or augmented from three sources inbound ATC transmissions aircraft status and database information and the pilot s outbound transmission Finally the ASR program 68 can be used to create text messages from the pilot and or co pilot The system 10 can transmit those text messages to ground stations or other air craft and receive other text messages from the ground stations or other aircraft Finally the TTS 70 read the received text messages to the pilot Having thus described the preferred embodiment of the invention what is claimed as new and desired to be protected by Letters Patent includes the following 1 A method of controlling a function of an aircraft the method comprising the steps of monitoring an outbound communication of the aircraft which relates to a request for or confirmation of the function using a speech recognition program to analyze at least a portion of the outbound communication to recognize a word or phrase corresponding to the function prompting a person on the aircraft to confirm that the word or phrase recognized by the speech recognition program does in fact correspond to the function and when the per
38. ses 66 may be of the type provided by Jeppesen FIG 2 illustrates the ASR and TTS programs 68 70 and other components of the audio panel 12 in more detail The ASR and TTS programs are stored in or on computer read able medium residing on or accessible by the avionics system 10 For example the ASR and TTS programs may be stored in memory residing in or accessible by the audio panel 12 or in another component of the avionics system 10 The ASR pro gram 68 can be any conventional speech recognition software such as the VoCon 3200 provided by Nuance Likewise the TTS program 70 can be any conventional text to speech soft ware such as the Real Speech Solo program provided by Nuance The computer programs preferably comprise an ordered listing of executable instructions for implementing logical functions in components of the avionics system 10 The computer programs can be embodied in any computer readable medium for use by or in connection with an instruc tion execution system apparatus or device such as a com puter based system processor containing system or other system that can fetch the instructions from the instruction execution system apparatus or device and execute the instructions In the context of this application a computer readable medium can be any means that can contain store communicate propagate or transport the program for use by or in connection with the instruction execution system appa ratus or device
39. son confirms that the word or phrase corre sponds to the function automatically performing the function 2 The method as set forth in claim 1 wherein the function is selected from the group consisting of changing a frequency of a radio on the aircraft setting a heading of the aircraft setting an altitude of the aircraft setting a holding pattern of the aircraft alerting an Expect Further Clearance instruction configuring an auto pilot of the aircraft performing a check list setting an approach of the aircraft setting an air speed of the aircraft setting or changing a flight plan for the aircraft drawing a VFR approach of the aircraft setting a barometric pressure correction setting of the aircraft changing a naviga tion setting of the aircraft setting a transponder code loading a departure procedure or arrival procedure and setting a descent or climb rate 3 The method as set forth in claim 1 wherein the person on the aircraft is a pilot or co pilot 4 The method as set forth in claim 1 wherein the prompt ing step is selected from the group consisting of the steps converting the recognized word or phrase to speech with a text to speech program and delivering the speech to the per son and displaying the recognized word or phrase on a dis play and awaiting confirmation from the person on the air craft 5 The method as set forth in claim 4 wherein the prompt ing step further includes the step of requiring the pers
40. speakers Operation The avionics system 10 described and illustrated herein may be used to assist pilots and co pilots in controlling certain aspects of their aircraft obtaining useful information about their aircraft performing operations pursuant to a checklist receiving status alerts about their aircraft recognizing rel evant communications directed to their aircraft and many other functions described herein and equivalents The system 10 may also be used by aircraft passengers to change between entertainment radios and or telephones and by flight atten dants to obtain useful information and to communicate with the pilots co pilots passengers and others Various opera tional embodiments of the invention are described below however the present invention is not limited to these particu lar exemplary embodiments In one embodiment the avionics system 10 may be used to automatically control a function of an aircraft in response to a pilot s or co pilot s confirmation of instructions from an external source such as an air traffic control ATC For example the system 10 may be used to automatically change the frequency of one of the radios 18 20 when a pilot or co pilot is instructed to do so by ATC When ATC assigns communication frequencies pilots or co pilots commonly read back the frequency assignments to ensure accurate com munication In an inbound radio communication over the radio 18 ATC may say something like N12
41. stem may also provide increased situational aware ness to the pilot by warning him or her when the system 10 detects any failure of avionics engine or airframe or approaching or entering restricted airspace in 3 dimensional space latitude longitude and altitude The system 10 can also take more positive action upon detecting a failure or other emergency such as turning on call outs when the radio is tuned to 121 5 and otherwise helping the pilot in an emer gency such as by finding the nearest airport setting way points tuning the radio to an emergency frequency or that of the nearest airport s tower and well as helping the pilot determine the nature of the emergency Rather than just listen to outbound transmissions the sys tem 10 may also listen to inbound transmissions from ATC For example the system may listen for the aircraft s tail number and attempt to recognize information associated with that tail number The ASR program 68 may use information stored in the databases 66 to aid in that recognition For example the ASR program 68 may try to match ATC trans missions with a pre planed flight plan or route the aircraft s current location the aircraft s heading or destination or any thing associated with the aircraft of flight from which certain information might be expected from ATC The system 10 can decode the ATC instructions and setup the autopilot or con firm autopilot settings highlighting anything not cor
42. the audio panel 12 may have inputs and outputs for a pilot s station referred to by the numeral 22 a co pilot s station referred to by the numeral 24 a flight atten dant s station referred to by the numeral 26 and a passenger s station referred to by the numeral 28 The particular stations illustrated and described herein are examples only any num ber of stations may be served by the audio panel The pilot s station 22 may include a speaker 30 a micro phone 32 a push to talk PTT switch 34 and a push to control PTC switch 36 Similarly the co pilot s station 24 may include a speaker 38 a microphone 40 a push to talk PTT switch 42 and a push to control PTC switch 44 The speakers 30 38 and microphones 32 40 are conventional and are preferably combined in aviation headsets A separate speaker 46 and microphone 48 may be positioned somewhere between the pilot s and co pilot s stations 22 24 to permit the pilot and co pilot to receive and transmit radio communica tions without their headsets The flight attendant s station 26 may also include a speaker 50 and a microphone 52 which may be stand alone compo nents or integrated into a headset and a push to control PTC switch 54 Similarly the passenger s station 28 may include a speaker 56 and a microphone 58 which may be stand alone components or integrated into a headset and a push to control PTC switch 60 The displays 14 16 are preferably large f
43. umerous control functions while operating aircraft Such monitoring and control functions are becoming increasingly more difficult and critical as the complexities of aircraft increase Avionics systems have been developed to automate some of the operational and monitoring requirements of air craft however pilots and co pilots still must manually moni tor and operate most aspects of their aircraft Accordingly there is a need for an improved avionics system that overcomes the limitations of the prior art More particularly there is a need for a system and method which reduces the amount of time pilots and co pilots spend on monitoring and control functions so they can concentrate on other required tasks SUMMARY OF THE INVENTION The present invention solves the above described prob lems and provides a distinct advance in the art of avionics systems More particularly the present invention provides a system and method which implement automatic speech rec ognition ASR and text to speech TTS programs to permit pilots co pilots and other persons to more quickly and easily perform control and monitoring tasks on aircraft The present invention is preferably implemented with an ASR program and TTS program which are integrated into an avionics system such as the G1000 integrated avionics system provided by Garmin International Inc The ASR and TTS programs may be incorporated into an audio panel or other component of the G1000 integrat
44. y turn on the runway lights if itis after sunset The system 10 can also respond to commands such as request weather to autono mously seek and present weather data from ATIS AWOS and or ASOS In doing so the system 10 can change an altimeter setting according to that recognized data and or display the data graphically or textually The system 10 can do so in the background without bothering the pilot or co pilot The system 10 may also be used to engine and airframe controls such as RPM manifold pressure mixture landing gear and flaps on command after checking that it is safe to do so and or after receiving confirmation The system 10 can also use the TTS 70 to read off fight variables or call outs such as US 7 912 592 B2 15 airspeed altitude vertical speed heading RPM fuel remain ing etc Such call outs may be configurable through the system 10 such that they can be turned on and off on com mand In this manner the system 10 can be used to control other warning systems such as a TAWS or TCAS The system 10 may also be used to find not only the closest airport but also the closest airport meeting some criteria such as rental car availability fuel type available available services etc The system 10 is preferably smart enough to recognize not only data and commands but also who those commands are from For example the system 10 may not respond to com mands from a passenger that would impact flight character
Download Pdf Manuals
Related Search