Home

Method of controlling a device to perform an activity

1. user engages the setup process through a wireless portable controller e g portable controller 108 not through a sta tionary PC based interface An advantage of the present invention is that the user is no longer required to setup devices by sitting in front of a personal computer computer client 106 having a fixed location which may not be anywhere near the CE devices at issue Having the PC based set up interface in a location that is not proximately located to the devices that are being setup can result in significant inconvenience for the user For example certain information requires physical interaction with the CE device such as identifying and label ing input cycles and determining IR delays which dictates how quickly commands can be sent to the device e g the time it takes a television to change channels upon receiving a channel entry command or the time it takes for the television to turn on upon receiving a power on command The present invention obviates the inconvenience of having to run back and forth between the device and the personal computer in order to input the requested set up information IV Example Computer System FIGS 1 10 are conceptual illustrations allowing an expla nation of the present invention It should be understood that embodiments ofthe present invention could be implemented in hardware firmware software or a combination thereof In such an embodiment the various components and steps woul
2. Inc San Jose Calif as a platform for the system compo nents A wireless access point 182 provides a central point for connectivity in a wireless network and always on connectiv ity necessary for tracking states of the system components Additionally a wireless access point 182 can provide a con nection point between a wired and wireless network In an embodiment using a server as discussed above a wireless access point 182 serves as a platform for server 208 In alter US 7 668 990 B2 9 native server based embodiments of the present invention server 208 can actually be located on a number of different device platforms in addition to a wireless access point such as wireless access point 182 including a bridge device such as bridges 130 130 a personal digital assistant a per sonal computer or the like System 100 also includes a plurality of infrared serial bridges 130 130 4 which comply with the IEEE 802 11 b standard for wireless communications Each infrared se rial bridge 130 interacts with one or more components shown infrared serial bridge 130 a interacts with television 104 a cable box 118 and media player 116 Infrared serial bridge 130 5 interacts with tuner 114 or any type of propri etary device that rely on infrared serial communications pro tocols as would be apparent to one skilled in the relevant art s Infrared serial bridge 130 c interacts with thermostat 112 Infrared
3. URL http www griffintechnology com downloads 6 282 714 Bl 8 2001 Ghori et al pdf manuals Total Remote user manual pdf printed Feb 23 6 288 716 B1 9 2001 Humpleman et al 2004 6 292 554 B1 9 2001 Oden et al International Search Report for International Application No PCT 6 317 028 B1 11 2001 Valiulis US03 20189 Issued on Oct 29 2003 6 317 718 Bl 11 2001 Fano Global Cach GC 100 Network Adapter Control amp Automate 6 321 092 11 2001 Fitch et al Common Devices Over A Network GC 100 Network Adapter URL 6 349 352 Bl 2 2002 Lea http wwwglobalcache com products gc index html Oct 11 2004 6 359 636 3 2002 Schindler et al 2 pages 6 363 434 3 2002 Eytchison GC 100 Key Features and Benefits GC 100 Network Adapter 6 388 399 B1 5 2002 Eckel et al URL http wwwglobalcache com products gc features html Oct 6 405 261 Bl 6 2002 Gaucher 11 2006 2 pages 6 463 343 10 2002 Emens et al Global Cach GC 100 Network Adapter Control amp Automate 6 496 135 12 2002 Darbee Common Devices Over A Network Product Data Sheet 2 pages 6 512 754 B2 1 2003 Feder et al 2004 6 563 430 5 2003 Kemink et al IPL TSFI244 Two Serial Four Flex and Four IR Port IP Link 6 587 067 B2 7 2003 Darbee et al Ethernet Control Interface With IR Learner Extron IPL T SF1244 6 597 374 Bl 7 2003 Baker et al IP Link Products URL http www extron com product product 6 600 421 B2 7 2003 Freeman asp i
4. tual plug and play protocol interface as described above are automatically discovered and identified in controller infor mation window 502 Controller information window 502 also specifies the room s or region s that a discovered wireless controller has been assigned In other embodiments a user can manually input or modify device information including the room region assignment about a wireless controller Referring back to FIG 3 at step 309 any non legacy devices within the designated room or region are detected and assigned As discussed system components having a plug and play protocol interface are automatically discovered and relevant device information about the discovered component is collected In an embodiment the user has the option of excluding the detected non legacy device from being con trolled within the controlled environment e g system 100 FIG 6 illustrates a non legacy device control screen 600 that can be included as part of a set up wizard for assigning non legacy devices according to embodiment of the present invention Device information window 602 is populated with all non legacy devices communicatively coupled to the con trolled environment e g system 100 Having a discovery protocol interface non legacy devices are automatically dis covered and identified in device information window 602 Device information window 602 also specifies the room s or region s that a discovered non legacy devic
5. gram Product for Managing Controlled Residential or Non Residential Environments U S patent application Ser No 10 382 897 and the application entitled Method System and Computer Program Product for Managing Controlled Residential or Non Residential Environments U S patent application Ser No 10 180 500 both of which are incorpo rated herein by reference in their entireties As described in these applications various methods and systems can be pro vided to manage the distribution of information including video audio voice text graphics control messages etc to other devices and or applications within the controlled envi ronment Such devices and or applications include but are not limited to communications equipment such as tele phones intercoms etc entertainment systems such as televisions CD DVD players gaming applications stereos etc monitoring systems such as security cameras baby monitors etc safety security systems such as fire alarms sprinkler systems locks on doors or windows etc personal computers such as desktops notebooks notepads personal digital assistants etc cooking appliances such as ovens coffee makers electrical food beverage warmers etc com fort systems such as heating and air conditioning HVAC humidifiers dehumidifiers air purifiers light switches light dimmers etc power outlets power supplies or the like example of such contro
6. 519 878 A 5 1996 Dolin Jr Continued OTHER PUBLICATIONS Total Remote Software Griffin Technology Downloads 1 page printed from Internet URL http www griffintechnology com software software totalremote html printed Feb 23 2004 Continued Primary Examiner Niketa I Patel 74 Attorney Agent or Firm Sterne Kessler Goldstein amp Fox P L L C 57 ABSTRACT Activity based or experienced based operations are created and defined during an initialization or setup session for one or more system component with minimum or no user interac tion The system components include legacy and or non legacy devices which are configured and managed within a controlled environment such as a home or business peer to peer or client server network During an initialization or setup session relevant device information and device meta data are collected via a conventional or discovery protocol interface For a legacy device that is not automatically dis coverable a user adds the legacy device to the controlled environment by inputting the device information The device information can also be automatically collected from an accessible database Once the device information has been collected the system components are configured to support one or more activity based or experienced based operations 27 Claims 12 Drawing Sheets 100 106 b 9 BB x Media Center Edition on
7. Frank et al 2003 0191826 A1 10 2003 Bellinger et al 6 032 202 A 2 2000 Lea et al 2003 0204582 Al 10 2003 Shimoda et al 6 049 711 4 2000 Ben Yehezkel et al 2003 0207668 Al 11 2003 McFarland et al 6 052 750 A 4 2000 Lea 2003 0208595 Al 11 2003 Gouge et al 6 085 236 7 2000 Lea 2003 0220988 Al 11 2003 Hymel A A A 6 131 028 10 2000 Whitington 2003 0224735 Al 12 2003 Moursund et al 6 148 205 11 2000 Cotton 2003 0224807 Al 12 2003 Sinha et al 6 154 745 11 2000 Kari et al 2003 0236890 A1 12 2003 Hurwitz et al 6 167 046 12 2000 Terada et al 2004 0003051 Al 1 2004 Krzyzanowski et al 6 170 007 BI 1 2001 Venkatraman et al 2004 0003073 Al 1 2004 Krzyznowski et al 6 198 479 B1 3 2001 Humpleman et al 2004 0046677 Al 3 2004 Dresti et al DEDOS BL 2004 0047347 Al 3 2004 Worry et al 6 199 136 B1 3 2001 Shteyn 2004 0054789 Al 3 2004 Breh et al 2004 0071136 Al 4 2004 Laumen et al 6 208 866 3 2001 Rouhollahzadeh et al 6218931 BI 4 2001 Asghar et al 2004 0098456 Al 5 2004 Krzyzanowski et al 6 222 530 Bl 4 2001 Sequeira 2005 0015764 Al 1 2005 Gaur 6 223 348 4 2001 Hayes et al 2005 0108369 Al 5 2005 Sather etal 709 220 6 243 707 Bl 6 2001 Humpleman et al 6 243 772 Bl 6 2001 Ghori et al OTHER PUBLICATIONS 6 259 707 7 2001 Dara Abrams et al 6 266 612 BL 7 2001 Dussell et al Total Remote v2 0 user guide 29 pages printed from 6 275 865 8 2001 Zou Internet
8. HOSE q 901 901 00 U S Patent Feb 23 2010 Sheet 3 of 12 US 7 668 990 B2 300 301 Add or Remove Rooms Assign WiFi IR Controllers to Rooms Assign Detected UPnP Networked Devices to Rooms Add Legacy Devices to Rooms Configure Devices 39 End 309 3 US 7 668 990 B2 Sheet 4 of 12 Feb 23 2010 U S Patent ev Sis 80v cOv x ro m ESI LUE I 202262 ee Sed 00 US 7 668 990 B2 Sheet 5 of 12 Feb 23 2010 U S Patent 80v Sls vOv cov US 7 668 990 B2 Sheet 6 of 12 Feb 23 2010 U S Patent 9 913 209 were SU UD PARIO 1 JEUN urgos eu OF NUSU BYE SEY yen pune SI 0 ix 77007777517 223 50 i E NETTE rr gt lt SE we RE ne 009 US 7 668 990 B2 Sheet 7 of 12 Feb 23 2010 U S Patent 9 Sls A E n ife corii noct ilt ME OOS gud 7 duan ur uoo waopdoup eq padnycltip aq po i 931p ji T ff 51004 oj sa3iasq TCI s t CPC RP MU SiS DR tire BEAT US 7
9. any type of device or peripheral that is capable of communicating via a discov ery protocol interface e g interfaces defined by the Univer sal Plug and Play UPnPTM Forum Having a discovery 20 25 30 35 40 45 50 55 60 65 4 protocol interface non legacy devices are specially adapted for discovery and control by other devices A legacy device on the contrary is defined herein to include any type of device or peripheral that is not capable of communicating via a discovery protocol interface Lacking a discovery protocol interface legacy devices therefore are not conventionally adapted for discovery and control by other devices According to embodiments of the present invention during the initialization or setup phase the system component is configured to perform one or more activity based or experi enced based operations with minimum or no user interaction For non legacy devices relevant information about the sys tem component is collected via a discovery protocol inter face For legacy devices a variety of system interfaces are provided for receiving the relevant information as described in greater detail below In an embodiment the activity based or experience based operations include one or more favorite settings for the system component For example a favorite setting can be established for television programming audio video record ings room temperature hot tub controls cloc
10. manufacturer name for each of the devices a model number of each of the devices and a device type for each of the devices wherein such device information is col lected via a network discovery protocol accessing metadata that specifies unique requirements per taining to control of the plurality of devices including at least one of a device activation requirement tuning requirement and input state selection requirement accessing a sequence of device specific commands based on said metadata and associated with the plurality of devices sending said sequence to the plurality of devices whereby each device specific command from said sequence is executable to control a corresponding device upon receipt of said device specific command at said corre sponding device defining a plurality of regions within a controlled environ ment and assigning each of the plurality of devices to at least one of said regions within the controlled environment 2 The method according to claim 1 wherein said accessing step comprises the step of querying a storage medium to access metadata for control ling the plurality of devices 3 The method according to claim 2 wherein said storage medium is coupled locally to the plurality of devices 4 The method according to claim 2 wherein said storage medium is remotely located from the plurality of devices 5 The method according to claim 1 further comprising the step of sending the sequence o
11. step of adding or removing the one or more system components over a network discovery protocol interface on a con troller 13 The method of claim 1 wherein said assigning step comprises the step of adding or removing the one or more system components over a user input interface on a controller 14 The method according to claim 1 wherein said network discovery protocol is Universal Plug and Play UPnP 15 The method of claim 1 wherein said defining step comprises the step of adding or removing regions from the controlled environ ment over a user input interface on a controller 16 The method of claim 1 wherein said assigning step comprises the step of adding or removing the one or more system components over a network discovery protocol interface on a con troller 17 The method of claim 1 wherein said assigning step comprises the step of adding or removing the one or more system components over a user input interface on a controller 18 A computer program product comprising a computer readable storage medium having a program stored thereon which when executed by a processor causes the processor to control a plurality of devices to perform an activity based or an experienced based operation comprising first computer readable program code means for causing the processor to collect device information pertaining to identification of the plurality of devices including at least one of a manufacturer name for
12. to the new state thereby placing the television in the proper input state Although the combination of improved computer process ing capabilities and global access to information has resulted in significant advancements in the information processing age there exists a need fora simple inexpensive yet versatile system that can integrate the functions of multiple residential devices connected to a residential network and facilitate the operations of these devices with minimal user intervention SUMMARY OF THE INVENTION A method system and computer program product are provided to configure and manage a plurality of system com ponents within a controlled environment such as a home business school etc as well as its surrounding areas The system components include legacy and or non legacy devices and the controlled environment includes a peer to peer communications system and or a client server network In an embodiment of the present invention activity based or experienced based operations are created and defined during an initialization or setup session for a system component with minimum or no user interaction The activ ity based or experienced based operations include favorite settings playlists security protocols context sensitive operations user specific operations follow me system con trols control macros including context based control mac ros or the like For non legacy devices relevant informatio
13. 668 990 B2 Sheet 8 of 12 Feb 23 2010 U S Patent cOL 2 91 E MEET Djuotu uQulAU oix veam Buyyb y kosas 4 5 lt 0 ppp saap Jo sodA oym 086 951590 SULUIY US 7 668 990 B2 Sheet 9 of 12 Feb 23 2010 U S Patent 008 8 Sis euo paps 244 eps DELIT Prog 08 od QUO pes A ATUM png US 7 668 990 B2 Sheet 10 of 12 Feb 23 2010 U S Patent 6 Sls spip p j ZG 994942 6 0 J 6 suBip 7 55820 p pajas duyuiu ag TET s US 7 668 990 B2 Sheet 11 of 12 Feb 23 2010 U S Patent Sup ny indu Aa 4 papi aap Ady pym uj ayy anod 103 ay 250 A S __ jit SHC 45 3ruossunq ec Mala oe nie adres ac 0001 U S Patent Feb 23 2010 Sheet 12 of 12 US 7 668 990 B2 1100 Processor 1104 1108 Display Interface 1102 Display 1130 Secondary Memory 1110 Commu
14. 8 990 B2 13 Apparatus U S patent application Ser No 10 783 017 which is incorporated herein by reference in its entirety FIG 9 and FIG 10 illustrates device control screens 900 and 1000 respectively that can be included as part of a set up wizard for specifying device metadata according to embodi ment of the present invention Device control screen 900 specifies the quantity of digits that are needed to change channels on a tuner As shown the tuner can accept two digits i e channels 01 to 99 three digits 1 channels 001 to 999 or four digits 1 channels 001 to 999 The quantity of channel digits depends on the make and model of the system component e g tuner 114 that is being configured Device control screen 1000 specifies the inputs and input order for controlling a system component For example a television can include two inputs AUX1 and AUX2 Input AUX1 can be used to receive input from a satellite receiver and input AUX2 can be used to receive input from a DVD player The present invention includes mechanisms for optimizing the collection of device metadata with little or no human intervention In an embodiment a series of questions or infor mational prompts that comprise the set up process can be dynamically optimized to avoid seeking irrelevant or unnec essary information In an embodiment device metadata is contained in an accessible database such as the metadata database descri
15. Industrial 20 25 30 35 40 45 50 55 60 65 10 Co Ltd of Osaka Japan employs a proprietary protocol for discovery and control of that device in a network In an embodiment of the present invention one or more bridges 130 a 130 e act as a plug and play proxy for a piece of legacy equipment and is herein referred to as a plug and play enhanced bridge 130 a 130 e The plug and play enhanced bridge 130 a 130 e therefore includes a virtual plug and play protocol interface that enables the plug and play enhanced bridge 130 a 130 e to advertise itself to other network entities as a conventional plug and play device exhibiting attributes similar to the legacy device that is com municatively connected to it When plug and play commands arrive from the network e g network 180 plug and play enhanced bridge 130 a 130 e translates them into an appro priate set of commands for controlling the legacy device such as IR commands or serial commands Note that although this embodiment is described with respect to plug and play virtu alization the present invention is not limited to plug and play virtualization but also encompasses legacy device virtualiza tion in accordance with other packet based discovery and control protocols including but not limited to Jini devel oped by Sun Microsystems of Santa Clara Calif or the Rendezvous protocol which is an open protocol devel oped in p
16. US007668990B2 a2 United States Patent Krzyzanowski et al US 7 668 990 B2 Feb 23 2010 10 Patent No 45 Date of Patent 54 75 73 21 22 65 63 51 52 58 METHOD OF CONTROLLING A DEVICE TO PERFORM AN ACTIVITY BASED OR AN EXPERIENCE BASED OPERATION Inventors Paul Krzyzanowski Fanwood NJ US Justin Flores Boca Raton FL US Glenn Harter Wellington FL US Assignee OpenPeak Inc Boca Raton FL US Notice Subject to any disclaimer the term of this patent is extended or adjusted under 35 U S C 154 b by 477 days Appl No 11 042 178 Filed Jan 26 2005 Prior Publication Data US 2005 0128048 A1 Jun 16 2005 Related U S Application Data Continuation in part of application No 10 783 511 filed on Feb 23 2004 and a continuation in part of application No 10 782 923 filed on Feb 23 2004 now Pat No 7 024 256 and a continuation in part of application No 10 783 017 filed on Feb 23 2004 now Pat No 7 129 855 and a continuation in part of application No 10 387 590 filed on Mar 14 2003 Int Cl GO6F 13 12 2006 01 GO6F 3 00 2006 01 GO6F 15 177 2006 01 USC o oen 710 72 710 8 710 62 709 220 709 221 709 222 Field of Classification Search 710 72 710 8 62 709 220 222 See application file for complete search history 106 a 56 References Cited U S PATENT DOCUMENTS 5 414 426 A 5 1995 O Donnell et al 5
17. ample television 104 can be a personal television enhanced to support the MSN service hosted by WebTV Networks Inc Mountain View Calif that supports the WEBTV services available from Microsoft Corporation Redmond Wash As shown television 104 can be con nected to cable set top box 118 DSS set top box 120 and or media player 116 e g PVR VCR or DVD player The aforementioned system components are not intended to be exhaustive Other devices including appliances appli cations and or the like can be implemented including but not limited to a refrigerator stove microwave toaster cof fee maker alarm clock humidifiers sprinkler system light ing light dimmers etc Inan embodiment portable controller 108 controls the operations and or functions of such compo nents such as on off timers modulation e g oven tempera tures etc pause snooze etc As discussed communications network 180 provides a transmission medium for communicating among the system components Communications network 180 is a wired and or 20 25 30 35 40 45 50 55 60 65 8 wireless local area network LAN Thus communications network 180 includes wired wireless or both transmission media including satellite terrestrial e g fiber optic copper UTP STP coaxial hybrid fiber coaxial HFC or the like radio microwave free space optics and or any other form or method of transmission A
18. and not limitation It will 20 25 30 35 40 45 50 55 60 65 18 be apparent to persons skilled in the relevant art s that vari ous changes in form and detail can be made therein without departing from the spirit and scope of the invention More over as previously discussed it should be understood that the method system and computer program product of the present invention should not be limited to a residential envi ronment The present invention can be implemented in other types of environments having a central processing system for distributing media and sending command and or control sig nals to a plurality of devices and or applications dispersed throughout a designated region In addition to a residence the designated region includes but is not limited to office com plexes suite of small offices production studios warehouses entertainment arenas health care facilities hotels vacation resorts aircrafts ships automobiles or the like Thus the present invention should not be limited by any of the above described exemplary embodiments but should be defined only in accordance with the following claims and their equivalents What is claimed is 1 A method of controlling a plurality of devices to perform an activity based or an experience based operation compris ing the steps of collecting device information pertaining to identification of the plurality of devices including at least one of a
19. art by Apple Computer Inc of Cupertino Calif Therefore in an embodiment bridges 130 a 130 e are provided to control legacy system components In another embodiment bridges 130 a 130 e are enhanced with a vir tual plug and play protocol interface to enable discovery and control of legacy system components within system 100 It should be noted that the wireless aspects of the present inven tion is not restricted to infrared communications In other embodiments bridges 130 a 130 e support other modes of wireless communications e g RF WiFi BLUETOOTH etc that enable discovery and or control of legacy system components In another embodiment a bridge 130 a 130 e is modified to enable discovery and or control of legacy sys tem components over a wired interface e g serial or parallel with other system components of system 100 In other embodiments a bridge 130 a 130 e can support a wired interface to a legacy system component and a wireless inter face to another system components a wireless interface to a legacy system component and a wired interface to other sys tem components wireless interfaces to both the legacy sys tem component and other system components and or wired interfaces to both the legacy system component and other system components Setup Optimization In an embodiment of the present invention activity based or experienced based operations are created and defined dur ing an init
20. available to the user and the manner in which such devices are interconnected In other words the setup process of the present invention allows the user to define activity based or experience based operations that can be executed with all possible media source renderer combina tions For example unlike other proprietary setup schemes that lock users into a particular source in defining an activity the present invention s setup process supports a number of activ ity based or experience based operations that utilize any available source such as Watch DVD from DVD player and Watch DVD from game console By contrast other propri etary schemes allow only a single Watch DVD experience that is confined to a single player source even though there may be two or more devices capable of playing a DVD The setup process of the present invention also supports activity based or experience based operations on multiple media renderers e g Watch DVD on TV and or Watch DVD on projection TV By contrast other proprietary schemes limit the user to one Watch DVD experience that is confined to a single rendering device even though two or more valid rendering devices may be available Unlike other proprietary setup schemes that limit a user s ability to define activities to one device per type the setup process of the present invention supports activity based or experience based operations that utilize source render
21. bed in the application entitled Device Con trol System Method and Apparatus U S patent application Ser No 10 783 017 which is incorporated herein by refer ence in its entirety As such the setup process can be tailored and streamlined based on the type and model number of the system component being added or configured For example the setup process would avoid asking for information that either is already known via the metadata database or that is not relevant to that particular system component such as types of inputs tuning rules power on delay etc Instead the setup process would be dynamically tailored to seek only that which is specific to and necessary to control the particular system component in light of its user specific configuration e g what devices are connected to the AV1 and AV2 inputs of a television Additionally certain system components such as some digital media adapters provide some functions that can only be controlled via IR while other functions can be controlled through a wired interface Based on the stored profile for such system components the IR controlled functions can be enu merated for and confirmed by the user during setup Referring back to FIG 3 at step 315 the system compo nents e g the assigned legacy and non legacy devices are configured for activity based or experienced based opera tions Once the device information including device meta data has been collected one or mor
22. condary memory 1110 The secondary memory 1110 can include for example a hard disk drive 1112 and or a removable storage drive 1114 representing a floppy disk drive a magnetic tape drive an optical disk drive etc The removable storage drive 1114 reads from and or writes to a removable storage unit 1118 in a well known manner Removable storage unit 1118 represents a floppy disk mag netic tape optical disk etc which is read by and written to removable storage drive 1114 As will be appreciated the removable storage unit 1118 includes a computer usable stor age medium having stored therein computer software e g programs or other instructions and or data US 7 668 990 B2 17 In alternative embodiments secondary memory 1110 includes other similar means for allowing software and or data to be loaded into computer system 1100 Such means include for example a removable storage unit 1122 and an interface 1120 Examples of such means include a program cartridge and cartridge interface such as that found in video game devices a removable memory chip such as an EPROM or PROM and associated socket and other remov able storage units 1122 and interfaces 1120 which allow software and data to be transferred from the removable stor age unit 1122 to computer system 1100 Computer system 1100 can also include a communications interface 1124 Communications interface 1124 allows soft ware and or data to be transferred between compu
23. control screens 600 700 and 800 various system components both legacy and non legacy devices can be added removed or configured for control by system 100 During the setup pro cess relevant device information e g device type make model region assignment etc is automatically discovered or added by user input For a legacy device that is not auto matically discoverable the user adds the legacy device to system 100 by identifying the device by device type make model etc and assigning the device to a particular room region via the set up process e g device control screens 600 700 and 800 setup process is also designed to acquire basic tional information about each new device such as tuning rules types and order of inputs etc The basic operational information is also referred to as device metadata and includes information regarding the unique control behavior s or operational idiosyncrasies of a particular device As described in greater detail below device metadata 15 executed to ensure a system component is placed in the appropriate state or mode for executing activity based or experienced based operations of the present invention Examples of device metadata include the input mode or state input types order of inputs and tuning rules for operating a device Device metadata 1s described in greater detail in the application entitled Device Control System Method and US 7 66
24. curity access warnings hot tub controls orthe like Another configuration can be established for home office activities which includes without limitation presets for a baby monitor a playlist of classical recordings coffee maker timer controls or the like A configuration can be established for morning rituals which includes without limitation presets for alarm snooze controls coffee maker timer controls lighting news broadcasts or the like In another embodiment the activity based or experience based operations are user specific For example user specific configuration can be established with security con trols set to block certain televisions programming web sites audio recordings or the like when in the company of minors However when in the company of adults a configuration can be established with a lower security setting In another embodiment the activity based or experience based operations include follow me system controls For instance follow me video is implemented to transfer a selected video production to various displays throughout the controlled environment As a user migrates from room to room within the controlled environment the user or a system US 7 668 990 B2 5 component in communications with the user is tracked to determine the user s current location and a display e g monitor or television positioned in the current location will automatically start to present a pre sel
25. d be implemented in hardware firmware and or soft ware to perform the functions of the present invention That is the same piece of hardware firmware or module of soft ware could perform one or more of the illustrated blocks 1 components or steps Additionally the present invention can be implemented in one or more computer systems or other processing systems capable of carrying out the functionality described herein Referring to FIG 11 an example computer system 1100 useful in implementing the present invention is shown Vari ous embodiments are described in terms of this exemplary computer system 1100 After reading this description it will become apparent to a person skilled in the relevant art s how to implement the invention using other computer systems and or computer architectures computer system 1100 includes one or more proces sors such as processor 1104 Processor 1104 can be a special purpose or a general purpose digital signal processor The processor 1104 is connected to a communication infrastruc ture 1106 e g a communications bus cross over bar or network Computer system 1100 can include a display interface 1102 that forwards graphics text and other data from the communication infrastructure 1106 or from a frame buffer not shown for display on the display unit 1130 Computer system 1100 also includes a main memory 1108 preferably random access memory RAM and can also include a se
26. d ipltsfi244 amp version print Oct 11 2004 6 pages 6 642 852 B2 11 2003 Dresti et al Creston Introduces RMC2e Mini Control System Home Toys 6 647 426 82 11 2003 Mohammed New Release URL http www hometoys com releases mar03 6 728 784 Bl 4 2004 Mattaway creston_02 html Oct 14 2004 2 pages 6 741 853 5 2004 Jiang et al Troy Serial Server URL http www troygroup com Connectiv 6 748 343 B2 6 2004 Alexander et al ity products SerialDeviceConnectivity SerialServers asp down 6 792 469 9 2004 Callahan et al loaded Jul 9 2004 2 pages 6 914 551 B2 7 2005 Vidal Troy Serial Servers URL http www troygroup com Connectiv 2001 0000194 Al 4 2001 Sequeira ity products DataSheets 2001 0018663 A1 8 2001 Dussell et al Troy 20Serial 20Server 20data 20sheet pdf downloaded Jul 2001 0036192 Al 11 2001 Chiles et al 9 2004 1 page 2001 0041561 1 11 2001 Ventulett et al EtherWind M 802 1 IbYM Wireless Print Server URL http www 2001 0049275 Al 12 2001 Pierry et al troygroup com wireless documents datasheets 2001 0049846 A1 12 2001 Guzzi al TROYU 20EtherWind pdf Nov 19 2003 2 pages US 7 668 990 B2 Page3 EtherWind IEEE 802 11b Wireless Print Server URL http www troygroup com wireless products wireless etherwind asp Nov 19 2003 3 pages Troy Wireless Introduces EtherWind Plus 802 11b Wireless OEM Board Level Product New Product Enables Wireless and Ethernet Connectivity on OEM D
27. d operation comprising the steps of collecting device information pertaining to identification of the plurality of devices including at least one of a manufacturer name for each of the devices a model number of each ofthe devices and a device type for each of the devices wherein such device information is col lected via user input determining the specific metadata relating to unique requirements pertaining to control of the plurality of devices including at least one of a device activation requirement tuning requirement and input state selec tion requirement necessary to provide control of the plurality of devices based on the collected device infor mation querying the user for the specific metadata accessing a sequence of device specific commands based on said metadata and associated with the plurality of devices and sending said sequence to the plurality of devices whereby each device specific command from said sequence is executable to control a corresponding device upon receipt of said device specific command at said corre sponding device 20 The method according to claim 19 wherein said deter mining step further comprises the step of processing the received information to dynamically tailor a series of questions or informational prompts to collect additional information pertaining to the plurality of devices 21 The method according to claim 19 further comprising the step of sending the sequence of de
28. des 430 a 114 130 b Audio Video Siream 112 130 d Eg lutron and X10 130 c 140 US 7 668 990 B2 Page 2 U S PATENT DOCUMENTS 2001 0053274 Al 12 2001 Roelofs et al 5 537 463 7 1996 Escobosa et al 2001 0054060 Al 2001 0055954 AI 12 2001 Fillebrown et al 12 2001 Cheng A 5 552917 A 9 1996 Darbee et al 2002 0006788 Al 1 2002 Knutsson et al 5 570 085 10 1996 Bertsch 2002 0019984 Al 2 2002 Rakib 5 689 353 A 11 1997 Darbee et al 2002 0022991 1 2 2002 Sharoodetal 705 14 5 771 388 6 1998 Mondrik et al 2002 0031120 Al 3 2002 Rakib 5 802 467 9 1998 Salazar et al 2002 0033760 Al 3 2002 Kobayashi 5 805 812 A 9 1998 Fish et al 2002 0034292 Al 3 2002 Tuoriniemi et al 5 875 108 2 1999 Hoffberg et al 2002 0035403 Al 3 2002 Clark et al 5 898 386 A 4 1999 Kaihatsu 2002 0035404 Al 3 2002 Ficco etal 5 926 108 A 7 1999 Wicks et al 2002 0035605 Al 3 2002 McDowell et al 5 930 699 7 1999 Bhatia 2002 0035621 Al 3 2002 Zintelet al 5 953 144 A 9 1999 Darbee et al 2002 0036795 Al 3 2002 Fujitani 5 956 025 A 9 1999 Goulden et al 2002 0068984 Al 6 2002 Alexander et al 5 956 487 9 1999 Venkatraman et al 2002 0087746 Al 7 2002 Ludtke et al 5 959 751 A 9 1999 Darbee et al 2003 0103088 Al 6 2003 Dresti et al 5 963 624 10 1999 Pope 2003 0105854 6 2003 Thorsteinsson et al 6 005 861 A 12 1999 Humpleman 2003 0163626 A1 8 2003 Chen et al 6 026 150 A 2 2000
29. e activity based or expe rience based operations can be created or defined both automatically or by the user As discussed above activity based or experienced based operations include favorite set tings playlists security protocols context sensitive opera tions user specific operations follow me system controls control macros including context based control macros and the like Examples of activity based or experience based operations are provided in U S Pat No 6 792 323 to Krzyza nowski et al which 1s incorporated herein by reference in its entirety and the application entitled Method System and Computer Program Product for Automatically Managing Components within a Controlled Environment U S patent application Ser No 10 782 923 which is incorporated herein by reference in its entirety Ifthe activity based or experienced based operations have already been defined by the system default operations or a 0 5 20 40 45 50 55 65 14 user new activity based or experienced based operation is established after a new system component has been added to the controlled environment as described above with refer ence to steps 303 312 For example if a context based con trol macro already exists for playing a DVD movie and a new television is added to a bedroom the existing control macro is updated to specify the new television as an optional system component for displaying the DVD m
30. e database As such the setup process can be tailored and streamlined based on the type and model number of the system component being added or con figured The setup process is dynamically tailored to seek only that which is specific to and necessary to control the particular system component in light of its user specific con figuration Once the device information including device metadata has been collected the system components e g the assigned legacy and non legacy devices are configured to support one or more activity based or experienced based operations BRIEF DESCRIPTION OF THE DRAWINGS FIGURES The accompanying drawings which are incorporated herein and form part ofthe specification illustrate the present invention and together with the description further serve to explain the principles of the invention and to enable one skilled in the pertinent art s to make and use the invention In the drawings generally like reference numbers indicate iden tical or functionally or structurally similar elements Addi tionally generally the leftmost digit s ofa reference number identifies the drawing in which the reference number first appears FIG 1 illustrates a residential controlled system according to an embodiment of the present invention FIG 2 illustrates a residential controlled system according to another embodiment of the present invention FIG 3 illustrates an operational flow for initializing system componen
31. e has been assigned In other embodiments a user can manually input or modify device information including room region assign ment about a non legacy device Referring back to FIG 3 at step 312 any legacy devices are assigned to the designated room or region In an embodiment of the present invention one or more legacy devices are communicatively coupled to a plug and play enhanced wire less controller such as a plug and play enhanced bridge 20 25 30 35 40 45 50 55 60 65 12 130 a 130 e described above As such a virtual plug and play protocol interface enables the coupled legacy device to be discovered and relevant device information about the discovered legacy device is collected In an embodiment a user has the option of excluding a discovered legacy device from being controlled In another embodiment a user can manually assign a legacy device e g legacy devices not coupled to a plug and play enhanced wireless controller to the controlled environment e g system 100 FIG 7 and FIG 8 illustrate a legacy device control screen 700 and legacy device control screen 800 which can be included as part of a set up wizard for assigning legacy devices according to embodiment of the present invention Legacy device control screen 700 can be used to support manual assignment of legacy devices and legacy control screen 800 can be used to support both manually assigned and automatically discovered l
32. e system components exchange audio video voice other data and or control messages directly with each other and without being centrally managed In an embodiment using a wired transmission medium communications network 180 is an Ethernet LAN having a CAT 5 cable or the like distributed to each system compo nent such as television 104 media player 116 etc The sys tem component includes an audio video AV connector that is responsive to receive the cable In an embodiment com munications network 180 includes a telephone line and or powerline such as the communications technologies made available from the Home Phone Networking Alliance Home PNA or the like In an embodiment using a wireless transmission medium communications network 180 supports the IEEE standard 802 11 a which specifies a wireless Ethernet protocol for large sized video In another wireless embodiment commu nications network 180 supports the IEEE standard 802 11 b which specifies a wireless Ethernet protocol for small size video In another wireless embodiment communications net work 180 supports the IEEE standard 802 11 g In another embodiment the BLUETOOTH wireless technology de veloped by Bluetooth SIG Inc is used to support short range wireless interfaces with system 100 As shown in FIG 1 and FIG 2 communications network 180 includes a wireless network access point 182 such as those available from Linksys a division of Cisco Systems
33. each of the devices amodel number of each ofthe devices and a device type for each ofthe devices wherein such device information is collected via a network discovery protocol second computer readable program code means for caus ing the processor to access metadata that specifies unique requirements pertaining to control of the plural ity of devices including at least one of a device activa tion requirement tuning requirement and input state selection requirement third computer readable program code means for causing the processor to access a sequence of device specific commands based on said metadata and associated with the plurality of devices 10 15 20 25 30 40 45 50 55 60 65 20 fourth computer readable program code means for causing the processor to send said sequence to said plurality of devices wherein each device specific command from said sequence is executable to control a corresponding device upon receipt of said device specific command at said corresponding device fifth computer readable program code means for causing the processor to define a plurality of regions within a controlled environment and sixth computer readable program code means for causing the processor to assign each ofthe plurality of devices to at least one of said regions within said controlled envi ronment 19 A method of controlling a plurality of devices to per form an activity based or an experience base
34. ected video production Similarly the activity based or experience based operations can include follow me audio follow me lighting or the like In another embodiment the activity based or experience based operations are defined by one or more control macros A control macro includes a set of commands that when executed enables a controller to control multiple operations and or functions of one or more system components In an embodiment the set of commands are stored in a central location and are associated with a generic command When activated the generic command retrieves and transmits the set of commands to the appropriate system component s for execution In other words the present invention enables a remote or portable controller to transmit a single high level request to a centralized command center which can be coupled to another computer The centralized command cen ter interprets the single request and finds the corresponding sequence of commands that needs to be transmitted to the appropriate system component s In an embodiment the control macro is defined to execute automatically upon the occurrence of a commencement parameter The commencement parameter can be a pre defined time date state event or the like or any combination thereof For example a pre defined sequence of control com mands can be executed upon the occurrence of a pre set time i e clock based and or date 1 calendar based a
35. eference in its entirety BACKGROUND OF THE INVENTION 1 Field of the Invention The present invention relates generally to control of con sumer electronics and more specifically to a process for creating activity or experience based operations for control ling consumer electronics 2 Related Art Theinstallation of computer networks for homes and busi nesses presents several challenges For example installing a computer network can be time extensive and expensive to deploy Additionally there is no easy method to integrate for example home computer networks with other residential devices such as televisions stereos DVD players and other home electronics Being able to efficiently distribute digital audio video AV data among personal computers and other AV devices such as televisions DVD players PVRs etc is complicated by differing and evolving communications stan dards and or formats Another significant challenge is being able to effectively control networked residential devices Although a remote control unit RCU can be trained to send signals to compo nents of an entertainment center such as a television stereo and DVD player a conventional RCU cannot communicate and control multiple personal computers and other analog and or digital devices at a residence In addition a conven tional RCU cannot communicate and control multiple resi dential devices without user intervention Even with a con venti
36. egacy control devices Referring to FIG 7 a device information window 702 specifies the device type for a legacy device that is being manually assigned The device types audio video light ing environmental and cameras are provided for illus trative purposes and other device types are considered to be within the scope of the present invention Upon specifying a device type legacy device control screen 800 is presented as part of the setup wizard As shown legacy device control screen 800 provides an example for adding removing or configuring an audio video device In an embodiment supporting a manual device assignment the audio device device type is specified in legacy device control screen 700 In another embodiment supporting automatic discovery of legacy devices e g coupled to a plug and play enhanced wireless controller legacy device control screen 800 would be presented upon discovery of the legacy device As shown in device informa tion window 802 the discovered legacy device or manually assigned device if appropriate is an audio video device that has been discovered or manually assigned if appropri ate to the living room The device information fields 804a 8044 are automatically populated or manually completed if appropriate with the relevant device information e g assigned room region device type make model etc Using a setup wizard such as illustrated by device
37. ention accommodates system components that are accessible from multiple rooms or regions shared or that are connected to a switching device that sends its output to multiple rooms or regions e g matrix switcher multi zone audio video system etc Accordingly the user can define a per room region activ ity based or experience based operation that is not necessar ily limited to system components that are physically located in that particular room or region For example a watch DVD in the master bedroom experience can be defined using a DVD player located in the living room that is a shared device or that is accessible to the television e g television 104 in the master bedroom via a switcher Based on the rooms regions and experience categories that are defined the present invention allows for a control inter face to be generated that allows a user to navigate to the selected room region select an experience category e g audio video and select a specific activity within the cat egory e g watch a movie on DVD player 1 Control inter face examples are described in greater detail in the applica tion entitled User Interface for Multi Device Control U S patent application Ser No 10 783 511 which is incorpo rated herein by reference in its entirety As described above the setup process of the present inven tion contemplates use of networked devices such as plug and play media players e g media p
38. er devices of the same type such as Watch DVD on TV 1 and Watch DVD on TV 2 or Watch DVD 1 on TV and Watch DVD 2 on TV By contrast other proprietary setup schemes do not let the user define activities for more than one device of the same type The present invention s setup process contemplates use in a wireless network in which a single controller e g portable controller 108 can control any system component in any US 7 668 990 B2 15 room or region from any location in a controlled environment e g system 100 Accordingly each system component is assigned to a specific room or region The user can then define activity based or experience based operations for each room orregion using the system components available in that room or region The present invention contemplates control of multiple system components of the same type e g two televisions in the same room The user can define a plurality of the same activity based or experience based operations in the same room using different sets and combinations of intercon nected system components For example the user can setup multiple Watch DVD experiences using different DVD players and or different TVs Watch Bedroom DVD player 1 on Bedroom TV 2 or Watch Bedroom DVD player 2 on Bedroom TV 1 Thus multiple user defined activity based or experience based operations can be created on a per room basis In addition the present inv
39. evices URL http www troygroup com wireless company news 081302 asp Nov 19 2003 2 pages EtherWind PlusTM 802 115 OEM Module URL http www troygroup com wireless documents datasheets TROY 20EtherWind 20Plus 20data 20sheet pdf Nov 19 2003 2 pages Ether Wind Plus IEEE 802 11b OEM Connectivity Module URL http www troygroup com wireless products wireless etherwindplus asp Nov 19 2003 2 pages Pronto RF Extender RFX 6000 Pronto accessory for operating devices inside a closed cabinet closet or even in another room URL http www pronto philips com index cfm id 577f Nov 19 2003 2 pages Pronto RF Extender RFX 6000 RF extender for TSU6000 amp 17803000 URL http www remotecontrol philips com library documents SPEC RFX6000 pdf Nov 19 2003 2 pages Harmony Remote Overview URL http www harmonyremote com product_overview htm downloaded from the World Wide Web on Feb 9 2004 2 pages Harmony Remote Features URL http www harmonyremote com product features htm downloaded from the World Wide Web on Feb 9 2004 5 pages Specifications for Harmony Remote URL http www harmonyremote com product specifications htm downloaded from the World Wide Web on Feb 9 2004 2 pages Harmony Remote Frequently Asked Questions URL http www harmonyremote com faq htm downloaded from the World Wide Web on Feb 9 2004 2 pages Harmony Remote 587 6599 published at
40. f device specific commands to a wired interface coupled to the plurality of devices 6 The method according to claim 1 further comprising the step of sending the sequence of device specific commands to a wireless interface coupled to the plurality of devices 7 The method according to claim 1 further comprising the step of performing the activity based or the experienced based operation to implement one or more favorite settings defined for a user US 7 668 990 B2 19 8 The method according to claim 1 further comprising the step of performing the activity based or the experienced based operation to play a playlist of media and or multimedia 9 The method according to claim 1 further comprising the step of performing the activity based or the experienced based operation to administer a security protocol established for the plurality of devices 10 The method according to claim 1 further comprising the step of performing the activity based or the experienced based operation to synchronize an operation of one of the plurality of devices with an operation of one or more additional of the plurality of devices receiving a sequence of commands from said controller 11 The method of claim 1 wherein said defining step comprises the step of adding or removing regions from the controlled environ ment over a user input interface on a controller 12 The method of claim 1 wherein said assigning step comprises the
41. hose defined by the UPnPTM Forum As would be appreciated by one skilled in the relevant art s the UPnPTM Forum is an open industry standard that uses Internet and Web protocols to enable devices such as personal computers peripherals intelligent appliances and wireless devices to be connected to a controlled environment such as system 100 and to become automatically aware of each other In accordance with a plug and play protocol such as the UPnP Forum when a user plugs non legacy device into a controlled environment e g system 100 the device con figures itself acquires a TCP IP address and uses a discovery protocol based on the Hypertext Transfer Protocol HTTP to announce its presence to other devices on within the con trolled environment e g system 100 Other plug and play compliant network entities can then negotiate a communica tions protocol with the device determine its capabilities and control it Note that although this embodiment is described with the UPnP protocol the present invention is not limited to the UPnP protocol but also encompasses other standards based discovery protocols such as the Rendez vous protocol developed in part by Apple Computer Inc of Cupertino Calif The present invention also encompasses proprietary discovery protocols designed for specific network devices For example the Panasonic amp KX HCM250 Wire less Network Camera sold by Matsushita Electric
42. http www harmonyremote com downloaded from the World Wide Web on Feb 9 2004 1 page Harmony Remote 587 7487 published at http www harmonyremote com downloaded from the World Wide Web on Feb 9 2004 1 page Harmony Remote 551 768 published at http www harmonyremote com downloaded from the World Wide Web on Feb 9 2004 1 page Harmony Remote 551 659 User Manual dated Oct 16 2003 published at http www harmonyremote com downloaded from the World Wide Web on Feb 9 2004 20 pages Harmony Remote SST 768 User Manual dated Mar 25 2003 published at http www harmonyremote com downloaded from the World Wide Web on Feb 9 2004 37 pages cited by examiner US 7 668 990 B2 Sheet 1 of 12 Feb 23 2010 U S Patent ojog 427 908 01 a e uong 63 B8E gt i 2 6 0 OLL 281 dian 02 Ew t i 231091 DIp9w dx 4950224 72 ERR P E M i 40901 2 90 9 701 001 US 7 668 990 B2 Sheet 2 of 12 Feb 23 2010 U S Patent Sls 5 051 e 9 051 01 vang 62 B897 0 801 092094 0040201 802 EOIN 021 201 BE Meu mec 3 8 t 22 3 Ts NM dX
43. ialization or setup session with minimum or no user interaction and are executed with the function and or opera tion of various system components This can be explained with reference to FIG 3 where flowchart 300 represents the general operational flow of an embodiment of the present invention for initializing or setting up the system components in a controlled environment such as system 100 More spe cifically flowchart 300 shows an example ofa control flow for establishing activity based or experienced based operations during the initialization or setup phase Although the control flow is being described with reference to a residential con trolled environment it should be understood that flowchart 300 can also be expanded to support non residential environ ments The control flow of flowchart 300 begins at step 301 and passes immediately to step 303 At step 303 a room or region within the controlled environment e g system 100 is des US 7 668 990 B2 11 ignated In an embodiment a control screen is presented on a display device to enable a user to perform this operation The control screen therefore enables the user to add remove and or configure a system component for the controlled envi ronment In an embodiment a set up wizard allows the user to navigate among a plurality of control screens to add or con figure system components FIGS 4a 46 illustrate a room control screen 400 in accor dance with an embodi
44. k alarms light dimmer settings web sites news broadcasts financial chan nels or the like In another embodiment the activity based or experience based operations include one or more playlists for the sys tem component For example a series of video or audio recordings can be prepared and or stored for playback on for example a television monitor or audio client A sequence of graphic images or photographs can be prepared and or stored for playback on for example a monitor or computer client A playlist of other forms of media and or multimedia can also be created according to embodiments of the present invention In another embodiment the activity based or experience based operations include one or more security protocols for the system component For example the system component can be configured to block certain content from being accessed by a designated user Non adult users for instance can be prevented from accessing designated television chan nels web sites areas such as lockable rooms drawers safes etc or the like In another embodiment the activity based or experience based operations are context sensitive The system compo nent can have one or more configurations based on a particu lar context Depending on the type of system component a configuration can be established for evening entertainment which includes without limitation presets for lighting audio video presentations se
45. layer 116 and servers e g server 208 that can be automatically discovered when they are powered on and connected to the controlled environ ment Once a networked device is found Applicants set up process will automatically display the device for the user and the user will be asked to assign that device to a particular room if it is not automatically positioned in accordance with the location positioning mechanisms of the present invention As described above the setup process of the present inven tion also accommodates use of a bridge device that converts commands from a packet based format e g Ethernet or 802 11 to a format that is intelligible by most CE devices in existence today IR or Serial As disclosed in the application entitled Legacy Device Bridge for Residential or Non Resi dential Networks U S patent application Ser No 10 387 590 the bridge device allows IR controlled CD devices which require direct line of sight connectivity to a remote control unit to be connected to a WiFi network Once con nected to a controlled environment e g system 100 via the bridge device the user is not required to be in the same room as the device in order to control it During the setup process the user can assign a bridge device to control a particular CE device See device control screen 500 in FIG 5 20 30 35 40 45 50 55 60 65 16 According to an embodiment of the present invention
46. lled environments is shown in FIG 1 which illustrates a residential controlled system 100 according to an embodiment of the present invention System US 7 668 990 B2 7 100 includes a communications network 180 that intercon nects a plurality of system components The system compo nents include a positioning unit 102 two televisions 104 shown as television 104 a and 104 0 two computer cli ents 106 shown as computer client 106 a and computer client 106 5 a portable controller 108 a lighting device 110 a thermostat 112 for a HVAC system a tuner 114 a media player 116 a cable box 118 a digital satellite system DSS box 120 and a wireless lighting control interface such as a Lutron or X10 transceiver 140 Other devices and or applications can also be included as system components Positioning unit 102 designates spatial locations within the residence that serves as the hosting environment for system 100 Positioning unit 102 is coupled to the other system components e g portable controller 108 via a wired and or wireless interface Positioning unit 102 is operable to desig nate a flooror room within the residence Positioning unit 102 is also operable to designate a specific location or region within a floor or room Moreover positioning unit 102 can be situated outside ofthe residence to thereby designate external areas of the residence Computer client 106 includes a wired and or wireless per sonal computer pe
47. m component s at the time of activating the context based control macro The above mentioned activity based or experience based operations have been provided by way of example and not limitation Other types of activity based or experienced based operations can be used and are intended to be within the scope of the present invention Additional examples of activ Ity based or experience based operations including further descriptions of the above mentioned operations are described in the application entitled Method System and Computer Program Product for Automatically Managing Components within a Controlled Environment U S patent application Ser No 10 782 923 which is incorporated 0 5 20 40 45 6 herein by reference in its entirety and in U S Pat 6 792 323 to Krzyzanowski et al which is incorporated herein by reference in its entirety As described in greater detail below activity based or experienced based operations are created and defined during an initialization or setup session and are executed with the function and or operation of system components including consumer electronic CE devices The activity based or experienced based operations support the control of multiple devices of the same type e g two DVD players The activ or experienced based operations also support the segmentation of devices among multiple rooms 2 a tele vision in a living room versus a
48. ment of the present invention Room control screen 400 can be included as part of a set up wizard for adding and or configuring a system component Room control screen 400 allows the user to designate a room or a region within a room from a compilation 402 of available rooms or regions The user can also add a new room or region by entering a description in room descriptor field 404 and activating the add button 406 Upon designating a room or region the designated room or region appears in the active room window 408 As shown in FIG 45 the living room has been designated in the active room window 408 As can be seen the descriptor living room is also highlighted in the compilation 402 of availabilities Referring back to FIG 3 at step 306 any wireless control lers are assigned to the designated room from step 303 Wire less controllers include the bridges 130 130 including the plug and play enhanced bridge 130 130 described above FIG 5 illustrates wireless controller device control screen 500 that can be included as part of a set up wizard for assigning wireless controllers according to embodiment of the present invention Controller information window 502 is populated with all wireless controllers communicatively coupled to the controlled environment e g system 100 In an embodiment the wireless controllers e g wireless con trollers having a plug and play protocol interface or a
49. mputer readable program code means for causing the processor to collect device information pertaining to identification of the devices including at least one of a manufacturer name for each of the devices a model number of each ofthe devices and a devicetype for each of the devices wherein such device information is col lected via user input 22 second computer readable program code means for caus ing the processor to determine the specific metadata relating to unique requirements pertaining to control of the plurality of devices including at least one ofa device activation requirement tuning requirement and input state selection requirement necessary to provide control of the device based on the collected device information third computer readable program code means for causing the processor to query the user for the specific metadata fourth computer readable program code means for causing the processor to access a sequence of device specific commands based on said metadata and associated with the plurality of devices and fifth computer readable program code means for causing the processor to send said sequence to said plurality of devices wherein each device specific command from said sequence is executable to control a corresponding device upon receipt of said device specific command at said corresponding device
50. n about the system component is collected via a discovery protocol inter face For legacy devices a variety of system interfaces are provided for receiving the relevant information In an embodiment the system components both legacy and non legacy devices are added removed or configured via a setup wizard During this initialization or setup process relevant device information e g device type make model region assignment etc is automatically discovered or added by user input For a legacy device that is not automatically discoverable the user adds the legacy device by identifying the device by device type make model etc and assigning the device to a particular room region The initialization or setup process is also designed to acquire device metadata which includes basic operational information or idiosyncrasies about each new device such as tuning rules types and order of inputs etc The concept of metadata is further described in the application entitled Device Control System Method and Apparatus U S patent application Ser No 10 783 017 The present inven tion includes mechanisms for optimizing the collection of device metadata with little or no human intervention In an embodiment a series of questions or informational prompts are dynamically optimized to avoid seeking irrelevant or US 7 668 990 B2 3 unnecessary information In an embodiment device metadata is contained in an accessibl
51. nication Infrastructure 1106 Hard Disk Drive 1112 Removable Storage Removable Drive 1114 Storage Unit 1118 Removable Interface 1120 Storage Unit 1122 Communications Interface 1124 Communications Path 1126 FIG 11 US 7 668 990 B2 1 METHOD OF CONTROLLING A DEVICE TO PERFORM AN ACTIVITY BASED OR AN EXPERIENCE BASED OPERATION CROSS REFERENCE TO RELATED APPLICATIONS This application is a continuation in part of U S patent application Ser No 10 783 511 filed Feb 23 2004 by Kil lian et al entitled User Interface for Multi Device Control incorporated herein by reference in its entirety This application is also a continuation in part of U S patent application Ser No 10 782 923 filed Feb 23 2004 by Krzyzanowski et al entitled Method System and Com puter Program Product for Automatically Managing Compo nents within a Controlled Environment incorporated herein by reference in its entirety This application is also a continuation in part of U S patent application Ser No 10 783 017 filed Feb 23 2004 by Krzyzanowski et al entitled Device Control System Method and Apparatus incorporated herein by reference in its entirety This application is also a continuation in part of U S patent application Ser No 10 387 590 filed Mar 14 2003 by Krzyzanowski et al Legacy Device Bridge for Residen tial or Non Residential Networks incorporated herein by r
52. onal RCU a user must manually activate a hard key to send a command in the form of infrared signals to a con trollable consumer electronic CE device having direct line of sight connectivity with the RCU conventional RCU suffers from other serious deficien cies that limit its usefulness without constant user interven tion For instance in the set up process of a conventional RCU the user can program the device by assigning a particu lar key with a series of key strokes that are stored in the RCU s 20 25 30 35 40 45 50 55 60 65 2 memory However the user cannot inform the RCU ofcritical operational information of the controllable devices e g tele vision DVD player etc Such critical operational information includes the input state 1 what signals are coming into what input For example ifa television is set to the tuner input instead ofthe input for receiving DVD data when a RCU is operated to play a DVD installed in a DVD player the television will display the broadcast channel to which it is tuned at the time and not the desired DVD The user must physically select the proper input by cycling through the available inputs using the input selection button The lack ofinput state information also precludes the user from switching between different activities e g watching DVD watching broadcast TV watching cable without also physically changing from the current input state
53. ovie upon activation of the control macro After creating or defining the activity based or experi enced based operation s the control flow ends as indicated at step 395 Thus the present invention includes a setup process that provides far more flexibility than other propri etary processes for setting up a controlled environment e g system 100 For example the Harmony remote controller available from Logitech supports a setup scheme whereby device control information is gathered according to certain discrete user activities e g watching a DVD watching tele vision and listening to a radio The Harmony remotes however have several limitations For example because the Harmony remotes IR based they require direct line of sight connectivity with the devices they control and there fore can only control devices in a single room Moreover the Harmony remotes only allow one set of devices to be designated to participate in an activity Ifthe user has multiple devices of a particular type in a room e g more than one TV only one device of that type can be designated to par ticipate in a user activity In addition the setup interface used by the Harmony remotes is only available through a per sonal computer having connectivity to a website via the Inter net However the present invention allows a user to define any number or any type ofexperiences that is desired subject only to the kinds of devices
54. rsonal digital assistant PDA enhanced telephone personal television or other data processing device linked to communications network 180 As a personal computer computer client 106 can be a desktop notebook notepad or the like Portable controller 108 is a wired and or wireless data processing device that enables a user to interact with send control messages to and or manage the distribution of infor mation including audio video voice and other data among the other system components Portable controller 108 can be a portable version ofthe devices listed as computer client 106 For example portable controller 108 can be a personal note book or notepad computer PDA enhanced telephone or other device linked to communications network 180 and including a display with the ability to interact with the other system components Hence portable controller 108 enables a user to remotely control the operations of various components of system 100 In an embodiment the display for portable controller 108 is capable of receiving video and or audio from the other system components In an embodiment portable controller 108 includes a flash ROM that enables wireless downloads and or uploads Television 104 is a conventional television In an embodi ment television 104 is enhanced to support interactive and or personal services Personal services include virtual record ing programming pausing rewinding live broadcasts or the like For ex
55. s shown in FIG 1 communications network 180 sup ports peer to peer communications However the present invention also supports client server networks FIG 2 illus trates another embodiment of residential controlled system 100 that includes at least one server 208 Server 208 polices all traffic among the other system components The exchange of information among the system components is routed or otherwise controlled via server 208 As such server 208 interacts with other system components to directly or indi rectly distribute data including audio and or video voice and or control messages over communications network 180 In an embodiment server 208 commands and controls the operations and or functions of one or more of the other sys tems components The functions managed by server 208 include video serving audio serving telephony messaging file sharing Internet access and security According to embodiments of the present invention a user operates por tablecontroller 108 to establish or reconfigure these functions and or receive media from server 208 or other systems com ponents or indirectly from the system components via server 208 Inan embodiment portable controller 108 includes several functions of server 208 as described herein and manages the distribution of information including audio video voice and other data among the other system components In another embodiment supporting peer to peer communica tions th
56. serial bridge 130 d interacts with lighting device 110 A wireless Ethernet bridge 130 e interacts with television 104 5 and a DSS box 120 Wireless Ethernet bridge 130 e can support any IP addressable device As such television 104 5 and DSS box 120 are next generation plug and play devices that have IP addresses Therefore the present invention can integrate legacy devices e g CE devices that rely on infrared serial commu nications protocols as well as devices having a discovery protocol interface e g devices defined by the UPNP Forum as system components An example of a controlled environment implementing an IEEE 802 1 1 b infrared serial bridge is described in the application entitled Legacy Device Bridge for Residential or Non Residential Networks U S patent application Ser No 10 387 590 which is incorpo rated herein by reference in its entirety In an embodiment of the present invention a virtual plug and play protocol interface enables a legacy device to func tion similarly to a device having a conventional plug and play protocol interface For example one or more bridges 130 a 130 e can be modified to include a virtual plug and play protocol interface that enables a legacy device such as a conventional CE device to be advertised to and controlled by non legacy system components as if the legacy device were compliant with for example a packet based discovery and control protocol such as t
57. t invention as discussed herein In particular the computer programs when executed enable the processor 1104 to implement the pro cesses of the present invention such as the method s imple mented using for example portable controller 108 control screens 400 1000 and or other system components of system 100 described above such as method 300 for example Accordingly such computer programs represent controllers of the computer system 1100 In an embodiment where the invention is implemented using software the software can be stored in a computer program product and loaded into computer system 1100 using removable storage drive 1114 hard drive 1112 inter faces 1120 or communications interface 1124 The control logic software when executed by the processor 1104 causes the processor 1104 to perform the functions of the invention as described herein In another embodiment the invention is implemented pri marily in hardware using for example hardware components such as application specific integrated circuits ASICs Implementation of the hardware state machine so as to per form the functions described herein will be apparent to per sons skilled in the relevant art s In yet another embodiment the invention is implemented using a combination of both hardware and software While various embodiments of the present invention have been described above it should be understood that they have been presented by way of example
58. television in a bedroom the control of non IR networked devices such as devices having a discovery protocol interface the control of devices beyond line of sight via a wireless network and setup directly on a remote control device as opposed to a fixed stationary inter face II Example Controlled Environment The present invention can be implemented in any type of controlled environment such as a residence business school or the like A residential controlled environment includes the confines of a home apartment mobile home houseboat or other types of residences However in embodiments a resi dential environment includes the surrounding area ofthe resi dence as well as any shelters constructs improvements or the like within a designated perimeter In other embodiments the present invention is imple mented in a non residential environment A non residential environment includes but is not limited to an office complex suite of small offices production studio warehouse enter tainment arena school or university health care facility hotel vacation resort aircraft ship automobile orthe like In embodiments the controlled environment for a non residen tial embodiment includes not only the actual confines of the aforementioned structures but also their surroundings within a designated perimeter Examples of a controlled environment are described in the application entitled Method System and Computer Pro
59. ter system 1100 and external devices Examples of communications interface 1124 include a modem a network interface such as an Ethernet card a communications port a PCMCIA slot and card etc Software and data transferred via communica tions interface 1124 are in the form of signals 1128 which can be electronic electromagnetic optical or other signals capable of being received by communications interface 1124 These signals 1128 are provided to communications interface 1124 via a communications path 1 channel 1126 Com munications path 1126 carries signals 1128 and can be imple mented using wire or cable fiber optics a phone line a cellular phone link an RF link free space optics and other communications channels In this document the term computer program product is used to generally refer to computer readable storage media such as removable storage unit 1118 removable storage unit 1122 and a hard disk installed in hard disk drive 1112 These computer program products are means for providing software to computer system 1100 The invention in an embodiment is directed to such computer program products Computer programs also called computer control logic or computer readable program code are stored in main memory 1108 and or secondary memory 1110 Computer programs can also be received via communications interface 1124 Such computer programs when executed enable the com puter system 1100 to implement the presen
60. ts in a controlled environment according to an embodiment of the present invention FIGS 4a 4b illustrate a room control screen according to an embodiment of the present invention FIG 5 illustrates wireless controller device control screen according to an embodiment of the present invention FIG 6 illustrates a non legacy device control screen according to an embodiment of the present invention FIG 7 illustrates a legacy device control screen according to an embodiment of the present invention FIG 8 illustrates a legacy device control screen according to another embodiment of the present invention FIG 9 illustrates a device control screen for specifying device metadata according to an embodiment of the present invention FIG 10 illustrates a device control screen for specifying device metadata according to another embodiment of the present invention FIG 11 is an example computer system useful for imple menting the present invention DETAILED DESCRIPTION OF THE INVENTION I Overview The present invention describes methodologies and or techniques for configuring and managing a system compo nent of a controlled environment such as a residence busi ness school or the like The controlled environment can be implemented with a peer to peer communications system and or a client server network The system component includes legacy as well as non legacy devices legacy device is defined herein to include
61. utomati cally 1 without direct user intervention user can desire to have the bedroom television automatically turned on and tuned to a particular channel at 6 30 am as part of his or her wake up routine In another example a pre defined sequence of control commands can be executed upon the occurrence of a particular event or the occurrence of a par ticular system or component state control macro can be automatically executed when for example a television in the family room has been powered on In an embodiment the control macro is defined to be auto matically particularized for application in a specific context Herein referred to as a context based control macro a con trol macro can be particularized for use in any context e g any region or any device of comparable type In other words a specific system component or combination of components 1s not identified in the control macro A context based control macro is defined by specifying the type of system compo nent that is required to perform the activity based or experi enced based operation Upon activation the control macro is interpreted to select the appropriate system component s In an embodiment the system component is selected by deter mining the current location of the user or a system compo nent in communications with the user In another embodi ment the user interacts with a remote or portable controller to specify the appropriate syste
62. vice specific commands to a wired interface coupled to the device 22 The method according to claim 19 further comprising the step of sending the sequence of device specific commands to a wireless interface coupled to the device 23 The method according to claim 19 further comprising the step of performing the activity based or the experienced based operation to implement one or more favorite settings defined for a user 24 The method according to claim 19 further comprising the step of performing the activity based or the experienced based operation to play a playlist of media and or multimedia 25 The method according to claim 19 further comprising the step of performing the activity based or the experienced based operation to administer a security protocol established for the device US 7 668 990 B2 21 26 The method according to claim 19 further comprising the step of performing the activity based or the experienced based operation to synchronize an operation of one of the plurality of devices with an operation of one or more additional of the plurality of devices receiving a sequence of commands from said controller 27 computer program product comprising a computer readable storage medium having a program stored thereon which when executed by a processor causes the processor to control a plurality of devices to perform an activity based or an experienced based operation comprising first co

Method of controlling a device to perform an activity

Contents

Download Pdf Manuals

Related Search

Related Contents