Home

Automatically Generating High-Quality User

Contents

1. PUC Servers Devices Generation Appendix F Gallery of PUC Interfaces coe ro San P lt gt PUC Servers Devices Generation UC v4 0 1133 UH 4x 8 21 amp Power jSettings m Fax Header v Number Color m Name ca Position Status Outside Image d Mark Fax Mark bd aa 345 Tray Front Power Settings_ Bak Auto Redial v mm C Redial Busy Numbers Color eye Pause H fA Status at a4 Tray settings ___ Power jSettings v Black Fax Line bd Color Date Tons E Method Status ly Correct Error for Fax Tray Fax Speed Front _ gt copy Fox Scan Photo aT PUC Servers Devices Generation A PUC v4 0 1133 UH Settings oume i Power Black w pei Cene xs 21 Dialing ELI i o 3 2 Ringing R u I
2. Tray Front v PUC Servers Devices Generation E X UC v4 0 1133 UH 4x 8 21 amp Canon PIXMA MP780 All In One aa PUC Servers Devices Generation E PUC v4 0 1133 UH 4x 8 21 PUC v4 0 1133 UH 4x 8 21 amp PUC v4 0 1133 UH 4x 8 21 amp To use the scan features connect your Canon all in one to a computer Fror or lt E Servers Devices Generation PUC v4 0 1133 UH 4x 8 21 amp To use the photo features connect a PicBridge camera to your Canon all in one Js v4 0 1133 UH 4x 8 21 amp g H g H rli Speed Dials Power i LQualty ri Print 1 Touch Dial List Nort as oon ae 1 Touch Speed Dias p Toxchsoeedoas eh uae 7 Auto B amp W Tel No S 81266 Status Tel No H 4122687482 4 5 ay aT Ph reer renee T Front v Speed Dials x _____ Print 1 Touch Dial List Print Coded Speed Dial List Tel No alja 401814266 Tel No 419237482
3. Light Dark Quantity Image Type i a ee boar 2 Sided Copying m Finisher 1 gt 1 0 SRI Nr gt Brightness Book Copy E jd off 1 gt Back Men c Complex Copier Consistent d Simple Copier Consistent with Simple Copier with Complex Copier Figure 1 6 Copier interfaces generated with and without consistency on the Smartphone platform Chapter 1 Introduction 10 appliances may share some similar functions there are also many unique functions which must be included in the user interfaces I have developed techniques for preserving the us ability of unique functions as changes are made for consistency Examples of interfaces generated with consistency are shown in Figure 1 5 for the PocketPC and in Figure 1 6 for the Smartphone Notice that there are several differences between the original interfaces for the complex and simple copiers The visual organization of the two interfaces is different on the PocketPC and the structural organization differs on the Smart phone Labels are quite different such as Start and Stop being used on the complex copier and Copy and Cancel used on simple copier Some functions are located differ ently between the interfaces such as the 2 sided copying function which is located with the quantity function on the complex copier but with the sorting functions on the simple copier There are also unique functions such as the Halftone function on the complex
4. name false gt lt xs complexType gt lt xs complexType name ValueDependencyType gt lt xs choice gt lt xs element name constant type valueAttribNoContent gt lt xs element name ref value type stateAttribNoContent gt lt xs choice gt lt xs attribute name state type xs string use required gt lt xs complexType gt lt xs complexType name NotType gt lt xs choice minOccurs 1 maxOccurs 1 gt lt XS lt XS lt XS lt XS lt XS lt XS lt XS lt XS elemen elemen elemen elemen elemen elemen elemen elemen t Ch Gh Ee Gk eho ch eh lt xs choice gt name or type DependencyContent gt name and type DependencyContent gt name apply over type ApplyOverType gt name defined type stateAttribNoContent gt name undefined type stateAttribNoContent gt name equals type ValueDependencyType gt name greaterthan type ValueDependencyType gt name lessthan type ValueDependencyType gt lt xs complexType gt lt xs complexType name ApplyOverType gt lt xs choice minOccurs 1 maxOccurs unbounded gt lt XS lt XS lt XS lt XS lt XS lt XS lt XS lt XS lt XS elemen elemen elemen elemen elemen elemen elemen elemen elemen t t Ge R Gi a GE E G a GE lt xs choice gt lt xs attribute lt xs attribute lt xs
5. Barakonyi 2004 Istvan Barakonyi Thomas Psik and Dieter Schmalstieg Agents That Talk And Hit Back Animated Agents in Augmented Reality Proceedings of the Third IEEE and ACM International Symposium on Mixed and Augmented Reality ISMAR Washington D C Nov 2 5 2004 pp 141 150 Barnard 1981 P J Barnard N V Hammond J Morton and J B Long Consistency and compatibility in human computer dialogue nteractional Journal of Man Machine Studies 1981 15 pp 87 134 Baudisch 2004 Patrick Baudisch John Pruitt and Steve Ball Flat Volume Control Im proving Usability by Hiding the Volume Control Hierarchy in the User Interface Proceedings of CHI Vienna Austria April 24 29 2004 pp 255 262 Bojanic 2006 Peter Bojanic The Joy of XUL 2000 http developer mozilla org en docs The Joy of XUL Brouwer Janse 1992 Maddy D Brouwer Janse Raymond W Bennett Takaya Endo Floris L van Nes Hugo J Strubbe and Donald R Gentner Interfaces for consumer products how to camouflage the computer CHI 1992 Human factors in computing systems Monterey CA May 3 7 1992 pp 287 290 Card 1983 Stuart K Card Thomas P Moran and Allen Newell The Psychology of Human Computer Interaction Hillsdale NJ Lawrence Erlbaum Associates 1983 CMU 1998 CMU Carnegie Mellon Pronuncing Dictionary 1998 http www speech cs cmu edu cei bin cmudict CMU 2006 CMU Speech at CMU Pittsburgh PA 2006 2006
6. lt xs schema gt C 2 Knowledge Base Schema lt xml version 1 0 encoding utf 8 gt lt xs schema targetNamespace http www cs cmu edu pebbles puc kb elementFormDefault qualified xmins http www cs cmu edu pebbles puc kb xmlns mstns http www cs cmu edu pebbles puc kb xmins xs http www w3 org 2001 XMLSchema gt lt Top Level Element gt lt xs element name puc knowledgebase gt lt xs complexType gt lt xs sequence gt lt xs element name spec store table type SpecStoreTableType minOccurs 1 maxOccurs 1 gt lt xs element name spec map type SpecMapType minOccurs 0 maxOccurs unbounded gt lt xs sequence gt lt xs complexType gt lt xs element gt lt Attribute Types gt lt xs simpleType name WrapAttributeType gt lt xs restriction base xs string gt lt xs enumeration value allowed gt lt xs enumeration value not allowed gt lt xs restriction gt lt xs simpleType gt lt xs simpleType name IndexAttributeType gt lt xs restriction base xs string gt lt xs enumeration value 1 indexed gt lt xs enumeration value 0 indexed gt lt xs restriction gt lt xs simpleType gt lt xs simpleType name SourceAttributeType gt lt xs restriction base xs string gt lt xs enumeration value manual gt lt xs enumeration value regular automatch gt lt xs enumeration value transitive automatch gt Appendix C Other PUC XML Language Schemas
7. lt value labels gt tag Integer type for variables that take the form of integers Minimum maximum and increment values can be defined using the lt min gt lt max gt and lt incr gt tags tag specifies a dictionary to be associated with the specific value of a variable pendency information for this state variable Specifies a special group that represents a list of data The variables _ contained in this group have multiple values for every item in the list _ List selection type for variables that specify a selection in a list in a _ different location within the spec than where that list is defined _ Specifies a label dictionary for the specific value of a variable _ Defines the minimum value a numeric variable may take _ Defines the maximum value a numeric variable may take _ elsewhere in the specification _ Negates the value of a dependency equation Defines an OR relation with the dependencies that are contained within 218 Element Name sobjects gt Defines the section in which groups and objects related to a content soutputs gt Defines the section that describes the output ports of an appliance lt output ports gt Defines the output ports that are used by a pass through or source lt pass through gt _ Defines a pass through content flow lt phonetic gt Provides pronunciation information for speech interfaces that are lt pointpos gt Defines the position of the deci
8. and each group may contain any number of state vari ables commands and other groups Designers are encouraged to make the group tree as deep as possible in order to help space constrained interface generators These generators TapeRecordable i aa l WE TVIVCR Channel Key EH sere State Variable Eo n Command Add B Group FH e List Group Figure 5 3 The group tree for the sample VCR specification Chapter 5 Specification Language I2 can use the extra detail in the group tree to decide how to split a small number of controls across two screens Interface generators for larger screens can ignore the deeper branches in the group tree and put all of the controls on one panel 5 2 1 5 Complex Data Structures Our specification language uses the group tree to specify complex type structures often seen in programming languages such as records lists and unions We chose this approach be cause we felt it simplified our language and followed the principle of one way to specify anything If we had chosen to specify complex types within state variables then authors could have specified related data either as a single variable with a record data type or as mul tiple variables within a group To support complex types we have added two special group elements Figure 5 4 shows an example of the list group element added for specifying lists Specify ing a list group is similar to sp
9. 9 6 Discussion The most important concept behind the design of the PUC s interface aggregation is its use of a content flow model to help users accomplish their high level goals This approach seems to work well for the constrained domains of home theaters and presentation rooms and I believe that it can be extended to support many more features than we have discussed here For example with more detailed modeling of content types Huddle should not only be able to find content flows for users goals but also to find the optimal path for the particular con tent that the user is viewing This is a particularly important problem as the types of content within the home theater grow to encompass numerous high definition video and audio stan dards which may be supported at varying levels by different appliances and different types of wires A problem I have been considering is how content flows can help the PUC understand that the lights should be dimmed in order to view a projected PowerPoint presentation An ex treme solution would be to extend the content flows all the way to the final content sink at the user s eyes and ears although this would require extensive modeling of each room its lighting and the user s perceptual capabilities in order to be successful A more practical ap proach may be to provide basic models of which lights and projectors interact perhaps with a few environment content sinks for the most important lo
10. Appendix B Specification Language Reference 221 lt groupings gt lt groupings gt lt groupings gt Contains the entire group tree Placement Inside the lt spec gt element May Contain coro ALiSt oroup lt union grou p gt lt group gt lt group name GroupA priority 10 gt lt group gt Defines the nodes of the group tree Group nodes may be assigned a label dictionary with the lt labels gt tag Group nodes may also specify dependencies for all their members using the lt active if gt tag These dependencies are applied to the rest of the member s dependencies with an AND logical operation Each group has a unique name that is its local name concatenated with the names of all its parent groups Placement Inside the lt groupings gt lt group gt lt list group gt lt types gt or lt union group gt elements Parameters e name required The local name of this group e type name required if within types section optional otherwise defines a type with the given name e is a The Smart Template that represents this group and its children e priority The priority this group should be assigned relative to other objects in its parent group Appendix B Specification Language Reference 222 May Contain lt labels gt lt active if gt lt state gt lt command gt lt explanation gt lt group gt lt liast group gt lt ni n group gt lt list group gt lt list group name
11. Chapter 5 Specification Language 80 several specifications including the navigation system spec I decided to limit the language s expressiveness to list operations that can be specified using normal commands and state vari ables This decision eliminates most direct manipulation operations but has been sufficient for all of the specifications we have written It may even be possible to support certain unre stricted direct manipulation operations such as arbitrary moves through the use of a Smart Template though this has not been implemented in the current PUC system 5 3 2 Learnability and Ease of Use We have evaluated the learnability and ease of use of the specification language in one for mal authoring study and many informal experiences with users both inside and outside of the PUC research group The formal study was conducted with three subjects who learned the language from reading a tutorial document see Appendix D and doing exercises on their own for approximately 1 5 hours Subjects were then asked to write a specification for a low end Panasonic VCR which took on average 6 hours to complete The focus of this study was on the consistency of the resulting specifications and not learnability per se so the details of the study prepara tion and its specific results are not discussed here see Chapter 6 for those details The subjects were able to learn the language sufficiently in the short 1 5 hour period to write
12. May Contain lt incr gt lt max gt lt min gt lt specific values important gt lt specific values important gt lt specific values important gt Defines that the user will want to modify this integer type to any of the specific val ues that is supports Thus a slider or other control that enables only imprecise changes of the value should not be used Placement Inside the lt integer gt element Appendix B Specification Language Reference 232 lt list selection gt lt list selection gt lt list selection gt List selection type for variables that represent and independent selection within a list elsewhere in the specification This is particular useful in situations where the some aspect of the appliance is configured as a list and then an item from this list must be selected often during normal operation e g the channels on a VCR or TV Placement Inside the lt type gt element May Contain lt active if gt lt incr gt lt max gt lt min gt lt 1 nere lt iner gt lt max gt lt max gt lt min gt lt min gt Describe minimum maximum and increment values that the variable may take The increment may not be defined for the floating point type Placement Inside the lt fixedpt gt lt floatingpt gt except for incr lt integer gt lt list group gt except for incr and lt string gt elements May Contain lt constant gt lt ref value gt
13. Polo Chau for helping with the design of the flow based interface and the evaluation of the PUC interfaces discussed in Chapter 10 Kevin Lit wack for building adaptors for many real world appliances and helping find problems with an early version of the specification language and Michael Higgins and Joseph Hughes from Maya Design for helping with the initial design of the specification language and the initial implemenation of the PUC server side infrastructure I would also like to thank Thomas K Harris Stefanie Tomko and Roni Rosenfeld for adopting the PUC technologies as the basis for their speech interface generator Many thanks also to Htet Htet Aung Mathilde Pignol Rajesh Seenichamy and Pegeen Shen for their contributions to developing the language documentation writing specifications and building appliance adaptors xxvii I am equally indebted to the corporate and government sponsors who generously funded my research or provided equipment the National Science Foundation the Pittsburgh Digital Greenhouse Microsoft General Motors Intel Mitsubishi Electric Research Labs Lantronix Lutron and VividLogic Many others have influenced this work including other students of Brad s other students in the HCI Ph D program and students and faculty doing related work at other universities They are Daniel Avrahami Laura Dabbish James Fogarty Darren Gergle Andy Ko Des ney Tan Luis von Ahn Jake Wobbrock Gregory Abo
14. valid specifications for the VCR This suggests that the language is very easy to learn Informally we can draw some conclusions from the people who have learned and used the specification while working in the PUC research group Over the course of six years nine different people have used the language to write specifications for a number of different ap pliances Each picked up the basics of the language in a day and was proficient within about two weeks Several people from the Technical University of Vienna and ISTI Pisa have used the PUC system and also learned the specification language see Chapter 11 for more details Al though their specifications have not been as complex on average as those written by members of the PUC research team they seemed able to learn the language from the online documen tation easily and without needing to ask many questions via e mail In all cases the most difficult aspects of specification writing seem to be identifying the vari ables and commands of an appliance and organizing the variables and commands into the group hierarchy I believe these tasks are inherently difficult however and do not represent a Chapter 5 Specification Language 81 weakness in the specification language Experienced authors seem to develop a strategy where they start by identifying all of the variables and commands with little focus on organization and then specify the group hierarchy after all variable and comman
15. with the manufacturer s interfaces for the appliances My evaluations of the PUC system suggest that my interface generators can produce inter faces which are indeed faster than the manufacturers interfaces for today s computerized appliances Going forward I believe the research described in this dissertation has implications for the future of user interface design and research For design it suggests that automatic design should be considered in products where interfaces may be constrained by external factors or individual user customization may have substantial benefits For research it suggests that an important direction for future work is developing new techniques that use automatic genera tion to create interfaces that are customized to each individual Chapter 11 Conclusion 195 Chapter 11 Conclusion 196 APPENDIX A Sample VCR Specification This appendix contains the full specification for the sample VCR example discussed in Chapter 5 lt xml version 1 0 encoding utf 8 gt lt spec xmlns http www cs cmu edu pebbles puc name SimpleVCR version PUC 2 3 guid 6924EAF4 67E3 431a BDB9 9FDCE83AC679 gt lt labels gt lt label gt Simple VCR lt label gt lt labels gt lt groupings gt lt group name Base gt lt state name Power gt lt apply type type name On0ffType gt lt labels gt lt label gt Power lt label gt lt labels gt lt state gt lt group name Powered
16. All of the moving rules will also copy any appropriate missing structure from the previous specification into the new specification For example suppose that there had been an addi tional group named DateTime between the clock group and the Base Status group in the Mitsubishi DVCR specification If this had been the case when generating the Samsung in terface to be consistent with the Mitsubishi the moving rule that moved the Clock group to the Base Status group would have created a new group named DateTime in the Base Status group and placed the Clock group into the new DateTime group Any labels from the DateTime group in the Mitsubishi specification would have been replicated in the newly created group Recreating the missing structure gives the interface generator more options for adding organization when creating the concrete interface The Move Single Moded Item rule handles moves that cross the boundaries of mutually Mitsubishi DVCR Samsung DVD VCR Samsung DVD VCR first modification a Mitsubishi DVCR Samsung DVD VCR Samsung DVD VCR first modification second modification b Figure 8 6 Containment stacks for the previous specification Mitsubishi DVCR the new specification Samsung DVD VCR and the results of two consecutive movements a Shows the movement of the clock group and b shows how the rule chains with the movement of the clock channel state a Desktop PUC with Uniform v1 0
17. Chapter 11 Conclusion 184 are parameterized allowing the same templates to be used across specifications for different appliances with different functionality Templates are also described using the primitive ele ments of the specification language which allows interface generators to render a template even if the generator is not pre programmed to understand that template To support consistency a language was developed for describing semantic similarities be tween appliance specifications This language describes the similarities in relative terms e g this is the same as that which is sufficient for ensuring consistency and may be easier to spec ify and use than other ontology techniques The PUC has rules that achieve consistency between generated interfaces These rules ensure both functional and structural consistency and make use of heuristics to ensure that the us ability of unique functions is not harmed by changes for consistency Evaluations of these rules showed that users can be twice as fast when using new interfaces that are generated to be consistent as compared to interfaces generated without the consistency rules Content flow is a useful model for describing the tasks that users may wish to perform with a system of multiple connected appliances such as a home theater or presentation room The Flow Based Interface allows users to specify and execute their high level tasks with a multi appliance system Four aggr
18. Consistency should only be explicitly considered if it is beneficial to the user s tasks I agree with Grudin s position and the PUC s consistency generation rules always favor usability of the interface over consistency Chapter 6 Consistency 85 My generation techniques are also able to avoid at least two of Grudin s issues with consis tency First consider Grudin s illustrative example of the placement of knives throughout a house Placing all knives in one drawer would be the consistent design Grudin argues be cause all knives would be in one central easy to find location The usable design however is to place knives in the location where they will be used e g table knives in the kitchen putty knives in the garage swiss army knives with the camping equipment etc In this example there are many different placements for knives and although Grudin has declared one par ticular design consistent the best particular design for any given user is the configuration that user expects I call this configuration the personally consistent design because it is consis tent with the user s personal view of where knives should be located around the home The PUC interface generators attempt to achieve personal consistency meaning that functions should have the appearance and location that the user expects In this way the PUC at tempts to avoid the inconsistency that occurs when the designer and the competent user employ d
19. Defines an or relation with the dependencies that are contained within Placement Inside the lt active if gt lt and gt lt apply over gt lt not gt or lt or gt elements May Contain lt and gt lt apply over gt lt defined gt lt equals gt lt false gt lt greaterthan gt lt lessthan gt lt not gt lt or gt lt true gt lt undefined gt lt not gt AnOte lt nor Defines a not relation with the dependencies that are contained within Placement Inside the lt active if gt lt and gt lt apply over gt or lt or gt elements May Contain lt and gt lt apply over gt lt defined gt lt equals gt lt false gt lt greaterthan gt lt lessthan gt lt not gt lt or gt lt true gt lt undefined gt Appendix B Specification Language Reference 240 lt true gt xtrue gt Specifies a true value for a dependency formula Placement Inside the lt active if gt lt and gt lt apply over gt lt not gt or lt or gt elements lt false gt lt false gt Specifies a false value for a dependency formula Placement Inside the lt active if gt lt and gt lt apply over gt lt not gt or lt or gt elements lt ref value gt lt ref value state StateName gt Used to define a numeric value for any of the value space parameter tags except the lt pointpos gt tag that depends on the value of a numeric state variable E g a ref value can be
20. Instructions This appendix contains the tutorial document created for teaching the PUC specification language to the subjects of the specification authoring study The document starts on the next page to preserve formatting 263 Specification Authoring Study We are developing the Personal Universal Controller PUC system which will improve the user interfaces for common home and office appliances by moving the interface to a hand held computer The system relies on a specification language that is capable of describing the complete functionality of any appliance that the user may encounter The handheld com puter uses the specification of an appliance to automatically generate a user interface that remotely controls that appliance One of the benefits of this system is that a user interface generated for a new appliance can be made consistent with interfaces that the user has al ready used for similar appliances For example the interface generated by my handheld for controlling a VCR in a conference center would be made consistent with the interface that I use for controlling my VCR at home In this study you will create functional specifications that are suitable for interface generation by our PUC system You will start by learning how to specify an appliance through reading and working a small specification for a to do list application Then you will write a specifi cation for an appliance that we have provided you with Mitsubis
21. sive proprietary database of appliance state information Synchronization helps decrease the Figure 2 1 The Philips Pronto TSU Figure 2 2 Two Harmony remote control devices 9600 remote control device the 890 and 1000 models from left to right Chapter 2 Related Work 19 tediousness and time consuming nature of programming the remote controls but only for appliances where some user has already uploaded the codes For other appliances the pro gramming process is just as time consuming when using these advanced universal remotes 2 1 2 Commercial Standards A number of industry groups have formed to create standards for remotely controlling de vices Four of the most prominent are the Microsoft led Universal Plug and Play UPnP UPnP 2005 initiative the UPnP affiliated Digital Living Network Alliance DLNA DLNA 2006 the Home Audio Video Interoperability HAVi initiative which is led by eight of the world s leading manufacturers of audio visual electronics HAVi 2003 and the INCITS V2 effort INCITS V2 2003 which is a collaboration between the National Insti tute for Standards and Technology NIST and a consortium of researchers from industry and academia The goal of all of these standards initiatives is to create a flexible communica tion infrastructure that makes it easier for people to control the appliances in their environment and for appliances to interoperate with each other Industry standards are also
22. 1ame defined type stateAttribNoContent gt lt xs elemen lt xs elemen lt xs element name undefined type stateAttribNoContent gt lt xs element name egquals type ValueDependencyType gt lt xs element name greaterthan type ValueDependencyType gt Cre CCR och Er wGk ct GE cer lt xs element name lessthan type ValueDependencyType gt Appendix B Specification Language Reference 212 lt xs element name true gt lt xs element name false gt lt xs choice gt lt xs attribute name ignore type IgnoreType use optional gt lt xs complexType gt lt xs complexType name DependencyContent gt lt xs choice minOccurs 1 maxOccurs unbounded gt lt XS lt XS lt XS lt XS lt XS lt XS lt XS lt XS lt XS lt XS lt XS elemen elemen elemen elemen elemen elemen elemen elemen elemen elemen elemen t Cho GR Gt Gk GF Gh TGR cet Sgr eb lt xs choice gt name or type DependencyContent gt name and type DependencyContent gt name not type NotType gt name apply over type ApplyOverType gt name defined type stateAttribNoContent gt name undefined type stateAttribNoContent gt name equals type ValueDependencyType gt name greaterthan type ValueDependencyType gt name lessthan type ValueDependencyType gt name true gt
23. 6 9 gen eee Mee nner eer Reet E 14 LA MDGS EHEC OVN EW onnan a E eases 15 2 Related Work 17 Dek COto Or Ap Nani CES ioes ah itaca eis eana E ENT 18 2AL TCOMMETEIA TOAN EE EERE ac ad aa Pst oad al elias E IS DW COWIE CIAL SIAMAAVUS sissies ttc dis AAEE T OE 20 Ddadl INCI IV Z SHANNA asses coer elas T ee C O 20 DZD AMONU PIE GNA LAVE EOR 21 24 23 Digital Living Network Aliante snl niin vsyty Gintcles loeeoateereeialeueins 22 2 1 2 4 Home Audio Video Interoperability oossoo 22 E E AN E EEE A A N N A 23 R EA E S GH A A AA E A A AA 23 Dede KCI VI eae ges cis at va EEE E E ETONE 23 2 1 3 1 Universal Interactor ooooossnnneeeennnnsseneennnssssssnreneressssssererreessess 23 2 1 3 2 IBM Universal Information Appliance ns 24 L AN OT REEE E O EEAO 24 2 ABA EE E EE E E E E I E EE A 25 DTD Ubiguitons TENAN eisa EEEE AEE AE TR RE 25 2 1 3 6 Analyses of Actual Remote Control Usage 26 Del Def DOGO TLD site E E A E 26 2L 38 ROUNO seis satiate a TEO E tal ietoetieh NETOA ORA 26 2 2 Automatic amp Guided User Interface Design sssssessesssssssersssrsserssressressresreses 27 22A Baty MOG BAe SYVIN aE EE AEE E TEE EE 28 DT T E A ESR 28 DDD Geese es 29 62a Fe O AEA E A T OAR 29 DD AA TUMUNUN E OE AEI TOO EA 30 22D MATIA E aie Ri 30 TOs HOS a A 30 De I TIN T oa oan se pes ag A 31 2 2 2 Model Based Systems for Very Large Interfaces and Platform Independence31 DN MVNO Dd Senia Vania ddan fa aedsateses andouats 32 DD OI
24. AT amp T 1825 Telephone E AIWA Shelf Stereo En AIWA Shelf Stereo 3 38a wages e Vole Mai ee Pow aoe oo recat aiboxes i 2 3 Seoc o BB ne N wed 03 20 PM volume E Repeat Random Mai lace a N Thu 02 01 PM N Thu 03 36 PM a er gt fd e Dolby Noise Reduction Disc Track 8 7 See s n form em AREIE Dialing Messages setup Rec Tape Radio Video CD MD Help Network Ej Help Network E Recording Help Ej a b d Figure 3 3 Screenshots of the implemented phone a b and stereo c d interfaces for the PocketPC Chapter 3 Preliminary User Studies 40 phone digital answering machine see Figure 3 1b These two appliances were chosen be cause both are common readily available and combine several functions into a single unit I owned the Aiwa shelf stereo that was used and the AT amp T telephone was the standard unit installed in many offices at Carnegie Mellon Aiwa brand stereos seemed to be particularly common at least among the subject population because ten of the twenty five subjects in the user studies owned Aiwa systems Two sets of interfaces were designed for these appliances low fidelity paper prototype de signs for the PalmOS platform see Figure 3 2 and functionally equivalent high fidelity designs that were implemented in Visual Basic for Microsoft s PocketPC platform see Figure 3 3 A different platform was used for the hig
25. Bibliography 319 Polson 1988 Peter G Polson The consequences of consistent and inconsistent user in terfaces Cognitive science and its applications for human computer interaction 1988 Hillsdale NJ Lawrence Erlbaum Ponnekanti 2001 S R Ponnekanti B Lee A Fox P Hanrahan and T Winograd TCrafter A service framework for ubiquitous computing environments UBI COMP 2007 Atlanta Georgia 2001 pp 56 75 Puerta 2002 A Puerta Eisenstein J XIML A Common Representation for Interaction Data 7th International Conference on Intelligent User Interfaces San Francisco 2002 pp 214 215 Puerta 1997 Angel R Puerta A Model Based Interface Development Environment IEEE Software July August 1997 14 4 pp 41 47 Reisner 1981 Phyllis Reisner Formal Grammar and Human Factors Design an Interac tive Graphics System IEEE Transactions on Software Engineering March 1981 1981 SE 7 2 pp 229 240 Reisner 1990 Phyllis Reisner What is inconsistency INTERACT 1990 pp 175 181 Rheinfrank 1996 J Rheinfrank and S Evenson Design Languages Bringing Design to Software T Winograd Ed 1996 New York Addison Wesley ACM Press pp 63 80 Rich 2005 Charles Rich Candy Sidner Neal Lesh Andrew Garland Shane Booth and Markus Chimant DiamondHelp A Graphical User Interface Framework for Hu man Computer Collaboration IEEE International Conference on Distribu
26. Chapter 3 Preliminary User Studies 53 devices will have play and stop not all will fast forward rewind next track or previous track Some appliances will add their own less common functionality such as the play new button on some answering machines Describing every convention and how it should be applied for an appliance would require a lot of detailed specification and might violate our third requirement of not including any layout information in our specifications Instead I have developed an innovative flexible standardization technique call Smart Templates which allows conventions to be standard ized in advance for appliance specifications to easily describe how conventions might be applied in the appliance user interface and interface generators to appropriately render a convention based on the appliance the controller device and various properties of the user such as locale Smart Templates are discussed in more detail in Chapter 7 Chapter 3 Preliminary User Studies 54 CHAPTER 4 System Implementation I have fully implemented the PUC system allowing users to control real appliances through automatically generated interfaces on real handheld devices The chapter overviews the PUC architecture and discusses aspects of the implementation that are not discussed elsewhere 4 1 Architecture The overall architecture of the PUC system is shown in Figure 4 1 The PUC system has two main entities the appliances th
27. E Center Level i 10 i 4 Tint 4 Es a 100 CHE T M Suhwennfer FT P rs al COE PUC Servers Devices Generation E PUC Servers Devices Generation E Figure 9 5 Two shots of the Active Flow Setup AUI for the DVD player to receiver and television flow Note that the interface is organized by appliance as shown by the tabs at the bottom of the screen Chapter 9 Aggregating User Interfaces 157 and video content respectively However we found that for appliances which handled both audio and video content that this approach relied too much on the knowledge base s ability to identify sub groups that corresponded to Audio and Video It is possible that this ap proach might be viable with improvement to the consistency sub system 9 5 3 General Setup The General Setup AUI see Figure 9 6 combines setup functions across all of the appliances that are not related to any content flow These functions typically include things such as pa rental content restrictions time functions software upgrade controls and the configuration of defaults The functions for the AUI are extracted by iterating over specifications for each of the appliances and eliminating all of the functions that were used in the previous two AUIs An additional filtering step removes any functions that are not in the Setup group g step y g The General Setup AUI is organized first by any high level collections of functions that we can identify as existing on
28. General Setup and Merged Functions It is important to note that the user also has access at any time to the full interfaces for each appliance Thus it is not our goal to provide access to the full set of appliance functionality through any particular aggregate interface but instead to provide interfaces to meaningful sets of functionality from the appliances in the system for the user s current task 95 1 Active Flow Controls The Active Flow Controls AUI combines commonly used functions that are related to the currently active flows Figure 1 8 and Figure 9 4 show examples of active flow control aggre gates generated when the active flow is playing a DVD to the receiver and television Chapter 9 Aggregating User Interfaces 155 AF puc v4 0 700 UH 46 5 37 amp Puc v4 0 965 Y d 11 20 amp FIPUC v4 0 700 UH 45 41 amp AJ puc v4 0 700 UH d 5 40 amp 2 15 2 voume M E start Resume Ena I Volume a mM Volume mf ea Slide Notes a k LJ Mute Slide notes from PowerPoint appear C Mute C Mute _ e o i daki g EE i jl L gt gt JL ee ant channe 63 PREET ant channa 3 EEE Disc Meru Resolution 1024x768 7 video Format video Format Tie Wen Can mt E gt Keystone ELT a n S z seeders pe oI PUC Servers Devices Generation E PUC Servers Devices Generation E PUC Servers Devices Generation E PUC Server
29. ListGroupA priority 10 gt lt list group gt Defines a special node of the group tree that represents a list A list group has the all the same qualities of a regular group but also may contain some extra elements for describing the features of the list A list group automatically creates two states within itself The Length state stores the current length of the list If this state has an undefined value then there are no items in the list The Selection state stores the current selection s If multiple se lections are allowed then Selection is treated like another list group allowing list operators like lt apply over gt to be applied to it Three elements are provided lt item count gt lt min gt and lt max gt to allow the speci fication writer to pre specify constraints on the size of the list Placement Inside the lt groupings gt lt group gt lt list group gt lt types gt or lt union group gt elements Parameters e name required The local name of this group e type name required if within types section optional otherwise defines a type with the given name e is a The Smart Template that represents this group and its children e priority The priority this group should be assigned relative to other objects in its parent group May Contain Appendix B Specification Language Reference 223 lt labels gt lt active if gt lt state gt lt command gt lt explanation gt l
30. NOCLIICACLON as well as its value is sent seuenceeeaeneneeseneseeaeneeeeseneeeeseseeeeseseeeeseseeeeseseeeeseseeeeseseeseseseeeeseseeseseseeseseseeeeseseeseseseeeeseseesesedeeseseseeseseeeeeeseseeseseseeseseeeeseseseeseseseeseseseeseseseeeeseeeeseseseeseseseeseseseeseseseeeeseseeseseseeseseeeeseseseeeeseeeeseseeeeeeseseeseseseeseseeeeseseseeseseeeeseseeeeseseeeeseseseeseseseeseseseeseseseeseseseeeeseeeeseseeeeseseeeeseseseeseseseeseseseeseseeeeseseeeseseeeeseseeeeseseeeeeee0e000500088 binary state _ This message may be sent in two different contexts It is sent when binary Cchange notification data has changed on the appliance and in response to a state value request message from the controller _ This message contains the appliance specification It is sent after receiving a spec request message _ This message contains a string message that needs to be delivered to the user _ of each controller server information This message is sent by the service discovery manager to let a controller _ know which appliances are connected to this server and whether the set of appliances has changed XML Binary data length in bytes XML data length in bytes SS Binary data optional S 4 bytes 4 bytes arbitrary arbitrary often zero Figure 4 5 Message format for the PUC communication protocol Chapter 4 System Implementation 63 has changed rather than resending the entire list We also considered adding a versioning sy
31. One state with one of four primitive types may be used or multiple states with an integer type may be used If one state is used the Smart Template must be applied directly to that state see_Ex ample 2 above The allowable types are Integer The value of the state variable should contain the number of seconds in the time du ration Fixed Point The pointpos may be set as necessary The value of the state variable should contain the number of seconds in the time duration The decimal component represents the fractions of a second that have elapsed Floating Point The value of the state variable should contain the number of seconds in the time du ration with the decimal component representing the fractions of seconds in the duration String Appendix D Specification Authoring Study Instructions 293 The value of the state variable must be of the form hh mm ss fff mm ss fff hh mm ss or mm ss where h hours m minutes s seconds and f fractions of a second Note that it is not optional to omit digits if they are not significant In other words 1 45 56 is not valid but both 01 45 56 and 01 45 560 are Also note that three digits are required for fractions of second component if it is used If multiple states are used to represent the time duration they must have the following form Hours This state must have an integer type A minimum and maximum may be optionally specified Minutes This state must have an int
32. Recently aggregate user interfaces known better as mashups have become popular among the Web 2 0 community Merrill 2006 The PUC s combination of functionality from mul tiple appliances can be seen as a type of mashup The PUC s functionality differs from most mashups however In part this is because the PUC s mashups are produced automatically whereas current web mashups are produced manually by skilled web programmers Another difference is that the PUC s aggregate interfaces are generated to perform a particular task and combine operations rather than data Most web mashups combine one or more data col lections with a set of interaction techniques for visualizing that data often a map While these visualizations can be useful their value is in exploring the data in new ways rather than allowing the user to manipulating multiple web applications simultaneously to accomplish a specific task Chapter 2 Related Work 37 Chapter 2 Related Work 38 CHAPTER 3 Preliminary User Studies The previous chapter showed that automatically generating high quality user interfaces is a difficult problem As a first step toward automatically generating remote control interfaces we hand designed control panels for two appliances evaluated them for quality conducted two user studies and then attempted to extract the features of these control panels that con tributed most to their usability This approach helped us understand t
33. Supporting Dynamically Available Services in Client Side Applications Proceedings of ESEC FSE Helsinki Finland September 1 5 2003 pp 379 382 Bibliography 316 Harris 2004 Thomas K Harris and Roni Rosenfeld A Universal Speech Interface for Appliances International Conference on Speech and Language Processing ICSLP Jeju Ko rea 2004 HAVi 2003 HAVi Home Audio Video Interoperability 2003 2003 bttp www havi org Hayes 1985 Philip J Hayes Pedro A Szekely and Richard A Lerner Design Alternatives for User Interface Management Systems Based on Experience with COUSIN Hz man Factors in Computing Systems San Francisco CA Apr 1985 1985 pp 169 175 Herman 2006 Ivan Herman and Jim Hendler Web Ontology Language OWL W3C Semantic Web Activity 2006 http www w3 ore 2004 OWL Hodes 1997 Todd D Hodes Randy H Katz Edouard Servan Schreiber and Lawrence Rowe Composable ad hoc mobile services for universal interaction Proceedings of the Third annual ACM IEEE international Conference on Mobile computing and networking ACM Mobicom 97 Budapest Hungary September 26 30 1997 pp 1 12 IMTC 2006 Georgia Tech IMTC Projects V2 Universal Remote Console Standard Atlanta GA 2006 2006 http www imtc gatech edu projects technoloey images v2 winamp html INCITS V2 2003 INCITS V2 Universal Remote Console Specification Alternate Interface Ac cess Protocol Washington D
34. a5 419237482 Ta Tel No Front gt 417685239 PUC Servers Devices Generation A PUC v4 0 1133 UH lt 7 4x 8 18 Copy Fax Scan PUC Servers Devices Generation Ej dg v4 0 1133 UH 4x 8 18 amp HRecewe Fax v Remote Receiving Rew 1D 0 PUC v4 0 1133 UH x 8 18 x PUC v4 0 1133 UH 7 4x 8 18 j Speed Dal Power Eee Print Reports v Tahe sens sous E Tel No Da HE 44 g g ee Number Tel No E 4 EE 4 ils eae 4122687482 T Tray _ Print Coded Speed Dial List Tel No Front Front v 4122687239 Print 1 Touch Dial List ra a PE Ae Fen Use as Fax No 4 Report Settings PUC Servers Devices Generation E PUC v4 0 1133 UH 4x 8 18 amp PUC v4 0 1133 UH 4x 8 18 amp j PUC v4 0 1133 UH 4x 8 18 amp PUC v4 0 1133 UH 4x 8 18 amp Power Fax Setup H Power Volume Power Maintenance Status TRE Language Engish z Status Key Pres E E Status R
35. as in the case of the When mapping on the Samsung DVD VCR shown in Figure 8 9 Each set of adjacent objects with the same mapping becomes a block which is stored in the list in the order they would appear in a generated interface see Figure 8 9 Consecutive unmapped objects are also stored as blocks in the list The block lists are processed by the reordering rules resulting in a new block list that specifies the final ordering for the group Mitsubishi DVCR Samsung DVD VCR original Recordings original modified Channel Speed Figure 8 9 Block lists created for the timed recordings groups of the Mitsubishi DVCR and Samsung DVD VCR VCR and Type are unmapped blocks in the block lists Chapter 8 Interface Generation 132 The current reordering phase has a rule that starts by searching the block lists from the new and previous specifications to find blocks with the same mapping These blocks are re ordered to match the previous specification Unmapped blocks are moved with the block that precedes them in accordance with the second unique function heuristic For example notice in Figure 8 9 that the Type block followed the When block to its new location Un mapped blocks at the beginning or end of a block list are not re ordered in accordance with the third unique function heuristic see section 8 4 1 8 5 Generating the Concrete Interface Once the abstract user interface has been created and modified
36. clusive state and the state must have labels for all of its values The first requirement is necessary because the overlapping panel control will also control the possible values of the state variable If there was a not a set of controls for one value of the variable then no panel would be created and the user could not change the state variable to that value blank panels are also not allowed The second requirement ensures that proper labels are available for each of the panels The third requirement is that tabs have not been used yet in the interface because multiple sets of tabs can be quite confusing When tabs do exist structure is added by the third rule discussed below The second organizing rule is a special case rule for handling the power button The power button is a unique situation because typically it is the first item in the specification and all of the other functions in the specification are only active when the power variable is set to one of its particular values usually Boolean true If this situation is found the second rule will create two overlapping panels at the very top of the interface Both panels will have a power button on them which allows the user to turn the appliance on or off In the off state a spe cial appliance screen is shown with a label showing the appliance name and a large power button The on state shows all of the controls for the appliance appropriately organized see Figure 8 10a b Th
37. e How can specifications vary for different appliances that share similar functions e How can specifications vary for different authors These studies focus on specifications for VCRs which require the use of many specification language features that are usually needed to specify complex appliances e g lists smart tem plates etc 6 2 1 Study 1 The first study addressed the question of how specifications can vary for different appliances with similar functions by examining specifications that I wrote for three different VCRs Two of the VCRs were complicated with many features a Mitsubishi HD HS2000U Digital VCR and a Samsung DVD V1000 DVD VCR combo player and the final VCR was the cheapest model that we could find at our local Best Buy store a Panasonic PV V4525S I Chapter 6 Consistency 88 wrote these specifications to ensure that the quality was high I did not recruit other authors for this study because I wanted to control for differences that might arise between authors I also tried to follow the manufacturers designs for their appliances as much as possible to be sure that any differences in specifications are due to differences in the appliances All three specifications took me a total of about one week to complete In order to analyze the VCR specifications I identified the state variables and commands hereafter referred to as objects which seemed to be shared across the appliances Some ob jects were identi
38. gt lt greaterthan gt lt defined state ToDo List Selection gt active ift gt lt command gt Figure D 8 The XML code for describing the add and remove commands for the to do list Note the group that contains these items Also note that dependencies have been provided for the Delete command This will be discussed in the Dependency Information section below Appendix D Specification Authoring Study Instructions 282 The Smart Template used in this case is called list commands You may wish to read the y description of this template in Appendix A in order to understand its usage A group that is tagged with this template may contain any number of commands that are tagged with more specialized templates such as list add list move after or list clear In this case we only wish to add commands for adding and removing items from the list so we will add two commands that implement the list add and list remove templates The cards for these commands are shown in Figure and the XML is shown in Figure You should note that the list XX smart templates do not have restrictions on the labels or dependencies of these commands so this information can vary depending on what is being described If there were two ways to add data to this list for example then there could be two commands implemented the list add Smart Template each with different labels Even if there is only one command of ea
39. is Text SchemaAWSDL Authentic Browser tah Untitled1 xml XMLSpy v2005 rel 3U Registered to Jeffrey Nichols Carnegie Mellon University 1998 2005 Altova GmbH Visual Studio NET 2003 To create a new XML document in Visual Studio open the File menu and the New fed Save anyway Gv Revalidate Ln 107 Col 412 sub menu and then select File item The following dialog box will be displayed Categories Templates Visual C cript Text File Style Sheet SLT File Appendix D Specification Authoring Study Instructions HTML Page AML File Bitmap File Frameset XML Schema Icon File 775 2 Select XML File in the right pane and click Open A new XML file will be created in Visual Studio To validate your XML file with the PUC schema you will need to copy the PUC schema file to a particular location in the Visual Studio directory structure and then add a specific state ment to the top of your new file The schema file see the link above must be placed in lt Visual Studio Directory gt Common 7 Packages schemas xml Where lt Visual Studio Directory gt is the location of your Visual Studio installation Typi cally this is C Program Files Microsoft Visual Studio NET 2003 but it may vary depending on your installation Once you have copied the schema file you must start your specification with the following line lt s
40. label gt lt max gt lt labels gt lt integer gt lt command gt lt type gt lt labels gt lt label gt Channel lt label gt lt label gt Chan lt label gt lt labels gt lt state gt a b Figure 5 1 Examples of a a state variable representing the current channel tuned by the VCR and b a com mand for ejecting the tape currently in the VCR Chapter 5 Specification Language 70 type with 4 items can have a value of 1 through 4 Enumerated types must have labels de fined for each of their values The types of fixed point floating point and integer all contain numeric values Integers do not have a decimal component while fixed point and floating point both do Fixed point values have a fixed number of digits to the right of the decimal point as defined by the re quired Decimal Places field Floating point values have an arbitrary number of digits on either side of the decimal point The fixed point and integer types also have an optional in crement field that can be used to further restrict the values that the state variable may con tain If an increment is specified then a minimum value must also be specified When these parameters are specified the value of the state variable must be equal to the minimum n increment where n is some integer The string type contains a string value Specification authors can specify a minimum maxi mum and average length for the string contained in thi
41. lt map gt lt map index 2 gt lt labels gt lt label gt Play lt label gt lt labels gt lt map gt lt map index 3 gt lt labels gt lt label gt Pause lt label gt lt labels gt lt map gt lt map index 4 gt lt labels gt lt label gt Rewind lt label gt Appendix A Sample VCR Specification 198 lt labels gt lt map gt lt map index 5 gt lt labels gt lt label gt Fast Forward lt label gt lt labels gt lt map gt lt map index 6 gt lt labels gt lt label gt Record lt label gt lt labels gt lt active 1if gt lt equals state TapeRecordable gt lt constant value true gt lt equals gt lt active if gt lt map gt lt value labels gt lt type gt lt labels gt lt label gt Mode lt label gt lt labels gt lt state gt lt command name Eject gt lt labels gt lt label gt Eject lt label gt lt labels gt lt command gt lt group gt lt state name Channel is a channel gt lt type type name ChannelType gt lt integer gt lt min gt lt constant value 2 gt lt min gt lt max gt lt constant value 128 gt lt max gt lt integer gt lt type gt lt labels gt lt label gt Channel lt label gt lt labels gt lt state gt lt state name TV VCR gt lt type gt lt boolean gt lt value labels gt lt map index true gt lt labels gt lt label gt VCR lt label gt lt labels gt lt
42. lt xs complexType name TypesStateType gt lt xsS sequence gt lt xs choice minOccurs 1 maxOccurs 1 gt lt xs element name type type PrimitiveType gt lt xs element name apply type type ApplyPrimitiveType gt lt xs choice gt lt xs element name labels type LabelDictionary minOccurs 0 maxOccurs 1 gt lt xs element name active if type DependencyContent minOccurs 0 maxOccurs 1 gt gt lt xs element name required if type DependencyContent minOccurs 0 maxOccurs 1 gt lt xs element name default value type StaticOrReference gt lt xs sequence gt lt xs attributeGroup ref TypesObjectAttribs gt lt xs attribute name access type AccessType use optional gt lt xs complexType gt Appendix B Specification Language Reference 208 lt xs complexType name TypesPrimitiveType gt lt xS Sequence gt lt xs choice minOccurs 1 maxOccurs 1 gt lt xs element name binary type BinaryType gt lt xs element name boolean gt lt xs element name enumerated type EnumeratedType gt lt xs element name fixedpt type FixedPtType gt lt xs element name floatingpt type FloatingPtType gt lt xs element name integer type IntegerType gt lt xs element name list selection type ListSelectionType gt PEAGE yeh uche ar or set 5 lt xs element name string type StringType gt lt xs choice gt
43. lt xs elemen 1ame State type StateType gt lt xs element name command type CommandType gt lt xs element name explanation type CommandType gt Gh nek 9 he seh ack oer 2 lt xs element name apply type type ObjectApplyType gt lt xs choice gt lt xs sequence gt lt xs attributeGroup ref TypesObjectAttribs gt lt xs complexType gt lt xs complexType name TypesUnionGroupType gt lt xs sequence gt lt xs element name labels type LabelDictionary minOccurs 0 maxOccurs 1 gt Appendix B Specification Language Reference 207 lt xs element name active if type ActivelfType minOccurs 0 maxOccurs 1 gt lt xs choice minOccurs 1 maxOccurs unbounded gt lt xs element name group type GroupType gt lt xs element name list group type ListGroupType gt lt xs element name union group type UnionGroupType gt lt xs element name state type StateType gt lt xs element name command type CommandType gt lt xs element name explanation type CommandType gt Gi Gh 4c ey seto ch lt xs element name apply type type ObjectApplyType gt lt xs choice gt lt xs sequence gt lt xs attributeGroup ref TypesObjectAttribs gt lt xs attribute name access type AccessType gt lt xs complexType gt lt xs complexType name TypesListGroupType gt lt xs sequence gt lt xs element name labels type LabelDiction
44. sion PUC 2 2 gt 9 Labels for specification Sem lt labels gt lt label gt PUC To Do List Application lt label gt lt label gt To Do List App lt label gt lt label gt To Do List lt label gt lt labels gt Eps Groups gt lt Groupings gt lt group name ToDo priority 10 gt lt labels gt lt label gt To Do List lt label gt lt label gt List lt label gt lt labels gt lt list group name List priority 10 gt selections nimher one gt lt state name Completed priority 5 gt lt Lype lt boolean gt lt value labels gt lt map index true gt lt labels gt lt label gt Done lt label gt lt labels gt lt map gt lt map index false gt lt labels gt lt label gt Incomplete lt label gt lt labels gt lt map gt lt value labels gt lt type gt lt labels gt lt label gt Completed lt label gt lt labels gt lt state gt Appendix D Specification Authoring Study Instructions 296 lt state name Category priority 4 gt lt Cypa gt lt list selection list Setup Categories List gt lt type gt lt labels gt lt label gt Category lt label gt lt labels gt lt state gt lt state name Description priority 8 gt Lype lt string gt lt type gt lt labels gt lt label gt Description lt label gt lt label gt Desc lt label gt lt labels gt lt state gt lt state name CompletionDate is a date priority 3 gt LyYp
45. true gt lt xs sequence gt lt xs element name undefined minOccurs 0 maxOccurs 1 gt lt xs sequence gt lt xs complexType gt lt xs complexType name NewValueType mixed true gt lt xs sequence gt lt xs element name undefined minOccurs 0 maxOccurs 1 gt lt xs sequence gt lt xs attribute name State type xs string gt lt xs complexType gt lt xs complexType name AnyListContentType gt lt xs choice gt lt xs element name data type ListDataType gt lt xs element name change type ChangeDataType gt lt xs element name insert type InsertType gt lt xs element name delete type DeleteType gt lt xs element name replace type ReplaceType gt lt xs sequence gt lt xs element name value type NewValueType minOccurs 1 maxOccurs unbounded gt lt xs sequence gt lt xs choice gt lt xs complexType gt lt xs complexType name AfterChangeOpListType gt lt xs choice gt lt xs element name data type ListDataType gt lt xs sequence gt lt xs element name value type NewValueType minOccurs 1 maxOccurs unbounded gt lt xs sequence gt lt xs choice gt lt xs complexType gt lt xs complexType name ListDataType gt lt xsS sequence gt lt xs element name el type AfterChangeOpListType minOccurs 1 maxOccurs unbounded gt lt xs sequence gt gt lt xs attribute name state type xs string use requir
46. which they may then instantiate in their specifications Specification authors are not required to use any templates in their specifications but of course there is a strong benefit to doing so because of the potential increase in generated interface quality e Interface Generator Programmers build the software that automatically generates user interfaces These programmers might be employed by the controller device manufac turer or a third party Programmers learn about available Smart Templates from the Template Registrar and will choose which templates to implement based on the properties and requirements of their target controller device There is no requirement for an interface generator to support any Smart Template though some would be highly recommended such as the date and time templates Chapter 7 Handling Domain Specific and Conventional Knowledge 100 Template Registrar Interface Generator Specification Author Programmer gt Appliance Specification Interface Interface Generator User Controller Device Figure 7 1 A diagram showing the four different roles in the creation implementation and use of a Smart Template in the context of the interface generation process e The Template Registrar is a global entity that manages the list of all Smart Templates and defines the required and optional parameters of those templates New templates may be developed by the registrar or proposed by other ent
47. 00 10 00 01 00 00 00 I I T 05 00 00 00 Tasks total Built In HP PUC HP Built In Canon PUC Canon Figure 10 3 Results of the first block of tasks showing the Built In condition compared with the other two for each appliance Time Built In PUC Failures Built In PUC Table 10 1 Average completion time and total failure data for the first block of tasks The PUC condition is the combination of the AutoGen and Consistent AutoGen conditions N 8 for the Built In condition and N 16 for the PUC condition indicates a significant difference between the Built In and PUC conditions for that appliance p lt 0 05 and f indicates a marginally significant difference p lt 0 1 Completion times and total failures were compared with a one way analysis variance and failures per task were compared with a one tailed Fisher s Exact Test Chapter 10 Usability Evaluation 169 Subjects overall had more difficulty using the Canon interfaces as compared to the HP inter faces across all conditions F146 6 25 p lt 0 02 but we still see the same significant benefits for the PUC interface over the built in interface Again subjects were significantly faster us ing the PUC F 22 21 88 p lt 0 001 with average total completion times of 9 32 for the PUC interface and 20 33 for the built in interface again about half the time Subjects also failed significantly less often using the PUC Fi 6 57 p
48. 3 Speech The speech interface generator creates an interface from a PUC specification using USI in teraction techniques Shriver 2001 The goal of these universal speech techniques is to balance the flexibility of natural language interfaces which are difficult to implement and typically do not convey their limitations to users with the structure of specialized hierarchi cal menu based speech interfaces The structure is given by a set of common interaction primitives that are designed to be learnable in five minutes but applicable to all of the inter faces that a user encounters hese interaction primitives allow users to navigate manipulate and understand any interface built with the USI techniques Studies have been conducted to evaluate users acceptance of these interfaces and the transference of USI skills from one in terface to another These studies show users prefer USI based interfaces to natural language interfaces though work is still needed to help users learn and remember the interaction primitives Tomko 2004 The speech interface generator differs from the graphical interface generators in that it con nects to multiple appliances if multiple appliances can be found on the network requests their specifications and then automatically generates an interface that allows control of all of the appliances collectively This includes building a grammar for the parser and a language model and pronunciation dictionary for the
49. 31 2 2 2 1 Mobi D Mobi D Puerta 1997 is model based user interface development environment capable of producing very large scale user interfaces The Mobi D development process differs from previous systems in that a series of declarative models are created iteratively starting with models of the users and their tasks and ending with a presentation model that represents the final interface All of these models are stored together and many relations are described be tween the different models to assist the system and designer with their interface building and maintainenance tasks Mobi D also has many component tools for helping designers at vari ous phases of the interface design process from describing the users tasks to the final guided assembly of the user interface Assembly of interfaces in Mobi D is a highly structured proc ess where the system steps the user through each of the sub tasks associated with the interface and provides suggestions of appropriate controls for a particular task Mobi D and the rest of the model based systems for building large scale interfaces serve a different pur pose than the PUC system and can be seen as complementary It is conceivable that features from the PUC system such as automatic modifications for consistency or aggregation of user interfaces could be beneficial in larger scale interfaces Models of the interface will be neces sary to implement these features however and these models might al
50. 7 x va S Ca a Receiver N Q gt S By J VCR Fa Wiring diagram G FE es O 3 A Ee Zs g ey T Spec g DVD Port info ee Knowledge Base 2 Internal flow lt a pE DVD Receiver am oE VCR System Wide Content Flow Model gt gt gt gt gt gt gt T m gt ainim midan Flow based TAS ase User Interface FBI AU Figure 9 2 Architecture of the aggregate interface generation features Chapter 9 Aggregating User Interfaces 151 onto the screen The goal of this interface is to make high level tasks easy to execute with the multi appliance system The PUC also generates Aggregate User Interfaces that combine functions from multiple appliances into a single user interface Various types of aggregate user interfaces support dif ferent tasks within the multi appliance system The Active Flow Control aggregate combines the most common control functions associated with the active content flows into a single interface with the goal of making common content manipulations such as volume control easy to access The Setup aggregates make infrequently used configuration parameters easy for the user to access with the goal of supporting expert usage of the appliance system Finally aggregates can merge some functions that occur on multiple appliances into a single point of control on one interface This allows the user to do such things as set the current time in an ageregate an
51. Also note that dependencies have been provided for the Delete command This will be discussed in the Dependency Information section below Figure 11 10 The dependency card for the Delete command Note that extended names are used here for the state variables You may find it necessary to go back and make names more explicit on dependency cards after you have organized the variables in your specification eeseeeeseeeeenees 6202 Gu2oZ 284 XXIV List of Tables Table 4 1 Table 4 2 Table 4 3 Table 4 4 Table 5 1 Table 5 2 Table 6 1 Table 7 1 Table 8 1 Table 8 2 Table 8 3 Table 8 4 Table 10 1 Table 10 2 Appliance adaptors built by the PUC research team eee eeeeseeeeeseeeeneeeeenees 58 Appliance simulators built by the PUC research team eeeeceeeesceeeeeteeeeeees 58 Messages that may be sent by the controller device eeeeeeseeeeeseeeeeneeeeees 63 Messages that may be sent by the appliance to a controller device 63 Complete list of appliance specifications authored by the PUC Fert oe Be Gln Goo 1 n a eee ee oR E ET 80 Maximum and average counts of various aspects of the PUC SPECIFICATIONS Watten TOA Ate eira i a Ee 80 Mapping types for consistency in the PUC system ssesssesssesssrsseresseressrersersses 96 Description of all implemented Smart Templates in the PUC system 111 The PUG s functional consistency roles
52. An interface for Windows Media Player generated on a PocketPC b The full interface gener ated for the GMC Yukon Denali driver information console functionality in the menus see Chapter 3 Figure 8 1 shows two example interfaces gener ated for Windows Media Player and the driver information console DIC of a GMC Yukon Denali The input and output technologies used on the PocketPC a touch sensitive medium sized screen are similar to those of the desktop and a result the generation rules used for the PocketPC are also used for the desktop The main difference between the desktop and Pock etPC generators is the choice of controls which is slightly better on the desktop because more controls are available than on the PocketPC For the remainder of this chapter when ever I discuss rules for generating PocketPC interfaces it can be assumed that these same rules are also used for generating desktop interfaces unless otherwise noted 8 1 2 Smartphone The Smartphone interface generator uses Microsoft s Windows CE based Smartphone plat form This platform is also a set of hardware requirements and a Windows CE operating system Unlike the PocketPC the required screen is smaller 220x176 and it does not have any touch sensitivity Instead interaction takes place through a 4 way directional pad a nor Chapter 8 Interface Generation 115 mal phone keypad home and back buttons and two soft buttons with labels that are shown on the
53. Defined True when the specified state has any value e Undefined True when the specified state does not have any value These dependencies can be composed into Boolean formulas using AND and OR NOT may also be used Specifying dependencies on list state variables must be done using the special apply over op eration This element applies the dependencies it contains over some items in a list and returns a value depending on the value of its true if property The dependencies are applied over the set of items based on the items property which may be set to all or selected The true if property may be set to all any or none If true if is all then all of the dependencies con tained in the apply over element must be true for elements if true is returned If true if is any then all of the dependencies contained in the apply over must be true for one of the Appendix D Specification Authoring Study Instructions 269 elements If true if is none then the dependencies must not be satisfied by any element to return true Smart Templates Smart Templates are standardized in advance so that specification authors can specify high level conventions that the interface generators will understand A number of templates have already been defined as described in the Smart Templates Appendix at the end of this document To use a Smart Template in a specification the author must do two things Tag a group or appliance object with the name
54. Element Name lt selections gt Defines the number of selections allowed in a list and whether the user lt server side Specifies that this state variable will error corrected on the server side error correction gt lt sortable gt Specifies that this list group has not natural order and may be arbitrar lt source gt Defines a source of content lt spec gt Every specification begins with this tag lt specific values Defines that the specific values of a numeric type will be interesting to important gt o _ the user as opposed to the position of the value in the range lt state gt Defines a state variable appliance object lt string gt String type for variables that take the form of strings lt text to speech gt Defines a text to speech entry to be included within a label dictionary lt true gt True for use in dependency expressions lt type gt Describes the value space of a state variable ex Boolean Integer lt types gt The first section of the specification in which authors may define sundefined gt Used in conjunction with the lt active if gt element to define a de sunion group gt Specifies a special group in which only one of the children may have a lt value labels gt Contains one or more lt map gt tags that provide label dictionaries for Appendix B Specification Language Reference Description _is allowed to modify the selection ily sorted by the user inte
55. Jakob Nielsen Usability Engineering Boston Academic Press 1993 Nylander 2004 Stina Nylander Markus Bylund and Annika Waern The Ubiquitous In teractor Device Independent Access to Mobile Services Computer Aided Design of User Interfaces CADUTI Madeira Portgual 2004 pp 271 282 Olsen Jr 1989 Dan R Olsen Jr A Programming Language Basis for User Interface Management Human Factors in Computing Systems Austin TX Apr 1989 1989 pp 171 176 Olsen Jr 2000 Dan R Olsen Jr Sean Jefferies Travis Nielsen William Moyes and Paul Fredrickson Cross modal Interaction using Xweb Proceedings UIST 00 ACM SIGGRAPH Symposium on User Interface Software and Technology San Diego CA 2000 pp 191 200 Omojokun 2006 Olufisayo Omojokun Jeffrey S Pierce Charles L Isbell Jr and Prasun Dewan Comparing End User and Intelligent Remote Control Interface Genera tion Personal and Ubiquitous Computing April 2006 10 2 3 pp 136 143 OSGi 2006 OSGi The OSGi Alliance Home Page 2006 http www osetore Paterno 1997 Fabio Paterno C Mancini and S Meniconi ConcurTaskTrees A Dia grammatic Notation for Specifying Task Models INTERACT Sydney Australia 1997 pp 362 269 Polson 1986 Peter G Polson E Muncher and G Engelbeck A test of a common ele ments theory of transfer SIGCHI conference on Human factors in computing systems New York NY ACM Press 1986 pp 78 83
56. PUC generated interfaces were presented on a Microsoft PocketPC device see Figure 10 1 The discussion of the two user studies starts with a description of the interfaces that were compared and the common protocol used for both studies This is followed by sections pre senting and discussing the results for each of the studies Chapter 10 Usability Evaluation 162 Resize Options Zoom 25 400 gt Zoom 25400 EBB ooa Presets o Density K_i C Borderless Copy oo u o Color Saturation Density 4 __ m gt o Saten Deest a m gt Least vivid Mest vivid Enhancements a HP printer b Canon printer c HP printer d Canon printer without consistency without consistency consistent with consistent with Canon printer HP printer Figure 10 1 PocketPC interfaces generated by the Personal Universal Controller PUC for the two all in one printers discussed in this paper 10 1 Interfaces The studies compare PUC generated interfaces with the manufacturers human designed interfaces for the same appliances and compare PUC generated interfaces with and without consistency for the two different printers The manufacturers interfaces for the two all one printers are shown in Figure 10 2 PUC specifications of both all in one printers were needed in order for the PUC to generate interfaces I wrote the initial specification for the Canon printer and a staff member wrote the initial specification for the HP
57. San Jose CA 2002 pp 117 128 Merrill 2006 Duane Merrill Mashups The new breed of Web app 2006 http www 128 ibm com developerworks library x mashups html Microsoft 2006 Microsoft XAML 2006 http windowssdk msdn microsoft com en us library ms747122 aspx Miori 2006 Vittorio Miori Luca Tarrini Maurizio Manca and Gabriele Tolomei Do moNot a framework and a prototype for interoperability of domotic middlewares based on XML and Web Services International Conference on Consumer Electronics ICCE 06 January 7 11 2006 pp 117 118 Mori 2004 Giulio Mori Fabio Paterno and Carmen Santoro Design and Development of Multidevice User Interfaces through Multiple Logical Descriptions IEEE Transac tions on Software Engineering 2004 30 8 pp 1 14 Mori 2002 Guillo Mori Fabio Paterno and Carmen Santoro CTTE Support for Devel oping and Analyzing Task Models for Interactive System Design IEEE Transactions on Software Engineering September 2002 2002 28 9 pp 797 813 Myers 2000 Brad A Myers Scott E Hudson and Randy Pausch Past Present and Future of User Interface Software Tools ACM Transactions on Computer Human Interaction 2000 7 1 pp 3 28 Newman 2002 Mark W Newman Shahram Izadi W Keith Edwards Jana Z Sedivy and Trevor F Smith User Interfaces When and Where They are Needed An Infra structure for Recombinant Computing UIST O2 Paris France October
58. Tape Player Counter Total Distance Driven 144 1 Total hours engine run 39 Annual Mileage 144 1 00 01 23 Dolby Noise Reduction 4 gt before before Figure 8 13 Diagrams showing how the first a and second b rules for optimizing the list structure behave Black solid arrows indicate how the screens are connected and red dashed lines indicate changes made by the rules Note that in a some items were list only and thus were promoted to the top level list while the others were placed on a panel The items all happen to be labels so this panel is a summary pane Chapter 8 Interface Generation 141 The third rule looks for any remaining panel only and list or panel items that have not been assigned to an editing pane Every list or panel item is assigned to a list and each remaining panel only item is given its own editing pane as discussed above Now that all of the editing panes have been created and every item has been assigned to an editing pane or a list we can now optimize the number of items in a list The fourth and fifth rules are very similar to the first and second rules except that they manipulate only list items The fourth rule eliminates unneeded child lists by moving all of their items into the parent list if there is enough room This rule always promotes the most important items first because the list hierarchy is being traversed in priority order The fifth rule tries to break up lists that have mor
59. Templates have been defined e address e date e date time e four way dpad e four way dpad with enter e list commands e media controls e time absolute e time duration e 7zoom controls The format for each of these templates is described below address Overview Represents an address like might be entered into a navigation system Contents This template only supports the Multi State form In the future it might make sense to sup port a single state string form that contains a parsable address but this is not required now There are no required states in this template This allows the template to be used in a num ber of situations where a partial piece of an address is needed such as zip code by itself The different states in this template are StreetName This state represents the name of the street It may have a string or enumerated type It must support the use of completions and server side error correction if either of those features are available ApartmentNumber This state represents the number of an apartment or suite at this address This state may have a string or integer type When the integer is used it may have type restric Appendix D Specification Authoring Study Instructions 285 tions such as bounds or an increment The labels of this state must be taken into ac count in order to know whether the number is for a suite apartment etc StreetNumber This state represents the
60. UH Starting Done Attempting to open server at 192 168 55 100 Connected to appliance server 192 168 55 100 Server information updated for chupacabra PUC v4 0 1133 UH Starting Done Attempting to open server at 192 168 55 100 Connected to appliance server 192 168 55 100 Server information updated for chupacabra PUC v4 0 1133 UH Starting Done Attempting to open server at 192 168 55 100 Connected to appliance server a a zimp Open Server Clear Mapping Counts Clear Specification Cache Save Knowledge Base Logging Pane Options Exit Disconnect Multi Appliance Systems Take Screenshot Refresh i 192 168 55 100 Take Full Appliance Screenshot 128 237 236 211 WD CEA Copier A Copier B Reset Interface a b c d Figure 4 3 Screenshots of the menu interface for the PocketPC PUC interface generator The desktop inter face generator has a similar menu structure The backgrounds of all these screenshots show the logging panel where messages from the interface generator are displayed recently used server Appliance servers are machines through which multiple appliances may be attached These servers maintain a list of all the appliances that are connected to them and send this list to controller devices when they connect The Servers menu shown in Figure 4 3b shows the servers that the controller device is connected to with a hierarchical menu for each server showing the appliances th
61. Web and Grid Services 2003 pp 28 35 Staab 2003 Steffen Staab e al Web Services Been There Done That IEEE Intelligent Systems 2003 18 1 pp 72 85 Sukavirtya 1990 Piyawadee Sukavirtya and James D Foley Coupling A UI Framework with Automatic Generation of Context Sensitive Animated Help ACM SIG GRAPH Symposium on User Interface Software and Technology Snowbird Utah Oct 1990 1990 pp 152 160 Sukavirtya 1993 Piyawadee Sukavirtya James D Foley and Todd Griffith A Second Generation User Interface Design Environment The Model and The Runtime Ar chitecture Human Factors in Computing Systems Amsterdam The Netherlands Apr 1993 pp 375 382 Sun 2003 Sun Jini Connection Technology 2003 2003 Sycara 2003 Katia Sycara Massimo Paolucci Anupriya Ankolekar and Naveen Srinivasan Automated Discovery Interaction and Composition of Semantic Web Services Journal of Web Semantics 2003 1 1 Szekely 1995 P Szekely Sukaviriya P Castells P Muthukumarasamy J Salcher E Declarative Interface Models for User Interface Construction Tools the Master mind Approach 6th IFIP Working Conference on Engineering for Human Computer Interaction Grand Targhee Resort 1995 pp 120 150 Szekely 1992 Pedro Szekely Ping Luo and Robert Neches Facilitating the Exploration of Interface Design Alternatives The HUMANOID Model of Interface Design Human Factors in Comput
62. X Print Last Transaction Print Fax Log Method Volume Appendix F Gallery of PUC Interfaces PUC v4 0 1133 UH 4x 8 15 amp HP Photosmart 2610 All in One Figure F 1 The full interface for the HP All In One printer generated without consistency as seen by users in the usability study 310 Cere oe e Status aa WRB Print on Both sides fs 9 Front l C Collate C Two Pages on One Repeat Off X Image rowe l7 TIA PUC Servers Devices Generation E PUC v4 0 1133 UH 4x 8 17 PUC Servers Devices Generation PUC v4 0 1133 UH 7 4x 8 18 PUC v4 0 1133 UH 4x 8 18 amp Status To use the scan features connect your Canon all in one to a computer To use the photo features connect a PicBridge camera to your Canon all in one at 355 Tray Front v Use as Fax No Power Status Tel No 401814266 a4 Tel No
63. a In the before tree the node marked A represents a state variable that can have values from 1 3 The node marked P is the parent group The A formulas shown below the remaining groups show the dependencies of each group on the variable A The A node is not dependent at all on A b In the after tree a new mutual exclusion group has been added that contains the A state variable and child groups for each set of mutually ex clusive groups The nodes not dependent on A are moved into their own group under the same parent but outside of the mutex group Chapter 8 Interface Generation 122 8 3 2 Choosing Abstract Interaction Objects Abstract Interaction Objects AIOs are abstract representations of the controls like buttons and sliders used in the concrete user interface The idea was developed by Vanderdonckt et al Vanderdonckt 1999 to build models of user interfaces that could be ported across differ ent platforms and I use them for the same reason in the PUC s abstract user interface The PUC also uses AIOs to delay selection of the concrete control until more properties of the concrete interface are known such as the available layout space or context of use within a list on a panel etc The interface generator chooses an AIO for each state variable and command using a deci sion tree de Baar 1992 A different tree is designed for each interface generator because each platform supports a different set of controls that must b
64. a list of thumbnails of all the images and a single record view that _ shows a larger image along with any other fields in the list This template is used _ primarily in the interface for the photo browser appliance A eueeeeaeaeneeeeeeneeeeeseeeeseeeeeeseseseeseseeeeseseeeeseseeeeseseseeseseeeeseseseeseseesesesessesepeeeeseseeeeseseeseseseeeeseseeeeseseeeeseseeeeseseeeeseseeseseseeseseseeeeseseeeeseseeeeseseeeeseseeeeseseeseeseseeeeseseeeeseseeeeseseeseseseeeeseseeeeseseeeeseseeeeseseeeeseseeeeseseeeeseseeeeseseeeeseseeeeseseeseseseeeeseseeseseseeeeseseeeeseseeeeseseeseseseeeeseseeeeseseeeeseseeeeseseeeeseseeeeseseeeeseseeeeseseeseseseeeeseseeeeseseeeeseees list commands Represents the common operations that might be supported by a list such as add remove clear and move The use of a template allows these operations to be better _ integrated with the list controls particularly for moving items Chapter 7 Handling Domain Specific and Conventional Knowledge 111 media controls status icons time absolute Zoom controls fe 6 Represents the functions for controlling the playback of a variety of media includ _ ing common functions such as play stop and pause and less common functions _ such as play new for answering machines reverse play for audio tapes and multiple playback speeds for VCRs and DVD players Represents a phone number and or the controls for entering a phone number such _as the standard phone keypad Lik
65. and X10 lighting Chapter 8 Interface Generation 118 There were a few unique challenges in generating an interface using USI techniques from the PUC specification language The language makes a distinction between state variables and commands but what is best described as a variable in a visual interface e g speaker volume might be better thought of as a command in a spoken interface e g louder This might be addressed by adding support for verb labels in the specification language that specify command words that could be used to manipulate a state variable from a speech interface Secondly in a visual environment groupings of functionalities or widgets need not have a name such grouping can be implied by adjacency or by a visual cue such as a frame In speech interfaces grouping must be assigned a word or phrase if they are to be directly ac cessed The specification language does not require groups to have labels making appropriate names hard or impossible to find It may be necessary to require labels on each group for speech interfaces but more work is needed to determine if this is necessary Using a speech interface also has benefits For example choosing from a long list of names is easy with speech yet can be challenging for visual interfaces 8 2 General Concepts An overview of the PUC generation process for graphical interfaces is shown in Figure 8 4 The process is made up of six consecutive phases that fall
66. as repeatedly repeated invoke invoking a command on another change invoke Converts between the situation where one appliance has a state variable and the other has a state Coeds variable with a command that must be invoked before a variable change will take affect time end to Converts between the situations where one appliance uses a start time and an end time and an duration other appliance uses a start time and a duration Ensures that nodes mapped with a node mapping use the same label Chapter 8 Interface Generation 125 8 4 3 Structural Modifications The goal of the abstract structural phase is to ensure that functions are located in the same place in new interfaces This phase is divided into two sub phases moving and then re ordering Moving rules only need to ensure that functions are placed in similar groups and then the re ordering rules can ensure that the functions have a consistent ordering within their groups Both of these sub phases rely on mappings such as node template and list mappings that identify similar groups across specifications Uniform uses this information to rearrange groups so that they have the same structure as a previous specification 8 4 3 1 Moving The moving sub phase traverses the abstract user interface s group tree and searches for map pings There are two difficulties that may arise when processing mappings to determine whether a move is necessary First the new specification
67. asked to fill out a consent form and a two page ques tionnaire about their computer background and remote control use Then each subject worked with two interfaces in one of four possible combinations to control for order effects Each subject saw one actual interface and one handheld interface and one stereo interface and one phone interface neither necessarily in that order For each interface the user was asked to work through a set of tasks When finished a final questionnaire was given that asked whether the actual appliance or PDA interface was preferred and for any general com ments about the study and the interfaces 3 22 Evaluation In order to compare the interfaces for both appliances task lists were created for the stereo and phone Each list was designed to take about twenty minutes to complete on the actual appliance and the same tasks were used for both the handheld and actual interfaces About two thirds of the tasks on both lists were chosen to be easy usually requiring one or two but ton presses on the actual appliance Some examples of easy tasks are playing a tape on the stereo or listening to a particular message on the phone The remaining tasks required five or more button presses but were chosen to be tasks that a user was likely to perform in real life Chapter 3 Preliminary User Studies 42 These included programming a list of tracks for the CD player on the stereo or setting the time on the phone It w
68. attribute name or type DependencyContent gt name and type DependencyContent gt name not type NotType gt name apply over type ApplyOverType gt name defined type stateAttribNoContent gt name undefined type stateAttribNoContent gt name equals type ValueDependencyType gt name greaterthan type ValueDependencyType gt name lessthan type ValueDependencyType gt name list type xs string use required gt name items type ItemsType use optional gt name true if type TruelfType use optional gt lt xs complexType gt lt xs complexType name PortsGroup gt lt xs sequence gt lt xs element name inputs type PortsType minOccurs 0 maxOccurs 1 gt lt xs element name outputs type PortsType minOccurs 0 maxOccurs 1 gt Appendix B Specification Language Reference 215 lt xs element name input output type PortsType minOccurs 0 maxOccurs 1 gt lt xs sequence gt lt xs complexType gt lt xs complexType name PortsType gt lt xs choice minOccurs 1 maxOccurs unbounded gt lt xs element name port type PortType gt lt xs element name port group type PortGroupType gt lt xs choice gt lt xs complexType gt lt xs complexType name PortType gt lt xs attribute name name type xs string use required gt lt xs attribute name content type type ContentTypeAttrib use requi
69. be sure that users will perform well with either type of interface My goal is to allow an inexperienced user to walk into my home theater room and be able to use the system rather than wait for me to set things up It may also be interesting to compare the flow based aggregate interface designs with the interfaces produces by today s integrated interface products such as the Philips Pronto or Logitech Harmony universal remotes In the final usability study of the PUC interface it was noticed that when users failed they often failed because a control was not where they expected it When asked to perform a task the typical process was for users to navigate to the panel where they believed the function was If the function was not there then users would begin a heuristic search of the rest of the Chapter 11 Conclusion 191 interface looking for the function Because users were not performing an exhaustive search it was quite common for them to open the panel that contained the function but not find the function because they did not believe that the function would be on that panel When users were asked why they missed functions in this case a common response was if it s not there it s not there This suggests to me that users might benefit from a searchable user interface where users can provide one or more keywords and view a list of results from the current in terface with controls for using the functions embedded in the searc
70. being developed to enable the construction of distributed net work services Unlike the above standards this work is focused on generic services rather than specifically on electronic appliances The proposed technologies in these standards in cluding service discovery and description could be used to find and control appliances These standards include Sun Microsystem s JINI system Sun 2003 and the OSGi Alliance OSGi 2006 2 1 2 1 INCITS V2 Standard Recent government legislation requires that appliances purchased by the government or government entities be usable by people with a wide variety of disabilities Unfortu nately most appliances built today have no accessibility features The InterNational Committee for Information Technology Standards INCITS has begun the V2 standardization effort INCITS V2 2003 Figure 2 3 An interface generated for WinAmp ae on a PocketPC using the INCITS V2 frame which is currently developing standards for soak IMG 2000 Repedea wik permission Chapter 2 Related Work 20 a Universal Remote Console URC that would enable many appliances to be accessible through the Alternative Interface Access Protocol AIAP A URC controls an appliance by using AIAP to download a specification written in three parts a user interface socket that describes only the primitive elements of the appliance a presentation template that de scribes either an abstract or concrete user interf
71. c the Canon PIXMA MP780 All In One printer and d the HP 2610 All In One printer PocketPC has a medium sized touchscreen that allows interactions similar to a desktop com puter though with a smaller screen area see Figure 1 2 The Smartphone has a small screen with no touch sensitivity and a four way joystick for navigating around the interface As a result Smartphone interfaces are list based and typically have a much deeper hierarchy than PocketPC interfaces see Figure 1 3 The interface generators have been tested with a variety of complex appliance specifications including those for several VCRs the HP and Canon all in one printers mentioned above and even the navigation system for a GMC Yukon Denali SUV This latter specification for the navigation system is especially complex but is easily handled by the PocketPC interface generator see Figure 1 4 I have also evaluated the usability of the generated interfaces by comparing the PUC produced interfaces for the HP and Canon all in one printers see Figure 1 2c d with the manufacturers interfaces on the actual appliances see Figure 1 1b c The comparison showed that subjects on average were twice as fast and four times as suc cessful when using the PUC interfaces to complete a set of representative tasks The interface generators can also produce interfaces that are personally consistent based on a user s previous experiences When a new interface is generated for an a
72. can now click the play icon in the flow s title bar which invokes the planner to automatically activate this flow If a successful plan is found the appliances will be automati cally configured the play icon will turn green the stop icon will turn white and a bubble will appear to inform the user that the flow has been activated Figure 9 3d shows the inter face once the user has dismissed the bubble If a plan cannot be found a bubble will appear to help the user resolve the problem see Figure 1 7d One difficulty with planning algo rithms such as the GraphPlan algorithm that the PUC uses is that they cannot produce useful error messages when planning fails Therefore the PUC first uses two approximation checks to search for conflicting appliance variables and active flows allowing a more useful error message to be produced The first conflict check searches the dependencies of the newly specified flow to see if any read only variables have values that make activating the new flow impossible Such variables usually reflect the physical status of the appliance which the user can address once informed For example the DiscIn variable of the DVD player might be set to false when the user pressed the play icon in our previous example If this happened the system would then ask the user if they can rectify this problem Although the current language for this error message can be somewhat stilted we do provide predefined strings for common
73. choose controls and their arrangement Some initial research has been conducted on adapting SUPPLE to support the creation of consistent user interfaces for the same application across different platforms Gajos 2005b Like ITS SUPPLE cannot create consistent interfaces if the underlying appliance models differ DiamondHelp Rich 2005 attempts to address the consistency problem for consumer elec tronics devices by combining a task model based approach with a direct manipulation interface While one of DiamondHelp s goals is to provide consistency the current system relies on designers to create the direct manipulation portions of the interface and for each appliance to supply its own task model DiamondHelp does not provide a way to search for possible inconsistencies across devices or to automatically adjust the interfaces to help the user transfer knowledge between interfaces 6 2 Specification Authoring Study I started work on consistent interface generation by studying how inconsistencies can arise in the user interfaces created by the PUC The PUC interface generator uses a rule based proc ess see Chapter 8 that is guaranteed to produce the same interface given the same appliance specification so any inconsistencies arising in the interface will be due to differences in the specifications of two similar appliances In order to understand how specifications can differ I conducted two studies to investigate the following questions
74. control for the mutual exclusion state the top most overlapping panel is changed to display the panel with the currently active controls see Figure 8 10c d As the interface tree creation process traverses the abstract user interface tree it maintains a reference to the concrete panel that represents the current portion of the abstract interface A set of rules add rows to this panel whenever an AIO is encountered in the abstract interface The process of adding a new row requires two steps first an appropriate CIO must be cho sen for the AIO and then an appropriate row must be chosen for the CIO The CIO is chosen by querying the AIO with a set of criteria from which a platform specific method of the AIO will return the most appropriate CIO for that platform and criteria There are two criteria currently supported whether or not a read only CIO is needed and whether the CIO should work within a list Other criteria such as maximum size could be supported in the future Based on the CIO a set of three rules choose the appropriate row Any remaining rules are not applied once a rule has created a row The first rule checks to see if the current CIO is located in a labeled group with just one other CIO and that both of the CIOs are either but tons or checkboxes In this case a Labeled Two Column row is created for the two CIOs using the group label The second rule checks the CIO to see if requires a full width layout and if so creates a
75. data such as sounds or images A Smart Template must be used with this type to ensure proper interpretation and rendering This type may contain arbitrary tags that act are parameters that particular Smart Templates will recognize Placement Inside the lt type gt element May Contain Any arbitrary tag lt boolean gt lt boolean gt Boolean type takes on true or false values Placement Inside the lt type gt element lt enumerated gt lt enumerated gt lt enumerated gt Enumerated type Define the number of items this composite type contains using the lt item count gt element Note the lt value labels gt element must contain lt map gt elements that map each of the enumerated values with a label So if there are 5 enumerated values in a particular enumerated type denoted by lt item count gt 5 lt item count gt the value labels section must contain 5 different map pings of values to labels Enumerated type values are treated as integers that range between 1 and the number of items in the type Zero is not a valid value for an enumerated type This is impor tant when you specify an index to the lt map gt tag Placement Inside the lt t ype gt element Appendix B Specification Language Reference 230 May Contain lt item count gt lt item count gt lt item count gt lt item count gt Used to denote how many enumerated values there are in an lt enumerated gt typ
76. deac tivated by the new plan The FBI also provides a way to navigate to the other aggregate user interfaces and the PUC interfaces for the individual appliances In the upper right corner of the FBI is a Navigate pull down menu which allows the user to navigate to the different aggregate user interfaces that can be generated discussed next Double clicking on any appliance icon either in the appliance bar or in any flow allows the user to navigate to the full interface for that individ ual appliance 9 5 Ageregate User Interfaces The FBI provides an interface for users to accomplish high level goals within the multi p p g g appliance system such watching a DVD movie with a home theater There is still a need however to provide the user with finer grain control of the individual appliance functions p g pp For example the user may wish to pause the DVD while it is playing to take a phone call or go to the next slide in a PowerPoint presentation A user might also discover that the movie is too dark requiring adjustment of the brightness of the television or that the keystone set ting needs to be adjusted on the projector To address these problems the PUC provides the user with several Aggregate User Interfaces AUIs that combine functions from each of the appliances in the system to create useful task specific interfaces The PUC currently can generate four different AUIs Active Flow Control Active Flow Setup
77. e The final representation of any interface can be described using a tree format It is not reasonable to include the tree representation of one interface in the specification of an appliance however because the tree may differ for different form factors For example the tree will be very deeply branched on a small screen WAP cellular phone interface whereas the tree will be broader for a desktop PC interface We prefer specifications that define a group tree that is deeply branched This information can be used for small screen and large screen interfaces alike because some of the branches can be ignored in a large interface e Domain specific conventions are often used in appliance interfaces such as the stan dard number pad on a telephone or the standard play and stop icons used on media players Interfaces generated by our PUC system need to include these conventions in their generated interfaces and we have developed Smart Templates to help identify pieces of a specification where conventions should be applied Each of these items is described in more detail below Appliance Objects Three types of appliance objects are supported in the specification language e States Variables that represent data that is stored within the appliance Examples might be the radio station on a stereo the number of rings until an answering ma chine picks up and the time that an alarm clock is set for Each variable has a type and the UI ge
78. e enumerated e fixed point e floating point e integer e _list selection string Each of these types has a different set of parameters that can be specified for it The Boolean type is for variables that have a value of true or false The enumerated type is for small collections of values that all have some string label Inter nally these values are represented by numbers starting with 1 For example an enumerated type with 4 items can have a value of 1 through 4 Enumerated types must have labels de fined for each of their values The types of fixed point floating point and integer all contain numeric values Integers do not have a decimal component while fixed point and floating point both do Fixed point values have a fixed number of digits to the right of the decimal point as defined by the re quired Decimal Places field Floating point values have an arbitrary number of digits on either side of the decimal point The fixed point and integer types also have an optional in crement field that can be used to further restrict the values that the state variable may contain If an increment is specified then a minimum value must also be specified When these parameters are specified the value of the state variable must be equal to the minimum n increment where n is some integer The list selection type is a special type for linking the value of a state variable to a selection within a list that is somewhere in th
79. eae TONT 53 3 6 8 Shared High Level Semantic Knowledge anoss 53 4 System Implementation AE ADEC oeoa E T E E ec ev egl eae 55 AD Controlling Applinet esne s a ta tasdpavsiepreaateen 57 4 3 Generating Interfaces on Controller Devices ssssesessesesesssessesessresssressresssesses 59 4 3 1 PocketPC and Desktop Implementation asooo 59 BOD Smartphone Tmplementati n ssisisirireserriai isinen iii akaa 60 AA COMMUNION ensia E slau renon ue niendeatoaacheees 62 5 Specification Language Daly Degn Prine ipl es iat E gas caseeeneteea E eee 66 UI Ae DESE ata a sa ata aca ae eae 67 5 2 1 Functional Language Elements Xi DD del AAV PUANCO OO CAS sasidas inna E TT 69 5 2 1 2 Type Information occeccecccecssessscssssssssesssessssssssesssssssessssssssesseesee 70 ILAI Label Informatio soriire EERE ATE 71 DE Ga eae eee ee 72 Jala KOMP AIA OUUTE ois ae EEE T T E 73 5 2 1 6 Dependency AI OVINALIO Ns scies neice cetataie may tscsauasaeeesagoenaeaiees 74 Di LF SUNG TCM DIAL EEEE TEE A duane daacats 75 922 Content low Lanenace Elements srairiirar a ainda 76 Teede AO REE N E T E T 76 5 2 2 2 Internal Flows Sources Sinks and Passthroughs assesses 77 5 3 Evaluation of Specification Language ssssesssessssesssesssessreserssersserssressreseresees 79 DA E EE E E E AO E 79 5 3 2 Learn bility and Ease of Use srissisnoiiinoerassoneninnirnanai iei 8l Consistency 83 Gal Wridetstariding Consistency acneei a E T 83 OLA PUAN C
80. element e priority optional if this element is not inside a state element Appendix B Specification Language Reference 228 access optional if this element is not inside a state element and the type be ing applied is a state variable or a union group lt default value gt lt default value gt lt default value gt Specifies a default value for a state variable In an interface generator these values would be used when the UI needs to prompt the user for a new value and could also be used for demonstrational purposes Placement Inside the lt state gt element May Contain lt constant gt lt ref value gt lt required if gt lt required if gt lt required if gt Contains dependency information that defines when the value of a state variable is required for successful operation of the appliance The content of this element is the same as the lt active if gt element If this element is not specified then the object will not be required To ensure that an object is required create this element con taining a lt true gt element Placement Inside the lt state gt element May Contain lt and gt lt apply over gt lt defined gt lt equals gt lt false gt lt greaterthan gt lt lessthan gt lt not gt lt or gt lt true gt lt undefined gt lt binary gt lt binary gt Appendix B Specification Language Reference 229 Binary type contains any kind of a binary
81. focus is on technology DLNA s focus is strictly on improving the user experience DLNA guidelines cover not only the technical layer to allow easy inte gration between appliances but also the design of the user interfaces Products can only receive DLNA certification if they comply with the technical and UI guidelines The PUC philosophy differs from DLNA in that the PUC allows users to decide how to integrate func tionality between their appliances and supports the inclusion of new appliances that cannot be envisioned today DLNA is based around a set of general use cases for appliances that can be envisioned today which may require revision in the future if new classes of appliances be come commonplace 2 1 2 4 Home Audio Video Interoperability Home Audio Video Interoperability HAVi is the only platform designed specifically for consumer electronics devices like televisions and VCRs and only works over an IEEE 1394 Firewire network Televisions that feature HAVi are available today from RCA and Mitsu bishi Electric and Mitsubishi at one time produced a VCR that supported HAVi HAVi s user interface concept is that the television because it is only appliance with a large screen should control all the other appliances in a home network There are three ways that a HAVi controller might control another appliance 1 every type of appliance that might be con trolled has a standardized interface specified by the HAVi working committe
82. full width row for the CIO The third rule creates a Labeled One Col umn row for the CIO using a label if any from the AIO As mentioned above these rules prefer the Labeled row formats because they create a simple regular grid of labels and con trols If a Labeled row format does not provide sufficient space for one of its controls then the format may be changed by one of the layout fixing rules discussed in the next section After the abstract user interface has been completely traversed all panel structure has been added and all CIOs have been created and placed into rows then the concrete sizing of all panels rows and CIOs is determined Size is determined by a two pass depth first traversal of the concrete interface tree The first pass calculates the minimum and preferred size for all panels based on the CIOs they contain The second pass determines the actual size of each panel based on available screen space and the minimum and preferred sizings During the second pass if an actual panel size must be smaller than the panel s minimum size or if a row cannot be as wide as its minimum width then a layout problem is noted by the interface generator These layout problems are addressed by the layout fixing rules discussed in the next section Chapter 8 Interface Generation 136 swt se r Odometer 144 1 Hourmeter 39 Stopwatch 0 0 Annual Mileage 144 1 Personalization g
83. g HP followed by the PUC interface without consistency for the other e g Canon e Consistent AutoGen PUC interface without consistency for one appliance e g HP followed by the PUC interface for the other appliance e g Canon generated to be consistent with the first interface e g HP Chapter 10 Usability Evaluation 166 The Consistent AutoGen configuration is designed to fulfill the assumption of the PUC s consistency algorithms which assume that users will receive a benefit from consistency when they encounter a new device because they are familiar with a previous interface These three configurations allow testing of both usability and consistency Usability is tested by comparing the Built in configuration with either of the others Consistency is tested by comparing the AutoGen and Consistent AutoGen configurations To test each of these con figurations with both of the possible orderings HP followed by Canon and vice versa a 3x2 between subjects study design was used A within subjects design is not possible because learning must be carefully controlled to compare performance for both the usability and con sistency studies 10 2 1 Tasks Eight tasks were designed for subjects to perform during each block of the study The tasks were chosen to be realistic for an all in one printer cover a wide range of difficulties and be as independent from each other as possible so success or failure on one task would
84. generators has been that their interface designs do not conform to domain specific design patterns that users are accustomed to For exam ple an automated tool is unlikely to produce a standard telephone keypad layout This problem is challenging for two reasons the user interface conventions used by designers must be described and the interface generators must be able to recognize where to apply the con ventions through analysis of the interface specification Some systems Wiecha 1990 have dealt with this problem by defining specific rules for each application that apply the appro priate design conventions Other systems Kim 1993 rely on human designers to add design conventions to the interfaces after they are automatically generated Neither of these solu tions is acceptable for the PUC system Defining specific rules for each appliance will not scale and a PUC device cannot rely on user modifications because its user is not likely to be a trained interface designer Even if the user was trained he or she is unlikely to have the The work in this chapter was originally published in Jeffrey Nichols Brad A Myers and Kevin Litwack Improving Automatic Interface Generation with Smart Templates in Proceedings of Intelligent User Interfaces IUI Funchal Port gual 2004 pp 286 288 99 time or desire to modify each interface after it is generated especially if the interface was generated when needed to perform a specific ta
85. gt lt xs element name labels type LabelDictionary minOccurs 1 maxOccurs 1 gt lt xs element name active if type ActivelfType minOccurs 0 maxOccurs 1 gt lt xsvall gt lt xs attribute name index type xs string gt Appendix B Specification Language Reference 211 lt xs complexType gt lt xs complexType name StringType gt lt xs all gt lt xs element name min type StaticOrReference minOccurs 0 maxOccurs 1 gt lt xs element name max type StaticOrReference minOccurs 0 maxOccurs 1 gt lt xs element name average type StaticOrReference minOccurs 0 maxOccurs 1 gt lt xs all gt lt xs complexType gt lt xs complexType name ListSelectionType gt lt xs sequence gt lt xs element name active if type ActivelfType minOccurs 0 maxOccurs 1 gt lt xs sequence gt lt xs attribute name list type xs string gt lt xs complexType gt lt xs complexType name FloatingPtType gt lt xsS sequence gt lt xs element name min type StaticOrReference minOccurs 0 maxOccurs 1 gt lt xs element name max type StaticOrReference minOccurs 0 maxOccurs 1 gt lt xs element name specific values important minOccurs 0 maxOccurs 1 gt lt xs sequence gt lt xs complexType gt lt xs complexType name IntegerType gt lt xsS sequence gt lt xs element name min type StaticOrReference minOccurs 0 maxOccurs 1 gt lt xs element name max ty
86. has been the basis for consistency three times for itself the answering machine and the DVD player and the Cheap VCR has been the basis for consistency once just for itself The answering machine and DVD player were generated to be consistent with the Panasonic VCR and thus both have counts of zero secce00e A diagram showing the four different roles in the creation implementation and use of a Smart Template in the context of the E 72 feiu 12 sates 75 ee 77 ee 78 dene 96 interface PENELIN PLOCEO S iois EEEa EE E 101 Three specification snippets showing instantiations different Smart Templates a The instantiation of the media controls template for the play controls on Windows Media Player b The instantiation of the time duration template for the counter function on the Sony DV Camcorder c The instantiation of the time duration template for the song length function on Windows Media Player arimeari a tanh E E avs etetmauaveeeteets 102 Renders of the media controls Smart Template on different platforms and for different appliances a Media controls rendered with Smart Templates disabled for a Windows Media Player interface on the PocketPC platform b Media controls rendered for the same interface with Smart Templates enabled on each of our three XIX Figure 7 4 Figure 7 5 Figure 8 1 Figure 8 2 Figure 8 3 Figure 8 4 platforms At the top is the desktop the middle is PocketPC a
87. in the interface generator but a future version could download icons from the ap pliances or the Internet Additional flows may be added to the screen by pressing the Add Flow button at the top of the screen and the scrollbar on the right allows for scrolling when Chapter 9 Aggregating User Interfaces 152 more flows have been added to the list than can be shown I envision two usage scenarios for this interface the user creates flows for each of the common tasks and switches among them or the user uses just one flow and modifies it as necessary to suit the current task When a user wishes to begin a flow an icon is dragged from the appliance bar to one of the empty spaces in the blank flow The empty space highlights when the icon is dragged over it indicating that the appliance icon may be placed there Once the DVD icon has been placed in the source location see Figure 9 3b content type icons appear on left side of the arrow and the icons in the appliance bar corresponding to appliances that cannot be sinks for the DVD player source are grayed out This includes icons that correspond only to sources such as the broadcast television and radio icons and the VCR icons that cannot be sinks for DVD content because of our home theater s particular wiring configuration The user can now see that the receiver and the television are the only available appliances that will work with the DVD player In this scenario the user first dra
88. in your specification Organizing the Specification Once all of the state variables commands and lists have been defined you should consider further organizing your description to make it more useful for an interface generator In gen eral you should try to make the hierarchy as deep as possible so that an interface generator for a small screen device will be able to make intelligent decisions about how to separate the pieces of the user interface In this specification I chose to have two main groups There is a group that contains the to do list which consists of the list containing the to do list data and the group of list com mands for modifying that data The group of list commands also contains the sorting function The second main group is called Setup and contains the display preference set ting the reminder frequency setting and the list of categories You will find in your specifications that you have many more functions than were included in this to do list application which will required much more organization As we mentioned above organization data is very important to the interface generator and may affect the qual ity of an interface generated from your specification You should be sure to put some effort into designing an interface hierarchy that is accurate and intuitive for your appliance Appendix D Specification Authoring Study Instructions 284 Appendix D 1 Smart Templates The following Smart
89. interaction and ubiquitous com puting communities on automatically generating aggregate interfaces The Web Services and Semantic Web communities have also been investigating web service composition though most of this work focuses on infrastructure issues and does not address the user interface 2 1 Control of Appliances A number of systems have been created for controlling appliances Commercial products have been available for years that allow limited control for certain electronic appliances and recently companies have begun forming standards groups to agree on new solutions for con trolling appliances such as HAVi HAVi 2003 JINI Sun 2003 and UPnP UPnP 2005 Another standards group INCITS V2 INCITS V2 2003 was formed to examine stan dardizing appliance control in order to benefit people with handicaps There have also been several research projects that have explored this area such as Xweb Olsen Jr 2000 Crafter Ponnekanti 2001 and Roadie Lieberman 2006 2 1 1 Commercial Products For years many companies have been selling so called universal remote controls which re place the standard remote controls for televisions VCRs and stereos with a single remote control unit A one way infrared protocol is used to issue commands to the appliances Typi cal universal remote control products have physical buttons that correspond to the most common subset of features found on the appliances that the universal remote c
90. into two categories Each of the generation phases creates a new interface data structure based on the previous structure tak ing the attributes of the appliance and the controller device into account Each of the consistency phases modify the current interface data structure to match elements of previous interfaces that the user has previously seen These consistency phases make use of the knowl edge base to find matching elements and choose the appropriate modification There are three types of interface data structures used in the generation process The first is the specification structure created by parsing a specification document written in XML The second is the abstract user interface which contains more complete knowledge about how the interface should be organized and an abstract choice of controls for each of the appli ance s functions The third is the concrete user interface which varies depending on the target platform for interface generation Because the first two structures are common across all generation platforms all of the rules for creating the abstract user interface and the consis tency rules that operate on the abstract user interface can be shared among the different platforms Chapter 8 Interface Generation 119 Four of the six phases are designed to produce consistent user interfaces The mapping phase analyses the new appliance specification to automatically find similarties to previous specifi cations the
91. menus without help be cause they did not think to press the button that makes the menus visible This was in spite of the fact that more than half used Palm devices regularly and were aware of the menu sys tem Although the study was successful there was concern that the prototype interfaces benefited from the close interaction of the subject and the experimenter In the paper proto type portion of the study the experimenter provided all feedback to the user including verbal hints when the user requested them Because of these issues a new study was con ducted with full implementations of the interfaces so that the experimenter would be a passive observer instead of an active participant 3 4 Study 2 The second study improved upon on the first study by replacing the paper prototypes with the full fidelity PocketPC prototypes One very important issue with these interfaces was their use of several conventions that are specific to the PocketPC operating system In particular there is a standard OK button for exiting dialog boxes that is displayed in the top right corner of the screen Users in pilot tests did not discover this feature and thus were unable to exit from certain screens in the inter face The interfaces were not changed because of our goal to use the conventions of the controlling device Instead a tutorial program was created and presented to subjects before they began the study The tutorial covers the OK button text ent
92. models of other model based systems XIML was de veloped by RedWhale Software and is being used to support that company s user interface consulting work They have shown that the language is useful for porting applications across different platforms and storing information from all aspects of a user interface design project It may be possible to express the information in the PUC specification language within an XIML document but the language also supports many other types of information that will not be needed such as concrete descriptions of user interfaces 2 2 2 4 IBM PIMA and MDAT The IBM PIMA project creates specialized interfaces for different platforms including PDAs and phones from a generic model of the application Banavar 2004b This work incorpo rates automatic generation techniques but differs from my work in two ways 1 layout information can be included in PIMA s generic application model to help specialize the in terface to different platforms and 2 designers typically must tweak the specialized interfaces after generation The PUC system uses Smart Templates and more sophisticated interface generation rules to address these issues The work on PIMA has been integrated into the Multi Device Authoring Technology MDAT project Banavar 2004a which fo cuses on the particular issues of authoring web pages that may be rendered on multiple platforms 2 2 2 5 UIML and TIDE The User Interface Markup Langauge U
93. more of its outputs For example the InFocus televi sion in our scenario has the capability of taking the audio it receives as an input and mak ing it available as an output for other appliances Tuning appliances such as cable television set top boxes are also represented as passthroughs which usually take a multi channel input from an antenna and output single channel data The passthrough structure is particularly important because it allows Huddle to track the flow of content from its origination point through multiple appliances to its final destina tion Previous systems such as Speakeasy Newman 2002 and Ligature Foltz 2001 have used only sources and sinks to model the path of data within a system Using their approach it is difficult to know whether the content a device is receiving as input is being redirected through an output which makes determining the full content flow impossible Without knowledge of the full content flow from start to finish the task the user is trying to perform could not be determined and a useful interface could not be generated for it 9 3 Aggregation Architecture The PUC controller device performs all of the aggregation of content flow information from the appliance specifications and generates aggregate user interfaces based on this information Even when an aggregate interface is available the PUC still allows users to use the interfaces generated for the individual appliances An overall vie
94. more than one appliance and then by appliance We first attempt to identify these high level collections with the knowledge base but also search for any top level groups within the Setup groups that may have the same name Groups with the same name may not be identified by Uniform because their contents are not identified as being similar enough These groups are often catch all groups such as Preferences which have a large variance in the types of functions that they contain AF puc v4 0 700 UH lt P d 417 3 WD Preferences ik Audio English Language Subtitle English Max Channel 125 Language Language Disc Menu English X YCR Lock Language Parental Limits Mo Parental ka Reset to Factory Defaults PUC Servers Devices Generation E PUC Servers Devices Generation E Figure 9 6 Two shots of the General Setup AUI for our home theater setup Note that in both shots the tabs at the bottom of the screen represent high level concepts within which the functions are organized by appliance combo boxes at top Chapter 9 Aggregating User Interfaces 158 9 5 4 Merging Controls There are a few settings across a system of appliances where a single value should be set once and then migrated to all appliances rather than requiring the user to laboriously set the value on each appliance Examples include the time on the clock the language e g English and the sleep timer that turns off the appliance a
95. much work for end users however who will typically pay professional system integrators to create their integrated in terfaces For the PUC my goal is to support the generation of aggregate interfaces with minimal effort on the part of the end user Ideally any work required by the end user should also scale well with the number of appliances connected in a multi appliance system I found that the content flows between appliances in a multi appliance system were useful for understanding the system and generating aggregate interfaces In particular content flows have two important properties 1 Content flows seem to be closely related to user goals with multi appliance systems For example in a home theater the user may want to watch a DVD movie which involves seeing the video on the television and hearing the audio through the stereo s speakers To accomplish this each of the appliances in the home theater must be configured to allow content from the DVD player to flow to the appropriate places 2 The content flows of a system can be described as the separate flows within each ap pliance combined with a wiring diagram showing how all of the appliances are connected This is an important property because it divides the modeling work among the manufacturers of the appliances The only system specific input needed by the PUC s aggregation algorithms is a diagram showing how the appliances are connected which can be supplied by a
96. occurs on the appliance The PUC does support text only alert dialog boxes that are displayed at the request of the appliance but these dialogs are not pre specificed in the specification language and appear in response to a special alert information messasge sent through the communication protocol Specifying these mes sages explicitly would allow the language of the dialog box to be modified by the interface generator his might be appropriate for example if a part of the interface mentioned in the dialog box has been modified by the consistency rules The PUC s consistent interface designs are based on seven basic requirements as discussed in section 6 3 These requirements are based on current work in consistency but are very high level and would likely benefit from some elaboration In particular it seems that there are several different dimensions along which consistency might be achieved Lexical consistency requires interfaces to use the same labels for the same functions Functional and structural consistency as defined earlier mean respectively that the same functions use the same con trols and functions are located in the same location A related but different dimension to structural consistency is navigational consistency which requires that users take the same steps to navigate to a function Navigational consistency implies structural consistency but for example two interfaces could be structurally consistent but not na
97. of the Smart Template Ensure that the group or appliance object conforms to the restrictions that have been speci fied for that Smart Template The appendix at the end of this document describes the restrictions for each Smart Template You may find that the Smart Templates that we have already defined are not expressive enough to use with the appliance that you are specifying If you encounter this situation please mark the group or object with the appropriate smart template name but specify con tents for the template that match the functionality of the appliance This will help us improve and extend our Smart Templates in the future 2 Authoring an Appliance Specification with XML Extensive documentation for the XML language is available on the web here http www pebbles hcii cmu edu puc specification html Documentation on Smart Templates is available here http www pebbles hcii cmu edu puc highlevel types html XML Editors Appendix D Specification Authoring Study Instructions 270 Using a special XML editor will make authoring specifications a much easier process We support two editors in this study XMLSpy Home Edition and Visual Studio NET 2003 You may use a different editor but we will not necessarily be able to help you if you have problems An important reason to use an XML editor is that you can ensure that your specification has the proper format by validating it against a schema To use this feature of your XM
98. on a OTD or Schema O Schema Othenvise please click the cancel button to begin with a new empty document Be sure that you have selected Schema and then click OK If you have not yet downloaded the PUC schema see the link above you should do so now and note the loca tion in which you save the file You will now see the following dialog box XMLSpy Choose a file Make path relative to Untitled ml Please choose a file from your hard disk or select one of the other windows curently open in SMLS py Click the Browse button and find the PUC schema file on your local hard drive Click OK to create the new file Your application screen should look something like this Appendix D Specification Authoring Study Instructions 273 x py Untitedt samt tc File Edit XML DTD Schema Schema design XSL XQuery Authentic Yiew Browser Tools ADVANCED Window Help T IOME F PEIE 4 a 8 om TE BE i lt xml yersion 1 0 encoding UTF 8 gt lt spec xmins http Avvwywy cs cmu edu pebbles puc xmins xsi http Awww vwv3 org 2001 KMLSchema instance xsi schemaLocation hittp JAvwyy cs cmu edu pebbles puc e ieffreynipucispecs puc xsd gt wn x average binary boolean command completions available constant detault value defined enumerated Text SchemaAWSDL Authentic Browser xh Untitled1 xml XMLSpy v2005 rel 3U Registered to Je
99. on their control panel including directional pads ok buttons and number pads see Figure 10 2 whose behavior changes depending upon the function selected through the printers menu This was a particular problem for the manufacturer s interface on the Canon which has many modes in which certain buttons cannot be used and for which there is no feedback Users must experiment to determine which buttons can be pressed in which situations The PUC addresses the feed back problem by graying out controls that are not currently available The PUC s screen allows for longer and better labels to be shown for each function The screen also allows for a two dimensional layout that can give clues to the organization of the interface For example the tab control allows users to see immediately that there are multiple groups of controls and what those groups are Also the functions displayed in the main por tion of any given screen are grouped by functionality which decreases the number of functions on any one screen and may make the interface easier to parse In order to improve the built in interfaces to the same usability as the PUC manufacturers would probably need to invest in larger screens for their appliances These screens would al low the organization of the interface to be clearer and hopefully eliminate some of the need Chapter 10 Usability Evaluation 175 for multi purpose buttons Any physical buttons that are not always functi
100. onseren 125 Consistency moving rules implemented in the PUC sssssessssrssesesesesseesssres 127 The row layouts supported by the PocketPC interface generator 006 134 The layout fixing rules used by the PocketPC interface generator 0 137 Average completion time and total failure data for the first block of tasks The PUC condition is the combination of the AutoGen and Consistent AutoGen conditions N 8 for the Built In condition and N 16 for the PUC condition indicates a significant difference between the Built In and PUC conditions for that appliance p lt 0 05 and indicates a marginally significant difference p lt 0 1 Completion times and total failures were compared with a one way analysis variance and failures per task were compared with a one tailed Fishers Exact Testaaminen a N E N E Ta 169 Average completion time and total failure data for the second block of tasks N 8 for all conditions indicates a significant difference between that row s condition and the Consistent AutoGen condition for that appliance p lt 0 05 and f indicates a marginally significant difference p lt 0 1 Completion times and total failures were compared with a one way analysis variance and failures per task were compared with a one tailed Fisher s Exact Test ccccesssceeesseceeeeneeeeeeneeeees 172 XXV XXVI1 Acknowledgements Many people have touched my life helped me find direction an
101. onto the UI device e The controller device is pre programmed with interfaces at the factory or by users in their homes The advantage to this approach is that all of the interfaces are hand designed specifically for the controller device and the appliances that it can control Most of today s universal remote controls are pre programmed with the codes for controlling a variety of home entertainment appliances though some can be taught additional codes for other appliances and others have programming environments that allow users to create their own interfaces If the manufacturer of the controller device does not provide an interface however it is often very tedious and time consuming for users to program their own interfaces e The controller device downloads complete user interfaces from appliances or the inter net This approach which is used by JINI Sun 2003 and Speakeasy Newman 2002 has the advantage that the controller device does not need to know in advance Chapter 1 Introduction 2 about every appliance it might control The appliance must be able to provide inter faces for each of the different controller devices it might encounter however such as a standard graphical interface for use with touch screens and normal desktops a list based interface for use with phones and a speech interface for devices that have a built in speech recognizer Controller devices with unique designs such as a future watch with a ci
102. printer Different writers were used for the two specifica tions because these specifications are used for the consistency user study and I wanted the specifications to contain similarities and differences that might be found in a realistic sce nario where the specifications were written separately by different manufacturers The specifications were also written using an approach that actual specification writers are expected to take Writers were generally faithful to the design of the actual appliances but also took advantage of the features of the PUC specification language For example the lan guage allows for multiple labels for each function and we added extra labels with further detail where necessary The PUC language also calls for writers to include as much organiza tional detail as possible in order to support generation on devices with small screens and we Chapter 10 Usability Evaluation 163 b Canon PIXMA MP780 4 Copy Menu 2 Reduce Enlarge 4 Paper Size 5 Paper Type P i Y e a HP Photosmart 2610 Figure 10 2 The all in one printers used in our studies with a larger view of the built in user interface Chapter 10 Usability Evaluation 164 also followed this guideline The initial specifications were tested with the interface genera tors to ensure correctness and went through several iterations before they were deemed of high enough quality to be used for the studies Note that this testing is
103. problems such as there being no disc in the DVD player see Figure 1 7d After the PUC checks for variable conflicts it checks to see if any currently active flows con flict with the new flow To perform this check it examines the dependency information associated with the newly specified flow and the dependencies for any active flows looking for variables that must have more than one value for the flows to be active simultaneously If this situation is found then the PUC immediately goes back to the user to ask which of the conflicting flows the user wants to use Once the PUC has found that no obvious conflicts exist it executes the planning algorithm to find a valid plan for activating the new flow If a plan is found then the system will carry out that plan to create the right configuration of variables that will activate the new flow and maintain the state of any existing flows The planning algorithm may still fail however such as when second order dependencies conflict In our experience these conflicts are rare but when they occur the PUC asks the user to choose between finding a plan that activates the Chapter 9 Aggregating User Interfaces 154 specified flow and disables the currently active flows or finding a plan that activates the speci fied flow without considering the effects on other flows If a plan is found in either case the user is prompted again before carrying out the plan to make it clear which flows will be
104. recognizer The generated grammar is compati ble with the Phoenix parser Ward 1990 which is used by the USI library to parse user utterances A grammar is generated for each appliance that contains phrases for query and control Query phrases include determining what groups are available what states and com mands are available what values the states can take as well as what values they currently hold Control phrases allow the user to navigate between groups issue commands and change states All of the appliance specific grammars together with an appliance independent USI grammar are compiled into a single combined grammar representing eve rything that the speech interface will understand Note that this is different from the interface aggregation for multiple connected appliances discussed in Chapter 9 because the combined speech interface is organized by appliance with no re organization for functions from different appliances that would be used together The speech interface generator has Chapter 8 Interface Generation 117 been implemented and tested with the Audiophase shelf stereo the Sony camcorder and multiple X10 devices Figure 8 3 shows an example of a user interacting with this system The speech interface translates the group tree from each appliance specification into a USI interaction tree This tree is a more concrete representation of what interactions can take place between the system and the user which consists o
105. reguired gt lt xs attribute name port type xs string use required gt lt xs complexType gt lt xs complexType name ExternalSourceType gt lt xs attribute name name type xs string use required gt lt xs complexType gt lt xs complexType name WiresType gt lt xs sequence gt lt xs element name connect type ConnectType minOccurs 1 maxOccurs unbounded gt lt xs sequence gt lt xs complexType gt lt xs complexType name ConnectType gt lt xs sequence gt lt xs element name start type ApplianceConnectType minOccurs 1 maxOccurs 1 gt lt xs element name end type ApplianceConnectType minOccurs 1 maxOccurs 1 gt lt xs sequence gt lt xs complexType gt lt xs complexType name ApplianceConnectType gt lt xs attribute name name type xs string use required gt lt xs attribute name port type xs string use optional gt lt xs complexType gt lt xs complexType name PreferredFlowsType gt lt xs sequence gt lt xs element name flow type PreferredFlowType minOccurs 1 maxOccurs unbounded gt lt xs sequence gt lt xs complexType gt lt xs complexType name PreferredFlowType gt lt xs attribute name appliance type xs string use required gt lt xs attribute name name type xs string use required gt lt xs complexType gt lt xs schema gt Appendix C Other PUC XML Language Schemas 262 APPENDIX D Specification Authoring Study
106. screen The list structure must still reflect the proper ties of the appliance however and should not deviate significantly from the initial structure e Editing panes are necessary in the Smartphone interface because many functions cannot be manipulated in the list For example a state variable with an enumerated type might be edited with a combo box or slider neither of which is supported in a Smartphone list interface Other functions can only be instantiated as list items be cause there is not a corresponding control that can be used on an editing pane Commands such as Seek are good example of this because the Smartphone does Chapter 8 Interface Generation 139 not allow on screen buttons such as those used to invoke commands in our Pock etPC interfaces The Smartphone interface generator optimizes navigation using a rule based approach The rules are applied iteratively during a depth first traversal of the list hierarchy Rules are also applied bottom up so the rules are applied to all of the children of a list before being applied to that list The children of each list are traversed in priority order which is a measure of importance that the specification author defines for each function and group in the appliance specification Traversing in this way ensures that the rules have more flexibility for optimiz ing the most important functions of an interface There are currently five rules for optimizing navi
107. see Chapter 6 If a previous function is found then a set of functional consistency rules are used to transform the new specification into a form that is consistent with the previous specifica tion In order to determine the particular rule to apply the PUC iterates through the rules until it finds a rule that matches the mapping The PUC will use the first rule that is found so the rules must be carefully ordered to ensure that those with more specific matching con ditions will be tested before those with more generic matching conditions The PUC currently implements seven functional consistency rules as shown in Table 8 1 Each rule modifies a portion of the new specification to match the previous specification For example the change invoke to change rule will convert between a state variable and a state variable with a command that must be invoked before a variable change will take effect As part of this conversion a command in the new specification must be hidden or a new command must be added If the command is hidden the converted state variable will automatically invoke the command when its value is set by the user If a command is added then the ap pliance will not be informed of a variable change until the user invokes the new command in the interface Table 8 1 The PUC s functional consistency rules Description invoke to invoke change to Converts between the situation where changing a variable on appliance is the same
108. shows the standard layouts and icons for Chapter 3 Preliminary User Studies 49 play buttons on a CD player and d uses the standard red circle icon for record These con ventions should not be specified as a part of the functional description of the appliance however because they are not always applicable The dialing pad convention does not make any sense for a speech interface for example I have developed a solution for addressing this problem called Smart Templates Nichols 2004b which is discussed in Chapter 7 3 6 Requirements The hand designed interface work led to the development of a list of requirements that the PUC system must fulfill in order to generate high quality interfaces This section describes those requirements and briefly discusses how they are fulfilled or not by the PUC and other systems 3 6 1 Two Way Communication One of the most important requirements for any PUC type system is two way communica tion between the controller and the appliance This is an obvious requirement for any system where the controller downloads an appliance specification before constructing an interface that issues commands back to the appliance It is important that this two way communica tion be maintained through out the entire session however so that the controller and appliance can keep their state synchronized State synchronization allows graphical interfaces to display information about the current state of the inte
109. state variable is used it has the following restrictions String The state variable must have a string type The value of the state variable must be a parseable date time string in one of the ISO 8601 international standard formats If two groups are used then one of those groups must be tagged with date and the other must be tagged with time absolute These groups must conform to the restrictions for these other Smart Templates four way dpad Overview This template represents a four way directional control Appendix D Specification Authoring Study Instructions 287 Contents This template may only be applied to a group that includes four commands The commands must be Left The command for the left button in the directional control Right The command for the right button in the directional control Op The command for the top button in the directional control Bottom The command for the bottom button in the directional control four way dpad with enter Overview This template represents a four way directional pad with an enter function Contents This template must include the four way dpad smart template and one extra command Enter The enter command list commands Overview This template is for commands that manipulate list data such as adding deleting or moving list items This is the first example of a nested template which means that the template must contains other templates In this case there i
110. study suggested that it would valuable to create a tool to assist users in creating new specifications Existing tools for writing XML are helpful for writing specifications but the language has some properties that are not conducive for direct XML editing For exam ple my typical process for writing a specification is to first enumerate all of the state variables and commands and then organize all of the appliance objects into an appropriate group hier archy Reorganizing hierarchies is very difficult in XML and might be better supported by a direct manipulation interaction Object names also change as the organization changes and a tool should manage names automatically Chapter 11 Conclusion 187 A third direction to explore with the specification language would be to analyze whether there is any additional information that should be added The latest version of the IN CITS V2 interface description language has two features that may be useful to add The INCITS V2 design separates labeling information from the functional interface description and places the labels in a separate resource file The particular resource file used during in terface generation may be changed to support different locales for example or perhaps even users with different cognitive abilities The INCITS V2 functional specification also includes a notification construct which represents a quick dialog box style interaction that is dis played when an event
111. subjects happened to own a stereo of the same brand used in this study 3 3 2 Results The results of the study indicate all p lt 0 001 that subjects made fewer missteps and asked for help less using the prototype handheld interfaces than using the actual appliances see Figure 3 4 This indicates that the prototype handheld interfaces were more intuitive to use than the actual interfaces Time was not recorded for this study because we believed that delays created by the paper prototypes would dominate the time required to complete all the tasks Even so informal measurements suggested that subjects needed about one half the time to complete all of the tasks using the prototypes as compared to the actual appliances Missteps Help Requests per Trial per Trial NO Q gI oO Missteps Requests Stereo Telephone Stereo Telephone L Existing Appliance Interface L_ Prototype Handheld Interface Figure 3 4 Box plots showing the range of missteps and help requests uses of external help for each appli ance and interface type Chapter 3 Preliminary User Studies 44 3 3 3 Discussion Users had great difficulty using the actual appliances but were able to understand and oper ate the paper prototype interfaces with reasonable ease One exception was found in the prototype stereo interface which made use of the Palm s built in menu system None of our subjects navigated to screens that were only accessible through the
112. tem like the home theater system described in the previous section often depends on how its component pieces are connected together For example video performance will be bad if a DVD player is connected through a VCR or it may not be possible to record video from one VCR onto another if the two are not connected properly The PUC sys tem has sufficient information to reason about such problems and conceivably could help users find solutions for their particular systems but this is not something that I have looked at as part of my thesis work Service Discovery A PUC controller device must be able to discover appliances in the environment that the user may wish to control Efficiently performing this task without centralized servers has been a focus of several research projects and those techniques have become common enough to be included in commercial systems such as UPnP UPnP Chapter 1 Introduction 13 2005 and JINI Sun 2003 The PUC system relies on existing techniques and does not further this research End User Programming and Macros Facilitating end user programming tasks such as the creation of user specified macros would be an interesting direction for the PUC re search This area is not unique to the PUC system however and many others are exploring end user programming in other contexts I am confident that their advances could be applicable to the PUC system in the future Security Security is an importan
113. that it was important for subjects to feel in control of the appliances Unfortunately it was not possible to use the PocketPC to actually control either appliance so instead software was created to simulate the appearance of con trolling the appliances A laptop with external speakers was connected to the PocketPC via a wireless network allowing the user s actions to be transmitted to the laptop The laptop then Chapter 3 Preliminary User Studies 4 simulated control by generating auditory feedback that was consistent with what would be expected if the PocketPC were actually controlling either of the appliances 3 2 User Studies Both studies were between subjects comparisons of the hand designed PDA interfaces and the interfaces on the actual appliances The performance of the subjects was measured using several metrics including the time to complete a task the number of errors made while at tempting to complete a task and how often external help was required to complete a task The purpose of these studies was to discover how users performed using the hand designed interfaces versus the interfaces of the actual appliances and discover what aspects of the hand designed interfaces were difficult to use The studies of the low and high fidelity interfaces were very similar so I will discuss the common procedure and evaluation methods before discussing the studies in detail 3 2 1 Procedure When each subject arrived they were
114. they started Before writing the VCR specification subjects were trained on the lan guage through a written document with several exercises and examples from a to do list application I chose to use written training to ensure that the learning experience was the same for each subject The to do list application was used as an example because it incorpo rated every feature of the language but was different enough from the VCR that it was unlikely to affect the subjects specifications Training and authoring took a substantial amount of time about 1 5 hours and 5 hours respectively so we allowed subjects to take the materials and VCR home with them and complete the study over the course of two weeks Subjects were paid for their participation 15 for completing the training and 50 upon re turning the VCR and turning in a valid specification All three of the subjects specifications contained two top level groups for setup functions and basic controls All also had a group for timed recordings but not all placed the group in the same location Two of the three made timed recordings a top level group while the other chose to place it in the basic controls group Two of the three had an advanced controls group with one placing this group at the top level and the other putting it inside the basic controls group Within the common groups the subjects used different strategies to further organize the functions For example one subject organize
115. this appliance Placement Inside the lt content flow gt or lt content group gt elements Parameters e content type required The type of content produced by this source May Contain ACCLVSe Li gt lt inp t ports gt lt processing gt lt output p rts gt ob jects lt recorder gt lt recorder name Tape content type av gt lt recorder gt Describes a content sink that records a content stream received as an input Placement Inside the lt content flow gt or lt content group gt elements Parameters e name required A name for the source e content type required The type of content produced by this source May Contain lt agtiye 1f gt lt input po rts gt lt object gt Appendix B Specification Language Reference 247 lt renderer gt lt renderer name Screen content type video gt lt renderer gt Describes a content sink that renders a content stream received as an input for the user Placement Inside the lt content flow gt or lt content group gt elements Parameters e name required A name for the source e content type required The type of content produced by this source May Contain lt aclcive 1Li gt lt input ports gt lt ob Jecte gt lt input ports gt lt input ports gt lt input ports gt Describes the input ports that may be used by a content flow Placement Inside the lt pass through gt lt reco
116. to ensure consistency it is time to generate the concrete user interface In the next two sections I describe how the con crete interface is generated for the PocketPC desktop and Smartphone platforms 8 5 1 PocketPC and Desktop The PocketPC and desktop interface generators use the same process for generating their concrete interfaces which is split into two steps The first step generates an initial version of the user interface with only the most important structural elements The second step looks for layout problems in the initial version of the generated interface such as panels that re quire scroll bars to display all of their content and controls that are not wide enough to display their labels Where problems are found rules attempt to fix the problems by adding organization or changing the layout Using this approach the user interface is built with the minimum amount of organization needed for the size of the screen and the number of appli ance functions Minimal organization is beneficial because it allows users to see all of the functionality of the interface in fewer screens and reduces the number of clicks necessary to navigate between functions 8 5 1 1 Creating the Initial Interface The concrete interface is represented by an interface tree that describes the panel structure of the user interface The children of the root of this tree represent the different windows used in the interface and the leaf nodes repres
117. two additional messages that allow appliances to identify them selves to other appliances In our current design we assume a lossless underlying networking technology such as TCP IP and so our protocol does not include any messages whose sole purpose is to acknowledge the receipt of another message The protocol also does not define a strict ordering among any of the messages both controller devices and appliances must be prepared to receive any message at any time All messages in the PUC protocol are composed of two chunks of a data an XML chunk followed by an optional binary chunk The binary chunk is used to transmit data typically images which cannot be easily converted to text and transferred within the XML chunk Both chunks of data are preceded in the message by two 4 byte fields the first containing the length of both data chunks and the second containing the length of only the XML chunk see Figure 4 5 The data in the XML chunk must conform to our communication protocol schema which defines each message type and each type s parameters This schema can be found in Appen dix C 1 and detailed documentation on the language is available online at http www pebbles hcii cmu edu puc protocol spec html1 Messages are designed around state variables and commands which are the basic functional elements of the PUC specification for more detail see section 5 2 1 1 Controller devices may send messages requesting changes to a state
118. type xs string use optional gt name spec2 guid type xs string use required gt name spec2 name type xs string use optional gt lt xs complexType gt lt xs complexType name GroupType gt lt xS sequence gt lt xs element name label type xs string minOccurs 0 maxOccurs 1 gt lt xs element name source type SourceType minOccurs 1 maxOccurs 1 gt lt xs element name cost type MappingCostType minOccurs 1 maxOccurs 1 gt lt xs choice minOccurs 1 maxOccurs unbounded gt lt XS lt XS lt XS lt XS element name mapping type MappingType gt element name state mapping type StateMappingType gt element name template mapping type TemplateMappingType gt element name location mapping type LocationMappingType gt lt xs choice gt lt xs sequence gt lt xs complexType gt lt xs complexType name MappingCostType gt lt xs attribute name one to two type CostAttribType use reguired gt lt xs attribute name two to one type CostAttribType use reguired gt lt xs complexType gt lt xs complexType name SourceType gt lt xs attribute name type type SourceAttributeType use reguired gt lt xs attribute name confidence type ConfidenceType use optional gt lt xs complexType gt lt xs complexType name LocationMappingType gt lt xS sequence gt lt xs element name label type xs stri
119. used to set the maximum of a numer state variable to be the value of an other state variable Placement Inside the lt max gt lt min gt lt average gt and lt incr gt elements Parameters e state required The name of a state variable with a numeric type Appendix B Specification Language Reference 241 lt constant gt lt Constanu value i12 gt Used to define a constant value in any location where a reference would also be ac cepted i e the lt ref value gt element Placement Inside the lt max gt lt min gt lt average gt and lt incr gt elements Parameters e value required The constant value lt completions available gt lt completions available gt Used to specify that completions are available for this state variable Placement Inside lt state gt lt server side error corrections gt lt server side error corrections gt Used to specify that this variable will be automatically error corrected on the server side Placement Inside lt state gt Appendix B Specification Language Reference 242 lt ports gt lt portse lt ports Defines the section in which the physical ports of the appliance are described Placement Inside the lt spec gt element May Contain Inputs gt lt Inp t o tput lt outputs gt lt inputs gt lt inputs gt lt inputs gt Defines the physical input ports of an appliance Placement Inside the l
120. users have seen This is the only phase that adds new information to the knowledge base The functional and abstract structural phases modify the abstract user inter face to ensure that functions appearing in previous interfaces are manipulated the same way and appear in the same location in the new interface The concrete phase is used to clean up visual consistency problems created by the interface generator s transformation of the ab stract user interface to the concrete interface such as changing the orientation of panels or adding additional organization I discovered in the creation of the consistency algorithms that most of the work needed to ensure consistency can take place in the third and fourth phases which operate on the abstract user interface This means that most of the consistency rules can be shared across platforms without modification Input Appliance a J Specification T Ta Generation Consistency Rule Phases Rule Phases Mapping Phase Abstract UI Phase 2 3 Functional w Previous Specifications Phase ie Functional Mappings Between Specicatons 4 Abstract _ _ __ Previously Generated Uls Structural Phase Concrete Ul 5 Phase Output nA Consistent User Interface Figure 8 4 PUC interface generation process diagram Chapter 8 Interface Generation 120 8 3 Generating the Abstract Interface The abstract interface is created from the appli
121. value in such a system e Would manufacturers support such a system For many years usability has not been an important criterion for consumers when purchas ing a new appliance Brouwer Janse 1992 Price and features have been the most important Chapter 11 Conclusion 179 factors as well as appearance and brand For consumers to purchase and use products with PUC technology usability will need to become a more important factor in buying decisions I think there is reason to believe that consumers buying habits may be changing however The iPod for example dominates its market in large part because it is highly usable Many reviews of technology products particularly on web sites such a CNET and Engadget com routinely evaluate usability along with the usual factors like price and performance Of course before customers can decide whether or not they would like to buy products that include the PUC manufacturers will need to include the technology in their products There are several issues that manufacturers might have with this technology most importantly the loss of control over their products interfaces Manufacturers will at least want to include branding marks in their interfaces and are likely to want some assurance that customers will not attribute fault to the manufacturer if a poor interface generator produces a bad interface for their product The current PUC system does not have support for including brand marks th
122. variable or the invocation of a command In response appliances will often send one or more state change notifications These three mes sages are most common sent in the PUC protocol State change notifications may also be sent when the controller device requests an update of an appliance s entire state There is also a message for a controller device to request an appliance s specification and another message used by the appliance to send the specification document The PUC s support for binary data and for complex data formats such as lists also intro duces the possibility that large amounts of data will need to be sent to the controller device whenever the appliance needs to update its state To address this problem for lists we have included four list operations that allow the appliance to only update the portion of a list that Chapter 4 System Implementation 62 Table 4 3 Messages that may be sent by the controller device Description sega uueueauuugueaeueeuegenesuegusegueaagegueguseuegusegueguseaeguseuegagugaegugegueauseaeauseeaegueueaeauguegegagugaegaguggeseyeeeegeguegggaeuguesugueaeguseegegugeegeeusuegesuguesegugueaegeseagaueaeaeguseaegueueaesueueaesaseaegueueagaueueaeauseeaeaueaeaesusaeaususaeaesusueaesueugeesaeueaugaeaeaesaeueaesaeeaesueaeausaeueaeseueaegugueaesueaeaeaueueaesueueaesaeueaesasuuaesasueaegugueausagugagsasuuaugusagagsagagaugasgeaggaggggggeggggggaggy _ This message requests the appliance to change the desi
123. which could be dis rupted when the function is moved Furthermore in my experience with the home theater appliances I noticed that different appliances sometimes share the same high level structure e g Control and Setup groups very high in the group hierarchy but very little of the same low level hierarchy In this situation a moving rule may move a function from deep in the hierarchy to be a child of a very high level group Often this will make the moved function more prominent than is appropriate More work is needed to understand the situations in which structural consistency rules help and hurt so that this knowledge can be integrated into the consistency rules One possibility may be to revisit the ideas of sparse branch and significant consistency that I have discussed elsewhere Nichols 2005 but there is still a need for software to be able to reliably detect these types of consistency In particular a threshold must be deteremined between the number of similar and unique functions in two groups of different appliance specifications that is sufficient to specify whether the similaral ity of those groups is sparse or significant My consistency algorithms also rely entirely on the similarity information that is contained within the PUC knowledge base In this work I have assumed that the knowledge base does not contain any false mappings which is unlikely if the similarity information is collected by an automated schema mapping algo
124. 1 3 HH Ring Standard Iaa UC v4 0 1133 UH 4x 8 21 amp Maintenance Print Nozzle Check Patter Cean Print Cartridges Deep Clean Print Head Align Print Cartridges Manually Align Print Head Auto Color Calibration Oean Sheet Feeder Roller Cean Bottom Plate Figure F 4 The full interfaces for the Canon All In One printer generated to be consistent with the HP printer as seen by users in the usability study 313 Appendix F Gallery of PUC Interfaces 314 Bibliography Ali 2002 Mir Farooq Ali Manuel A Perez Quinones Marc Abrams and Eric Shell Building Multi Platform User Interfaces with UIML Computer Aided Design of User Interfaces Valenciennes France May 2002 pp 255 266 Association 1996 1394 Trade Association A V C Digital Interface Command Set 1996 http www 1394ta ore Banavar 2004a Guruduth Banavar et al An Authoring Technology for Multidevice Web Applications IEEE Pervasive Computing July Sept 2004a 3 3 pp 83 93 Banavar 2004b Guruduth Banavar Lawrence D Bergman Yves Gaeremynck Danny So roker and Jeremy Sussman Tooling and system support for authoring multi device applications The Journal of Systems and Software 2004b 69 3 pp 227 242
125. 10 1 Combined with the built in interfaces for the two printers this results in the six total interfaces used in the studies Complete screenshots of these interfaces can be viewed in Appendix F 10 2 Protocol The subjects who used the PUC interfaces first had a short tutorial on the interface of the PocketPC handheld device This was necessary because the PUC s design assumes that users will be familiar with the device they are using and the PocketPC has several interface quirks that can frustrate users who are not aware of them e g the Ok button in dialog boxes is lo cated in the title bar at the top of the screen Since the intention of the PUC is to work on people s own personal devices it is reasonable to expect that they will be familiar with the user interface of the device itself Chapter 10 Usability Evaluation 165 All subjects performed a block of eight tasks on one of the six interfaces just described After completing all of the tasks the subjects received instruction on the quickest method of per forming each of the tasks they had just performed After receiving instruction on a task subjects were required to perform the task again until they did not make errors Additional instruction was available for the tasks as needed by the subject Once the instruction period was completed successfully the subject performed a second block of the same eight tasks on a different interface for the other all in one printer The g
126. 25 3 PUC 4 0 1103 Y d 12 19 3 PUC 4 0 1103 Y 4 12 18 3 Resize Presets om Fit To Page o 60 Custom Fi Borderless _ Borderless Copy Repeat Image con Foe sen Po PUC Servers Devices Generation E z Puc v4 0 1102 Y4 11 145 amp Resize Options Status dis BA repeat Image Image Size x sea rots aT PUC Servers Devices Generation E Copy a HP printer without consistency oja PUC Servers Devices Generation E Copy Puc v4 0 1102 Y4 12 00 amp 7 a Tray Two Pages on One a Image Columns Lo o bhi copy Fax Scan Photo 4 gt PUC Servers Devices Generation E b Canon printer without consistency E No of Columns E c Canon printer consistent with HP Figure 8 8 A demonstration of the Bring Together Split Dependents rule The top image shows the loca tion of the resize mode function and the bottom shows the location of the repeat image function and its parameters In column c the dialog box shown on the bottom is opened by pressing the Repeat Image button shown in the top image Chapter 8 Interface Generation 131 specification contains the unmoded functionality then this functionality must be moved near the state variable If the previous specification contained unmoded functionality then the movi
127. 256 lt xsi restriction gt lt xs simpleType gt lt xs simpleType name ConfidenceType gt lt xs restriction base xs float gt lt xs miniInclusive value 0 gt lt xs maxInclusive value 1 gt lt xs restriction gt lt xs simpleType gt lt xs simpleType name SimilarityAttributeType gt lt xs restriction base xs string gt lt xs enumeration value sparse gt lt xs enumeration value branch gt lt xs enumeration value significant gt lt xs restriction gt lt xs simpleType gt lt xs simpleType name CostAttribType gt lt xs restriction base xs string gt lt xs enumeration value equivalent gt lt xs enumeration value fully contained gt lt xs enumeration value overlapping gt lt xs enumeration value no overlap gt lt xs restriction gt lt xs simpleType gt lt xs simpleType name MapTypeAttributeType gt lt xs restriction base xs string gt lt xs enumeration value function gt lt xs enumeration value organization gt lt xs restriction gt lt xs simpleType gt lt Spec Store Table Types gt lt xs complexType name SpecStoreTableType gt lt xs sequence gt lt xs element name spec type SpecType minOccurs 0 maxOccurs unbounded gt lt xs sequence gt lt xs complexType gt lt xs complexType name SpecType gt lt xS sequence gt lt xs element name location type xs string minOccurs 1 maxOccurs 1 gt name generated interface loca
128. 27 30 2002 pp 171 180 Bibliography 318 Nichols 2003 J Nichols Myers B A Studying The Use Of Handhelds to Control Smart Appliances 23rd International Conference on Distributed Computing Systems Workshops ICDCS 03 Providence RI May 19 22 2003 pp 274 279 Nichols 2002a J Nichols Myers B A Higgins M Hughes J Harris T K Rosenfeld R Shriver S Requirements for Automatically Generating Multi Modal Interfaces for Complex Appliances ICMI Pittsburgh PA 2002a Nichols 2002b Jeffrey Nichols Brad A Myers Michael Higgins Joseph Hughes Thomas K Harris Roni Rosenfeld and Mathilde Pignol Generating Remote Control Inter faces for Complex Appliances UIST 2002 Paris France 2002b pp 161 170 http www cs cmu edu pebbles papers PebblesPU Cuist pdf Nichols 2004a Jeffrey Nichols and Brad Myers Report on the INCITS V2_ALAP URC Standard 2004a http www cs cmu edu jeffreyn papers cmu puc v2 report pdf Nichols 2004b Jeffrey Nichols Brad A Myers and Kevin Litwack Improving Automatic Interface Generation with Smart Templates ntel igent User Interfaces Funchal Portu gal 2004b pp 286 288 Nichols 2005 Jeffrey Nichols and Brad A Myers Generating Consistent User Interfaces for Appliances Workshop on Multi User and Ubiquitous User Interfaces MU31 San Diego CA 2005 pp 9 10 http www jeffreynichols com papers nichols mu3i pdf Nielsen 1993
129. 295 aaz a Desktop PUC with Uniform v1 0 295 BAS a Desktop PUC with Uniform v1 0 295 BA PUC Servers Devices Generation PUC Servers Devices Generation PUC Servers Devices Generation Power On x Power Setup DvDcD MP3 VCR Power Status Setup DVD CD MP3 VCR On zi Progress Clock hd On E Tape Recordable No Status Controls Setup Timed Recording Analog Only TV View Complete Clock Set Marua zl wE ee 01 30 00 PM Recor English No Time Zone Easten Protected Daylight Savings Time No Yes TY View Clock Set Maul vcr 01 30 00 PM Time Zone Eaten amp l Clock Set Channel lt Auto Clock Set Manual od 01 30 00 PM Daylight Savings Time No Time Zone Eastern Lel Lal kt Daylight Savings N o Month April Year Ki Date a Digitally Copy Protected On Month Apri Year al Date a Digitally Copy Protected Off Month April Year a Date 1 J lle l a Mitsubishi DVCR b Samsung DVD VCR c Samsung DVD VCR Without Consistency Without Consistency Consistent with Mitsubishi DVCR Figure 8 7 User interfaces generated by the desktop PUC for a Mitsubishi DVCR and a Samsung DVD VCR without consistency and the Samsugn DVD VCR generated to be consistent with the Mitsubishi DVCR Note that the clock functions are located under the Status tab for the Mitsubishi DVCR under Setup f
130. A Foltz Ligature Gesture Based Configuration of the E21 Intelligent Environment Proceedings of the MIT Student Oxygen Workshop 2001 Gajos 2004 K Gajos Weld D SUPPLE Automatically Generating User Interfaces Intelligent User Interfaces Funchal Portugal 2004 pp 93 100 Gajos 2005a Krzysztof Gajos David Christianson Raphael Hoffmann Tal Shaked Kiera Henning Jing Jing Long and Daniel S Weld Fast And Robust Interface Genera tion for Ubiquitous Applications Seventh International Conference on Ubiquitous Computing UBICOMP Tokyo Japan 2005a pp 37 55 Gajos 2005b Krzysztof Gajos Anthony Wu and Daniel S Weld Cross Device Consis tency in Automatically Generated User Interfaces Proceedings of the 2nd Workshop on Mulu User and Ubiquitous User Interfaces San Diego January 9 2005b pp 7 8 Gajos 2006 Krzysztof Z Gajos Jing Jing Long and Daniel S Weld Automatically Gen erating Custom User Interfaces for Users With Physical Disabilities ASSETS Portland OR 2006 p To appear Gomes 2003 Lee Gomes Appliances Have Become Like PCs Too Complex for Their Own Good The Wall Street Journal OnLine May 12 2003 http www pebbles hcit cmu edu puc localmedia wsj 20030512 pdrf Grudin 1989 Jonathan Grudin The Case Against User Interface Consistency CACM CACM Oct 1989 1989 32 10 pp 1164 1173 Hall 2003 Richard S Hall and Humberto Cervantes Gravity
131. Add is a list add gt lt labels gt lt label gt Add lt label gt lt labels gt lt command gt lt command name Delete is a list remove gt lt labels gt lt label gt Delete lt label gt lt labels gt lt command gt lt command name Clear is a list clear gt lt labels gt lt label gt Clear All lt label gt lt label gt Clear lt label gt lt labels gt lt command gt lt group gt lt group gt lt group gt lt group gt lt groupings gt lt ports gt lt inputs gt lt port name VHF UHF Antenna content type multi channel av physical type coax gt lt inputs gt lt outputs gt lt port name VHF UHF Antenna content type multi channel av physical type coax gt lt port group name Output content type av gt lt port name Video content type video physical type RCA gt lt port group name Audio content type component audio gt lt port name Right content type component audio right physical type RCA gt lt port name Left content type component audio left physical type RCA gt lt port group gt lt port group gt lt outputs gt lt ports gt lt content flow gt lt pass through content type av gt lt active 1if gt lt equals state Base Power gt lt constant value false gt lt equals gt lt active if gt lt input ports gt lt port name VHF UHF Antenna gt lt input ports gt lt output ports gt lt por
132. Automatically Generating High Quality User Interfaces for Appliances Jeffrey Nichols December 2006 CMU HCI 06 109 Human Computer Interaction Institute School of Computer Science Carnegie Mellon University Pittsburgh Pennsylvania 15213 Thesis Committee Brad A Myers Chair Scott E Hudson John Zimmerman Dan R Olsen Jr Brigham Young University Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy Copyright 2006 Jeffrey Nichols All rights reserved This research was supported in part by the National Science Foundation through the author s Graduate Re search Fellowship and grants IIS 0117658 and IS 0534349 Microsoft General Motors the Pittsburgh Digital Greenhouse and Intel Equipment supporting this research was generously donated by Mitsubishi Elec tric Research Laboratories VividLogic Lantronix Lutron IBM Canada Symbol Technologies Hewlett Packard and Lucent The views and conclusions contained herein are those of the author and should not be interpreted as necessarily representing the official policies or endorsements either expressed or implied of any sponsoring party or the U S Government Keywords Automatic interface generation aggregate user interfaces handheld computers personal digital assistants mobile phones home theater appliances personal universal con troller PUC user interface description languages UIDLs remote controls multi modal inte
133. Base Setup for the DVD VCR and Base Status for the DVCR which are different These items in each specification are also adjacent and the move will not cause an item to cross a mutual exclusive group boundary which means that the Move Single Item rule will be applied The moving rule will take the clock group and move it to the DVD VCR s Base Status group which is similar to the group with the same name on the DVCR see Figure 8 6a Note that the moving algorithm also chains appropriately For ex ample the clock group on the DVD VCR contains a variable that specifies the channel from which clock information can be extracted The DVCR has a similar state variable but is lo cated in the setup group instead of the clock group Before the PUC moved the clock group the clock channel variable had the same containment in both specifications but afterward this is no longer true When the algorithm is applied to this channel variable the difference in containment is found because the containment stacks are calculated from the variable s current location The algorithm will then move the channel variable back to its consistent location in the setup group see Figure 8 6b Note that this movement is visible in the gen erated interfaces Figure 8 7b shows the clock group under the Setup tab with the clock set variable and Figure 8 7c shows the clock group in the Status tab without the clock set vari able Chapter 8 Interface Generation 128
134. C December 31 2003 ISO 1988 ISO Information Processing Systems Open Systems Interconnection LOTOS A Formal Description Technique Based on Temporal Ordering of Observational Behavior 1988 Ivory 2001 Melody Y Ivory and Marti A Hearst The State of the Art in Automating Us ability Evaluation of User Interfaces ACM Computing Surveys December 2001 33 4 pp 470 516 John 1996 Bonnie E John and David E Kieras The GOMS Family of User Interface Analysis Techniques Comparison and Contrast ACM Transactions on Computer Human Interaction 1996 3 4 pp 320 351 John 2004 Bonnie E John Konstantine Prevas Dario D Salvucci and Ken Koedinger Predictive Human Performance Modeling Made Easy CHI Vienna Austria April 24 29 2004 pp 455 462 Kellogg 1987 Wendy A Kellogg Conceptual consistency in the user interface Effects on user performance Proceedings of LINTERACT S7 Conference on Human Computer In teraction Stuttgart September 1 4 1987 Kieras 1995 David E Kieras Scott D Wood Kasen Abotel and Anthony Hornof GLEAN A Computer Based Tool for Rapid GOMS Model Usability Evaluation of User Interface Designs Ezghth Annual Symposium on User Interface Software and Technol ogy Pittsburgh PA Nov 1995 pp 91 100 Kim 2004 Jihie Kim Marc Spraragen and Yolanda Gil An Intelligent Assistant for In teractive Workflow Composition Intelligent User Interfaces IUI
135. ESESSSSESESOSSESESOESESESOESESESOESESESSESESESOSSESESSESESESOESESESOESESESSSSESESSESESESSESESESSESESESSSSESESSESESESSESESESSESESESSESESESSESESESEESESESSESESESESSESESSSSESESEESESEES Home Entertainment Appliances Office Appliances Audiophase Shelf Stereo DirecTV D10 300 Receiver Gefen 2 1 HDMI Switchbox InFocus 61MD10 Television JVC 3 Disc DVD Player Mitsubishi HD H2000U DVCR Panasonic PV V4525S VCR Philips DVD Player DVDP642 Samsung DVD V1000 DVD VCR Combo Player Sony A V Receiver Sony Camcorder Lighting Controls Intel UPnP Light Lutron RadioRA Lighting X10 Lighting GMC 2003 Yukon Denali Systems Driver Information Console AT amp T 1825 Telephone Answering Machine Canon PIXMA 750 All In One Printer Epson PowerLite 770 Projector HP Photosmart 2610 All In One Printer Complex Copier Simple Copier Desktop Applications Laptop Video Controls Microsoft PowerPoint Microsoft Windows Media Player 9 PUC Photo Browser PUC To Do List Task Manager Alarm Clocks Equity Industries 31006 Alarm Clock Timex T150G Weather Alarm Clock Other Axis UPnP Pan Tilt Zoom Surveilance Camera Simulated Elevator Climate Control System Navigation System Table 5 2 Maximum and average counts of various aspects of the PUC specifications written to date Functional Labeled Smart Tem Max Tree Ave Tree Elements Groups Groups plates Depth Depth Css eso ns ean a8 Not including GM navigation system specification
136. However the modification algorithms may also need extra information about the concrete interface such as links that specify which labels go with which controls A second challenge is to understand how to modify an inter face in a way that is consistent with the original hand design For example the interface layout may be based on a grid that all changes should take into account Thirdly such a sys tem needs to work with real interfaces that are implemented using current interface toolkits such as Java Swing or the NET Framework This may present many issues For example how should an automatic modification system deal with custom controls which are quite common in most existing interfaces It would also be useful to understand if there are any features that toolkits might include to help with automatic modification The implications of supporting consistency in hand designed interfaces could be quite inter esting especially for web applications Today there are many competing web sites such as MySpace Orkut and Friendster that support basically the same functionality with slightly different user populations With automatic modification to aggregate similar functionality and ensure consistency it might be possible to automatically create a mash up of several similar web sites that provides access to all of the data from each of these web sites through a unified web interface Understanding the pros and cons of this approach and finding app
137. IML Abrams 1999 claims to provide a highly device independent method for user interface design but it differs from the PUC in its tight coupling with the interface UIML specifications can define the types of components to use in an interface and the code to execute when events occur The TIDE interface design pro gram Ali 2002 has been implemented to address some of these issues with UIML TIDE requires the designer to specify the interface more generically first using a task model Then the task model is mapped with the designer s assistance into a generic UIML model which Chapter 2 Related Work 33 is then further refined into a specific user interface This process is very similar to that of PIMA mentioned above 2 2 2 6 TERESA Transformation Environment for interactive Systems representations TERESA Mori 2004 is a semi automatic system for transforming user interfaces between different plat forms It allows designers to build their systems at an abstract level using the ConcurTaskTrees modeling language Paterno 1997 and then transform that model into other models at different abstraction levels including a concrete user interface for several dif ferent platforms TERESA s current focus is on web applications though in principle it could be extended to other environments The current system is able to automatically gener ate interfaces but in practice it seems to require designer involvement at each level of
138. Items gt lt active if gt lt equals state Base Power gt lt constant value true gt lt equals gt lt active if gt lt group name Status is a status icon group gt lt state name TapeIn access read only is a tape in status indicator gt 197 lt apply type type name YesNoType gt lt labels gt lt label gt Tape In lt label gt lt labels gt lt state gt lt state name TapeRecordable access read only is a tape recordable status indicator gt lt apply type type name YesNoType gt lt labels gt lt label gt Tape Recordable lt label gt lt labels gt lt active if gt lt equals state TapeIn gt lt constant value true gt lt equals gt lt active if gt lt state gt lt group gt lt group name Controls gt lt labels gt lt label gt Controls lt label gt lt labels gt lt group name PlayControls is a media controls gt lt active 1if gt lt equals state TapeIn gt lt constant value true gt lt equals gt lt active if gt lt labels gt lt label gt Play Controls lt label gt lt label gt Play Mode lt label gt lt text to speech text Play Mode recording playmode au gt lt labels gt lt state name Mode gt lt type gt lt enumerated gt lt item count gt 6 lt item count gt lt enumerated gt lt value labels gt lt map index 1 gt lt labels gt lt label gt Stop lt label gt lt labels gt
139. L editor you will need to download the schema for the PUC specification language which is available here http www pebbles hcii cmu edu puc puc xsd We strongly recommend using the schema with your editor as it guarantees that your document will be correctly formatted XMLSpy Home Edition XMLSpy Home Edition can be freely downloaded from the following link http origin altova com download_components html Near the top of this page you will find links for downloading the installer and requesting a free license Once you have the XMLSpy application installed and running you should screen like this Appendix D Specification Authoring Study Instructions Zk x Altova XMLSpy adog i File Edit amp uthentic Tools ADVANCED Window Help Element ax Attribute Ax Entity ax XMLSpy v2005 rel 3U Registered to Jeffrey Nichols Carnegie Mellon University 1998 2005 Altova GmbH CAP NUM SCRL Before you create a new file open the Tools menu at the top of the page and select the Op tions item A dialog box will open containing a set of seven tabbed panes Click on the tab for the Editing pane Once you have done this you should see the following screen except the red circle Schema fonts Text fonts Table wiew Smart table detection for rep elements Load entry helpers upon opening file Build table for any repeated elements Attributes _ Elements Show single table
140. Madeira Portugal 2004 pp 125 131 Bibliography 317 Kim 1993 Won Chul Kim and James D Foley Providing High level Control and Expert Assistance in the User Interface Presentation Design Human Factors in Computing Systems Amsterdam The Netherlands Apr 1993 pp 430 437 Lieberman 2006 Henry Lieberman and Jose Espinosa A Goal Oriented Interface to Consumer Electronics using Planning and Commonsense Reasoning Intelligent User Interfaces Sydney Australia 2006 pp 226 233 Limbourg 2004 Quentin Limbourg Jean Vanderdonckt Benjamin Michotte Laurent Bouillon and Victor Lopez Jaquero UsiXML a Language Supporting Multi Path Development of User Interfaces 9th IFIP Working Conference on Engineering for Hu man Computer Interaction jointly with 11th International Workshop on Design Specification and Verification of Interactive Systems EHCI DSV1S 2004 Hamburg Germany 2004 pp 200 220 Madhaven 2001 Jayant Madhaven Philips A Bernstein and Erhard Rahm Generic Schema Matching with Cupid 2744 VLDB Conference 2001 Mahajan 1997 R Mahajan and B Shneiderman Visual and Textual Consistency Checking Tools for Graphical User Interfaces IEEE Transactions on Software Engineering 1997 23 11 pp 722 735 Melnik 2002 Sergey Melnik Hector Garcia Molina and Erhard Rahm Similarity Flood ing A Versatile Graph Matching Algorithm and its Application to Schema Matching 18th ICDE
141. NS ciir on E ER A E OE AR 126 OA E VO E ETE E E NO E T E 126 BAL ROE tears TE E A E TA O 132 8 Generating the Concrete IMr aC aa a a ae 133 EAL LOREAL AUAN OD ri EAR E T OOTO 133 S dhd Creating the INIA INET soisi enea E 133 EIL IG Lajon LTOS water a a a E Gapetcetoeent ss 138 E OI DOTE PO E N E E ETETE 138 8 6 Modifying the Concrete Interface for Consistency sssssesssesessserssersssrserssee 143 Bey RESUS aii Discuss oien a ar NON 145 Aggregating User Interfaces 147 9 1 E TE o AAE EANA ES sp A 148 xiii 9 2 Content Flow for Understanding Systems of Appliances ce eeseeeeseeeeeneeee 149 OD GOVE CATOM A NEC ennaa aston naseseraseatheeecauetnagse a siaziae 150 ee Nye Bs d UR BAG n a ode nes teengesacn R 152 OD Apor aae cer Ime a ces aia eset eeepc esas Pascoe a eee ae 155 OSL ACHE TOW E OnO aE E T T T A 155 DDD AUDE TOW SO UD EEEE AT EN E I TEON IJ DED GI SI tea tee E E EE E EE ees 158 ITA VICI EOI EEEE E EE A T EA 159 I DBUS Oese a Gacen antaiieson accused nto aneecned 160 10 Usability Evaluation 161 TOSE cS ae alc eee te 163 WO A EOROC Cl epitaph EE E SEE OE EEEN AE NE EERTE AOE TEITE 165 DOS LAEE TE E EEr E E E EA r TA EO 167 LO PCPA S E A E E A ENO 168 104 Evaluation of Usability saree E E EE 168 TT MRO POT EE ETE EE A EA E E Sets seen E E E 168 LOAD DISCUSSION OF Usab N oari e E E E E A N 170 105 Eyaluation of C OnsistencCy ecs E E 171 TOS T R eee A E E A E OPE RT E T 171 103 2 Discussion of Consiste
142. OM AI E ious A E E R A EA NOTE 86 OA APPS COMIN EERE ATE ETNE OE 87 6 2 Specitieation Authoring Study eissa rona a AE 88 a aes ee 88 C2 ae 90 DD MOTTO cere tea ar en ik ease stent 91 63 Require ndents tor Consis ten yan rr Heanes ENE S 91 6 4 Understanding and Finding Similarities between Specifications 0 0 cee 93 CAT K NODA TE DIO aac sete eee E eae eee ean asda 93 CALLI Map hino NADIS o EEE alec E O OT a 95 6 4 2 Automatically Finding Map pin icine nein aati 97 Handling Domain Specific and Conventional Knowledge 7 1 Rok cna a a etc ee 100 xil FZ SOSsie mand Urennnirieta oriri toeetiee E ate 101 7 2 1 Implementing a Smart Template for an Interface Generator eseccceee 107 Poo momma empare AD a2 cies sree 8 toett a teiaated N 108 Taa MEISE Us O EE cases wise E E aaa 109 Interface Generation 113 Bob Generation PAON Senen n a T ict N E 114 EAL Poka CUNA LIN O EEEE EN A ta OOE tea ates 114 a E S O A T A E O 115 E EE AE E E E T EAR 117 s2 Genen Conconi era E E E reer 119 8 3 Generating the Abstract interlace usiessa rene ecees 121 8 3 1 Mutual Exclusive Dependency Information ns 121 8 3 2 Choosing Abstract Interaction Objects weed vein Saasesuietevineioe teeta owes 123 8 4 Modifying the Abstract Interface for Consistency ceseesseceesseeeeeseeeesaeeeens 123 8 4 1 Heuristics for Unique FUNCTIONS oireisiin a eE E 124 SA2 Funional AOA VAT Ons Tiran EE T A EE T A 125 OD Smutural Modif ti O
143. SSSESSUSUAUAUSESESESSSESSSAUAUAUSESESESESSSSEAUAUAUSESESSSESESESAUAUSSESESESESESSSAUAUSESESESESESSSSSAUAUAESESESESESESSAESESESESESESESEEEES Date Represents calendar dates Many date formats are supported to match the internal _ implementation of a particular appliance including integers strings and multi variable formats This template also allows for use of the special date manipulation _ control from each controller device s platform date time _ Merged template of date and time absolute The individual templates are used to transform the specification format into a common format that can then be used with a special control if available that allows the simultaneous manipulation of _ date and time dimmer A specialized template that represents a dimmer control for a lighting source This _ template supports an arbitrary scale with optional support for direct on and off _ functions This template is primarily used by the Lutron RadioRA lighting control scececsesecccncsscccccccscscecccenscseeecacnsccsceaaensucsseeacccscssseaaecusssseaaecgssseaaqagsnss sprstessesensnenanansnsnnenenanananansnsnsesenanananaansnsesesesananaesnsesesesesesasGsasAesesesesesesasasanAeseseseneeesa4aQanAeSeSeseseseiasanaeeseeeseseiesaiaiaeseeesesesesesaiaQaseeQeseseseseiasaiaeeeseQesesesesaianaeseQesesesesesaeaiaeseseseseseseiaianeeseseseseseiesananaesnecseieseisiaiaeeseseeeeeseiseanaeeeseseseseeeeenseeeeeeeeeeenny four way dpad Represents a four way directional c
144. Smartphone s screen see the left side of Figure 8 2 The Smartphone interface generator creates interfaces that follow Microsofts Smartphone user interface guidelines As for the PocketPC I chose this approach so that our interfaces would be consistent with other Smartphone applications allowing users to leverage their knowledge of their Smartphone to control appliances The guidelines stipulate that most in terfaces should use a list based hierarchy that leads to summary panes for viewing data or editing panes for modifying data The generator follows these guidelines and focuses on op timizing the structure of the lists so the hierarchy is shallow and each list requires only one screen Figure 8 2 shows a Smartphone and the full generated interface for the GMC Yukon Denali DIC The interface generator tries to keep as much of the interface in the list format as possible but sometimes a variable cannot be manipulated within the constraints of the list In this case an editing pane is created that contains the appropriate controls The interface generator may also create summary panes when a number of read only state variables are grouped together Note that the list hierarchy created by the interface generator is static so that users can learn the hierarchy over time and remember the locations of commonly used functions The user interface guidelines also state that the left soft button should always be used for in voking the most commonl
145. Templates disabled for a Windows Media Player interface on the PocketPC platform b Media controls rendered for the same interface with Smart Templates enabled on each of our three platforms At the top is the desktop the middle is PocketPC and the bottom shows Smartphone The Smartphone control maintains consistency for the user by copying the layout for the Smartphone version of Windows Media Player the only media player application we could find on that platform This interface overloads pause and stop onto the play button c Different configurations of media playback controls auto matically generated for several different appliances Chapter 7 Handling Domain Specific and Conventional Knowledge 104 out the buttons on one or more lines depending on space enlarging buttons of greater im portance such as Play and using a grid to create aesthetically pleasing arrangements see Figure 7 3c The PUC interface generators can also robustly deal with content within a Smart Template that does not conform to the restrictions specified by the template registrar When content is encountered that the interface generator does not understand this content is passed back for rendering by the normal interface generation process This approach is also used in other cases when the Smart Template rules understand the content but are not able to produce a rendering better than the normal generator For example the time absolute template has support for op
146. Thomas K Harris Describing Appliance User Interfaces Abstractly with XML in Proceedings of the Workshop on Developing User Interfaces with XML Advances on User Interface Description Languages Gallipoli Italy May 25 2004 pp 9 16 65 detail highlighting the language s unique aspects Finally I will conclude with some infor mal evaluation and observations about the specification language and its effectiveness This design of the language is shown through an example specification for a basic VCR ap pliance specification This VCR has five functions that users can manipulate power the common media controls including record and eject channel TV VCR and a list of timed recordings that will take place in the future There are also two status indicators for deter mining whether a tape is in the VCR and whether that tape is recordable The VCR has only one physical input a standard television antenna and two physical outputs an antenna passthrough and the standard three wire yellow red white plugs for composite video and ste reo audio All of these features can be described in the specification language as will be shown below The full specification for the simple VCR can be found in Appendix A snip pets of the simple VCR s specification will be shown as each of the language s features is described This chapter focuses on the conceptual aspects of the language with limited discussion of syntax Readers interested in a
147. UC 3 0 39 a t9 Yy PUC 3 0 39 a t9 Yy PUC 3 0 39 si abc Yy PUC 3 0 39 D Ty PUC Log PUC Log Alarm Warning Image Type PUC 3 0 39 d Headlamps and Horn 4 Text 4 gt Starting Done Automatic Locking i i A i ctlanecie code oer onofPak sl MMN 3 Logging Pane 2 Clear Mapping Counts 4 Options 3 Clear Specification C Automatic Unlocking 5 Recent Servers gt 4 Save Knowledge Base Unlock Al i Park 2 6 Current Servers gt 6 Current Servers gt Seat Position Recall 7 Devices 7 Devices gt Seat Position Recall Off 4 gt Open Mem Bak Mem a b re d e Figure 4 4 Screenshots of interfaces for the Smartphone interface generator showing the menu based interface a c and custom controls built for the Smartphone d e Smartphone devices have two soft buttons underneath the screen with the leftmost button invoking a common command and the rightmost button opening a single menu This menu has hierarchical items that include the same functionalaity as the PocketPC and desktop gen erators The initial view of the Smartphone interface see Figure 4 4a shows a logging panel with the two initial soft button labels at the bottom From this screen the user can press the left soft button to open a connection to an appliance server or the right soft button to open the menu see Figure 4 4b The menu has hierarchical items corresponding to the server device and generation menus of the PocketPC and desktop generators R
148. When the user modifies a fax setup function the rule causes the interface generator to automatically change to the fax mode apply the user s change and then switch back to setup mode All of this can be done quickly in the background without causing any noticeable confusing changes in the user interface The Move One To Many Split Items rule handles the situation where one item in a speci fication is mapped to multiple split items in another specification such as for the front display functions in the Mitsubishi and Samsung VCRs discussed above The behavior of this rule differs depending on direction in which consistency must be ensured If the new specification contains the split items and the previous specification contains a single item then all of the split items are moved to a location similar to that of the single item If the situation is reversed then the single item is duplicated in all of the locations where the split items were found in the previous specification A future version of this rule might try to modify the single item to match each of the split situations but I could not find a generaliz able way to do this from the information currently available in the mappings For example when making the Samsung interface consistent with the Mitsubishi rules with more informa tion might be able to split the Samsung s overlapping brightness and content control for the front display into two independent controls to match the f
149. a logical operation block such as lt and gt or lt or gt tags Placement Inside the lt command gt lt explanation gt lt group gt lt list group gt lt union group gt lt map gt lt list selection gt or lt state gt elements Parameters Appendix B Specification Language Reference 236 e ignore stop dependency inheritance through the group tree The possible options are to omit the option parent and all May Contain lt and gt lt apply over gt lt defined gt lt equals gt lt false gt lt greaterthan gt lt lessthan gt lt not gt lt or gt lt true gt lt undefined gt lt apply over gt lt apply over list SomeList items all true if any gt lt apply over gt Used to apply dependency information to lists of information There are three different ways that dependencies can be applied The apply over block can be true if the dependencies are true for any item in the list if they are true for all items in the list or if they are true for no items in the list The particular choice is chosen with the true if attribute Placement Inside the lt active if gt lt and gt lt apply over gt lt not gt or lt or gt elements Parameters e ist required The name of the list group that the dependencies are applied to it e true if Sets when the apply over returns true The possible options are any all or none May Contain lt and gt
150. a question of whether the PUC would be equally successful for expert users As users become experts they are less likely to make mistakes which would probably benefit the harder to use Built In appliance interfaces more than the PUC interfaces How ever fewer steps are required to navigate to and use most functions in the PUC interfaces Furthermore the PUC interfaces provide more visual context for the user s current location in the interface We believe that these features would allow users to become experts with the PUC interface faster than the Built In interfaces and perform faster once experts The results of the second study suggest this may be true as discussed in the next section 10 5 Evaluation of Consistency To evaluate consistency the completion times of interfaces in the AutoGen and Consistent AutoGen conditions are compared for the second block of tasks We also compare the Built In condition to the Consistent AutoGen condition to see how consistency might further improve today s appliance interfaces 10 5 1 Results Figure 10 4 shows the average completion times for each task in the second block for the AutoGen and Consistent Auto Gen conditions Table 10 2 shows the same data in more de tail and includes the Built In condition and failure data for all the conditions Again one way ANOVAs are used to compare the completion times of the various conditions Failures are not discussed here because nearly all subjects w
151. abstraction to create a usable interface 2 2 2 7 USIXML The USer Interface eXtensible Markup Language USIXML Limbourg 2004 allows the specification of many different types of user interface models including task domain pres entation and context of use models with a substantial support for describing relationships between all of the supported models The explicit goal of this language is to support all fea tures and goals of previously developed UIDL s and as such it has many many features USIXML appears to be complete enough to specify all of the features in the PUC specifica tion language but it is not clear how easy the language is to author or whether it is concise enough to produce specifications that can be easily handled by a resource constrained device 2 2 2 8 XAML and XUL The eXtensible Application Markup Language XAML Microsoft 2006 and the XML User interface Language XUL Bojanic 2006 are two different languages for specifying a user interface developed by Microsoft and Mozilla respectively XAML will be used in the next major revision of the NET Framework and eventually in the Vista operating system to describe most graphics content that is rendered to the screen XUL is currently used to define the user interfaces of all Mozilla browsers Documents written in either language are similar to the presentation models used by many model based systems which abstract some plat form specific elements but are typica
152. ace and a set of resource descriptions that give human readable labels and help information for the user interface The URC will either then automatically generate an interface from an abstract presentation template or display one of the interfaces specified in a concrete presentation template An example WinAmp in terface generated from by a URC using the V2 framework is shown in Figure 2 3 I have provided feedback to the V2 group in the past that led to the current design of their specifi cation A detailed report is available analyzing the similarities and differences between the V2 and PUC systems Nichols 2004a 2 1 2 2 Universal Plug and Play Universal Plug and Play UPnP is designed both to allow user control and appliance inter operation The two important units within UPnP are the service and the control point which are similar to the appliance and controller respectively in the PUC system Each UPnP service has a downloadable description formatted in XML that lists the functions and state variables of that service Control points connect to services download their descriptions and then call functions and receive event notifications from the services One important differ ence between the PUC and UPnP is the automatic generation of user interfaces UPnP chose to avoid automatic generation and instead relies on standardized appliance descriptions A standardized description allows a control point to know in advance what
153. agine you find the copies that the device produces too dark Improve this by changing one setting of the device You do not need to produce any actual copy The device remembers the current date and time However you suspect they are incorrect Find out at where you can change them Actual change is not necessary Configure the device so that it will print out an error report only when it has problem receiving a fax it should NOT print out any error report when it has problem sending a fax Configure the device so that when faxing it will ALWAYS automatically redial a number that is busy Configure the device so that any received fax larger than the default paper size will fit on one page Appendix E Usability Study Instructions On the device you can assign speed dials to fax numbers that you use often There are two types of speed dials One is called One touch Speed Dial the other is called Coded Speed Dial You will only use the Coded Speed Dials for this task Imagine a document has already been properly placed on the device Fax it using the third Coded Speed Dial in black and white On the device you can assign speed dials to fax numbers that you use often Some speed dials have already been set up for you Imagine a document has already been properly placed on the device Fax it using the third speed dial in black and white 307 Appendix E Usability Study Instructions 308 APPENDIX F G
154. ajos 2006 Figure 2 4 shows some example interfaces generated by SUPPLE SUPPLE s approach allows it to manage the trade offs in an interface design by exploring the entire design space This is somewhat more flexible than the PUC s rule based approach but also requires exponentially more processing as more variables and interface elements are con sidered This means that SUPPLE s performance will degrade as the complexity of the user Cipsanagm Classroom Light Bank AN Controts Vant Light Hank AN Cominus Went Loaf Center Right Projector Len Projector w Light Light nf Light Power Off Dp pE Power On owo Leve Level Level itii jan l 1 npu Center Coupuia T i Computer 1 A Low Low Light Line TA aE on ae it gt gt Computer 2 Video gt Med CHT Mec 7 z al nf Screen Light Lewel Video _ High On ax F s gt High Screen Lowered Figure 2 4 Examples interfaces for a classroom controller generated by Supple for different devices a a standard desktop computer with a mouse and b a touchscreen The classroom has three sets of lights with variable brightness an A C system and an LCD projector with a corresponding motorized screen Gajos 2004 Reproduced with permission Chapter 2 Related Work 35 interface increases Another difference is SUPPLE s interface description which contains some of the same information as the PUC specification language but does not currently h
155. akes it easier for appliance specification writers to use the templates because there is no need to modify the appliance s internal data representation in order to interface with the controller infrastructure For ex ample some appliances may not be able to export their playback state and thus would want to use the option of specify each playback function with a command Another example is for the time duration template Windows Media Player makes the duration of a song avail able as a single integer while our Sony DV Camcorder makes the playback counter available as a string see Figure 7 2b and c for specifications Both of these representations can be ac commodated by the time duration Smart Template which allows the PUC to be implemented more cleanly with these appliances because no translation is needed to suit the requirements of the PUC Note that the regular specification language is used to specify how the smart template is represented No new techniques need to be learned by specification authors to use and customize smart templates Chapter 7 Handling Domain Specific and Conventional Knowledge 103 Each interface generator implements special generation rules for each template This allows each template to be rendered appropriately for its controller device using bits of hand designed interfaces specifically created the template and the controller device Sometimes these hand designed interfaces include platform specific controls th
156. allery of PUC Interfaces This appendix will contain a number of screenshots of interface generated by the PUC and any interfaces used by the appliance simulators that have been created 309 PUC v4 0 1133 UH 4x 8 15 amp Copies 1 4 e Color Resize Options Width PUC v4 0 1133 UH 4x 8 15 amp To use the scan features connect your HP all in one to a computer Power To use the photo features insert a memory card or Badk connect a PicBridge camera to your HP all in one Language Power Save Mode After 12 hours X Delay F Reset Everything PUC v4 0 1133 UH 4x 8 15 amp 1 Mailroom 81266 2 Brandon 412268744 3 Jeff 4122687239 4 Laura 4122683472 PUC v4 0 1133 UH 4x 8 15 amp PUC v4 0 1133 UH 4x 8 15 amp Quality Standard x Darkness M_M Header Info Redial Busy Numbers Redial if No Answer Fax Receiving Auto Answer Rings to Answer after 5 rings Ad Answer Ring Pattern Paper Letter X Size Ft Fax To Page PUC v4 0 1133 UH 4x 8 15 amp Set IP Automatic v ppo Address Subnet 255 255 0 0 Mask Gateway Host HP2610 Print Error Every error
157. als state Base PoweredItems Controls PlayControls Mode gt lt constant values 1 gt lt Stop gt lt equals gt lt equals state Base PoweredItems Controls PlayControls Mode gt lt constant value 6 gt lt Record gt lt equals gt lt or gt lt not gt lt active if gt lt output ports gt lt port group name Output gt lt port name VHF UHF Antenna channel 3 gt lt output ports gt lt objects gt lt group name Base PowerediItems Controls gt lt objects gt lt source gt Figure 5 7 The description of the video tape source content flow from the example VCR specification Note that dependencies from the content groups that contain the source flow are ANDed with the source s own de pendencies Chapter 5 Specification Language 78 5 3 Evaluation of Specification Language Before discussing interface generation there are several important questions to ask about the specification language e Is the language complete enough to specify the functionality of every appliance e Is the language easy to learn and use 5 3 1 Completeness Members of the PUC research group including myself several undergrads a masters student and two staff members have used the specification language to author specifications for 33 different appliances see Table 5 1 We have tried to cover a large range of appliance types and to write specifications for several highly complex appliances includ
158. ame labels as a previously learned copier interface see Figure 1 5 or that the clock is easier to set on a new VCR interface because the clock controls are lo cated at the same place in the interface hierarchy In order to facilitate this knowledge transfer between interfaces Polson suggests that tasks must have similar steps and there must be sufficient external cues in both applications Polson 1988 To facilitate this the PUC has the following requirements for its consistent user interfaces R 1 Interfaces should manipulate similar functions in the same way R 2 Interfaces should locate similar functions in the same place These two requirements help to ensure that user tasks will have similar steps which can fa cilitate knowledge transfer They also illustrate a fundamental separation in the PUC between functional consistency and structural consistency Two interfaces are functionally con sistent if the same set of controls is used for similar functions Two interfaces are structurally consistent if similar functions can be found in similar organizational groups These two types of consistency are independent and are addressed separately by the PUC Chapter 6 Consistency 91 In order for knowledge transfer to occur there also must be sufficient external cues to indi cate that the applications are the same In many cases the PUC gets an important external cue for free because users are often aware of the type of appliance they are usin
159. amil Nozsle GhecitPattern 00 Gean Print Head a4 11 23 03 gt BB sin 8 8 pena 144 344 jaa Auto Head Align Tray Tray Ringing E A Tray Front El auto eB amp w Font Font Front _ Deea D Ring Pitch Standard X Ao Color Calbratk off hook L Oean Sheet Feeder Roller lt Cean Bottom Plate Scan Photo Setup 4 gt Scan Photo Setup gt Scan Photo Setup 4 gt PUC Servers Devices Generation PUC Servers Devices Generation Ej PUC Servers Devices Generation PUC Servers Devices Generation E leerer exis Q Power Status 1 Auto Redial v LD Auto Redial Pause 13 Fax Setup Line Type Line Touch Tone Type V Fx Send Error H Fx Re v4 0 1133 UH 7 4x 8 18 amp Canon PIXMA MP780 All In One Power PUC Servers Devices Generation Figure F 2 The full interface for the Canon All In One printer generated without consistency as seen by us ers in the usability study Appendix F Gallery of PUC Interfaces 311 EG v4 0 1133 UH qx 8 24 PUC v4 0 1133 UH 4x 8 24 amp UC v4 0 1133 UH 4x 8 24 amp cos FFI i Arranging Copies v ees None Actual Size X image Poster 2 pages X Size Width PUC v4 0 1133 UH 4x 8 24 amp 1 Mailroom 81266 Mode Yes 2 Bra
160. an one context An example of the first situation can be found in the Mitsubishi DVCR and Samsung DVD VCR specs where the front display setting is controlled by one state variable on the Samsung but by two variables on the Mitsubishi that separately control the brightness Chapter 8 Interface Generation 126 and content of the front display An example of the second situation can be found on the HP and Canon all in one printers The HP specification has global buttons for starting the copy fax or scan functions depending on the current mode The Canon has separate buttons for starting copies starting faxes and starting scans Each set of buttons on the Canon is mutu ally exclusive of the others however because the printer can only be in one of these modes at any given time There are five different rules for moving items One of these rules handles the simplest case of moving items that are adjacent in both specifications and do not cross any mutually exclu Table 8 2 Consistency moving rules implemented in the PUC Name Split Mutex Description Move Single Item _ Moves adjacent items to a location in the new specifica tion that is similar to the location of similar items in the _ previous specification Move Single Moded Item Performs the same move as the Move Single Item rule 2 but handles situations when this move will cause items to move across a mutually exclusive group boundary For example the Cano
161. an control For televisions this is usually limited to channel up and down volume up and down a num ber pad for manually entering channels and a power button For example my mother has a universal remote for her television and VCR but must use the original remote for the TV in order to access the television s configuration menus Some universal remotes avoid this prob lem with a teaching feature which allows the user to assign buttons on the universal remote to a particular code that is recorded from the original remote control The Philips Pronto see Figure 2 1 was one of the first LCD based universal remote control products In addition to being able to program new infrared codes for new appliances users can also design the panels of the controls that they use Using the Pronto it is easy for ex ample to create a specialized screen for watching movies that has the DVD player and stereo Chapter 2 Related Work 18 volume controls and another for watching television that only has controls for the cable box channels Users can even associate multiple codes with a single button allowing them for example to create a macro for playing a DVD that turns on the DVD player and television switches to the appropriate channel and plays the DVD The problem with the Pronto as with the other universal remotes is that all of the programming must be done manually which can be a tedious and time consuming task especially for a large numb
162. ance specification The structure of the inter face is taken from the specification s group tree and modified by an analysis of dependency information Abstract interaction objects AIOs are assigned for all state variables and com mands and placed within the structure of the abstract interface The result is a set of abstract controls within a hierarchical tree that can be transformed into concrete interfaces for the various platforms The two most importants parts of this process the analysis of dependency information for structure and the assignment of AIOs are discussed below 8 3 1 Mutual Exclusive Dependency Information The interface generator uses the specification s group tree as the basis for the structure of the abstract interface but this tree is just one specification author s interpretation of the structure that the appliance should have Some structure may be missing and the relative importance of different pieces of the existing structure may not be clear For example an appliance speci fication for a shelf stereo might have a group node whose branches represent the main modes of that stereo CD Tape Radio etc The generator needs to discover that this group is of particular importantance for this appliance and create an interface structure that will always make these important branches clear For most appliance modes it is best for each mode to have its control placed on a separate overlapping panel to save space and i
163. and a copier fax machine telephone answering machine vending machine and CD player at school That does not count the mobile phone and wristwatch that I usually carry around with me nor does it count the three normal computers that I use daily or many of the computerized compo nents that my car would have if it had been built in the last ten years All of these devices have different interfaces even for functions that are common across most of them such as setting the internal clock Even devices that are functionally similar like my car stereo home stereo and the media player on my computer at school have vastly different interfaces Prob lems like these are encountered by most users of today s computerized devices Users must spend time learning how to use every device in their environment even when they are simi lar to other devices that they already know how to use Part of the usability problem with today s computerized appliances is created by the many trade offs that manufacturers must balance They would like to produce highly usable appli ances but those appliances must also reach market in a timely fashion be cheap to manufacture and have the new features that users want The declining cost of microproces sors allows manufacturers to cheaply and quickly add more computation and new features to their appliances but unfortunately the cost of building high quality user interfaces for these new features is still very ex
164. and the items in the set are placed on it The list item that opens the editing pane is labeled with the common portion of the label associated with that set see Figure 8 13b We originally considered hav ing an additional rule that moved all panel only items onto a single editing pane if no sets were found and labeling the item that opened the pane with the label of the parent group concatenated with the term Controls We decided against this rule however because we believed the user would have a hard time guessing what functions were on the panel given this label and because the navigation cost in terms of the number of key presses for giving each panel only item its own editing pane is not much different than having a panel of unre lated controls PUC v3 0 Audiophase lt Yj PUC v3 0 DIC cx Yi 1 General 1 General 2 Total Distance Driven 3 Total hours engine run 4 Annual Mileage 144 1 4 Personal Trip 5 Stopwatch 0 0 5 Reset Personal Trip 6 Start 6 Business Trip 7 Stop 7 Reset Business Trip 8 Personal Trip PUC v3 0 Demotion Te lt T PUC v3 0 Audiophase CX Yy 1 Power On 1 Power On 2 Volume Up 2 Volume Up 3 Volume Down 3 Volume Down 4 Mute Off 4 Mute Off 5 X Bass On 5 X Bass On 6 CD Player Controls 6 CD Player 7 CD Player Track 7 Tape Player 8 CD Player Disc i 9 More PUC v3 0 Demotion Te T PUC v3 0 Audiophase abc Yy PUC v3 0 Audiophase abc Yy 1 Tape Player Controls 2
165. anging one setting of the device 7 Given a page with a picture determine how to produce one page with several in stances of the same picture repeated 8 The device remembers the current date and time Determine where in the interface these values can be changed but changing them is not required The tasks were carefully written to not use language that favored any of the user interfaces being tested In some cases this was easy because all interfaces used the same terminology In other cases words were used that did not appear in any of the interfaces We also used exam ple documents rather than language to demonstrate the goal of task 7 10 3 Participants Forty eight subjects twenty eight male and twenty female volunteered for the study through a centralized sign up web page managed by Carnegie Mellon University Most subjects were students at either CMU or the University of Pittsburgh and had an average age of 25 and a median age of 23 We also had 3 subjects older than 40 years Subjects were paid 15 for their time which varied from about forty minutes to an hour and a half depending on the configuration of interfaces being used Subjects were randomly assigned to conditions 10 4 Evaluation of Usability To evaluate the usability of the PUC interfaces the task completion times and failures for the Built in condition were compared with the other two conditions For this analysis the data from the first block in eac
166. ared New interface generation layers will be needed to address differences in the platforms but again the cost of implementing a template may be reduced substantially The layering approach does not remove all of the complexity from the implementation of a Smart Template however Some differences in the specification of a template must be re flected in the generation for that template which requires the complexity of the template to be addressed in both layers 7 3 Smart Template Library The PUC research team and I have implemented 15 different Smart Templates as shown in Table 7 1 These templates collectively illustrate all of the features discussed in previous sec tions A full description of all Smart Templates is available on the web at http www pebbles hcii cmu edu puc highlevel types html As we have built Smart Templates we have found two uses for templates that improve the PUC interfaces but differ somewhat from our initial intentions e The PUC supports state variables with a binary type These variables could contain images sounds or other data that cannot be easily communicated through the PUC s text based communication protocol Smart Templates were chosen to handle this binary information for several reasons First it did not seem appropriate for all interface generators to handle all kinds of binary data Smart Templates were attrac tive then because they are optional by design although in this case the lack
167. ariable This value was chosen much later in the specification process after I knew the other items that would be in the same group as this state variables In most cases it will be difficult to assign a priority value when you are initially defining an object because the contents of the group that contains that object will not be known The XML for this state variable has all of the same components as the card formatted ac cording to the schema see Figure Appendix D Specification Authoring Study Instructions 278 lt state name ReminderFregquency priority 5 gt Cy pes lt enumerated gt lt item count gt 3 lt item count gt lt enumerated gt lt value labels gt lt map index 1 gt lt labels gt lt label gt Weekly lt label gt lt labels gt lt map gt lt map index 2 gt lt labels gt lt label gt Daily lt label gt lt labels gt lt map gt lt map index 3 gt lt labels gt lt label gt Hourly lt label gt lt labels gt lt map gt lt value labels gt lt type gt lt labels gt lt label gt Reminder Frequency lt label gt lt label gt Reminder Freq lt label gt lt label gt Reminders lt label gt lt labels gt lt state gt Figure D 2 The XML code for the reminder frequency state variable The display preference setting is also a state variable and has a definition that is very similar to that shown for the reminder frequency variable As an exercise I suggest writing y
168. ary minOccurs 0 maxOccurs 1 gt lt xs element name active if type ActivelfType minOccurs 0 maxOccurs 1 gt lt xs choice minOccurs 1 maxOccurs unbounded gt lt xs choice minOccurs 0 maxOccurs 1 gt lt xs sequence gt lt xs element name min type StaticOrReference minOccurs 0 maxOccurs 1 gt lt xs element name max type StaticOrReference minOccurs 0 maxOccurs 1 gt lt xs sequence gt lt xs element name item count type xs integer gt lt xs choice gt lt xs element name selections type SelectionTypeType minOccurs 0 maxOccurs 1 lt xs element name group type GroupType gt lt xs element name list group type ListGroupType gt lt xs element name union group type UnionGroupType gt lt xs elemen 1ame State type StateType gt lt xs element name command type CommandType gt lt xs element name explanation type CommandType gt Gr ARAA yete eh eer yeh m lt xs element name apply type type ObjectApplyType gt lt xs choice gt lt xs sequence gt lt xs attributeGroup ref TypesObjectAttribs gt lt xs complexType gt lt xs complexType name TypesCommandType gt lt xS sequence gt lt xs element name labels type LabelDictionary gt lt xs element name active if type ActivelfType minOccurs 0 maxOccurs 1 gt lt xs sequence gt lt xs attributeGroup ref TypesObjectAttribs gt lt xs complexType gt
169. as anticipated that some subjects would not be able to complete some of the more diffi cult tasks If a subject gave up while working with the actual phone or stereo they were given the user manual and asked to complete the task Subjects working on the prototype inter faces were allowed to press the Help button available in some form on every screen On the paper interfaces for the first study a verbal hint was given whereas the Visual Basic inter faces for the second study presented a scrollable screen of text indexed by topic The performance of each subject on both lists of tasks was recorded using three metrics time to complete the tasks number of missteps made while completing the tasks and the number of times external help was needed to complete a task The time to complete the tasks was measured from the press of the first button to the press of the button that completed the last task External help is any use of the manual for an actual appliance or the help screen on the PDA or any verbal hint from the experimenter For the purposes of this study a misstep is defined as the pressing of a button that does not advance progress on the current task Repeated pressings of the same button were not counted as additional missteps Sometimes a subject would try something that did not work try something else and then repeat the first thing again If the interface had given no feed back either visibly or audibly the repeated inco
170. as designed specifically to produce interfaces for appliances It could be used to generate other kinds of interfaces but is limited in the scope of the applications that could be supported Any application that only needs dialog box style controls in its interfaces could be generated The PUC may also be able to produce interfaces for applications that need to display large amounts of data using its support for complex data structures although this capability has not been rigorously tested by the appliances tested so far Applications that manipulate long text strings such as e mail editors could theoretically be handled by the as PUC framework though the current interface generators have limited support for rendering large text boxes in an interface Some application interfaces cannot be generated by the PUC In particular the PUC cannot support generation for applications that must use direct manipulation such as painting or circuit design applications Support for direct manipulation was a limitation of previous model based systems as well and those that were able to support direct manipulation re quired substantial extra modeling effort to provide enough information to allow the interface generator to produce a reasonable interface These systems required designer involvement in the process as well to guide the generation or fix any mistakes The limitations of the PUC system illustrate the fundamental trade off between modeling e
171. as enabled through the standard IR protocol used by the stereo s own remote control We have also considered building general adaptors for existing appliance control protocols such as UPnP and HAVi UPnP and HAVi both have their own appliance description lan guages and the idea was to build a gateway for each that would translate that protocol s appliance description into a PUC specification With the translated specification and map ping from one description language to another the gateway could translate between the protocols on the fly Unfortunately we were not able to build these general adaptors for two reasons one practical and one more fundamental The practical reason we chose not to build a general purpose adaptor is that few appliances are designed to use these protocols and those that are export only a few functions through the protocol This makes the value in cre ating a general adaptor fairly low Fundamentally the problems with building a general adaptor were even worse The description languages for UPnP and especially for HAVi were not nearly detailed enough to produce a reasonable PUC specification For HAVi the prob Chapter 4 System Implementation 57 lem is that descriptions are limited to a low level description of the desired user interface in cluding pixel locations and sizes for every control Without links between the labels and their associated controls it is very difficult to create a reasonable PUC spec
172. as several limitations however which Huddle overcomes First a ge neric aggregator in Crafter is only able to generate an interface for the common properties and functions shared by its service interfaces and none of the unique functions that may be implemented by a specific service Second ICrafter s generic aggregators are not able to in clude any design conventions that might be specific to the services For example a play button would be appropriate if a DataProducer was a DVD player but not if the producer was a camera For Crafter to produce interfaces with unique functions and appropriate de sign conventions a special purpose interface aggregator would need to be built for the specific appliances involved In contrast Huddle s interface aggregation is faithful to the spe cific appliance interfaces that are being aggregated and includes all functionality of the connected appliances 2 1 3 4 Xweb The Xweb Olsen Jr 2000 project is working to separate the functionality of the appliance from the device upon which it is displayed Xweb defines an XML language from which user interfaces can be created Unlike the PUC specification language Xweb s language uses only a tree for specifying structural information about an appliance Their approach seems to work well for interfaces that have no modes but it is unclear how well it would work for re mote control interfaces where modes are commonplace Xweb also supports the construct
173. at are connected to that server If the user selects an appliance from this hierarchical menu then the controller device connects directly to the ap pliance downloads the appliance specification and generates the user interface Appliances connected to the controller device as displayed in the Devices menu shown in Figure 4 3c The Generation menu shown in Figure 4 3d gives access to various options that control the generation of user interfaces including resetting the consistency system and opening the fea tures of the multi appliance system To support the generation of interfaces it was also necessary to implement a number of cus tom controls to cover functionality that was not present in the NET Compact Framework For example the Compact Framework does not have a panel that automatically adds scroll bars when controls are placed off the panel s viewable area It was also necessary to imple ment a date time picker a slider the combo box organizing panel used for navigation between the three panels on the left half of Figure 8 1b and a few different list widgets see Figure 1 2b for a shot of a list widget displaying multiple steps in a route 4 3 2 Smartphone Implementation The Smartphone interface generation also has a menu based interface however the structure of the menus are slightly different because of differences in the Smartphone s menu design Chapter 4 System Implementation 60 P
174. at are consistent with other user interfaces on the same device In the case of our time duration Smart Template implementation each platform has a different standard control for manipulating time that our interface generators use Unfortunately none of our platforms have built in controls for media playback so our media controls Smart Template use renderings that I hand designed to be appropriate for each of the different platforms The Smartphone media controls implementation does mimic the interface used by the Smartphone version of Windows Media Player however and thus is consistent with another application on that de vice see Figure 7 3b Smart Templates are also able to intelligently choose a rendering based upon the contents of the template For example each implementation of the time duration template only ren ders the time units that are meaningful and each implementation of our media controls Smart Template renders buttons for only the functions that are available The media controls implementations on the PocketPC and desktop extend this by intelligently laying Mini Disc Player a gO Mode Play o Play Control P P il 1 aa P ja gt Machine Pay Contro DE im I e me Media Player gt an Tape player 4 Ot s j a a b G Figure 7 3 Renders of the media controls Smart Template on different platforms and for different appli ances a Media controls rendered with Smart
175. at provide some service to the user and controller devices that automatically generate and present remote control interfaces for the appliances Appliances and controller devices communicate using a peer to peer approach which allows any con troller device to control multiple appliances simultaneously and any appliance to be controlled by multiple controller devices simultaneously Two way communication is re quired because controllers must be able to send commands and receive both specifications and appliance state information from the appliances Appliances and controller devices communicate via a custom communication protocol that was designed to be operable across many different network layers such as Wi Fi Bluetooth or Zigbee although our current implementations only support TCP IP Our protocol is 55 based on XML and its messages are designed around the functional elements that are sup ported by the PUC specification language Unlike many other appliance control systems such as UPnP the PUC does not support automatic discovery of appliances Currently the user of a controller device must specify a server to connect to either via an IP address or a name and then the protocol can relay the name of any appliances that are controllable through that server I considered including discovery features in the protocol but decided this feature was outside the scope of my research In the future some discovery mechanism could be integra
176. ate interface allows the user to set the current time for example and have this value migrated automatically to every appliance in the system In order to demonstrate the feasibility of the PUC system and to ensure that the PUC sup ports all of the complexities of today s computerized appliances I thought that it was important for the PUC to control real appliances To accomplish this the PUC system has a communication protocol to allow handheld devices to interact with appliances and a set of appliances adaptors that translate between the PUC protocol and the proprietary protocols of existing appliances The PUC is currently able to control 9 existing appliances see Table 4 1Error Reference source not found for a complete list and it would be easy to increase this number as more appliances are built with support for external control 1 2 Outside the Scope My work on the PUC system could have been taken in many directions This section de scribes issues related to the PUC system that I do not explore in this dissertation Help systems When users encounter problems with an automatically generated inter face they should be able to access help information that is generated based upon the properties of the interface Such automated help systems have been created in the past such as the Cartoonist system for the UIDE environment Sukaviriya 1990 Automated trouble shooting for complex systems The functionality of a complex sys
177. atically appear when they later choose to modify their to do list items The cards and XML for the category state variable are shown in Figure and Figure respectively State Variable Type Name O Boolean Object e O Enumerated Of State Variable O Command O Explanation Number of Items Fill in values and labels on label card O Read only Name Cates haley O Fixed Point Decimal Places Min Max Increment Priority tt 0 10 O Floating Point Smart Template g ss nn nn f Vin Ma x Labels Cate S 6 ry O Integer Min Max increment if List Selection nS Se List Name S ctl o G weg or a5 f item dependency attach card Wiite additional labels on back List O String Figure D 5 The object and state variable type cards for the Category state variable that is contained in the to do list group Note the use of the list selection type on the state variable type card lt state name Category priority 4 gt LYpe gt lt list selection list Setup Categories List gt lt type gt lt labels gt lt label gt Category lt label gt lt labels gt lt state gt Figure D 6 The XML code for the Category state variable that is contained in the to do list group Note the use of the list selection type in this variable Providing a User Interface for Modifying List Data Appendix D Specification Authoring Study Instructions 281 The description of the to do list application includes functio
178. ation e type name required if within types section optional otherwise defines a type with the given name e priority The priority this state should be assigned relative to other objects in the same group May Contain lt active if gt lt labels gt Appendix B Specification Language Reference 22 lt type gt lt type name TypeName gt lt type gt Describes the value space of a state variable ex Boolean Integer etc and the la bels that its values take Placement Inside the lt state gt and lt types gt tag Parameters type name The name of the type object Required if this element is con tained in a types block May Contain lt binary gt lt boolean gt lt enumerated gt lt fixedpt gt lt floatingpt gt lt integer gt lt list selection gt lt string gt lt value labels gt lt apply type gt lt apply type name TypeName gt Allows the re use of an existing type block within a specification Using this element is exactly the same as cutting and pasting the type block No data will be shared with other states or groups that apply the same type The type must have been declared earlier in the specification document Placement Inside a lt state gt lt group gt lt list group gt or lt union group gt element Parameters type name required The name of the appliance object to reference e name optional if this element is not inside a state
179. ation value component video gt lt xs enumeration value av gt lt xs enumeration value multi channel audio gt lt xs enumeration value multi channel video gt lt xs enumeration value multi channel av gt lt xs restriction gt lt xs simpleType gt lt Elements gt lt xs complexType name LabelDictionary gt lt xS sequence gt Appendix B Specification Language Reference 206 lt xs choice minOccurs 1 maxOccurs unbounded gt lt xs element name label type xs string minOccurs 1 gt lt xs element name ref value type stateAttribNoContent gt lt xs element name phonetic type xs string gt lt xs element name text to speech type TextToSpeechType gt lt xs choice gt lt xs sequence gt lt xs complexType gt lt xs complexType name stateAttribNoContent gt lt xs attribute name state type xs string use required gt lt xs complexType gt lt xs complexType name TextToSpeechType gt lt xs attribute name text type xs string use required gt lt xs attribute name recording type xs string gt lt xs complexType gt lt Aggregate Spec Types gt lt xs complexType name AggregateSpectTag gt lt xs attribute name name type xs string use required gt lt xs complexType gt lt xs complexType name ApplianceReference gt lt xs attribute name appliance type xs string use required gt lt xsr
180. ator itself rather than for controlling appliances This decision was influended by the preliminary user studies which showed that users did not typically look for appliance Chapter 8 Interface Generation 114 i PUC v4 0 1121 UH 4x 3 37 X i PUC v4 0 1121 UH x 3 37 X PUC v4 0 1121 UH 4x 3 37 X Alarm Warning Headlamps and Horn v a Odometer 144 1 Average MPG 13 5 Automatic Locking Fuel Range 10 5 Hourmeter 39 Instant MPG 19 2 Lock Doors Out of Park M Annual Mileage 144 1 GM Oil Life 65 ae Unlock Allin Park A Seat Poston Recalo 7 jerar Perimeter Lighting O Perimeter Lighting O Ov Remote Lock Lamp and Horn v Feedback Ay Puc v2 04 Play Control a Remote Unlock gt aT Fuel Information Personalization KIO Fuel Information Personalization dafe Personalization Settings le a PUC Servers Devices Generation PUC Servers Devices Generation Current Title The Old Apartment Ki mE PUC v4 0 1121 UH x 3 37 X PUC v4 0 1121 UH 4x 3 37 X Volume Personal Trip l Business Trip l Reset Personal Trip Current 62 7 Current 81 4 Distance Distance Fuel used 5 0 Fuel used 10 0 L Mute Status Stopped Version 9 0 0 3008 Average 20 5 Average 21 0 Economy Economy Average 40 Average 45 Speed in MPH Speed in MPH Trip to Annual 35 0 Trip to Annual 65 0 a b Figure 8 1 a
181. attribute name object type xs string use reguired gt lt xs complexType gt lt Types Section gt lt xs complexType name TypesGroup gt lt xs choice minOccurs 1 maxOccurs unbounded gt lt xs element name group type TypesGroupType gt lt xs element name list group type TypesListGroupType gt lt xs element name union group type TypesUnionGroupType gt lt xs element name state type TypesStateType gt lt xs element name command type TypesCommandType gt lt xs element name explanation type TypesCommandType gt Cro Gk wet eto o eb lt xs element name type type TypesPrimitiveType gt lt xs2choice gt lt xs complexType gt lt xs attributeGroup name TypesObjectAttribs gt lt xs attribute name type name type xs string use required gt lt xs attribute name is a type xs string use optional gt lt xs attribute name priority type PriorityType use optional gt lt xs attributeGroup gt lt xs complexType name TypesGroupType gt lt xs sequence gt lt xs element name active if type ActivelfType minOccurs 0 maxOccurs 1 gt lt xs element name labels type LabelDictionary minOccurs 0 maxOccurs 1 gt lt xs choice minOccurs 1 maxOccurs unbounded gt lt xs element name group type GroupType gt lt xs element name list group type ListGroupType gt lt xs element name union group type UnionGroupType gt
182. ave a written syntax Instead the description is defined by run time objects created by a program mer much like the second generation UIDE system 2 3 Ageregate User Interfaces While there has been a great deal of research on automatically combining services especially in the Web Services and Semantic Web communities there has been very little work on combining the user interfaces Work in Web Services typically takes the infrastructure as a given and focuses on how existing standards such as WSDL UDDI and DAML S Sycara 2003 can be used to automatically discover and compose services Unlike in my work Web Service composition research seems to rarely consider creating user interfaces for combined services Srivastava 2003 One exception is an article Staab 2003 which briefly mentions control issues There has been work on user interfaces for specifying how services should be connected e g Kim 2004 that could be leveraged in the future to build an editor that specifies how appliances are connected Several systems have explored the infrastructure issues that are involved in connecting and configuring systems of multiple appliances One such system is Speakeasy Newman 2002 which uses mobile code to allow arbitrary devices and services to interact and also to distrib ute user interfaces to the handheld devices from which users interact While Speakeasy might be able to automatically provide a wiring diagram for the PUC s inter
183. ay Contain Second use only lt active if gt lt channel gt lt objects gt Appendix B Specification Language Reference 245 lt content flow gt lt content flow gt lt content flow gt Defines the section in which the internal content flows of an appliance are described Placement Inside the lt spec gt element May Contain lt content group gt lt source gt lt pass through gt lt recorder gt lt renderer gt lt content group gt lt content group gt lt content group gt Defines a group of content flows for the purpose of defining some dependencies that apply to all of the content flows in the group Placement Inside the lt content flow gt or lt content group gt elements May Contain Aclive 1Li coon lenl group gt source gt xpass through gt recorder lt renderer gt lt source gt lt source name Tape content type av gt lt source gt Describes a source of content within an appliance Placement Inside the lt content flow gt or lt content group gt elements Parameters e name required A name for the source e content type required The type of content produced by this source Appendix B Specification Language Reference 246 May Contain lt ACTCLVe LI lt cCuLcputc porits gt lt ob ects gt lt pass through gt lt pass through content type av gt lt pass through gt Describes a pass through for content on
184. be pressed and held to activate another function There is no text on the tele phone to indicate this A similar problem is also encountered on the stereo Setting the timer requires the user to press a combination of buttons each button press within four seconds of the last The stereo does not display an indicator to warn of this restriction and often users were confused when a prompt would disappear when they had not acted quickly enough The phone also suffered from an underlying technical separation between the telephone and the answering machine functions None of the buttons on the phone can be used with the answering machine Even the numeric codes must be set using arrow buttons rather than the phone keypad All but one subject tried to use the keypad buttons to set the code The ex ception had used a similar AT amp T phone in the past All of these problems were avoided in the PDA interfaces because there was room for labels that were more descriptive and certain multi step functions could be put on a separate screen or in a wizard Using different screens to separate infrequently used or complex functions can also be problematic however Other buttons or menu items must be provided so that the user can navigate between screens and the labels for these navigation elements must describe the general contents of the screen that they lead to This was particularly a problem for the handheld stereo interface which has more than ten scre
185. been created The interface generator has been used to successfully generate interfaces for all of the 33 specifications that the PUC team and I have authored As discussed in Chapter 5 this collec tion of specifications covers a wide range of appliance types and includes many complex specifications Chapter 10 shows that the generated interfaces can be more usable than the human designed interfaces for two complex all in one printers which I believe are represen tative of the many complex appliances with which the PUC system could be used That chapter also shows that the consistency algorithms shown here have usability benefits The interface generator does have some trade offs The rule based approach that I chose has several benefits including allowing for timely generation of interfaces on low resource hand held devices and the ability to always produce the same output given the same input but it also has several drawbacks common to all rule based systems In particular new rules must be designed carefully to avoid unintended interactions with existing rules Interface genera Chapter 8 Interface Generation 145 tion also involves balancing many trade offs and rules are poorly suited to making decisions that optimally balance these trade offs I have used two different rule based approaches to iteratively improve interfaces during generation the PocketPC and desktop interface genera tors start simple and add complexity while the Sma
186. ber of selections available in a list and whether or not a user may change the current selection s Placement Inside the lt list group gt element Parameters e number required The number of allowed selections Possible values are one and multiple e access Defines how users interact with the ChildUsed variable Possible val ues are read only and read write lt sortable gt lt sortable gt Specifies that this list group has not natural order and can be arbitrarily sorted by the user interface Placement Inside the lt list group gt element lt state gt lt state name StateName access read write priority 10 gt lt state gt Defines a state variable appliance object Placement Inside lt group gt lt list group gt lt types gt or lt union group gt Parameters e name required The name of the state variable Appendix B Specification Language Reference 225 e type name required if within types section optional otherwise defines a type with the given name e priority The priority this state should be assigned relative to other objects in the same group e is a The Smart Template that represents this group and its children e access Defines how users interact with the ChildUsed variable Possible val ues are read only and read write May Contain lt active if gt lt apply type gt lt completions available gt lt default value gt lt labels gt lt requir
187. boxes from Visual Basic and Visual C 4 0 but in principle the tool could be run on any user interface that is translated into Sherlock s textual interface de Chapter 6 Consistency 86 scription language The output of Sherlock is a collection of tables that display each of the metrics Designers look for consistency problems by searching the tables for outlier values GLEAN Kieras 1995 makes predictions about human performance on a user interface based on a GOMSL GOMS Goals Operators Methods and Selection Rules Language model The use of GOMSL is important because of its unique ability to accurately predict transfer times between tasks John 1996 which is useful for finding consistency problems between similar tasks Unfortunately a GOMSL model must be created for the user interface before any analysis can be conducted There is no way to automatically generate such a model though recent work has shown that it is possible for other types of GOMS models John 2004 6 1 2 Applying Consistency A number of methods have been developed to help ensure interface consistency Platform interface guidelines and toolkits like those developed by Apple for the Macintosh help de signers make their applications consistent with others on the same platform Usually these guidelines work best for common functions such as by defining a standard menu structure that makes it easy to open and save files and access clipboard functions More s
188. cal such as the counter and status variables that tracked whether a tape was in the VCR Many objects were similar but not completely identical For example the only differences between some objects were the labels such as for the TV VCR or VCR TV Boolean states that are present on each VCR Other objects contained some of the same val ues but also supported other features that were not present across all of the VCRs For example all of the VCRs have a state variable that specifies whether the coax input is coming from the antenna or cable The Panasonic VCR supports only these two options and the Samsung adds an extra option called Auto in which it will automatically select the appro priate value The Mitsubishi VCR does not have the auto value but it supports two additional input types not found on the other VCRs cable box and digital cable Other functions shared across all of the VCRs were specified quite differently For example each of our VCRs supports a timed recording feature to specify TV programs that the user wishes to record in the future The way to specify the time that the program would be re corded differed across devices The Mitsubishi and Panasonic VCRs both have variables for the start time and stop time of the recording while the Samsung has a matching variable for start time but a different variable that specifies the duration of the recording In this case the underlying data is quite
189. can be defined within the lt value labels gt tag _ pendency information for this state variable with a fixed decimal point location Minimum maximum and incre _ ment values can be defined using the lt min gt lt max gt and lt incr gt tags _ Minimum and maximum values can be defined using the lt min gt and lt max gt tags _ dependency information for this state variable _ This restriction means that the variable must have a value equals to its _ minimum n increment where n is an integer ance Zi Element Name lt inp t ports gt lt item count gt Used to denote how many values there are in an lt enumerated gt type or lt integer gt lt label gt Defines a label to be included within a label dictionary lt labels gt Defines a label dictionary for an appliance object or group The lt map gt lt lessthan gt Used in conjunction with the lt active if gt tag to define lessthan de lt list group gt lt list selection gt lt map gt lt min gt lt max gt lt modifies state gt Allows a command to specify that its invocation will modify a state i te or lt object gt Defines an object that is related to a content flow Appendix B Specification Language Reference Description _ Defines the input ports that are used by a pass through recorder or _ renderer content flow a lt list group gt Labels for the items can be defined within the
190. can be shown on the screen at once but we have rules that will attempt to address this problem later in the generation process If a mutually exclusive set of functions is encountered then usually no additional action is required because of the changes already made to the abstract interface structure The result of the list building process is each set of functions being available from a separate list that is accessed from the same parent list see Figure 8 12a In the case where the user cannot choose which set of functions is enabled through the interface such as when the appliance has a read only mode the interface genera tor may create overlapping lists that are switched based on the state of the appliance see Figure 8 12b c Optimizing the list structure for navigation ensures that users spend less time finding features in the interface and more time using those features The challenge of optimizing is balancing the structure that has already been built with the constraints of Smartphone user interfaces There are two constraints of the Smartphone interface that need to be addressed e Navigation is particularly important in Smartphone interfaces because only nine items can be shown on each list screen and users constantly navigate up and down the list hierarchy This means that the Smartphone interface generator should try to make the depth of the list hierarchy as shallow as possible and place the maximum number of functions onto each
191. cates sata accurate OEEO 32 Dt IV Me serail date realness tetera E EEA A 33 2224 LBM PIMA and MDAT sas cssicesasiethavivteedssestessavivietisiestetaxteacs 33 PL a fag UTM band ee a PE RR ET OE ERE eR ETRE 33 D220 TERESA ass ic cies sais oda wat aalan nossa deat A ose ane 34 Vee USAN Deen err ene ere eee ee een ee O at eee ne eee 34 D2 DiGy AAM Lana A U Dorrie EEEE ATE EONO 34 AE E a a P E A AA S IES A E AAA renee er 35 23 Agarose User MENICE eT E a E E ene Rass 36 Preliminary User Studies 39 DL tdatid Desieaned User INCE ACES na a ERE a a 39 I2 Ol US S A T A AET 42 BZA TLIO UNE sAirr AEE E A N TEAT E O N 42 P L O ee ee nO ee eee ee OE NOO ON 42 I e E T S 43 FL L IIE anita tet Ss Sea pacha tek aos cet tah fe tat nate aie esses E E 44 DoD TRIS etait sl tas la sta sh E he te ee ab atin ats aint abner set eae ts 44 DDD LIICUSION scat vit iiae E E REAT EE E TEE ETER E 45 JA ST a E E S 45 IAL LAUDUN EEEE E E REEE T EE TE E 45 o PE tease ah E E E E A 46 SP D O E ta ee aes E N E 47 I ABa Or erae n AE 48 IO REeguremnen Oesa a N nde 50 LOL Lw Way COMMUNICATION sissies aE A 50 3 6 2 Simultaneous Multiple Controllers sossesseessessenseesserssrerssereseresserssee 50 3 6 3 No Specifre Layout InJormnati On iseit asi Taea aE EEEE TEE EREE 51 FOA ITITI OOUE EE EEE E EE E N E E 51 3 6 5 Actions as State Variables and Command iceccccccssccccsiccessiscssnseessneeeens 52 306 DnA CIC Tf OVINE ON E EE N a ad em EEE 52 RO OUE LADES ETE ase
192. cations in a room that can allow the PUC to reason about the interactions of appliances within the environment The content flow concept should prove extensible to other appliance domains such as video conferencing systems and even manufacturing processes The aggregation approach seems to be limited by the correspondence between the content flows and the tasks that the user wants to perform Where there is correspondence such as in the scenarios considered here this ap proach works well One scenario where the PUC may not be effective is in the kitchen where tasks often center on recipes It seems that many recipes use the same content flow through appliances which suggests that the content flow may not be descriptive enough to generate useful task based interfaces for kitchen appliances Chapter 9 Aggregating User Interfaces 160 CHAPTER 10 Usability Evaluation Two studies of the generated interfaces were conducted to examine the usability of interfaces generated by the PUC The first study compared the generated interfaces to existing human designed interfaces for the same functionality with the hypothesis that interface quality is no longer a limiting factor for automatically generated interfaces The results were that users of the automatically generated interfaces were twice as fast and four times more successful than users of the existing interfaces for a set of eight independent tasks with varying difficulty The sec
193. ccomplish a single task An obvious solution here is to integrate the appliances into a single monolithic appliance for which an interaction designer can carefully construct a good user interface In fact this solution can be seen for some consumer electronics such as for shelf stereos which integrate an amplifier with a CD player radio tuner and other audio devices The problem with this approach is that it does not allow for expandability and inno vation If all audio appliances had been integrated ten years ago then today there would be no place for devices like the iPod The PUC allows users to easily interact with systems of appliances which enables appliance manufacturers to pursue the design of new appliances that may be added to these systems There is also no guarantee that the interface produced by the manufacturer for the integrated appliance will be usable even if the functionality of mul tiple appliances is integrated into a single monolithic appliance In fact the combined appliance interface would probably be harder to design because of the many complicated functions that the integrated appliance would support beyond that of a single appliance The examples from current systems suggest that manufacturers might not succeed at this difficult design challenge Brouwer Janse 1992 Gomes 2003 For a system like the PUC to succeed as a product in the real world it is important to ask the following questions e Would consumers find
194. ce but the infrastructure is much more general The system is a set of APIs for discovering services downloading an object that represents the service making calls on the object using a remote procedure call protocol and finally releasing the object when it is no longer needed Like HAVi JINI also relies on the Java platform to send mobile code from the service to the computer that wishes to use the service This mechanism could be used for example to display a user interface that allows a human to control a service It would be possible to implement a system like the PUC on top of the JINI protocol but JINI by itself does not provide the user interface description fea tures that the PUC does 2 1 2 6 OSGi OSGi OSGi 2006 is a dynamic module system for the Java programming language which provides several standard primitives that allow developers to produce applications from a set of small reusable components The OSGi Service Platform allows modules to be composed across devices and networks and for the composition to be changed dynamically to support new tasks This platform also allows for modules to be automatically discovered and inte grated into a currently running system These features allow OSGi to be used as the infrastructure in a variety of smart home projects as listed on the web http www osgi org markets smarthome asp 2 1 3 Research Systems 2 1 3 1 Universal Interactor Hodes et al Hodes 1997 propose a similar
195. ces This included questions such as why are these elements grouped together or why are these widgets never shown at the same time These are questions that might sug gest what information should be contained in the specification language The prototype interfaces showed that finding groups of similar functions is important for constructing a good interface These groups define how elements are placed relative to each other and which elements can be separated across multiple screens The different screens of the tab components are the best examples of grouping in our prototype interfaces see Figure 3 2 and Figure 3 3 Grouping is also used to separate the mode random and repeat ele ments from the rest of the elements of the stereo CD player interface see Figure 3 3c These elements are used in all of the CD player s modes while the other components are only used in half the modes Unfortunately the visual groups cannot be specified explicitly because their members may vary between target platforms For example on a device with a small screen it might be nec essary to separate the display of the current disc and track from the controls for playing a CD It would not be appropriate if the PUC separated the play and stop buttons however We noted that grouping information could generally be specified as a tree and that the same tree could be used for interfaces of many different physical sizes provided the tree had suffi cie
196. ch type it is important to fill out all of the information for each object and group because an interface generator might not know about the Smart Template that you have used If this is the case and you have included all of the information the generator can still create an interface If information is missing then this would not be possible Another variable that uses a Smart Template in this specification is the variable for comple tion date in the to do list As an exercise I suggest that you find the appropriate Smart Template and apply it when specifying this state variable Dependency Information If you looked at XML code for the Delete command above in Figure you may have no ticed that this command includes dependency information Specifically the Delete command is only available if there are more than zero items in the to do list and an item in the to do list is selected The card for this dependency formula see Figure should be at tached to the object card for the Delete command Appendix D Specification Authoring Study Instructions 283 Dependencies and or State Name gt lt def Value or State Name a ToDo List Length O To Ars o List Sele Aching Figure D 9 The dependency card for the Delete command Note that extended names are used here for the state variables You may find it necessary to go back and make names more explicit on dependency cards after you have organized the variables
197. chols Brandon Rothrock Duen Horng Chau and Brad A Myers Huddle Automatically Generating Interfaces for Systems of Multiple Appliances in Proceedings of the 19 annual ACM symposium on User Interface Software and Technology UIST Montreux Switzerland Oct 15 18 2006 pp 279 288 147 Broadcast Television ji Radio Y Oig Speakers Sony Receiver DEES ley A A A VCR w w J Panasonic Panasonic Philips Projector VCR 1 VCR 2 DVD Player i E Audio Audio Video 4 n Video Ee InFocus Television a Home Theater PowerPoint _ N S 7 Laptop External Task Manager Video Control a Media Player b Presentation Room Figure 9 1 Configuration of appliances in two multi appliance system scenarios a a home theater and b a presentation room 9 1 Scenarios Throughout this chapter I will use two scenarios to demonstrate the PUC s interface aggrega tion features a home theater and a presentation room The home theater setup see Figure 9 la includes five appliances an InFocus television 61md10 a Sony audio receiver STR DE935 with attached speakers a Philips DVD player DVDP642 and two identical Panasonic VCRs PV V4525S This setup supports many common tasks such as watching television watching a movie from either a DVD or videotape and listening to the radio It also supports a number of more complicated tasks such as
198. cify the type of the state variable shown in Figureb The reminder frequency can be set to three different val ues so I have picked an enumerated type for this state that contains three items Labels must be specified for each of the values of the enumerated type which is done by attaching three label cards to the type card see Figurec State Variable Type Name f O Boolean Object M State Variable O Command O Explanation bf Enumerated 2 O Read only Name Remi nde cFrea uenGy O Fixed Point Priority S 0 10 Number of Items Fill in values and labels on label card Decima Places Min Max Increment O Floating Point Min Max Labels Reweln dec Erea lad l g O Integer R Min Max Increment Remi ders D List Selection WKeminders Smart Template List Name if item dependency attach card Write additional labels on back O String a b Labels Labels Labels Value T Value 2 Value 3 Labels Week Y Labels Jaily Labels Hou ch A Figure D 1 The cards that specify the reminder frequency setting All of these cards would be attached together with a paper clip You may have noticed that the object card shown for the reminder frequency setting has a priority value of 5 specified for it You may wonder where this value came from especially because priority values are chosen relative to the other members of the group that contains this state v
199. code between each of the interface generators This includes the code the implements the communication protocol parses the specification language and performs other common tasks Some interface generation rules can be shared among the platforms as well but each platform also has its own unique generation rules These rules will be discussed in detail in Chapter 8 Each of the graphical generators has its own custom interface for allowing the user to connect to appliances Some custom controls were also implemented for the various generators to support the generation of list interfaces and add common controls that were not supported in the Compact Framework 4 3 1 PocketPC and Desktop Implementation The PocketPC and desktop generators have a similar menu based interface that allows users to connect to appliances and generate user interfaces The interface generator menus are used exclusively for generator functions and no appliance functionality is ever added to them This decision was based in part of the preliminary user studies see Chapter 3 that showed subjects rarely looked in menus for appliance functions There are four menus The PUC menu shown in Figure 4 3a allows users to connect to a new appliance server or choose a Chapter 4 System Implementation 59 PUC v4 0 1133 UH Starting Done Attempting to open server at 192 168 55 100 Connected to appliance server 192 168 55 100 Server information updated for PUC v4 0 1133
200. consistency rules would make changes to the new interface based on the pre vious products A report based on this information might help the designers understand how the new interface is inconsistent with the previous products and help them make improve ments Other evaluations might examine how a PUC specification for an appliance differs from its hand designed interface or identify areas of the interface structure that are particu larly deep or difficult to access because of dependencies The hurdle to using these techniques to evaluate a hand designed interface is providing the evaluation system with an abstract model of the interface that it can understand It seems less than ideal to ask the interface designer to provide this model as it would require extra work to produce much of the knowledge needed is already stored implicitly in the hand designed interface and it would require the interface designer to learn an abstract modeling language The last point is particularly important as most interface designers are not programmers and may not have the training to produce an adequate abstract model of their hand designed in terface Thus it seems an important area of future work is developing techniques to infer useful abstract interface models Ideally this model could be inferred from an existing con crete interface but this seems unlikely to succeed because many design decisions that influenced the final design may not be apparent from t
201. copier and the Book Copy function on the simple copier Note that the consistent interfaces for these copiers address the differences without disrupting the usability of the unique functions I have also conducted an evaluation of the rules for generating consistent interfaces again using the HP and Canon all in one printer appliances In this study users with no knowl edge of either appliance were asked to perform a set of tasks on a PUC interface for one of the appliances Following the tasks users were instructed on the correct method for perform ing each task on the first interface and then asked to complete the same tasks on a PUC interface for the other printer appliance This second interface was either generated normally without consistency or generated to be consistent with the first interface I found that users were twice as fast on average at performing tasks on the second interface when that interface was generated to be consistent as compared to the users of the normally generated interface The interface generator can also produce user interfaces for systems of multiple connected appliances such as home theaters or presentation rooms These interfaces are generated based on a full content flow model of the appliance system that is assembled from each ap pliance s specification and a wiring diagram for the system supplied by a third party such as a future wiring technology or the user The PUC provides two types of interfaces
202. copying from a tape in VCR 1 to a tape in VCR 2 or watching television on one channel while recording up to two other channels Sometimes tasks can be mixed such as watching a sporting event on television while listening to a radio broadcast of the play by play Certain tasks are impossible with this setup such as recording a DVD to videotape recording the radio or recording from a tape in VCR 2 to a tape in VCR 1 As we will show below the PUC s flow based interface makes it clear to the user which flows are not possible The presentation room configuration has three physical devices see Figure 9 1b a projec tor a VCR and a laptop The laptop s functions however have been separated into several independent logical appliances which include the PowerPoint and Windows Media Player applications the task manager and control of the external video port This configuration supports common presentation tasks such as showing slides showing video from the laptop and showing video from a VCR tape Chapter 9 Aggregating User Interfaces 148 9 2 Content Flow for Understanding Systems of Appliances In order to generate aggregate user interfaces the PUC needs some knowledge of how each appliance in a system relates to the others Current integrated user interfaces such as those that can be built with technology from Crestron or AMX require new configuration infor mation for each system of appliances This approach requires too
203. d be increasing the usability of an appliance adding accessibil ity of the appliance interface to users with disabilities the goal of INCITS V2 or integrating the appliance interface with other appliances that may be connected to it It re mains to be seen whether these issues will motivate manufacturers to adopt standards such as INCITS V2 and add these technologies to their appliances 11 2 Impact Although the PUC system has not been adopted by any manufacturers of computerized ap pliances it has affected the development of the INCITS V2 standard been used by two different research groups and its specification language was used as a component of a project in an undergraduate class at the University of Alabama The INCITS V2 group described in section 2 1 2 1 is developing a standard for moving the interfaces from appliances to a remote control device much like the PUC for the pur pose of providing accessible user interfaces for users with physical and cognitive disabilities The initial draft of the V2 specification did not sufficiently abstract the functionality of the appliance to allow for generation of interfaces on a wide variety of platforms and did not meet several of the requirements that I found were needed for a system that provides remote Chapter 11 Conclusion 182 control of appliances see section 3 6 I provided an initial analysis of the V2 technology Nichols 2002a that along with other interactions led
204. d functions based on whether they belonged to the TV and VCR using this method to organize within both the basic controls and setup groups Chapter 6 Consistency 90 The subjects also placed objects at different locations in their hierarchy For example the repeat play command was put in the advanced playback controls group of one specification and in the setup group in another one The functions were also defined differently in some cases as one subject used commands for the play stop and pause buttons while the other two used state variables 6 2 3 Discussion In these studies I found that specifications will have differences even if written by the same author or for the same appliance These differences may be found in the specification of similar functions and the organization of these functions The number and variety of differ ences was particularly surprising and demonstrates the challenges the PUC faces when creating consistent interfaces Next I will combine these results with prior work on interface consistency to synthesize a set of requirements for consistency in the PUC system 6 3 Requirements for Consistency For the PUC we define a consistent user interface as one that is easier to learn because it in corporates elements and organization that are familiar to the user from previous interfaces In the context of appliance interfaces this might mean that a new copier interface is easier to learn because it uses the s
205. d have this change be automatically broadcasted to each appliance in the system 9 4 Flow Based Interface The FBI is designed to allow users to quickly specify a flow from one source of content to one or more content sinks For example the user might specify a flow from the DVD Player s disc to the television s screen and the receiver s speakers When the user activates this flow the PUC inspects the dependencies of each of the flow s elements generates a plan to satisfy these dependencies and executes that plan to enable the flow If the flow cannot be enabled perhaps because of other active flows that the user has already specified the system will prompt the user with a dialog box and attempt to help the user resolve the problem Several examples of the FBI are shown in Figure 1 7 and Figure 9 3 To make the idea behind the FBI clear the interaction that a user would have with the inter face in order to start watching a DVD movie will be described Figure 9 3a shows the FBI in its initial blank state Near the top of the screen is a blank flow with empty spaces for a source and sink with an arrow between them At the bottom of the screen is the appliance bar which contains an icon corresponding to each appliance that has a source or sink in the system The appliance bar may grow upward to allow space for all of the available appliances in a system to be shown Appliance icons are currently assigned by a hard coded matching function
206. d kept me on the path to finishing this thesis I have tried to acknowledge as many of these people below as I could and my sincerest apologies to anyone I have accidentally forgotten I would first like to thank Brad Myers for his guidance and advice throughout my time at Carnegie Mellon Brad taught me what research is how to do it and how to present the re sults in a clear and understandable fashion I am still amazed that even at his busiest Brad was always able to find time to read one of my papers and give me an enormous amount of feedback on its content I only hope that I can be half as successful as a researcher and a men tor in my career as Brad has been in his I would like to thank the other members of my thesis committee Scott Hudson John Zim merman and Dan Olsen for providing invaluable comments along the way that shaped the direction of this work I would also like to thank Scott Hudson for all of his efforts in organizing the HCII s Ph D program and for putting up with all of the student feedback that I brought to him during my time as the student ombudsperson Scott was always willing to help address our prob lems and I do not think the program would be half as strong without his presence I was lucky to have the assistance of many great individuals over the course of PUC project In particular I would like to thank Brandon Rothrock for assisting with several aspects of the system implementation Duen Horng
207. different even though the function is identical We also analyzed the organization of the VCR specifications and found a few differences In general it seems that most of the same high level groups were shared between the specifica a Mitsubishi HD HS2000U b Samsung DVD V1000 c Panasonic PV V4525S Figure 6 1 VCRs used in the first study The Panasonic VCR in c was also used in the second study Chapter 6 Consistency 89 tions though the exact placement of those groups varied somewhat For example all of the VCRs have the Power state at the top level with groups for Status and Setup The Mitsubishi and Panasonic VCRs also have groups for Controls and Timed Recordings at the top level The Samsung DVD VCR has these same groups but they are located in a top level VCR group because this appliance also must support its DVD and MP3 players 6 2 2 Study 2 The second study examined the variations in specifications written for the same VCR by sev eral different authors For this study I was particularly interested in seeing how organization varied between specifications I used the Panasonic VCR from the first study and recruited 3 students in our university s electrical and computer engineering department to be subjects I chose these subjects because I expect that specification authors in industry would have this background Unlike in the first study these subjects had no knowledge of the PUC specification language when
208. do items e Detailed information about each item including a description category completion date and completion flag e Add and delete functions for to do list items e Sorting functionality for to do list on category completion date or completion flag e A customizable list of categories e Add and delete functions for category list items e Reminder frequency setting which may be weekly daily or hourly e Display preference for choosing whether or not to show completed tasks For a regular appliance you should be able to discover many more details about each of the various functions than I have presented above It is important that you explore the features through the user manual and using the appliance so that you can ensure that your specifica tion is as accurate as possible Defining a State Variable We will start by specifying state variables for the reminder frequency and display preference These are variables because each has a value that can be manipulated by the user You could choose to have separate commands for these settings but in general it is better to use a state variable wherever possible For the card based method we start by defining a state variable with an object card Figurea shows the object card for the reminder frequency state variable Because this object card de Appendix D Specification Authoring Study Instructions 277 fines a state variable a state variable type card must be attached to spe
209. ds have been identified Of course identifying the variables and commands of an appliance may not be as difficult for the engineers that originally built the appliance Thus the specification language may be even easier for the makers of an appliance to use once learned than shown by the authoring study Chapter 5 Specification Language 82 CHAPTER 6 Consistency The PUC system is the first to automatically generate interfaces that are consistent with in terfaces the user has seen previously This chapter discusses the meaning of consistency both in general and for the PUC system and describes the infrastructure needed for the PUC to generate consistent user interfaces The rules for generating consistent interfaces are discussed along with the other interface generation rules in Chapter 8 6 1 Understanding Consistency Consistency has been a subject of research for the user interface community for many years and there has been much debate about what consistency is and how to apply it effectively According to Grudin a two day workshop of 15 experts was unable to produce a definition of consistency Grudin 1989 Reisner said that consistency is loosely defined as doing similar things in similar ways and that inconsistency occurs when the designer and the competent user employ different assignment rules Reisner 1990 Kellogg Kellogg 1987 The work in this chapter was previously published in Jeffrey Nic
210. e lt string gt lt type gt lt labels gt lt label gt Finish By lt label gt lt label gt Due Date lt label gt lt label gt Due lt label gt lt labels gt lt state gt lt list group gt lt group name Commands is a list commands priority 10 gt lt command name Add is a l1ist add priority 8 gt lt labels gt lt label gt Add To Do Item lt label gt lt label gt Add To Do lt label gt lt label gt Add lt label gt lt labels gt lt command gt lt command name Delete is a list remove priority gt lt labels gt lt label gt Delete To Do Item lt label gt lt label gt Delete To Do lt label gt lt label gt Delete lt label gt lt labels gt lt aCtive 1i gt lt greaterthan state ToDo List Length gt lt constant value 0 gt lt greaterthan gt Appendix D Specification Authoring Study Instructions 297 defined state ToDo List seleaction gt lt active if gt lt conmmand gt lt state name SortBy priority 5 gt L_ypee lt enumerated gt lt item count gt 3 lt item count gt lt enumerated gt lt value labels gt lt map index 1 gt lt labels gt lt label gt Category lt label gt lt labels gt lt map gt lt map index 2 gt lt labels gt lt label gt Completion Date lt label gt lt label gt Date lt label gt lt labels gt lt map gt lt map index 3 gt lt labels gt lt label gt Completed lt label gt lt labels gt
211. e turn to the PUC interface the appropriate information from the database is automatically filled into the appliance s fields This integration is particularly useful in the navigation sys tem interface where it allows users to quickly specify a destination to navigate to from their contacts see Figure 7 4c Another potential controller specific implementation would be to take advantage of any special physical buttons that a controller device possesses For example if a controller had two special buttons for volume up and down then the volume Smart Template could automatically allow those buttons to control the volume of the current ap Chapter 7 Handling Domain Specific and Conventional Knowledge 105 2 x 9 35 Dk gt x 9 35 Dk Clock 2 54 00 PM a Clock 14 54 00 a Hour Mode Hour Mode X Auto GPS Update Auto GPS Update PUC v4 0 1120 UH x 9 33 IX Current Location a 1400 Pennsylvania Ave NW Daylight Savings Time Location Address bd Time Zone 2116 Hampton St Pittsbu MAA Location POI Automobile Daylight Savings Time Time Zone Gas Station Texaco Pittsburgh w PUC Servers Devices Generation Destination 4 x 8 18 ok Destination a 4x 9 16 Dk Location Location Location aT Location addres Address Address Num Street Num Street 2116 Hampton St City State Pittsburgh 2116 Hampton St Pittsburgh PA 15218 Uni
212. e xs string use required gt lt xs attribute name spec2 state type xs string use required gt lt xs complexType gt lt xs complexType name ValueType gt lt xs sequence gt lt xs element name specl type SpecValueType minOccurs 0 maxOccurs 1 gt lt xs element name spec2 type SpecValueType minOccurs 0 maxOccurs 1 gt lt xs sequence gt lt xs attribute name specl type xs string use optional gt lt xs attribute name spec2 type xs string use optional gt lt xs complexType gt lt xs complexType name SpecValueType gt lt xs choice minOccurs 1 maxOccurs 1 gt lt xs element name constant type ConstantValueType gt lt xs element name range type RangeValueType gt lt xs element name defined gt lt xs element name undefined gt lt xs choice gt lt xs complexType gt lt xs complexType name RangeValueType gt lt xs attribute name start type xs string use required gt lt xs attribute name end type xs string use required gt lt xs complexType gt lt xs complexType name ConstantValueType gt lt xs attribute name value type xs string use required gt lt xs complexType gt lt xs complexType name MappingType gt lt xs sequence gt lt xs element name label type xs string minOccurs 0 maxOccurs 1 gt lt xs element name source type SourceType minOccurs 1 maxOccurs 1 gt lt xs element name cost ty
213. e accounted for in the tree Pa rameters used in the tree include whether the object is a state or a command the type of the state variable constraints on the type of the variables whether the variable is read only etc For example commands are always represented with a button AIO and read only state vari ables are usually represented by a label AIO 8 4 Modifying the Abstract Interface for Consistency After the abstract user interface is created two phases modify the interface to ensure consis tency with previous interfaces These phases are 1 Functional Phase each mapping is examined and the abstract user interface may be modified to ensure functional consistency 2 Abstract Structural Phase the organization of the abstract user interface is modified for consistency This phase helps to ensure structural consistency Following this phase the PUC transforms the abstract interface into a concrete user interface Although the functional phase precedes the abstract structural phase these two phases could be executed in the opposite order I chose this order for implementation reasons because it is easier to find functions in the abstract user interface before the structural phase moves them around All of the rules is these phases must simultaneously balance consistency with previous inter faces and the usability of the new interface This is particularly important when dealing with functions that are unique to the n
214. e and a HAVi controller could have a hand designed interface built in for each standardized type 2 every appliance can export a level 1 or data driven interface which is basically a description of a hand designed interface that includes buttons labels and even multiple panels and 3 every appliance can export a level 2 user interface which is a piece of mobile code written in the Java language which displays a remote control user interface when executed on a HAVi con troller None of the interface descriptions are abstract as the PUC appliance specification language is and only the second and third interface description options may allow the HAVi Chapter 2 Related Work 22 controller to access special features of the appliance The main advantage of HAVi over other proposed industry standards is its ability to control older legacy appliances using older pro tocols such as AV C Association 1996 The main disadvantage of HAVi is the size of its API which includes three levels of interface specification standardized templates for many types of appliances which must be built into any controller implementation a Java virtual machine and support for a number of legacy protocols 2 1 2 5 JINI Sun s JINI system was designed as a network infrastructure to make it easier for program mers to create distributed systems Like the PUC INCITS V2 HAVi and UPnP it could allow a controller device to manipulate an applian
215. e appliance description Usually this will be used be when the user has configured a set of common values in a setup portion of the interface and then wants to select one of those values elsewhere in the interface You must specify the name of the list that this variable will select from Optionally you may also specify some de pendencies on the values of that list in order to restrict the list items that may be selected The string type contains a string value Currently there are no parameters for this type Appendix D Specification Authoring Study Instructions 267 The Group Tree Proper structure is a very important part of any user interface In our language we use a hier archical group tree to specify structure The leaf nodes in the tree are appliance objects and the branch nodes are groups Each node in the tree has a name which must not be the same as any other child of its parent Thus each node has a locally unique name and a glob ally unique name can be constructed by pre pending the names of all a nodes parents We use a character to separate each name For example the locally unique name of a state variable might be PlayState but the globally unique name would be Stereo CD PlayState That same specification could also contain other states like Stereo Tape PlayState Note that the root name Stereo cannot be re used as a name anywhere in the specification It is often necessary to explicitly refer to state variabl
216. e channel Defines the channel to block May Contain lt active if gt lt channel gt lt objects gt lt objects gt lt objects gt Defines a set of groups and appliance objects that are related to the control of this content flow Placement Inside the lt pass through gt lt recorder gt lt renderer gt and lt port gt elements May Contain lt group gt lt object gt Appendix B Specification Language Reference 250 lt object gt lt object name Base Power gt Defines an object that is related to the control of a content flow This name must re solve to an appliance object within the specification and not a group Placement Inside the lt objects gt element Parameters e name required The name of an appliance object within the specification Appendix B Specification Language Reference 251 Appendix B Specification Language Reference 252 APPENDIX C Other PUC XML Language Schemas The PUC also uses XML as its communication protocol to store mappings in its knowledge base and to describe systems of multiple appliances Each of these XML languages is de scribed in this appendix C 1 Communication Protocol Schema lt xml version 1 0 encoding utf 8 gt lt xs schema targetNamespace http www cs cmu edu pebbles puc puc protocol elementFormDefault qualified xmlns http www cs cmu edu pebbles puc puc protocol xmlns mstns http www cs cm
217. e complication is not inherent to the appliance s functionality but instead due to the limited interface capabilities of the appliance The PUC is much less limited in its ability to provide an interface and as such tasks can be accomplished in the interface without as many complicated steps One of the biggest challenges throughout the PUC s consistency algorithms is appropriately dealing with the unique functions and organization found in similar appliances Chapter 8 Chapter 11 Conclusion 178 describes three heuristics to address this problem but the solutions are often limited because of a lack of useful semantic information about the unique functions In part this is due to the PUC s use of relative semantics to understand similarity the PUC only knows that two functions are similar not why they are similar what they do or how they relate to other functions in the appliance With better information the PUC could make more informed decisions about where to place functions and when to create new organization Of course better information would come at the cost of additional modeling for each appliance because it seems unlikely that detailed information could be automatically extracted from the appli ance specifications A large part of the multi appliance interface problem discussed in Chapter 9 arises from mul tiple appliances being connected together which requires the user to interact with multiple interfaces to a
218. e environ ment and focuses the designer on a tight design evaluate redesign cycle To support this cycle the system is explicitly designed such that the application can be run even if it is not fully specified The benefit of this is that designers can get immediate feedback and explore many alternatives in a short amount of time 2 2 1 5 Mastermind The MASTERMIND project Szekely 1995 started as collaboration to combine the best features of UIDE and HUMANOID In addition to modeling capabilities of those systems MASTERMIND also uses task models to inform its automatic interface designs Task mod els have since been used in nearly every new model based system MASTERMIND was also one of the first systems to explore the idea of generating different interfaces for desktop com puters handheld computers and pagers Szekely 1996 by using the model to decide which information or interface elements could be removed from the interface as the size decreased The interfaces generated for each different device used the same interaction techniques which is not true of the dramatically different PUC interfaces generated for the PocketPC as compared to the Smartphone 2 2 1 6 ITS ITS Wiecha 1990 is another model based interface system and was developed by research ers at IBM The ITS system differs from other model based systems in its explicit separation of concerns within its four layer specification ITS s layers consist of actions for modi
219. e for simplifying the user interface implementation process 2 2 1 2 Jade Jade Vander Zanden 1990 is another example of an early model based system for auto matically generating dialog box layouts based on a textual specification of the content Like the PUC s specification language Jade s textual specification contains no graphical informa tion which keeps each specification small and allows the look and feel of the generated interfaces to be independent of their content Unlike the PUC system Jade allows interface designers to manually edit its results to fix any problems in the automatically generated inter faces Most of the model based systems discussed in this section have similar features for allowing the interface designer to guide the generation process and or edit the final user in terfaces While the PUC system could allow manually editing it is important to remember that users of the PUC system are not trained designers and will rarely have the time or desire to modify a generated interface A213 GIDE Systems of the late 80 s and early 90 s such as UIDE Sukaviriya 1993 HUMANOID Szekely 1992 and ITS Wiecha 1990 expanded on these ideas with more complicated models that could generate more sophisticated user interfaces UIDE which stands for User Interface Design Environment is the earliest of these systems The knowledge base contains information about objects the actions that users can use to manipulate th
220. e list is converted back to rows The process used is similar to the one used for re ordering the abstract user interface Figure 8 14 shows an example of how this re ordering rule affects the Canon all in one printer interface when it is generated to be consis tent with the HP all in one printer The second and third rules address inconsistencies that may have arisen during the genera tion of the concrete interface The second rule adds overlapping panel organization to a user interface if it was used in the previous interface and more than one control can be placed on g PUC v4 0 1124 UH qx 4 25 X g PUC v4 0 1125 UH gt qx 4 37 X g PUC v4 0 1124 UH 4x 4 29 X Rese r Resize Options tr Resize 100 Resize Zoom 25 400 79 zoom a W u o H z5 300 f Sf Borderless OOOO O e Ea x 5 a _ Borderless Copy Tray Repeat Image PUC Servers Devices Generation PUC Servers Devices Generation PUC Servers Devices Generation a HP without consistency b Canon consistent with HP c Canon consistent with HP without concrete rules with concrete rules Figure 8 14 Interfaces generated for the HP and Canon all in one printers demonstrating the effects of the concrete interface re ordering rule Note the difference in the order of the Black Color and Cancel buttons Chapter 8 Interface Generation 144 each of the overlapping panels The exact type of overlapping panel widget is chosen based
221. e name of the template and the conforming to the template s restrictions within that section of the specification see Figure 7 2 When the interface generator encounters a section of a specification referencing a template that it knows about it can invoke special code written by the generator program mer to appropriately render the template If the generator encounters a template that it does not know about it will use its normal generation rules to render the template contents This is possible because every Smart Template is defined using the primitive elements of the speci fication language For example Figure 7 3a shows an instance of a media controls Smart Template rendered by a generator with no knowledge of that template and Figure 7 3b lt group name Controls is a media controls gt lt labels gt lt label gt Play Controls lt label gt lt labels gt lt group name Counter is a time duration gt lt state name Mode gt lt labels gt lt label gt Counter lt label gt lt labels gt lt type gt lt enumerated gt lt state name Hours gt lt item count gt 3 lt item count gt lt type gt lt enumerated gt lt integer gt lt value labels gt lt type gt lt map index 1 gt lt state gt lt labels gt lt label gt Stop lt label gt lt labels gt lt map gt lt state name Minutes gt lt type gt lt map index 2 gt lt integer gt lt labels gt lt label gt Play lt label gt lt
222. e or a fixed number of items in a lt list group gt Labels for the items can be defined within the lt value labels gt tag Placement Inside the lt enumerated gt and lt list group gt elements lt fixedpt gt lt fixedpt gt lt fixedpt gt Fixed Point type for variables that take the form of decimal values with a fixed decimal point The location of the fixed decimal point is specified with the lt pointpos gt element Minimum maximum and increment values can be defined using the lt min gt lt max gt and lt incr gt tags Placement Inside the lt type gt element May Contain lt DPOLNL DOSS Siner lt max gt lt n n gt lt pointpos gt lt pointpos gt lt pointpos gt Defines the position of the decimal point for a fixed point type Placement Inside the lt fixedpt gt tag Appendix B Specification Language Reference 231 lt floatingpt gt lt floatingpt gt lt floatingpt gt Floating Point type for variables that take the form of decimal values Minimum and maximum values can be defined using the lt min gt and lt max gt tags Placement Inside the lt type gt element May Contain lt max gt lt min gt lt integer gt lt integer gt lt integer gt Integer type for variables that take the form of integers Minimum maximum and increment values can be defined using the lt min gt lt max gt and lt incr gt tags Placement Inside the lt t ype gt element
223. e than the nine items that can be shown at once on the screen The method for doing this uses common label prefixes and suffixes just like the second rule Child lists are built in reverse priority order until the current list contains nine items or less I have experimented with several different methods for assigning a function to the most common soft button on the Smartphone Initially the PUC used this button to move up in the list hierarchy which duplicated the functionality of the physical back button This helped novice phone users navigate our interface but I felt that it might be more useful to assign common functions from the appliance to the button instead I investigated two ap proaches a static approach using priority information from the appliance specification and an adaptive approach based on recorded usage information The first method chooses a function for each screen by ranking each of the functions on that screen according to the priority information in the specification language If there is a tie the PUC chooses the function that occurs first in the appliance specification One function is chosen for each screen and these functions do not change once the interface is built The second method is adaptive which means that the function assigned to the soft button changes as the user interacts with the interface The PUC selects the function by searching the recorded usage information for the most likely next
224. e the address template on the PocketPC this template can integrate with the built in Outlook database to allow users to auto matically insert numbers into the fields of the PUC interface The status icons template is a grouping template in which one or more special _ templates representing a variety of status indicators typically represented as icons _ are contained Currently ten different indicators are supported by the PUC in _ cluding ones for specifying whether a tape is in the appliance whether the device is _ busy etc see Figure 7 5 for a complete list Represents an absolute time of day as opposed to a duration of time This tem _ plate is commonly used for the clock function of an appliance but also has many _ other uses The date template can also handle other related preference variables _ such as whether the user prefers a 12 or 24 hour time and whether daylight sav ings time is currently in effect Some of these preferences will affect the representation of other time absolute templates on the same appliance Many dif ferent representations for time are supported including integer string and multi variable representations i Represents a duration of time such as would be required for a stopwatch appli _ ance As for the time absolute template many different representations of a time _ duration are supported including interface string and multi variable representa Represents controls for zoomin
225. e third rule places the non dependent items and the mutual exclusion state in a panel as they normally would be Immediately after the mutual exclusion state an overlapping panels j PUC v4 0 1122 UH 4x 1 29 X PUC v4 0 1122 UH qx 1 27 X PUC v4 0 1122 UH g qx 1 29 X PUC v4 0 1122 UH qx 1 30 X HP Photosmart 2610 All in One Copies 1 al Resize Options Resize Options Zoom 25 400 Enlarge to Poster Size v 25 Zoom gu TP Poster 35 0 Width PUC Servers Devices Generation PUC Servers Devices Generation PUC Servers Devices Generation PUC Servers Devices Generation a b C d Figure 8 10 Screenshots of the HP all in one printer interface demonstrating the three mutual exclusion rules a The special power off screen generated by the second rule The remaining shots are of the power on view b The fax mode of the all in one printer This mode is accessed through tabs at the bottom of the screen cre ated by the first rule Another tab can be seen in screen shots c and d c The copy mode of the all in one printer with the resize options set to zoom 25 400 The resize options state is a mode with several different options created by the third rule d Another view of the copy mode with the resize options set to poster size Chapter 8 Interface Generation 135 row is added containing a panel for each of the mutually exclusive sets of controls When the user manipulates the
226. ecently connected servers are also placed in separate sub menu to limit the number of items in the menu The multi appliance interfaces have no yet been implemented for the Smartphone so only consis tency options are available in the generation menu see Figure 4 4c It was also necessary to implement custom controls for the Smartphone interface Although the interface guidelines for the Smartphone called for three types of interface views see sec tion 8 1 2 for a more detailed description no controls were provided in the Compact Framework for impelementing these views Thus it was necessary to implement a list widget for the typical list based interaction see Figure 1 3a c for several examples and a widget for panel based interactions that automatically scrolls as the user s input focus changes see Figure 4 4d It was also necessary to implement a custom slider widget because the Smart phone version of the Compact Framework did not even implement an interactive scrollbar An example use of the custom slider control is shown in Figure 4 4e for controlling the brightness of a copier Chapter 4 System Implementation 61 44 Communication The communication protocol defines 11 different messages that may be sent between an ap pliance and a controller device 6 of these messages may only originate from the controller devices see Table 4 3 whereas the other 5 may only originate from an appliance see Table 4 4 The protocol defines
227. ecify and store the mappings an XML based language was developed for speci fying mappings the schema for which can be found in Appendix C 2 The basic elements of the mapping language correspond to the six mapping types described in Table 6 1 and each mapping type has child elements that allow its parameters to be specified I chose not to use an ontological approach for the knowledge base for two reasons First cur rent ontology languages such as RDF W3C 2006 and OWL Herman 2006 seem to be most useful for understanding the relationships between concepts that can be described with nouns and adjectives For example an ontology might help a computer understand that a bear is an animal with fur and animals with fur are mammals The concepts neces sary to achieve consistency are not nouns however but actions verbs and actions are much more complex because they are made up of multiple steps which may be constrained in a particular order Differences between similar actions may occur in one or more steps or in the ordering constraints between steps The more general ontology languages such as OWL may be able to represent actions but there are two drawbacks specialized processing is re quired to analyze the ordering constraints between steps and the description of an action requires substantially more overhead than a language designed to described actions such as Concur TaskTrees Paterno 1997 Chapter 6 Cons
228. ecifying an array of records in a programming language and multiple list groups can be nested to create multi dimensional lists Each list group has an implicit length state variable named Length that always contains the current length of the list If this variable is undefined then the list currently has no members The specification may define bounds on the length of the list in order to help the interface generator create a better rendering An exact size may be specified or a minimum and or maximum size may be specified List groups also maintain an implicit structure to keep track of one or more list selections The number of selections allowed may be defined in the specification one and many are the only options currently and the default is one if nothing is specified If a list allows only lt list group name List gt lt labels gt lt label gt Timed Recording lt label gt lt labels gt lt min gt lt constant value 0 gt lt min gt lt max gt lt constant value 8 gt lt max gt lt selections access read write number one gt lt state name Channel gt lt state gt lt state name StartTime is a date time gt lt state gt lt state name Duration is a time duration gt lt state gt lt list group gt Figure 5 4 An example of a list group used in the VCR specification to describe the list of timed recordings that may be specified by the user Chapter 5 Specificatio
229. ed lt xs complexType gt lt xs complexType name ChangeDataType gt lt xs sequence gt lt xs element name el type AnyListContentType minOccurs 1 maxOccurs 1 gt lt xs sequence gt Appendix C Other PUC XML Language Schemas 255 lt xs attribute name state type xs string use required gt lt xs attribute name index type xs integer use required gt lt xs complexType gt lt xs complexType name DeleteType gt lt xs attribute name state type xs string use required gt lt xs attribute name begin type xs integer use required gt lt xs attribute name length type xs integer use reguired gt lt xs complexType gt lt xs complexType name InsertType gt lt xs sequence gt lt xs element name el type AfterChangeOpListType minOccurs 1 maxOccurs unbounded gt lt xs sequence gt lt xs attribute name state type xs string use required gt lt xs attribute name after type xs integer use required gt lt xs complexType gt lt xs complexType name ReplaceType gt lt xsS sequence gt lt xs element name el type AfterChangeOpListType minOccurs 1 maxOccurs unbounded gt lt xs sequence gt lt xs attribute name state type xs string use required gt lt xs attribute name begin type xs integer use required gt lt xs attribute name length type xs integer use reguired gt lt xs complexType gt
230. ed if gt lt server side error correction gt lt type gt lt command gt lt command name CommandName priority 10 gt lt command gt Defines a command appliance object Placement Inside lt group gt lt list group gt lt types gt or lt union group gt Parameters e name required The name of the command e type name required if within types section optional otherwise defines a type with the given name e priority The priority this state should be assigned relative to other objects in the same group May Contain lt active if gt lt labels gt lt modifies state gt Appendix B Specification Language Reference 226 lt modifies state gt lt modifies state state Base Power gt Specifies that this command will modify a state in this specification This may all the interface generator to appropriately display changes caused by invoking this com mand Placement Inside the the lt command gt element Parameters state required The name of the state that will be modified lt explanation gt lt explanation name ExplanationName priority 10 gt lt explanation gt Defines an explanation appliance object The labels block which is required for an explanation object defines the text that will be used for the explanation Placement Inside lt group gt lt list group gt lt types gt or lt union group gt Parameters e name required The name of the explan
231. ed on this standard After a standard has been agreed upon manufacturers may still add custom functions but these functions will not appear in most UPnP printer user interfaces Smart Templates however standardize functionality at a much finer grain level It should be much easier for manufacturers to agree on the features of media controls for example as compared to a printer or VCR Smart Tem plates also do not affect manufacturers ability to innovate New functions even if they extend the functionality of an existing Smart Template can always be added to an appliance and rendered by the normal interface generator It is not unusual for consumer electronics manufacturers to produce both appliances and high end remote controls that can be used with those appliances In this situation specifica tion authors and interface generator programmers might work for the same company and it is reasonable to believe that manufacturers might develop their own in house Smart Tem plates that improve their appliances interfaces when used with their remote controls However since these proprietary templates would not be understood by other interface gen erators the interfaces generated for that appliance could be sub par as compared to competing appliances This could be particularly bad if the proprietary template replaced a common template such a media controls preventing even basic conventions from being applied in the appliance interface Chap
232. egate interface generators were also created that use content flow and combine functionality from multiple appliances to produce useful task based interfaces PUC graphical interface generators have been implemented on several different platforms including the PocketPC Microsofts Smartphone and desktop computers I collaborated with another research group to produce a speech interface generator based on the PUC framework demonstrating the PUC specification language is sufficiently abstract to create interfaces in multiple modalities Finally the PUC interface generator was evaluated to show that subjects using interfaces produced by the system can be twice as fast and four times as successful as compared to inter faces currently available on today s computerized appliances thus proving the thesis underlying this work 11 4 Future Work There are many directions for future work that build off of this dissertation Chapter 11 Conclusion 185 So far I have built graphical interface generators for three different platforms but there are many more mobile platforms on which interface generation could be implemented It would be most interesting to build generators for platforms that are substantially different than those I have already looked at For example smart phones from most other manufacturers have similar interface styles to the Microsoft Smartphone platform that my interface genera tor supports One interesting platform to ex
233. egate user interfaces Work in the control of appliances spans products such as universal remote controls to ap pliance communication infrastructures being developed in both industry and academic research A few projects have examined how the user interfaces for consumer electronics might be improved User interface generation has been the subject of research for many years sometimes under the name of model based user interfaces because the interfaces are generated from models of the application domain tasks the user might perform and the target platform The original goal of this work was to allow programmers who were not typically trained to design inter faces to produce good user interfaces for their applications More recent work falls into two categories fully automatic generation for producing interfaces customized for each user and designer guided generation for producing and maintaining very large scale user interfaces Aggregate user interfaces are those that combine functionality from multiple sources to pro duce one user interface Also known as mash ups this idea has recently become a popular piece of the new Web 2 0 technologies where developers are combining data from multi 17 ple web sites on a single site to produce compelling visualizations and applications Web mash ups are one type of aggregate interface though most examples to date were built by hand Some work has been done in the human computer
234. eger type ranging between 0 and 60 if the Hours state is specified Minimum and maximum are optional if Hours is not specified This state must be included if both the Hours and Seconds states are specified seconds This state must have an integer type ranging between 0 and 60 if the Minutes state is specified Minimum and maximum are optional if Minutes is not specified This state must be included if both the Minutes and Fraction states are specified ErACCTON This state must have an integer type The ranges may be specified to fit the type of fraction being used Increments may be specified only for the least significant unit For example if a particular template defines an hours state and a minutes state then an increment may be specified only for the minutes state Another template that defines all four of the possible states may only have an increment specified for the fraction state zoom controls Overview This template represents controls for zooming something on the appliance Contents This template may include two commands only one of which must be included In The zoom in command Appendix D Specification Authoring Study Instructions 294 Out The zoom out command Appendix D Specification Authoring Study Instructions 295 Appendix D 2 Complete XML Specification for To Do List Application lt xml version 1 0 encoding ut 8 gt lt spec xmins http www cs cmu edu pebbles puc name ToDoApp ver
235. eir options and to build an interface that allows them to make their choice Also because interfaces are generated based on previous interfaces there is an opportunity to seed the consistency system with high quality interfaces for a variety of different common appliance types Future interfaces for actual appliances would then be modified by the con sistency rules to be more like the high quality interfaces hopefully producing a higher quality result than otherwise This concept could even be applied to individual functions ensuring that certain annoying situations such as unneeded confirmation buttons never occur in fu ture interfaces I have not experimented with this idea and it would be interesting to see how far this technique can be taken to produce improved interfaces The PUC currently does not provide any mechanism for end user customizations Although our target users are not trained interface designers they still may wish to modify the inter faces produced by the system This mechanism could take the form of an interface builder or might be designed more around the structure of the underlying specification language Any customizations should be noted by the consistency system and used to influence the design of future interfaces It should also be possible for users to create their own macros that automate certain functions of their appliances There are two important problems of multi appliance systems that the PUC does not c
236. element name sSimilar objects type xs integer minOccurs 1 maxOccurs unbounded gt lt xs element name branch type BranchType minOccurs 0 maxOccurs unbounded gt lt xs sequence gt lt xs attribute name name type xs string use optional gt lt xs attribute name guid type xs string use required gt lt xs attribute name Similarity type SimilarityAttributeType use reguired gt lt xs complexType gt lt xs complexType name BranchType gt lt xs attribute name spec group type xs string use required gt lt xs attribute name spec entry group type xs string use regquired gt lt xs complexType gt lt Spec Map Types gt lt xs complexType name SpecMapType gt lt xs sequence gt lt xs choice minOccurs 0 maxOccurs unbounded gt lt XS lt XS lt XS lt XS lt XS lt XS elemen elemen elemen elemen elemen elemen t name group type GroupType gt name mapping type MappingType gt name state mapping type StateMappingType gt 1ame l1ist mapping type ListMappingType gt name template mapping type TemplateMappingType gt Gr ore seh sch oer D name location mapping type LocationMappingType gt lt xs choice gt lt xs sequence gt lt xs attribute lt xs attribute lt xs attribute lt xs attribute name specl guid type xs string use required gt name specl name
237. ement must be true for all elements in order for true to be returned If true if is amy then true is returned as long as the dependency formula con tained in the apply over is true for at least one of the elements I have discovered that dependency information can also be useful for structuring graphical interfaces and for interpreting ambiguous or abbreviated phrases uttered to a speech inter face For example dependency information can help the speech interfaces interpret phrases by eliminating all possibilities that are not currently available The use of these formulas for interface generation is discussed later in Chapter 8 5 2 1 7 Smart Templates Smart templates allow the specification author to indicate that portions of a specification will have a high level semantic meaning for the user When an interface generator encounters these portions of the specification it will attempt to generate an interface that matches the user s expectations An author can specify that a Smart Template for a group state variable or command with the is a parameter For example in Figure 5 la the Channel state vari able has been marked with the channel Smart Template Smart Templates are described in detail in Chapter 7 Chapter 5 Specification Language 75 5 2 2 Content Flow Language Elements We have also found that information about content flow can be useful to describe the rela tionships between appliances that have been connected
238. en the control s for a function should be disabled It was also noticed that most of the functions of an appliance were manipulating some data in a definable way but some were not For example the tuning function of a radio is ma nipulating the current value of the radio station by a pre defined increment The seek function also manipulates the radio station value by changing it to the value of the next ra dio station with clear reception This manipulation is not something that can be defined based on the value of a variable and thus it would need to be represented differently in the specification language Each of the interfaces used different labels for some functions For example the Palm stereo interface see Figure 3 2c d used the label Vol to refer to volume whereas the PocketPC stereo interface see Figure 3 3c d used Volume This problem seems likely to be even worse for much smaller devices such as mobile phones or wrist watches Thus it seems im portant for the specification language to include multiple labels that an interface generator could choose between when designing its layouts Finally it was found that all of the interfaces used some conventional designs that would be difficult to specify in any language At least one example of a conventional design can be found in each of the panes in Figure 1 a shows a telephone keypad layout b uses stan dard icons for previous track and next track c
239. ens Many of the screens are accessi ble through the menu bar at the bottom of the screen Subjects in the study and think aloud participants before the study were very tentative about navigating the menus to find a par ticular function In tasks that required the subject to navigate to a screen from the menu bar the subject commonly opened the correct menu closed the menu did something wrong on the current screen and then opened the menu again before finally picking the correct item Chapter 3 Preliminary User Studies 47 The PDA stereo interface had other problems as well In particular the record function was difficult to represent in the interface because it was associated with tapes but needed to be available in all of the stereo s five playback modes tape radio CD etc Although a record button was available on every screen see Figure 3 3c d many subjects would get confused and incorrectly switch to the tape mode instead of pressing the record button The red circle next to the text label on the Rec button was added after pilot testing to make the button more visible because we thought that people tried the tape mode because they did not see the record button This change seemed to have little effect however 3 5 Analysis of Interfaces Once we were confident that our interfaces were usable the interfaces were analyzed to un derstand what functional information about the appliance was needed for designing the interfa
240. ent group gt lt pass through content type av gt lt active if gt lt equals state Base PoweredItems Controls TV VCR gt lt constant value false gt lt equals gt lt active if gt lt input ports gt lt port name VHF UHF Antenna gt lt input ports gt lt output ports gt lt port name VHF UHF Antenna gt lt output ports gt lt pass through gt lt recorder name Tape content type av gt lt active if gt lt equals state Base PoweredItems Status TapeIn gt Appendix A Sample VCR Specification 202 lt constant value true gt lt equals gt lt equals state Base PowerediItems Status TapeRecordable gt lt constant value true gt lt equals gt lt equals state Base PowerediItems Controls PlayControls Mode gt lt constant value 6 gt lt equals gt lt active if gt lt input ports gt lt port name VHF UHF Antenna gt lt channel state Base PoweredItems Controls Channel gt lt port gt lt input ports gt lt objects gt lt group name Base PoweredItems Controls gt lt objects gt lt recorder gt lt content group gt lt content flow gt lt spec gt Appendix A Sample VCR Specification 203 Appendix A Sample VCR Specification 204 APPENDIX B Specification Language Reference This reference appendix is based on the online documentation for the specification language which can be found here http www pebbles hcii cmu edu puc specificati
241. ent panels in the interface Branch nodes spec ify how their child panels are placed relative to each other either as a set of panels separated with vertical or horizontal edges or a set of overlapping panels Each panel described in a leaf node contains a list of rows which describe how controls should be placed relative to each Chapter 8 Interface Generation 133 other Rows have five different possible layouts as shown in Table 8 3 Of these layouts the two labeled formats are preferred because they create a simple grid with labels on the left and controls on the right Some controls because of their content must be displayed across the entire panel and thus use the full width row The full width row and the multiple items row are also used if the available screen space is limited The interface tree is assembled by traversing the structure of the abstract user interface and applying a set of concrete interface construction rules There are two types of these rules one set creates new panel structure based on mutually exclusive groups found in the specification and another set find the appropriate concrete interaction object CIO for each AIO and place that CIO into a row There are three rules that add structure to the concrete interface based on mutual exclusion Two of these rules look for particular properties of the mutually exclusive situation and cre ate structure if those properties are found If none of these rules create
242. er Knowing the circum stances in which an object will be enabled or disabled can provide a helpful hint for structuring the interface because items that are active in similar situations can be grouped and items can be placed on panels such that the widgets are not visible when the object would not be active Specifying the prior knowledge of the enabled property is discussed in more detail later 5 2 1 2 Type Information Each state variable must be specified with a type so that the interface generator can under stand how it may be manipulated For example the Channel state in Figure 5 la has an integer type We define seven primitive types that may be associated with a state variable e binary e boolean e enumerated e fixed point e floating point e integer e string Many of these types have parameters that can be used to restrict the values of the state vari able further For example the integer type can be specified with minimum maximum and increment parameters see Figure 5 1a The enumerated type is for small collections of values that all have some string label Inter nally these values are represented by numbers starting with 1 For example an enumerated lt state name Channel is a channel gt lt type type name ChannelType gt lt integer gt lt min gt lt constant value 2 gt lt min gt lt command name Eject gt lt max gt lt labels gt lt constant value 128 gt lt label gt Eject lt
243. er of appliances The Logitech Harmony remote see Figure 2 2 is unique among universal remotes because it internally tries to maintain a record of the current state for all of the appliances that it can control This has the limitation that the remote must know the state of the system when it is first used and that a control must be done via the Harmony remote afterwards but it has the advantage that the remote can hide functionality that is not available in the current state The user interface is further simplified using a task based interface shown on the small LCD screen which displays a list of tasks such as play movie in VCR or play DVD The list is based upon the appliances the user has and the current state of the system When one of these options is selected the remote sends the appropriate codes to all appliances and may also instruct the user to do certain tasks such as insert a DVD into the player Both of these remote control devices also synchronize with a desktop computer to make the task of programming easier This also allows users to download their remote control layouts from the device and share them with other users on the Internet Several communities have been created to share panels for the Pronto such as http www remotecentral com and http www prontoedit com Synchronization is also the basis for programming the Harmony remote which is done via a web site that gives the user access to Harmony s exten
244. ere able to complete all their tasks in the AutoGen and Consistent AutoGen conditions results of the analyses of failures are shown in Table 10 2 Chapter 10 Usability Evaluation 171 Subjects who used the Canon printer and then the HP were significantly faster for total task completion time using the consistent PUC interface compared to the normal PUC interface Fii4 10 01 p lt 0 007 and the built in interface Fii4 64 48 p lt 0 001 The total completion time for the consistent PUC interface was on average more than twice as fast as the normal PUC interface M 2 10 vs M 4 54 and more than four times faster than the built in interface M 2 10 vs M 8 55 After first using the HP printer subjects were also significantly faster using the consistent PUC interface for the Canon printer both compared with the normal PUC interface Fi 14 7 60 p lt 0 02 and the built in interface Fi14 16 89 p lt 0 002 The average total com 06 00 30 00 05 00 25 00 04 00 20 00 03 00 15 00 02 00 Time taken Time taken 10 00 id Tasks total 01 00 00 00 AutoGen HP Consistent HP AutoGen Canon Consistent Canon Figure 10 4 Results of the second block of tasks showing the AutoGen condition compared to the Consistent AutoGen condition for each appliance 1 2 3 4 5 6 7 8 Total Time HP AutoGen Consistent Built In Canon AutoGen Consistent Built In Failures HP AutoGen Consistent B
245. erface was a very difficult task and often resulted in low quality interfaces Myers 2000 Most systems moved to designer guided processes rather than use a fully automatic approach Chapter 2 Related Work 27 Two motivations suggested that continued research into model based approaches might be beneficial e Very large scale user interfaces assembled with existing techniques are difficult to im plement and later modify and detailed models of the user interface can help organize and partially automate the implementation process The models can then be used to help designers re visit the interface and make modifications for future versions e A recent need for device independent interfaces has also motivated new research in model based user interfaces and specifically on fully automated generation Work in this area has also begun to explore applications of automatic generation to create in terfaces that would not be practical through other approaches For example the PUC s consistency feature see Chapters 6 amp 9 generates interfaces that are person ally consistent with each user s previous experience While all of these systems discussed in this section generate interfaces to our knowledge no user studies have been conducted to evaluate the resulting interfaces The closest reported study is of SurrLe Gajos 2004 discussed below which asked subjects without any interface design training to produce interfaces for a presen
246. es John Gould and Sharon Greene ITS A Tool for Rapidly Developing Interactive Applications ACM Transactions on Information Systems Jal 1990 1990 8 3 pp 204 236 Bibliography 322
247. es a Windows Media Player 9 b the navigation system from a 2003 GMC Yukon Denali c the Canon PIXMA MP780 All In One printer and d the HP 2610 All In One printer Interfaces generated by the PUC on the Smartphone for several different appliances a Windows Media Player b a simulated elevator c d a shelf stereo with CD radio and tape functionality The full PocketPC interface generated for the navigation system from a 2003 GMC Yukon Denali The arrows show buttons that cause a GialOe DOM tO OPE leca A Copier interfaces generated with and without consistency on the Pocket PE patior a E E Copier interfaces generated with and without consistency on the Smartphone PATON enar neas E EE eae Several examples of the Flow Based Interface being used for various tasks a playing a DVD movie with the video shown on the television and audio coming through the stereo s speakers b watching a sporting event on television but listening to the play by play over the radio c selecting different sources of content during a presentation with a PowerPoint slideshow as the current source and d resolving a problem with the DVD player through the GWESTION ANSWER IMrErTaCE esaeria aa E ENE A E Several examples of aggregate user interfaces generated based on the user s current task a playing a DVD movie with the video shown on the television and audio coming through the stereo s speakers b presenting Powerp
248. es in a specification This may be done using any name that starts with a globally unique name Since the root name is unique the full name will always work For the above examples Tape PlayState and CD PlayState would also work assuming that there are no other groups or states named CD or Tape There are three types of groups normal list and union A normal group is used for putting related data together similar to a record in a programming language List and union groups have special behavior beyond that of normal groups which are discussed in the next section Lists Specifying a list group is similar to specifying an array of records in a programming language and multiple list groups can be nested to create multi dimensional lists Each list group has an implicit length state variable named Length that always contains the current length of the list If this variable is undefined then the list currently has no members The specifica tion may define bounds on the length of the list in order to help the interface generator create a better rendering An exact size may be specified or a minimum and or maximum size may be specified List groups also maintain an implicit structure to keep track of one or more list selections The number of selections allowed may be defined in the specification one and many are the only options currently and the default is one if nothing is specified If a list allows only one selecti
249. esigned interfaces were often based upon dependency in formation For example suppose that an interface was being created for a shelf stereo system with a tape and CD player When the power is off a screen with only a power button widget would be shown because none of the other objects would be enabled When the power is on a screen is shown with many widgets because most of the ob jects are active when the power is on We might also expect this interface to have a panel whose widgets change based upon whether the tape or CD player is active The final representation of any interface can be described using a tree format It is not reasonable to include the tree representation of one interface in the specification of an appliance however because the tree may differ for different form factors For example the tree will be very deeply branched on a small screen WAP cellular phone interface whereas the tree will be broader for a desktop PC interface The specifica tion language defines a group tree that is deeply branched It is expected that this information could be used for small screen and large screen interfaces alike because presumably some of the branches could be ignored in a large interface It was important to use domain specific conventions as much as possible in the hand designed interfaces so that users could leverage their knowledge of previous systems to use the interfaces There is a need for some way to include this informat
250. euaseisde seine soleisoeecioeetto O ee ea seeteoeeestoea 70 xviii Figure 5 2 Figure 5 3 Figure 5 4 Figure 5 5 Figure 5 6 Figure 5 7 Figure 6 1 Figure 6 2 Figure 7 1 Figure 7 2 Figure 7 3 The label dictionary for the playback controls group of the VCR This dictionary contains two textual labels and some text to speech TEL FOL EAE OM vase ree saw sche A a ke EE thebeats ibis The group tree for the sample VCR specification eeeeeseeeseeeeneeeeeees An example of a list group used in the VCR specification to describe the list of timed recordings that may be specified by the user 0 An example of a common type of dependency equation specifying that a variable or command is not available if the appliance s power is we I a ene ea nee ene aS eT NL SEE UEC OR DACRE Arne ene we Seem TOR Po The ports section of the example VCR specification csseceeesseeeeeeeeees The description of the video tape source content flow from the example VCR specification Note that dependencies from the content groups that contain the source flow are ANDed with the source s UW Er HE penden Seanair aped eau seciey sng toed E NA VCRs used in the first study The Panasonic VCR in c was also uscd im the second CHIC roae rn a ste N eats An example mapping graph for the media control functions e g play stop and pause on four appliances The node counts indicate that the Panasonic VCR
251. ew interface Changes for functional consistency can be Chapter 8 Interface Generation 123 applied without disrupting the usability on unique functions but structural changes have broader impact with potentially unintended consequences The rules for ensuring structural consistency must be carefully applied to ensure that the usability of unique functions is not substantially diminished 8 4 1 Heuristics for Unique Functions In order to address the issue of usability for unique functions three heuristics for the struc tural consistency rules are followed when making changes that involve unique functions Heuristics were necessary for two reasons First the best approach for dealing with a unique function is to analyze how that function is related to the other functions of the appliance for which we have mappings If these relationships can be understood then the unique function can be moved or changed with the functions that are most strongly related to it Unfortu nately understanding how a unique function relates to other elements of a specification is difficult because the specification language contains very few cues to discover this kind of information The second reason for using heuristics is the lack of any formal studies that suggest specific methods for balancing the usability of unique functions with consistency In particular my central premise is that unique functions should be kept with the consistent interface rather than f
252. exType name ListGroupType gt lt xS sequence gt lt xs element name appliance reference type ApplianceReference minOccurs 0 maxOccurs 1 gt lt xs element name labels type LabelDictionary minOccurs 0 maxOccurs 1 gt lt xs element name active if type ActivelfType minOccurs 0 maxOccurs 1 gt lt xs choice minOccurs 1 maxOccurs unbounded gt lt xs choice minOccurs 0 maxOccurs 1 gt lt xs sequence gt lt xs element name min type StaticOrReference minOccurs 0 maxOccurs 1 gt lt xs element name max type StaticOrReference minOccurs 0 maxOccurs 1 gt lt xs sequence gt lt xs element name item count type xs integer gt lt xs choice gt lt xs element name selections type SelectionTypeType minOccurs 0 maxOccurs 1 gt lt xs element name Sortable minOccurs 0 maxOccurs 1 gt lt xs element name group type GroupType gt lt xs element name list group type ListGroupType gt lt xs element name union group type UnionGroupType gt lt xs element name State type StateType gt lt xs element name command type CommandType gt lt xs element name explanation type ExplanationType gt Ge e a a ek cei seh Seb 2k lt xs element name apply type type ObjectApplyType gt lt xs choice gt lt xs sequence gt lt xs attributeGroup ref ApplianceObjectAttribs gt lt xs complexType gt lt xs complexType na
253. f nodes with phrasal representations Each node is either actionable or incomplete An actionable node contains a phrase that when uttered will cause some device state change An incomplete node contains a phrase that re quires more information before it can be acted upon uttering the phrase causes the system to prompt the user with completion options derived from that node s children Grouping in formation is represented in the structure of the tree and influences exploration disambiguation and scope User control system System control system system voice User Options System control x10 control audiophase User control x10 System control x10 x10 voice User options System global off all lights on all lights off User control audiophase System control audiophase stereo voice User Options System stereo power x bass volume User stereo power System lt stereo is turned on gt User Options System stereo power x bass volume User More System mode tuner CD User stereo power is options System on off User tuner is options System tuner radio band tuner seek forward tuner seek re verse User Tuner System Tuner User what is radio station System radio station is 106 7 User radio station is 98 5 System lt changes station to 98 5 gt Figure 8 3 An example interaction using the speech interface to control a shelf stereo
254. f the interface which allows interfaces to be modified for each particular user The consistency feature studied here is one example and our second study showed that consistency can be beneficial to users Manufacturers may ob ject to consistency however because branding may be removed from interfaces and worse still branding from a competitor may be added in its place Our position is that branding which affects the usability of an appliance such as custom labels for certain functions or par ticular sets of steps needed to complete particular tasks is not good for the user and the consistency system should be allowed to modify them However branding marks such as company names logos etc should be preserved appropriately Support for branding marks and consistency of those marks is a feature that may be added to the PUC system in the fu ture An important question is what allows the PUC to generate interfaces that are better than the built in interfaces on the appliances And what would be needed to improve the built in in terfaces I believe PUC interfaces are better than the appliance interfaces for many reasons First the PUC does not use the same instance of a control for multiple functions All but tons sliders etc presented in a PUC interface are used for only one function In contrast most appliances overload multiple functions on their buttons For example both printer in terfaces provide a number of multi purpose buttons
255. face aggregation fea ture it does not provide support for automatically generating user interfaces or for combining user interfaces for multiple appliances into a single aggregate user interface I am aware of only two systems that have provided automatic combination of user interfaces One is the Crafter system Ponnekanti 2001 discussed earlier which can aggregate user interfaces for services that implement specific programmatic service interfaces Interface ageregators were implemented for specific combinations of service interfaces which prevent Crafter users from aggregating services in ways that the developers did not anticipate in ad vance The second system is the Gravity project Hall 2003 which provides a mechanism for auto matically constructing an application user interface based on a dynamic set of building blocks These blocks might change based on the user s context such as the location envi Chapter 2 Related Work 36 ronment task etc It seems that the focus with Gravity is on the framework issues such as discovering available components and repairing the interface when a currently visible build ing block is no longer accessible Currently there seems to be no user interface integration between the building blocks as each block is displayed as a separate panel within the Gravity application An important focus of the PUC work is integrating functions from each appli ance into a single interface
256. face shown here is the configura tion dialog box of a word processor Now we do the actual tasks You will work on them one by one and please complete them on your own as I cannot offer help You have 5 min to complete each task If you complete a task within the 5 min you can proceed to the next one But if at the end of the 5 min you haven t completed the task I ll prompt you and ask you to move on You are required to keep trying within the 5 minutes until you finish the task or 5 mins has passed I ll explicitly tell you when you re done with a task I ll say You re done If I do not say anything that means you re not done and you should keep trying Here s the procedure for each task Before each task I ll ask you to turn around so that you do not look at the screen of the handheld computer Then I ll give you a card with the task instruction on it You can take s i 7 s your time to read the instruction because the time you spend on reading won t count to wards the 5 minutes You can keep the card while you do the task you can put it on the table And you can look at it if necessary And while you re reading the instruction I ll reset the interface so that it will always look the same when you start a new task Appendix E Usability Study Instructions 306 Tasks Imagine a document has already been properly placed on the device Make two black and white copies of it Do NOT make one copy at a time Im
257. ffort and ability of the generator to produce complex interfaces A highly detailed model may allow an interface generator to produce highly complex interfaces and perhaps even support direct manipulation but building a model with the necessary amount of detail may require substantial effort In the design of the PUC system I have explicitly tried to optimize this trade off for appliance interfaces The preliminary user studies allowed me to design a specification language that contains sufficient information to generate a usable interface but is still concise and easy to use I could accomplish this because appliance interfaces are a sub set of all possible interfaces Optimizing the specification language for conciseness and ease of use resulted in not includ ing some features that have become commonplace in other model based systems Most significantly the PUC specification language does not include a task model For many appli ances task information is not necessary because most of the tasks have only one step For example play the tape or increase the volume involve pressing only one button or sliding one slider Even more complex tasks such as programming a timed recording on a VCR can be rendered understandably by the PUC because there are few steps and few dependencies between the steps Although some of today s appliances have interfaces which may require many complicated inter dependent steps to complete tasks most of th
258. ffrey Nichols Carnegie Mellon University 1998 2005 Altova GmbH Ln 3 Col 33 CAP NUM SCRL You may now start writing your specification If you see a screen that looks more like the image below then you should repeat the instruc tions above for turning off the automatic append mandatory children feature Appendix D Specification Authoring Study Instructions 274 ih Eile Edit XML DTD Schema Schemadesign XSL XQuery Authentic View Browser Tools IDIS SNAUTS a lt xml version 1 0 encoding UTF 8 gt lt spec xmins http www cs cmu edu pebbles puc xmins xsi http ia wv3 org 2001 KMLSchema instance xsitschemaLocation http Avwyy cs cmu edu pebbles puc jeffreynipucispecsipuc xsd gt slabels slabel lt Jabel lt abels groupings gt lt group name gt lt group name gt lt group name gt lt group name gt lt group name gt lt group name gt lt group name gt lt group name gt lt group name gt lt group name lt group name gt lt group name gt lt group name gt lt group name lt qroup E Altova XMLSpy Untitled1 xml Jog ADVANCED Window M amg A apply over apply type average binary boolean command completions available constant default value gt defined enumerated Attributes Entities m This file is not valid The content for x element group
259. for inter acting with a system of appliances the flow based interface that allows users to specify their high level tasks see Figure 1 7 and aggregate interfaces that combine functions from multi ple appliances to allow users to perform low level actions during a task see Figure 1 8 Chapter 1 Introduction 1 The PUC system produces four different kinds of aggregate user interfaces Active Flow Control interfaces provide access to the most common control functions of the currently ac tive flows such as volume and playbacks controls Figure 1 8 shows four example of this type of aggregate user interface Active Flow Setup interface provide access to less common con figuration features for the currently active flows such as brightness and contrast for a television or speaker balance for a receiver The General Setup aggregate interface contains setup and configuration options that do not relate to any flow such as the parental control settings present on some appliances Finally similar functionality from multiple appliances content Flow E dx 1 21 3 A content Flow 4x 1039 amp Navigate Add Flow StopAl Navigate Xia Xj A x mr go 4 DYD needs to be inserted into aap 0 the DYD player in order to activate this flow Press OK after inserting a DYD to continue activating this flow Split x OC 4 Split Q a b G d Figure 1 7 Several examples of the Flow Based Interface being used for var
260. ften provide the right function at the right time Aggregate interfaces based on usage or context are likely to adapt to the user over time how ever which may create problems for users if the interface changes in an unexpected way and previously available functions are either removed or hidden An area that seems promising is producing interface adaptation methods that are high level and user driven This might mean that users give broad or abstract descriptions about the functionality they desire to have in an interface the system produces the interface and then the system and user engage in an iterative design process to produce a product that is similar to what the user desires This process might be driven by a visualization that allows users to see their actual usage and make design suggestions based on it For example the user might perform a task and realize that they are likely to perform that task in the future The user would then open a usage visualization make a selection corresponding to the task just performed and then the system would produce an interface specifically for that selection I would also like to conduct a formal evaluation of the interfaces generated for systems of multiple appliances Some informal evaluations with prototypes of the flow based interface were performed during its design and those results were incorporated into the final design However there has been no evaluation of the aggregate interfaces and I cannot
261. fter a selected number of minutes Other set tings that occur across appliances however should not be merged For example it is usually wrong to set the channel of the VCRs and television to the same value simultaneously or to set all the devices to be powered on at the same time Even setup functions cannot always be combined depending on the particular function and how similar the functions are across ap pliance types For example the DVD player and the television both have a contrast setting but it would be inappropriate to set both of them simultaneously The Merged Functions AUI handles the small number of functions that are appropriate to combine see Figure 9 7 As with previous aggregates the knowledge base is used to identify similar functions across appliances that should be merged As future work it could be interesting to explore how the volume function might be merged across appliances using the flat volume technique developed by Baudisch et al Baudisch 2004 While this work has been shown to apply quite well to volume it is unclear whether it would be applicable for other kinds of settings such as brightness or contrast AS PUC 4 0 700 UH tf 225 3 Current Time 12 00 00 AM Sleep Timer Turn Off In ar 30 00 55 PUC Servers Devices Generation E Figure 9 7 The merged function AUI featuring the clock language and sleep timer functions on a single panel Chapter 9 Aggregating User Interfaces 159
262. function from the last function that was used If there is no usage information the PUC uses the algorithm from the first method to select the function The soft button is currently changed every time the screen changes or the user invokes a function and experimenting with other times is the subject of future work Unlike with the first approach back may be assigned to the soft button if the usage infor mation suggests that the next thing the user is likely to do is move up in the hierarchy Chapter 8 Interface Generation 142 No formal evaluation has been conducted of either of these methods The non adaptive ap proach has the advantage that users can memorize the function that is assigned to it as they use the interface but the priority information in our specification is not always reliable and does not always pick the right function For example the power button is picked on the main screen of our shelf stereo though in fact this is not a function that seems to be used very often The adaptive approach would seem to fix this problem because it relies on actual usage data but the cognitive load of keeping track of which function is currently assigned to the button seems too high It seems to usually be faster to remember the keypad shortcut for each function rather than to read the label on the soft button It may be that the adaptive approach becomes beneficial after using the interface for a significant period of time but there are
263. functions and state variables a service will have which allows a hand designed user interface to be created in ad vance on a control point Similar to HAVi UPnP does allow services to specify additional functions and state variables beyond those in the standardized set but it is not clear how a control point would accommodate these additional functions or variables in its user inter face UPnP provides a way around this by allowing a control point to also download a web page and control the specialized functions of the service using standard web protocols but the solution results in two different user interfaces being displayed on the same controller device Several UPnP products are available today including gateway router products streaming AV products and a Pan Tilt video camera from Axis Communications UPnP currently has Chapter 2 Related Work 21 standardized twelve different service descriptions and more devices are likely to appear on the market as the number of standardized service specifications grows 21 2 3 Digital Living Network Alliance The Digital Living Network Alliance DLNA is developing a series of standards based on the UPnP standards to improve the usability of network connected appliances throughout the home A typical DLNA usage scenario would be viewing some content from a networked PC such as music video or pictures on a stereo or television located elsewhere in the house Unlike for UPnP where the
264. fying data stores dialog for specifying control flow style rules for defining the interface elements layout and language of the user interfaces and style programs that instantiate the style rules at run time The layers are designed to make it easier for experts in different areas to collabo rate on the interface design For example programmers would implement the actions and Chapter 2 Related Work 30 style programs while interface designers would write the style rules and application experts would specify the dialog An important focus of ITS is making the dialog and style rules lay ers highly usable so that non technical experts could be first class participants Wiecha 1990 in the design process The design process was also very iterative rules were expected to be continually refined until an acceptable user interface was created Unlike many of the other model based interface systems ITS was used to create several commercial applications including all of the kiosks at the EXPO 92 worlds fair in Seville Spain 2 2 1 7 TRIDENT TRIDENT Vanderdonckt 1995 a model based system built around the same time as MASTERMIND combines the ideas of an automatic interface generator with an automated design assistant Like other systems TRIDENT uses a task model an application model and a presentation model as the basis for creating interfaces The TRIDENT system established a set of steps for its interface generation process de
265. g and will have some memory of using similar appliances in the past To reinforce that cue we have the following requirements to help increase users perceptions of consistency between interfaces R 3 Interfaces should use familiar labels for similar functions R 4 Interfaces should maintain a similar visual appearance The studies show that situations may arise where these requirements cannot be followed For example similar functions may have different representations that cannot use the same con trol Unique functions may also affect the order in which controls appear on the screen or affect the layout if they require larger controls or have wider labels In these situations there is a fundamental trade off between maintaining consistency to a previous interface and appro priately rendering all of the new appliances functions To address this problem the PUC could go against Grudin Grudin 1989 and favor consistency In this case the PUC would move the unique functions to a separate panel so that they could not affect the layout This solution has many negative consequences for usability however important functions could be moved to a non intuitive location and the extra features of a similar function might ap pear to not exist It seems better to favor usability in these situations and therefore the PUC has the following requirement R 5 Usability of unique functions is more important than consistency for similar functions I ha
266. g in an appliance interface typically with the stan _ dard looking glass icons This template is intended primarily for appliances with content that cannot be displayed on the PUC oo oF Busy Camera Computer Copy Protected Black Ink Connected Connected AA ls os ad az X su i Memory Card Network Paper Loaded Tape In Player Color Ink Inserted Connected Figure 7 5 Icons currently supported by the PUC status icons Smart Template Chapter 7 Handling Domain Specific and Conventional Knowledge 112 CHAPTER 8 Interface Generation This chapter discusses the interface generation algorithms used by the PUC system All the interface generators use a rule based approach that operates in two stages In the first stage an appliance specification is converted to an abstract user interface with no platform specific details The second stage converts the abstract user interface into a platform specific concrete user interface The most important contributions of the interface generators are the use of mutual exclusions to determine the structure for the abstract interface and algorithms that manipulate new interfaces to that they are consistent with previous interfaces that the user has seen gt This work in this chapter was originally in published in three different papers Jeffrey Nichols Brad A Myers Michael Higgins Joseph Hughes Thomas K Harris Roni Rosenfeld Mathilde Pignol Generating Remote Control Interfaces f
267. g the consistent interfaces as compared to either of the other interfaces Much of this effect for both appliances is due to four tasks 2 3 6 and 7 This was expected because the normal PUC interfaces for these appliances were already consistent for tasks 1 and 8 and thus did not benefit from any change in the consistent interfaces We had hoped to see consistency effects for the remain ing tasks but other factors seem to have affected tasks 4 and 5 The change made to ensure consistency for task 5 copying involved changing the place ment of the copy and cancel buttons on one screen see Figure 10 1 Apparently the visual search for the new button placement did not affect subjects speed compared to the normal PUC interfaces One change was made to ensure consistency for task 4 changing the fax error printing The function needed for this task is located with other fax configuration functions which are lo cated in different places on the two appliances in the fax mode on the HP and in the setup section of the Canon interface The change for consistency performed by the PUC is to move all the configuration functions to the location where the user originally saw them From observations of subjects actions it appeared that this manipulation worked in the studies Unfortunately the error reporting function was also different between the two appli ances in a way that the PUC s consistency system could not manipulate When using t
268. gation which are applied in the order dis cussed here Each looks for a particular set of features in the list hierarchy and makes some change to the list if that set of features is found Some of these rules make decisions about whether a particular function will be displayed as a list item or as a control on an editing pane During this discussion functions that can only be displayed in a list will be called list only items Functions that must be displayed on an editing pane will be called panel only items and all other functions will be called list or panel items Restrictions on how a function may be displayed are based on the AIO that was selected for that function in the abstract user interface The first two rules minimize the number of editing panes that may be accessed from the cur rent list Neither of these rules is applied if the current list contains only one panel only item PUC v3 0 Audiophase C Ti PUC 3 0 Simulated El lt Yi I PUC v3 0 Simulated ElvZ Til PUC v2 0 3 Windows MC TI Power On 1 Floor 3 1 Floor 1 Called Play Control Yolume Up 2 Up Off 2 Floor 2 Off Volume Down 3 Floor 3 Off X Bass Off 4 Enter 4 Floor 4 Off CD 5 Floor 5 Called Radio 6 Security Code Tape 7 Open Doors Holding AUX 8 Close Doors Off 9 Exit 1 2 3 4 J 6 7 8 Figure 8 12 Example screens from automatically generated Smartphone interfaces a The opening screen for controlling a shelf s
269. gnated state to the _ value contained in the message This message requests the appliance to invoke a command This may cause _ state changes as side effects _ This message requests the appliance to send a copy of its specification It will send this via the device spec message seuenenenaeceseesenseeesenseeeseneeeeseseeeeseseeeeseseeeeseseeseseseeeeseseeeeseseeeeseseeseseseeseseseeeeseseeseseseeseseseesesen seseseseeseseeeeseseeeeseseseeseseseeseseseeseseseeeeseseeseseseeeeseseeeeseseeseseseeseseseeseseseeseseseeseseseeseseeeeeeseseeseseseeseseeeeseseseeeeseseeeeseeeeseseseeeeseeeeseseeeeseseeeeseseeeeseseseeseseseeseseeeeseseeeeseseeeeseeeeeeseseeeeseseeeeseseeeeeeseeeeseeeeeesesee0esee0e0e00e0000050505005050580505088 full state request This message requests the appliance to send state change notification _ messages for every state that it has sesh EE AEE EEEE A A A AA EEE EEA E EEE EE EEA EAEE E AEEA AA E A A A A E A E ees A A E AA A A A E A AA AA A E E A L AEEA state value request This message requests the appliance to send a binary state change notification message containing binary data for a particular state server information _ This message is sent by controllers to get the list of appliances connected to request _ a particular server Table 4 4 Messages that may be sent by the appliance to a controller device Name _ Description state change _ This message is sent whenever the appliance changes state The state name
270. gs the television to the empty sink space on the flow At this point the green play button will become enabled because this configuration corresponds to a valid flow Note that the television speakers can be a sink for audio con tent The asterisk above the arrow on the right side indicates that the flow based interface will infer the type of content to route to the television based on the specified sinks The user now wishes to add the receiver as an additional sink To do so the Split button is pressed underneath the arrow on the left This causes the flow to be split into two arrows and a new empty sink space to be created see Figure 9 3c To add the receiver s speakers as a sink the user can then drag the receiver to the empty space In this scenario the PUC is able to automatically infer that the TV speakers should not be used because an audio sink was added to the flow If the user wanted audio to come from both sets of speakers this wouldbe EF content Flow E 4x 6 07 amp EF content Flow 4x 6 08 3 EF content Flow E 4x 6 09 amp A content Flow 4x 6 12 amp Xj A Figure 9 3 An example of using the Flow Based Interface to configure a DVD player to play video through a television with the audio routed through the stereo s speakers Chapter 9 Aggregating User Interfaces 153 indicated by tapping the content type icon next to the television and selecting the au dio video content type The user
271. h condition is of the most interest because the second block is influenced differently by the subjects experiences in the first block 10 4 1 Results Figure 10 3 shows the average completion time for each of the tasks on each appliance com paring the Built In condition with the other two conditions combined which I will refer to as the PUC condition Note that data from the AutoGen and Consistent AutoGen condi Chapter 10 Usability Evaluation 168 tions can be combined here because the same interfaces are used in the first block of both conditions To compare completion times and failures in the first block we conducted sev eral one way analyses of variance ANOVAs For all of these analyses n 8 in the Built In condition and n 16 in the PUC condition Table 10 1 shows the data in more detail with analyses comparing user performance for each task On the HP appliance subjects were significantly faster for total task completion time using the PUC interface Fi 22 12 11 p lt 0 002 completing all of the tasks in less than half the time M 5 54 for the PUC interface vs M 13 12 for the built in interface Subjects also failed significantly less often using the PUC interface Fi 22 5 69 p lt 0 03 with a fifth as many failures using the PUC interface as compared to the built in interface 2 total failures for all users vs 9 06 00 30 00 05 00 25 00 04 00 20 00 03 00 Time taken Time taken D o 02
272. h fidelity designs because of the availability of Microsoft s eMbedded Visual Basic tool which made the implementation relatively painless Because of the complexity of both appliances the Palm prototypes required approximately 20 hours to create and were improved with heuristic analysis prior to the user study The PocketPC interfaces required more than fifty hours of design and implementation effort to create The PocketPC interfaces were improved through a combination of heuristic analysis techniques and think aloud studies with pilot users There were two goals with the design of these interfaces Functional completeness with the appliances A goal of the automatically generated in terfaces is to deal with the full complexity of the appliances and it was important that the hand designed interfaces would help in understanding how to address this goal Consistency with the conventions of the platform An anticipated advantage of a UI device is that interfaces would be easier to use because the user would already have some familiarity with the platform and could leverage this knowledge to better use the appli ance interfaces Although there was no screening to ensure that our user study subjects were experts with the Palm or PocketPC platforms an effort was made to ensure plat form consistency in order to understand how difficult this goal would be for the automatic interface generators The study of the low fidelity interfaces showed
273. h result interface Some tasks might be performed dramatically faster using this approach assuming the desired con trol appeared near the top of the search results There are a number of issues with this approach however The biggest is what if the search rarely returns the result that the user is looking for In this case the user may believe the function does not exist This concern does not worry me however because the number of functions in an interface is much lower than for example the number of pages on the web The interface search can be liberal about what it returns in its results and there is a fairly good chance that the desired function will appear Another worry is whether users can easily specify a search query using a handheld device where text entry is difficult and may be more time consuming than just searching the interface This is a definite concern and I hope that it can be addressed by creating query interfaces that can rely in part on selection from a list of appliance specific words Another approach would be to leverage the consistency system For example the user might open a previous interface indicate the function they are looking for and then have the search engine return results in the current interface that are similar Fi nally another worry is the situation in which the user searches for a function that the appliance does not have For the consistency based search this problem is easy to address because the
274. hat are contained lt apply over gt lt apply type gt Allows the re use of an existing type block within a specification lt bainary gt block gt Specifies that a channel should be blocked by an appliance that is pass lt boolean gt lt channel gt Used in the content flow descriptions to define a particular channel lt command gt Defines a command appliance object lt completions lt constant gt Defines a constant value in any place where a reference would also be lt content flow gt Defines the section that describes the internal content flows of an ap lt content group gt Groups two or more content flows for the purpose of specifying de Appendix B Specification Language Reference Element Index Description of objects Defines an AND relation with all dependencies that are contained within unless they are grouped within a logical operation block such as lt or gt within Applies dependency relations to a list of data Binary type encompasses any kind of binary data including images and sounds A Smart Template must be used to appropriately inter pret and render the binary data _ing through a multi channel content stream Boolean type takes on true or false values _ that content may flow over _ Specifies that this state variable has completions available from the available gt _ server accepted pliance pendencies that apply to both 216 Elemen
275. hat defines when that port is active for that flow Thus to activate a particular port with a particular flow both dependency formulas must be satisfied Channels are an important concept in content flow specifications When a pass through or sink receives a multi channel input a channel variable may be specified from the appliance that specifies the particular channel being tuned The language can also specify that one channel of a multi channel stream is being replaced by the appliance which is used by ex Chapter 5 Specification Language 77 ample VCR specification to describe that the output of the tape source can appear on chan nel 3 see Figure 5 7 The set of variables commands and groups that modify the behavior of the flow is impor tant for the generation of aggregate user interfaces For example the set allows the tint brightness and contrast functions of a television to be associated with the screen sink lt content group gt lt active if gt lt equals state Base Power gt lt constant value true gt lt equals gt lt active if gt lt content group gt lt active if gt lt equals state Base PoweredItems Controls TV VCR gt lt constant value true gt lt equals gt lt active if gt lt source name Tape content type av gt lt active if gt lt equals state Base PoweredItems Status TapeIn gt lt constant value true gt lt equals gt lt not gt lt or gt lt equ
276. have approximately the same meaning Thus the interface generator can use any label within a label dictionary interchangeably For example this allows a graphical inter Chapter 5 Specification Language 71 lt labels gt lt label gt Play Controls lt label gt lt label gt Play Mode lt label gt lt text to speech text Play Mode recording playmode au gt lt labels gt Figure 5 2 The label dictionary for the playback controls group of the VCR This dictionary contains two textual labels and some text to speech information face generator to use a longer more precise label if there is sufficient screen space but still have a reasonable label to use if space is tight Figure 5 2 shows the label dictionary for the Play Controls group of the VCR which has two textual labels and a text to speech label 5 2 1 4 Group Tree Interfaces are always more intuitive when similar elements are grouped close together and different elements are kept far apart Without grouping information the start time for a timed recording might be placed next to real time control for the current channel creating an unusable interface We avoid this by explicitly specifying grouping information using a hierarchical group tree We specify the group tree as an n ary tree that has a state variable or command at every leaf node see Figure 5 3 State variables and commands may be present at any level in the tree Each branching node is a group
277. he HP interface made to be consistent with the Canon interface users needed time to understand how the functions were different before they could make the correct change When using the Canon interface consistent with the HP interface the interface generator made the unfortu Chapter 10 Usability Evaluation 173 nate choice of placing the needed functions in a dialog box accessible by pressing a button The button to open the dialog was placed next to several other buttons which distracted subjects from the button they needed to find For tasks 2 and 6 we see a significant benefit for consistency for both appliances Tasks 3 and 7 both have a marginally significant benefit for consistency on just one appliance task 3 on the HP and task 7 on the Canon Similar to task 4 both tasks 3 and 7 are slightly different on the two appliances in ways that the PUC s consistency system cannot change We believe this means that subjects were not able to leverage all of their previous knowledge and had to spend some of their time thinking about how the appliances worked thus slowing them down It is important to note that there are no situations where the PUC s consistency algorithms make the interface significantly worse for users even for task 4 on the Canon interface gen erated to be consistent with the HP The consistency system is able to provide benefits when there are similarities between the appliances and it does not hurt the user when t
278. he extra depth in the abstract user interface Second the concrete interface generation may have used differ ent controls for organization or placed panels in different orientations than in the previous user interface Three concrete consistency rules for the PocketPC and desktop have been im Chapter 8 Interface Generation 143 plemented to address these issues There are currently no concrete consistency rules for the Smartphone interface generator because at least so far there has not been a need for them The difficulty for implementing these rules is to determine which panels correspond between the new interface and the previous interface The current approach for determining corre spondence is to link each panel to the highest level group it contains within the abstract user interface A panel in the new interface corresponds to a panel in the previous interface if the groups of both panels have a mapping between them This is not a perfect algorithm be cause sidebars for example often mix controls from several groups at different levels within the specification however it seems to be effective for the cases that have been tried to date The first concrete consistency rule addresses the re ordering of rows within panels When two panels are found to correspond between the previous and new interface the rows from both panels are converted into block lists the new block list is re ordered based on the previ ous list and then th
279. he features that a high quality remote control interface will have Nichols 2002a Nichols 2003 and then apply them in the interface generator software Nichols 2002b 3 1 Hand Designed User Interfaces Two common appliances were chosen as the focus of our hand designed interfaces the Aiwa CX NMT70 shelf stereo with its remote control see Figure 3 1a and the AT amp T 1825 tele The work in this chapter was originally pubished in Jeffrey Nichols and Brad A Myers Studying the Use of Handhelds to Control Smart Appliances in Proceedings of the International Workshop on Smart Appliances and Wearable Computing IWSAWC Providence RI May 19 22 2003 pp 274 279 and Jeffrey Nichols Brad A Myers Thomas K Harris Roni Rosenfeld Michael Higgins and Joseph Hughes Requirements for Automatically Generating Multi Modal Interfaces for Complex Appliances in Proceedings of the IEEE Fourth International Conference on Multimodal Interfaces ICMI Pitts burgh PA October 14 16 2002 pp 377 382 39 Figure 3 1 a The Aiwa CX NMT170 shelf stereo with its remote control and b the AT amp T 1825 office telephone digital answering machine used in our studies Mailbox FE 2 3 4 411 Vol Mode w Single Dise vo Wed 3 20 PT Bnei Cl anii 8 N Today 2 01 PM N Today 3 36 PM be CEA Figure 3 2 Paper prototypes of the phone a b and stereo c d interfaces for the Palm EJAT amp T 1825 Telephone 3 34a FRE
280. he final artifact For example imagine a panel containing several controls During the design process the designer may have chosen to remove a bounding rectangle that grouped a subset of the controls on the panel While this grouping may not have been necessary in this particular concrete interface it might be useful for the abstract model to contain information about this extra grouping for use in later analyses This suggests that it may be interesting to explore automatically inferring abstract interface models from the interface design process An abstract model of a hand designed interface might also enable an automated system to modify hand designed interfaces to support features that the PUC explored for automatically generated interfaces like aggregation and consistency There are a number of challenges to Chapter 11 Conclusion 193 building a system that automatically modifies existing interfaces First it will be important to understand what information is needed about a hand designed interface in order to make appropriate modifications and how this differs from the information needed to automatically generate a user interface It is possible that the information needed to modify an interface is smaller than that needed to automatically generate an interface because a concrete interface is already available to work from For example label information would not be needed in the model because it is already contained in the interface
281. he home such as a television or personal computer and uses two part design that would be difficult to adapt to today s mobile phones The task based portions of the user interface are automatically generated from task models but the direct manipulation portions are currently hand designed The unique as pect of DiamondHelp is its combination of two different interface styles which allows users to choose how to interact with the appliance while benefiting from structured support 2 1 3 8 Roadie Chapter 2 Related Work 26 The Roadie system Lieberman 2006 provides a goal oriented user interface for consumer electronics that may combine features of multiple appliances Like the PUC s flow based in terface see Chapter 9 Roadie uses a planning algorithm to automatically configure appliances to match user goals Unlike the PUC Roadie uses a database of commonsense knowledge to find and understand possible user goals within the system The user can specify the action they wish to perform using natural language and then Roadie will attempt to in terpret this action using its database and create a plan Because the possible actions are restricted to the contents of the commonsense database Roadie may not be able to support uncommon actions such as those related to an uncommon configuration of appliances or to a new class of appliance that has just been added to the system The PUC in contrast is able to acquire a model of the system from
282. he interfaces it produces but this generator is not discussed in detail because I was only peripherally involved in its con struction 8 1 1 PocketPC and Desktop The PocketPC interface generator is the PUC s primary development platform and most fea tures were implemented first on PocketPC before being migrated to the other platforms The platform is a set of hardware requirements for OEMs and a Windows CE based operating system that runs on top of compliant hardware The hardware platform requires a 240x320 or 480x640 screen with touch sensitivity A 4 way directional pad is also available along with four physical application buttons see Figure 8 la Our current interface generator focuses on the touchscreen and only the media controls Smart Template makes use of the physical buttons see Chapter 7 The PocketPC interface generator produces interfaces that appear much like other PocketPC applications so that PocketPC owners can leverage their existing knowledge to use the inter faces The generator attempts to avoid scrollbars whenever possible which causes many interfaces to include overlapping panels manipulated using tabs or another organizing con trol Dialog boxes are also created in certain situations These techniques allow all of the complex functionality of an appliance to be accessible through one user interface The Pock etPC has menus at the bottom of its screen but these are reserved for controlling the interface gener
283. here are dif ferences A question to ask is whether the benefits that appear to be from consistency could be due to some other factor in the generation process I do not believe this is likely because the rules added for consistent interface generation only make changes to the new interface based on differences with a previous interface that the user has seen These rules do not perform other modifications that might improve the user interface independent of consistency 10 6 Discussion The studies presented here do have some limitations I used only one type of appliance all in one printers and only tested two instances of this type As discussed earlier I believe that the all in one printers we chose are representative of complex appliances as a whole They also require the use of many of the PUC specification language s most advanced features such as lists and Smart Templates see Chapter 7 Although only two all in one printers were used they were carefully chosen to both be complex and representative of different common interface styles We also chose the HP in part because it had in my estimation the best interface of any all in one printer available These two studies together have shown that the PUC can generate interfaces that exceed the usability of the manufacturers own interfaces Using automatic generation to create appli Chapter 10 Usability Evaluation 174 ance interfaces allows flexibility in the design o
284. hi DVCR Finally you may optionally write an additional specification for an appliance that you own and have an owner s manual for You will author these specifications in an XML based language that we have developed 1 General Concepts for Describing Appliance Functionality This section describes the general concepts about our method for describing appliance func tionality These concepts will be applicable regardless of which authoring method you choose to use for this study The four most important concepts in our language are e The functions of an appliance can be represented by either state variables or state less commands State variables have specific type information that describes how they can be manipulated by the interface Commands and states are collectively called appli ance objects e The structure of the prototype interfaces were often based upon dependency informa tion For example suppose that an interface was being created for a shelf stereo Appendix D Specification Authoring Study Instructions 264 system with a tape and CD player When the power is off a screen with only a power button widget would be shown because none of the other objects would be enabled When the power is on a screen is shown with many widgets because most of the ob jects are active when the power is on We might also expect this interface to have a panel whose widgets change based upon whether the tape or CD player is active
285. his command is activated the previous track should be selected NOTE Other playback modes may be possible and should probably be considered These include e Reverse Play e Fast Forward while Playing vs while Stopped e Rewind same thing e Different speeds of fast forward and rewind e Play New for answering machines time absolute Overview This template specifies absolute time i e time of day This includes the time value along with parameters of the time such as the time zone 12 24 hour mode etc Contents This template supports both the single and multi state specification methods The single state method may only represent an absolute time value in 24 hour units This value may be rendered as a 12 hour time value with AM PM depending on the configuration of the user s device The multi state method may also include parameters for the time value as described below The single state form may have one of four primitive types Integer The value of the state variable should contain the number of seconds since midnight No increment is allowed String Appendix D Specification Authoring Study Instructions 291 The value of the state variable must be of the form hh mm ss or hh mm where h hours m minutes and s seconds Note that unlike the time duration tem plate digits may be omitted if they are not significant In other words 1 45 56 has the same meaning as 01 45 56 An arbitrary number of digits may be
286. hols Brandon Rothrock and Brad A Myers UNI FORM Automatically Generating Consistent Remote Control Interfaces in Proceedings of the Conference on Human Factors in Computing Systems CHI Montreal Quebec Canada April 22 27 2006 pp 611 620 83 proposed that consistency does not have one single definition but suggests that it can be de fined within a framework with two types of consistency at three different levels The two types of consistency are internal consistency within an application and external consistency between multiple applications The three different levels are conceptual communication and physical Grudin Grudin 1989 adds another type of consistency called analogical which is meant to describe consistency with the world outside of the computer system Each of these definitions gets close to the concept of consistency but none helps us opera tionalize consistency within an automatic system Fortunately there is a fair amount of work examining the factors that lead to consistency or lack thereof Most of this work breaks the user interface down into a formal model usually a set of production rules which describes the actions that users must take in order to accomplish their tasks Barnard Barnard 1981 found that positional consistency is important within a command p y p language meaning that common parameters between functions should always be placed in guag 8 y p the same position This wo
287. how to create better interfaces for everyday appliances Physical Today you ll be using two multi function printers to complete some tasks You ll be doing a set of 8 tasks using one of the printers and then you ll do the same set of tasks using the other printer Puc gt puc amp puc gt Uniform Today you ll be controlling two multi function printers to complete some tasks You ll be doing a set of 8 tasks by controlling one of the printers and then you ll do the same set of tasks by controlling the other printer You won t be directly interacting with the printers but you ll be controlling them using a handheld computer Q Used a handheld computer before Multi function printer Copier Fax machine The multi function printers that you re going to use can print copy fax and scan in black and white as well as in color This study will take about 90 minutes Pll record audio and video and the time that you take to complete the tasks However all this information will remain anonymous Before we start I would like you to sign a consent form and complete a questionnaire Appendix E Usability Study Instructions 305 Puc gt puc amp puc gt Uniform Now we re going to do two exercises They are just to famil iarize you with the interface of the handheld So you can take your time The first exercise is a Pocket PC tutorial this handheld computer is called Pocket PC For the second one I would like you to imagine the inter
288. http www speech cs cmu edu de Baar 1992 D J M J de Baar Foley J D Mullet K E Coupling Application Design and User Interface Design Conference on Human Factors and Computing Systems Mon terey California ACM Press 1992 pp 259 260 315 Denis 2003 Charles Denis and Laurent Karsenty Inter usability of multi device systems A conceptual framework Multiple User Interfaces A Seffah and H Javahery Eds 2003 John Wiley amp Sons pp 373 385 DLNA 2006 DLNA Digital Living Network Alliance Home Page 2006 http www dlna org Doan 2001 AnHai Doan Pedro Domingos and Alon Halevy Reconciling Schemas of Disparate Data Sources A Machine Learning Approach SIGMOD 2001 pp 509 520 Eisenstein 2001 Jacob Eisenstein Jean Vanderdonckt and Angel R Puerta Applying model based techniques to the development of UIs for mobile computers Intelligent User Interfaces Santa Fe 2001 pp 69 76 Eustice 1999 K F Eustice T J Lehman A Morales M C Munson S Edlund and M Guillen A Universal Information Appliance IBM Systems Journal October 1999 38 4 pp 575 601 http www research ibm com journal sj 384 eustice html Florins 2004 Murielle Florins Daniella G Trevisan and Jean Vanderdonckt The Conti nuity Property in Mixed Reality and Multiplatform Systems A Comparative Study CADUI 04 Funchal Portugal January 13 16 2004 pp 323 334 Foltz 2001 Mark
289. iance e The interaction layer creates a user interface based on the common data structure and allows the user to manipulate the template s data When changes are made by the user the template data structure is updated and the data translation layer is notified These layers separate the instantiation of a template in a specification from the generation of its interface The template data structure is the sole point of communication between the layers Using this approach is particularly beneficial in several situations e Templates that represent a single data type such as date or time may have many dif ferent formats in the specification In this situation implementing the data conversion is complex but the interface generation is basically the same no matter what the underlying data format is Separating the data conversion simplifies the in terface generation because the generation only needs deal with a single data format e Merged templates are easier to implement because the data conversion layers from the constituent templates can be re used to create a merged template data structure Chapter 7 Handling Domain Specific and Conventional Knowledge 107 This reduces the cost of implementing a merged template to the sub task of creating interface generation algorithms e If code can be shared across platforms as it can in my implementations for the Pock etPC and Smartphone then the data conversion layers can also be sh
290. idea to our PUC which they call a universal interactor that can adapt itself to control many devices Their approach uses two user inter Chapter 2 Related Work 23 face solutions hand designed interfaces implemented in Tcl Tk and interfaces generated from a language they developed called the Interface Definition Language IDL IDL fea tures a hierarchy of interface elements each with basic data types and supports a layer of indirection that might allow for example a light control panel to remap its switch to differ ent physical lights as the user moves between different rooms Unlike the PUC work this work seems to focus more on the system and infrastructure issues than the user interface It is not clear whether IDL could be used to describe a complex appliance and it seems that manually designed interfaces were typically used rather than those generated from an IDL description 2 1 3 2 IBM Universal Information Appliance An IBM project Eustice 1999 describes a Universal Information Appliance UIA that might be implemented on a PDA The UIA uses an XML based Mobile Document Appli ance Langauge MoDAL from which it creates a user interface panel for accessing information A MoDAL description is not abstract however as it specifies the type of widget the location and the size for each user interface element 2 1 3 3 ICrafter The Stanford Crafter Ponnekanti 2001 is a framework for distributing and co
291. ienced so it is possible for the PUC to generate consistently poor interfaces if users start with poor interfaces In order to address this the PUC must handle the following requirement R 7 Users must be able to choose to which appliance consistency is ensured This requirement affects the PUC s consistent interface generation at a fundamental level because its data structures must include information about each of the possible consistency choices and it must have some means to keep track of the current choice To support this I developed the mapping graph structure which is used by all of the PUC s consistency algo rithms and discussed in the next section Although the architecture supports the ability for users to choose consistency an interactive interface to support user choice is still the subject of future work 6 4 Understanding and Finding Similarities between Specifications In order to support consistency within the PUC the interface generator must understand how a new specification is similar to previous specifications for which interfaces have already been generated 6 4 1 Knowledge Base The knowledge base is an important piece of the PUC s architecture for supporting consis tency It stores previously generated specifications mappings between specifications and information about the interface designs built from those specifications The most important elements of the knowledge base are the mappings between functi
292. ifferent assignment rules Reisner 1990 Grudin also notes that ease of learning and ease of use may require conflicting designs which prevents a consistent design from being used between interfaces optimized for different pur poses Personal consistency helps here because the PUC can generate interfaces that suit each user s particular situation either a novice just learning to use an appliance s functions or an expert that is familiar with many appliances of the same type What happens when the novice user has learned the interface and wishes to become an expert though In this case the flexibility of automatically generating interfaces allows the PUC to regenerate the interface with the design for experts in place of the design for novices 6 1 1 Evaluating Consistency The process of evaluating the consistency of an interface can be time consuming and diffi cult A number of systems have been created to help automate the collection of usage data that can help in the evaluation of consistency Ivory 2001 Sherlock and GLEAN have spe cific features for automatically evaluating the consistency of an interface Sherlock Mahajan 1997 uses a heuristic approach to evaluate task independent qualities of user interfaces for consistency This includes looking at visual properties such as widget sizes and font choices and terminology issues such as inconsistent abbreviations and spelling errors The system is capable of evaluating dialog
293. ification For UPnP the descriptions provided functional information at the right level of abstraction but there was no grouping information and few human readable labels PUC specifications could be created from this information but the resulting interfaces would be poor at best Table 4 1 Appliance adaptors built by the PUC research team Home Entertainment Appliances Desktop Applications _ Audiophase Shelf Stereo Microsoft PowerPoint Sony Camcorder Microsoft Windows Media Player 9 Vicheng Couttols PUC Photo Browser Intel UPnP Light Other Lutron RadioRA Lighting Axis UPnP Pan Tilt Zoom Surveilance Camera X10 Lighting Table 4 2 Appliance simulators built by the PUC research team Home Entertainment Appliances Office Appliances Panasonic PV V4525S VCR Canon PIXMA 750 All In One Printer GMC 2003 Yukon Denali Systems HP Photosmart 2610 All In One Printer Driver Information Console Other Climate Control System Simulated Elevator Navigation System at PUC Debug Server v2 1 101 oz aD Phitips DVD DVP 642 Fie Reload Convert Statistics Device Status Fort Visible Active InFocus Television Inactive Sony Receiver Inactive Panasonic YCA Inactive Panasonic YCA Inactive Philips WWD DWPE42 Inactive Preview Mode Title Chapter Figure 4 2 The interface for the PUC Debug Server a The main window showing the appliances currently being simulated by this server b The interface for simulating a Philips DVD
294. ing 1 gt 1 v 2 Sided Copying 1 gt 1 v Brightness Ho mM p Basic Controls Page Control i i Basic Controls Page Control PUC Servers Devices Generation Ej PUC Servers Devices Generation Ej PUC Servers Devices Generation VEFE 27 a Complex Copier Without b Simple Copier Without c Complex Copier d Simple Copier Consistency Consistency Consistent with Consistent with Simple Copier Complex Copier Figure 1 5 Copier interfaces generated with and without consistency on the PocketPC platform PUC v3 0 33 Copier 2 m Yy PUC 3 0 33 Copier 2 Gai Yy 1 Copy 1 Start 2 Cancel 2 Stop 3 Paper Select Letter 8 3 Tray Letter 4 Main 4 of Copies 4 5 Fit To Page 5 Auto Zoom 6 Image Quality 6 Image Quality 7 Page Control 8 Confirm Sort Choice Quantity Image Type Text Brightness INC Finisher Off te 2 Sided Copying 4 gt 1 gt 1 41 gt Halftone Book Copy off O Two Page Separation 2 Sided Copying 1 gt 1 a Complex Copier Without Consistency b Simple Copier Without Consistency PUC 3 0 33 Copier 3 Gi Yy Start PUC v3 0 33 Copier m Yx 1 Copy Stop 2 Cancel Tray Letter 3 Paper Select Letter 8 of Copies 4 1 2 3 4 4 Main 5 Auto Zoom 6 7 8 5 Fit To Page 6 Image Quality 7 Sorting Image Quality Page Control Confirm Sort Choice
295. ing Systems Monterey CA May 1992 1992 pp 507 515 Szekely 1996 Pedro Szekely Retrospective and Challenges for Model Based Interface Development 2nd International Workshop on Computer Aided Design of User Interfaces Namur Namur University Press June 5 7 1996 pp 1 27 Tomko 2004 Stefanie Tomko and Roni Rosenfeld Speech Graffiti vs Natural Language Assessing the User Experience HLT NAACL Boston MA 2004 UPnP 2005 UPnP Universal Plug and Play Forum 2005 2005 http www upnp org Vander Zanden 1990 Brad Vander Zanden and Brad A Myers Automatic Look and Feel Independent Dialog Creation for Graphical User Interfaces Human Factors in Computing Systems Seattle WA Apr 1990 pp 27 34 Vanderdonckt 1995 J Vanderdonckt Knowledge Based Systems for Automated User Interface Generation the TRIDENT Experience Technical Report RP 95 010 Na mut Facultes Universitaires Notre Dame de la Paix Institut d Informatique 1995 Vanderdonckt 1999 J Vanderdonckt Advice Giving Systems for Selecting Interaction Objects User Interfaces to Data Intensive Systems 1999 pp 152 157 W3C 2006 W3C Resource Description Framework RDF 2006 http www w3 or Ward 1990 W Ward The CMU Air Travel Information Service Understanding Sponta neous Speech DARPA Speech and Natural Language Workshop 1990 Bibliography 321 Wiecha 1990 Charles Wiecha William Bennett Stephen Boi
296. ing a high end Mit subishi DVCR a Samsung DVD VCR combo player two all in one printers from HP and Canon and the navigation system from a GMC vehicle Table 5 2 shows some statistics for the specifications that have been written so far The table shows that PUC specifications on average are quite complex particularly the GM navigation system specification which is nearly twice as complex as any other specification All of these specifications cover all of the functions of their appliance giving us confidence that the language is capable of representing both the most common and most obscure functions of any appliance Although I cannot conclusively prove the language s completeness without writing a specifi cation for every possible appliance I believe there is sufficient evidence from the existing specifications to suggest that the language may be complete At the lowest level of description we have seen in all of the specifications that state variables and commands are adequate for describing the functional elements of an appliance At higher levels the hierarchical group tree has been sufficient for representing organization and the dependency formulas have been descriptive enough to specify behavior while being re strictive enough to facilitate analysis that can be applied in the generated interfaces The main difficulty in the language design came from supporting complex data structures particularly the lists that are found on ma
297. ing information for a system of appliances can be combined with the port and content flow information from each appliance to build a model of content flow through the entire system This content flow model is very useful for generating interfaces that aggregate func tions from multiple appliances The generation of aggregate user interfaces is described in detail in Chapter 9 5 2 2 1 Ports The input and output ports of an appliance define that appliance s relationship with the out side world In order to match the users intuitive understanding of ports specification authors are encouraged to create a port for each of the physical plugs that exist on the outside of an appliance Future tools could then use this information to help users correctly wire Chapter 5 Specification Language 76 lt ports gt lt inputs gt lt port name VHF UHF Antenna content type multi channel av physical type coax gt lt inputs gt lt outputs gt lt port name VHF UHF Antenna content type multi channel av physical type coax gt lt port group name Output content type av gt lt port name Video content type video physical type RCA gt lt port group name Audio content type component audio gt lt port name Right content type component audio right physical type RCA gt lt port name Left content type component audio left physical type RCA gt lt port group gt lt port group gt lt outputs gt lt po
298. ion forms that allow the service pro vider to supply hints about how the generated interface should appear This gives the service providers control over the generated interfaces and allows them to include brand marks and interactions 2 1 3 6 Analyses of Actual Remote Control Usage Omojokun et al Omojokun 2005 have collected usage data for consumer electronics in real home settings and applied a machine learning approach to discover the core set of func tionality that is used by a particular user and to cluster these functions into task groups They compared their automatic results to their users intuition and discovered that neither ap proach was sufficient for building a complete user interface In the future they propose to explore a mixed approach that combines automatic and user oriented approaches to design user interfaces My approach differs because the PUC interfaces include the full functionality for each appliance rather than a subset containing the most commonly used functions In the future I am interested in applying Omojokun s work to optimize the organization of the PUC s user interfaces to favor commonly used functions while still including the remaining functions 2 1 3 7 DiamondHelp DiamondHelp Rich 2005 combines a task based dialog interface with a direct manipula tion interface to bring usability and consistency to consumer electronics interfaces The interface is designed for display on a large screen in t
299. ion in the appliance specifications and the Smart Templates technique was developed to address this problem Each of these items is described in detail below Chapter 5 Specification Language 68 FAko Appliance Objects Three types of appliance objects are supported in the specification language States Variables that represent data that is stored within the appliance Examples might be the radio station on a stereo the number of rings until an answering ma chine picks up the time that an alarm clock is set for and the channel on a VCR Each variable has a type and the UI generator assumes that the value of a state may be changed to any value within that type at any time that the state is enabled For example the channel state variable see Figure 5 1a has an integer type and the in terface generator can infer the change channel function because it knows how to manipulate an integer type It is possible for state variables to be undefined i e with out any value This commonly happens just after an interface is generated before any values have been assigned but could occur for other reasons Commands Commands represent any function of an appliance that cannot be de scribed by variables alone They may be used in situations where invoking the command caused an unknown change to a known state variable such as the seek function on a radio or in situations where the state variable is not known manufac turer choice
300. ion of speech interfaces Their approach to speech interface design including em phasis on a fixed language and cross application skill transference is quite similar to the Universal Speech Interface approach as it is derived from a joint philosophy Rosenfeld 2001 Xweb s language design allows users to directly traverse and manipulate tree structures by speech however they report that this is a hard concept for users to grasp Olsen Jr 2000 The interfaces designed for the PUC using the Universal Speech Interface design differ by trying to stay closer to the way people might talk about the task itself and is somewhat closer to naturally generated speech 2 1 3 5 Ubiquitous Interactor The Ubiquitous Interactor UBI system Nylander 2004 is also working to separate presen tation from functionality for services Services in UBI are described using interaction acts somewhat like the abstract interaction objects used by the PUC interface generator which Chapter 2 Related Work 25 describe the interaction the user should have without providing any information about how the interaction should be presented The general description of the service can be augmented with service and device specific hints that are provided in a customization form The UBI interface generator combines the information from the interaction act and the customization forms to produce a final user interface The unique feature of UBI comes from its customizat
301. ion value none gt lt xs enumeration value parent gt lt xs enumeration value all gt lt xs restriction gt lt xs simpleType gt lt xs simpleType name AccessType gt lt xs restriction base xs string gt lt xs enumeration value read only gt lt xs enumeration value read write gt lt xs restriction gt lt xs simpleType gt lt xs simpleType name TypeNameType gt lt xs restriction base xs string gt lt xs enumeration value one gt lt xs enumeration value multiple gt lt xs restriction gt lt xs simpleType gt lt xs simpleType name PriorityType gt lt xs restriction base xs integer gt lt xs minInclusive value 0 gt lt xs maxInclusive value 10 gt lt xs restriction gt lt xs simpleType gt lt xs simpleType name TrueIfType gt lt xs restriction base xs string gt lt xs enumeration value any gt lt xs enumeration value all gt lt xs enumeration value none gt lt xs restriction gt lt xs simpleType gt lt xs simpleType name ItemsType gt lt xs restriction base xs string gt lt xs enumeration value all gt lt xs enumeration value selected gt lt xs restriction gt lt xs simpleType gt lt xs simpleType name ContentTypeAttrib gt lt xSs restriction base xs string gt lt xs enumeration value audio gt lt xs enumeration value component audio gt lt xs enumeration value video gt lt xs enumer
302. ious tasks a playing a DVD movie with the video shown on the television and audio coming through the stereo s speakers b watching a sporting event on television but listening to the play by play over the radio c selecting different sources of content dur ing a presentation with a PowerPoint slideshow as the current source and d resolving a problem with the DVD player through the question answer interface OT Yoo AF puc v4 0 700 UH 4 5 41 amp AJ Puc v4 0 700 UH 46 5 38 amp Cst Resume J Ea J Volume R mi gt Slide Notes 3 r Slide notes from PowerPoint appear C Mute deal E a or __EEEE Resolution 1024x768 n video Format av mue a Keystone 4 st Td 30 3 C fem 11 PUC Servers Devices Generation Ej PUC Servers Devices Generation Ej PUC Servers Devices Generation E a b C d Figure 1 8 Several examples of aggregate user interfaces generated based on the user s current task a playing a DVD movie with the video shown on the television and audio coming through the stereo s speakers b pre senting Powerpoint slides through a projector c watching broadcast television with audio playing through the television s speakers and d recording a tape in one VCR from a tape playing in another VCR Chapter 1 Introduction 12 are merged into a single interface control for certain functions such as the clock or language settings The merged aggreg
303. iple platforms with fea tures like consistency and aggregation Smart Templates also provide some consistency across interfaces for different appliances because they ensure that the same controls are used for similar kinds of functions see Figure 7 3c Chapter 7 Handling Domain Specific and Conventional Knowledge 109 Smart Templates do have some limitations however The complexity and size of a template is limited by the cost of implementing that template on an interface generator While the tem plate registrar could standardize a template for a very large set of functionality such as entire appliance like a VCR the cost of hand designing an interface flexible enough to accommo date the wide variety of functionality that a VCR may have is prohibitive As the functionality of a template becomes a large percentage of an entire appliance the benefits possible from other automatic generation techniques like consistency will also be lost be cause the majority of the user interface is no longer being generated automatically It is important to note that the standardization needed for Smart Templates is quite different from that being pursued by current industry projects like UPnP UPnP 2005 UPnP stan dardizes the specification for each appliance This means that all the manufacturers for printers for example must sit down in a room and agree on the set of functionality that all printers will have User interfaces for printers are bas
304. ir database Pressing OK from this dialog causes the selected information to be filled appropriately into the fields of the template entos SEC srei E EN Icons currently supported by the PUC status icons Smart Template a An interface for Windows Media Player generated on a PocketPC b The full interface generated for the GMC Yukon Denali driver information CONSOLE cccccccccccccccecceccecececcccececcuceececceucececcescucscecececceseccecs A Smartphone displaying a PUC interface for Windows Media Player and the automatically generated interface for the Driver Information Center in a 2003 GMC Yukon Denali SUV The user navigates through list panes a b to get to summary c e and editing panes E a PEPEE E T A A E EE anaes E remote An example interaction using the speech interface to control a shelf Stereo ana XPOS MENG operis E EEE OES PUC interface generation process diagram E TEE MEET EPA EAA E E EE 106 sa E2 me 116 118 XX Figure 8 5 Figure 8 6 Figure 8 7 Figure 8 8 Figure 8 9 Figure 8 10 A demonstration of changes made to the tree structure when mutual exclusion is found The circles represent nodes within the interface structure which could represent groups state variables or commands a In the before tree the node marked A represents a state variable that can have values from 1 3 The node marked P is the parent group The A formulas shown below the remaining g
305. is organized by appliance as shown by the tabs at the bottom of the screen cceeeeeeee 157 Two shots of the General Setup AUI for our home theater setup Note that in both shots the tabs at the bottom of the screen represent high level concepts within which the functions are organized by appliance combo boxes at top eseceessscceesseceeesseceeeseaeecessaeeeseeaeeceeeaeeeeeaees 158 The merged function AUI featuring the clock language and sleep MIME r MUMCHONS ON 4 SiN le panel erene venus tecaapenmaamenaseees 159 PocketPC interfaces generated by the Personal Universal Controller PUC for the two all in one printers discussed in this papet ceeee 163 The all in one printers used in our studies with a larger view of the built in SEE 1 tel ACE oerna A A T suensaeeeleescaueas 164 Results of the first block of tasks showing the Built In condition compared with the other two for each appliance eseesseceeeeeneeeeeeeeeeeenees 169 Results of the second block of tasks showing the AutoGen condition compared to the Consistent AutoGen condition for each appliance 172 The PUC being used to control a character in an augmented reality application as part of work performed with the PUC at the Technical University of Vienna Tracked PocketPC as a multi purpose interaction device left Tangible interface in a screenshot of the AR LEGO application right PDA screen capture of the LEGO robo
306. istency 94 Second ontologies often rely on hierarchy to understand how concepts generalize The bears are animals example from above shows a small aspect of how hierarchy allows for generalization To create a useful action ontology for appliances each appliance operation would need to be placed into a hierarchy of one or more generalized operations Determin ing each of these generalized operations and appropriately labeling them would be a significant and time consuming challenge and furthermore would probably not be useful unless it was complete for all appliances My solution with its one to one mappings does not require the entire appliance space to be mapped before being useful and can be incre mentally added to as more appliances are specified 6 4 1 1 Mapping Graphs Mappings between similar functions in multiple specifications are grouped together in a mapping graph The central purpose of a mapping graph is to help determine which appli ance should be used as the basis for consistency for a function Every mapping belongs to a mapping graph and there is a mapping graph for each set of similar functions in the knowl edge base For example the power media controls and VCR TV functions all have separate mapping graphs containing mappings specific to those functions An example mapping graph for the media controls function is shown in Figure 6 2 To find the specification to ensure consistency to the PUC s consistency a
307. ithms that apply knowledge of similarity between specifications and interfaces the user has previously seen to generate new interfaces that are personally consistent e Algorithms that use a model of the content flow between appliances to generate task based interfaces that combine functionality from multiple appliances An evaluation of the PUC system compared the automatically generated interfaces for two all in one printers with the manufacturer s interfaces for the same two appliances and found that users of the automatically generated interfaces were twice as fast and four times as suc cessful for both common and complex tasks The evaluation also shows that the PUC s consistency features allow users to be twice as fast when using a new appliance that is similar to an appliance they have previously encountered This evaluation is the first known user study of automatically generated interfaces compared to human designs and it shows that automatic generation of user interfaces for end users is now viable for interactive systems vi For my parents Vil vill Table of Contents ADSI 39s 52sec E TEE E ox aan gaan E endo OEN ENOTE eT v LICO E OOIE S i EEE E EE O AE OEE xvii IB se al 0 E AT TPE T IE AENEA T A PET ne EEE NE TT A E EAE ANT XXV PNCKVO WEG G ACT 09 ae E E ears ees xxvii 1 Introduction l kE ThePeroni Univeral Contolera ei ees 7 Pd Outside tie Scope eranen E T ahaa naoheratcaehaeiey 13 jG ames Bio da 610
308. ities 7 2 Design and Use There are three steps in the development and use of a Smart Template 1 The template registrar defines the new template 2 A specification author instantiates the template one or more times in an appliance specification 3 An interface generator which is already aware of the template uses special hand designed interface code to appropriately render the template as used in the specifica tion Smart Templates must be defined in advance by the template registrar to ensure there is some agreement between the specifications authors who instantiate templates in specifica tions and the interface generator programmers who write the code that renders the templates Agreement does not need to absolute however and flexibility is built in to the design to allow some benefits to the user even when both the specification authors and gen erator programmers to not adhere fully to the Smart Template standards The user will Chapter 7 Handling Domain Specific and Conventional Knowledge 101 always be able to access all appliance functionality and in the worst case the user will be pre sented with an interface rendered without any special Smart Template rules The registrar defines a new Smart Template by giving the template a name and defining a set of specification restrictions for the template A specification author instantiates a template by adding an is a attribute to a group variable or command with th
309. ix C Other PUC XML Language Schemas 260 lt xs attribute name value type xs string use optional gt lt xs attribute name wrap type WrapAttributeType use optional gt lt xs complexType gt lt xs complexType name ListMappingType gt lt xs sequence gt lt xs element name label type xs string minOccurs 0 maxOccurs 1 gt lt xs element name source type SourceType minOccurs 1 maxOccurs 1 gt lt xs sequence minOccurs 0 maxOccurs 1 gt lt xs element name Specl type ListIndexStateMapType minOccurs 1 maxOccurs 1 gt lt xs element name Spec2 type ListIndexStateMapType minOccurs 1 maxOccurs 1 gt lt xs sequence gt lt xs sequence gt lt xs attribute name specl list type xs string use required gt lt xs attribute name spec2 list type xs string use required gt lt xs complexType gt lt xs complexType name ListIndexStateMapType gt lt xs choice minOccurs 1 maxOccurs 1 gt lt xs element name list index type ListIndexMapType gt lt xs element name state type StateMapType gt lt xs choice gt lt xs complexType gt lt xs complexType name ListIndexMapType gt lt xs attribute name type type IndexAttributeType use optional gt lt xs complexType gt lt xs complexType name StateMapType gt lt xs attribute name name type xs string use required gt lt xs attribute name type type IndexAttributeType use optional gt lt x
310. knowledge base can inform the user that the current interface does not contain any similar functions For other forms of search this is an issue that will need to be ad dressed There are also many potential benefits to a searchable user interface For example instead of searching for a function the user could instead search for a task If the system could recognize the task and its steps then it could perform separate searches for each of the task steps and attempt to assemble an aggregate interface for the task based on these searches Heuristics based on the location of items in the specification might even allow multi function appli ances such as a combination VCR DVD recorder to produce grouped results for each of the appliance s main functions For example a timed record task search on the multi Chapter 11 Conclusion 192 function recorder might produce two results one for recording a future show onto a VCR tape and another for recording onto a DVD R My dissertation focuses on generating interfaces but there may some potential to use some of these ideas to evaluate hand designed user interfaces For example a manufacturer might like to compare the hand designed interface for a new appliance with previous products Given an abstract representation for these interfaces and information about their similarities it might be possible to build an evaluation process that would determine where and how the PUC s existing
311. labels gt lt min gt 0 lt min gt lt max gt 59 lt max gt lt map gt lt integer gt lt type gt lt map index 3 gt lt state gt lt labels gt lt label gt Pause lt label gt lt labels gt lt group gt lt map gt lt value labels gt b lt type gt lt state gt lt command name PreviousTrack gt lt labels gt lt label gt Prev lt label gt lt labels gt lt state gt lt state name SongLength is a time duration gt lt type gt lt command name NextTrack gt lt string gt lt labels gt lt label gt Next lt label gt lt labels gt lt type gt lt state gt lt labels gt lt label gt Length lt label gt lt labels gt lt group gt lt state gt a C Figure 7 2 Three specification snippets showing instantiations different Smart Templates a The instantiation of the media controls template for the play controls on Windows Media Player b The instantiation of the time duration template for the counter function on the Sony DV Camcorder c The instantiation of the time duration template for the song length function on Windows Media Player Chapter 7 Handling Domain Specific and Conventional Knowledge 102 shows the same instance rendered by generators on several different platforms that did know about the template The restrictions on the specification allow Smart Templates to be parameterized which al lows them to cover both the common and unique functions of an ap
312. les mov _ ing the dependent pieces to the appropriate locations Chapter 8 Interface Generation 127 sive group boundaries The remaining rules address particular combinations of split and mu tually exclusive situations Table 8 2 shows the different moving rules An important feature of the moving sub phase is its data structure called the containment stack The purpose of the containment stack is to keep track of similar parent groups as the sub phase traverses through the tree Two stacks are created one for the new appliance and another for the previous appliance Only mappings between these two specifications are in cluded in the containment stack so any entry in the stack is known to refer to an existing location in both specifications For example the containment stack for the clock group in the Mitsubishi DVCR and Samsung DVD VCR is shown in Figure 8 6a The clock is lo cated in a different group in these specifications which is reflected in the containment stack All of the moving rules check the containment stacks to see if the top most group mappings are different If the mappings are different and other requirements of the moving rule are met then the mapping s objects are moved to the group that corresponds to the previous specification s top most group mapping For example suppose that we are generating the Samsung DVD VCR interface to be consistent with the Mitsubishi DVCR The top most group mappings are
313. lgorithms start at the node that represents the appliance being generated and traverse the mapping graph to find the node that the user has seen most often Each node maintains a count of the times it has been used as the basis for consistency As discussed earlier it may be impossible to ensure consistency between similar functions if their specifications are too different and this is repre sented by the edges of the mapping graph For example in Figure 6 2 the Panasonic VCR represents play stop and pause as a state variable while the Cheap VCR uses only com mands It is not possible to convert between these representations so the mapping between them will have infinite cost Costs allow mapping graph traversals to ensure that consistency can be maintained between the endpoints of the traversal result You may wonder why I bother to include infinite cost edges in the mapping graph When no zero cost edges are available infinite cost edges may be traversed to ensure at least some consistency by using the labels of a similar function Chapter 6 Consistency 95 Table 6 1 Mapping types for consistency in the PUC system Description _ Allows a series of operations on one appliance to be matched with series of operations on another appliance with support for repetition The possible operations are invoking a command or changing the value of a state variable suuauusuucausausauesuesuussunsussuesuusaesaueseesensausauesueseesunsueesu
314. liance specification can easily define or ganization using a tree to group similar functions This makes the interface generation process easier because most concrete interfaces can also be represented as a tree The utility of trees for grouping seems to be universally accepted most current systems use some kind of tree for grouping functions although USIXML Limbourg 2004 uses a graph Chapter 3 Preliminary User Studies 51 3 6 5 Actions as State Variables and Commands Each action that the user can take must be represented in the appliance specification We found as have many others that state variables and commands are a succinct way to repre sent the manipulable elements of an appliance Some systems such as V2 and Microsoft s UPnP UPnP 2005 separate the state variables from the commands that act upon them This means that a specification for a radio might include a station variable and also tune up tune down seek up and seek down commands associated with the variable The PUC sys tem infers as many functions from the state variable as possible but still uses commands for those functions that cannot be inferred such as seek up and seek down Not every command can be associated with a state variable however and specification lan guages must support unassociated commands Unassociated commands are required for representing functions where there is no notion of state such as pressing the flash button on a telephone C
315. lica tions that it would benefit is the subject of future work In this dissertation I have explored two ways that automatically generated interfaces may support features that would be impractical to include in hand designed interfaces consis tency and aggregation An important area of future work is to find more areas where automatic generation may provide benefits beyond those of hand designed interfaces One area in particular that may be of significant value is the automatic generation of interfaces for users with physical or cognitive disabilities With a model of the user the automatic interface generator may be able to produce interfaces that are specifically designed to accommodate the user s disabilities The high variance in disabilities between different users makes this problem particularly challenging both because a model describing the users capabilities will Chapter 11 Conclusion 194 need to be quite broad and the rules necessary to generate an interface will need to take into account a wide range of possibilities Several researchers have recently started examining this direction but so far there are few results demonstrating success for a wide range of disabled users 11 5 Final Remarks This dissertation has attempted to demonstrate the following thesis A system can automatically generate user interfaces on a wide variety of platforms for remotely controlling appliances where the users performance is better than
316. lity of a database containing many instances of the data specified by the schema In contrast instance information is not available for PUC specifications during gen eration and is not likely to be of much use anyway as most PUC state variables have very few possible values We built two separate matching systems The first is based on our intuition about the PUC data structures and makes use of names label dictionaries and variable types The second is based on the similarity flooding technique Melnik 2002 developed for schema matching which also incorporates organization The first system performs the best finding about 60 of the mappings in our VCR test cases with about 20 of the total mappings found being false positives Currently neither system is successful enough to integrate into the PUC however we have found the results of these systems to be useful as a starting place for a hu man to create mappings between two specifications Future work in this area will require investigating other means to improve the matching al gorithms such as incorporating a thesaurus of appliance terminology or leveraging the existing mappings among other specifications We did not explore the latter approach be cause the size of our knowledge base is still quite small Chapter 6 Consistency rae Chapter 6 Consistency 98 CHAPTER 7 Handling Domain Specific and Conventional Knowledge A common problem for automatic interface
317. ller screens Labels also differ such as for the brightness function which is labeled darkness on the HP all in one printer exposure on the Canon all in one printer and light dark on the office copier The organization of functions is also different through the appliances As one exam ple the date and time are set on the HP printer through the fax configuration screen whereas they are accessed through the general setup screen on both of the other appliances The lack of consistency between interfaces prevents a user that is familiar with one of these appliances from leveraging that knowledge to use either of the other appliances 4 Copy Menu a M 2 Reduce Enlarge 4 Paper Size b G Figure 1 1 Physical interfaces for three different appliances with copier functionality a a Canon NP6035 office copy machine b a Canon PIXMA MP780 All In One Photo Printer and c a Hewlett Packard Photosmart 2610 All In One printer The latter two appliances also have fax and special photo printing capabilities Chapter 1 Introduction 6 UI devices are in unique position to ensure a consistent experience for their users because the devices will be used for controlling most appliances and can track all of interfaces that users see This knowledge combined with information showing how the new interface is similar to previous interfaces can be used to automatically generate new interfaces that use controls labels and organizati
318. lly fixed to one interface modality with constraints on the form factor and input techniques In this case both languages are designed for large Chapter 2 Related Work 34 screen graphical interfaces XUL has been shown to be beneficial for porting applications across various platforms of this type including Windows Linux and Macintosh The PUC specificiation language differs from these languages in that it describes appliance functional ity without any specific details of the user interface allowing the specification to apply for interfaces in different modalities and substantially different format factors with different in put techniques 2 2 2 9 SUPPLE Most automatic interface generation systems including the PUC use a rule based approach to create user interfaces SUPPLE Gajos 2004 instead uses a numeric optimization algorithm to find the optimal choice and arrangement of controls based on a cost function The devel opers of Supple have experimented with including a number of different factors in this cost function Common factors to all of their functions are the cost of navigation between any two controls and the cost of using a particular control for a function Additional costs have been included based on the common tasks that a user performs Gajos 2004 consistency between interfaces for the same application generated on different platforms Gajos 2005b and the physical abilities of the user for assistive technology G
319. lt 0 02 with 10 total failures for all users over all tasks using the PUC interface and 16 total failures using the built in inter face about 1 3 fewer failures on average We also performed the same analyses comparing the Built In condition and the combined PUC condition for the data from the second block of tasks All of these analyses were signifi cant and matched the results for the first block except for the number of failures over all tasks for the HP printer In this case there were too few failures to make this analysis possi ble zero failures for all 16 subjects using a PUC HP interface and only one failure for the 8 subjects using the built in HP interface 10 4 2 Discussion of Usability The results show that users perform faster over all eight tasks using the PUC interfaces as compared to the printers built in interfaces For the Canon printer the PUC interfaces are significantly faster for nearly all of individual tasks tasks 3 and 8 are marginally significant and only task 2 automatically re dialing a busy number was not found to be different at all Task 2 was also the task most failed by users of the PUC interfaces for the Canon by a wide margin I believe task 2 was particularly hard for users because the Canon printer has many configuration features for sending and receiving faxes which are complex seemingly overlap with unrelated functions and use language that is difficult to understand These function
320. lt map gt lt value labels gt lt type gt lt labels gt lt label gt Sort By lt label gt lt label gt Sort lt label gt lt labels gt lt active if gt lt greaterthan state ToDo List Length gt lt constant value 0 gt lt greaterthan gt lt active if gt lt state gt lt group gt x Group gt lt group name Setup priority 1 gt lt labels gt lt label gt Setup lt label gt lt labels gt lt state name DisplayPreference priority 8 gt lt LYpe lt enumerated gt lt item count gt 4 lt item count gt lt enumerated gt Appendix D Specification Authoring Study Instructions 298 lt value labels gt lt map index 1 gt lt labels gt lt label gt All Items lt label gt lt label gt All lt label gt lt labels gt lt map gt lt map index 2 gt lt labels gt lt label gt Incomplete Items lt label gt lt label gt Incomplete lt label gt lt labels gt lt map gt lt map index 3 gt lt labels gt lt label gt Past Due Items lt label gt lt label gt Past Due lt label gt lt labels gt lt map gt lt map index 4 gt lt labels gt lt label gt Completed Items lt label gt lt label gt Completed lt label gt lt labels gt lt map gt lt value labels gt lt type gt lt labels gt lt label gt Display Preference lt label gt lt label gt Display Pref lt label gt lt label gt Display lt label gt lt labels gt lt state gt lt state na
321. lt apply over gt lt defined gt lt equals gt lt false gt lt greaterthan gt lt lessthan gt lt not gt lt or gt lt true gt lt undefined gt lt defined gt lt defined state SomeState gt Used in conjunction with the lt active if gt tag to define that some appliance object depends on a state variable having a defined value Appendix B Specification Language Reference 237 Placement Inside the lt active if gt lt and gt lt apply over gt lt not gt or lt or gt elements Parameters e state required The name of the state that is depended upon lt undefined gt lt undertined state SomeState gt Used in conjunction with the lt active if gt tag to define that some appliance object depends on a state variable not having a defined value Placement Inside the lt active if gt lt and gt lt apply over gt lt not gt or lt or gt elements Parameters e state required The name of the state that is depended upon lt equals gt lt equals state SomeState gt value lt equals gt Used in conjunction with the lt active if gt tag to define equals dependency infor mation for this state variable Placement Inside the lt active if gt lt and gt lt apply over gt lt not gt or lt or gt elements Parameters state required The name of the state that is depended upon May Contain lt ref value gt lt constant gt Appendi
322. lt command name Delete is a list remove gt lt labels gt lt label gt Remove Category lt label gt lt label gt Remove lt label gt lt labels gt lt command gt lt group gt group gt lt group gt lt groupings gt Appendix D Specification Authoring Study Instructions 300 lt spec gt Appendix D Specification Authoring Study Instructions 301 Appendix D Specification Authoring Study Instructions 302 APPENDIX E Usability Study Instructions This appendix contains the informal document describing the process employed in the usab lity of the all in one printers The document starts on the next page to preserve formatting 303 Conditions D ae a Te res ee Ta k ee Setup of Equipment Canon MP780 eee HP 2610 Questionnaires e Background questionnaire p1 3 e Post questionnaire p4 Actions during Study before study Move pda printer in position ee ae Connect usb cable of pda to desktop Poin camer pda pres Warn up prince Add more pap Appendix E Usability Study Instructions 304 I sowie Start cameasi es Start multitrack stopwatch i Start activsync remote display before 1 device 1 task Start recording in camtasia Reset Multitrack stopwatch Stop recording in camtasia and save video file Save task times in Multitrack stopwatch After study Script after 1 device Co Save task times in Multitrack stopwatch Our research team is investigating
323. lt string gt lt string gt lt string gt String type for variables that take the form of strings You can specify parameters about the length of the string to help out the interface generators Placement Appendix B Specification Language Reference 233 Inside the lt type gt element May Contain lt min gt lt max gt lt average gt lt average gt lt average gt lt average gt Describe the average length of a string Placement Inside the lt string gt element May Contain lt constant gt lt ref value gt lt value labels gt lt value labels gt lt value labels gt Contains one or more lt map gt tags that provide label dictionaries for specific values that the variable might have Placement Inside the lt type gt element May Contain lt map gt lt map gt lt map index value gt lt map gt Specifies a label dictionary for the specific value of a variable specified by the index parameter Certain values of a variable can also be enabled based on dependency information which is specified in the enclosed lt active if gt element Appendix B Specification Language Reference 234 Placement Inside the lt value labels gt element Parameters e index required The value to associate this dictionary with May Contain lt labels gt lt active if gt lt labels gt lt labels gt lt labels gt Defines a label dictionary for an appliance
324. lt xs element name value labels type ValueLabelsType minOccurs 0 maxOccurs 1 gt lt xs sequence gt lt xs attribute name type name type xs string use required gt lt xs complexType gt lt Groupings Section gt lt xs complexType name MultipleGroups gt lt xs choice minOccurs 1 maxOccurs unbounded gt lt xs element name group type GroupType gt lt xs element name list group type ListGroupType gt lt xs element name union group type UnionGroupType gt lt xs element name apply type type ObjectApplyType gt lt xs choice gt lt xs complexType gt lt xs complexType name GroupType gt lt xs sequence gt lt xs element name active if type ActivelfType minOccurs 0 maxOccurs 1 lt xs element name labels type LabelDictionary minOccurs 0 maxOccurs 1 lt xs choice minOccurs 1 maxOccurs unbounded gt lt xs element name group type GroupType gt lt xs element name list group type ListGroupType gt lt xs element name union group type UnionGroupType gt lt xs element name state type StateType gt lt xs element name command type CommandType gt lt xs element name explanation type ExplanationType gt Gr Gr get EE Ack Ace lt xs element name apply type type ObjectApplyType gt lt xs choice gt lt xs sequence gt lt xs attributeGroup ref ApplianceObjectAttribs gt lt xs complexType gt lt xs at
325. mal point for a fixed point type Sport Defines a port of an appliance lt port group gt Defines a group of ports of an appliance sports gt Defines the section in which all ports of an appliance are described lt processing gt lt recorder gt lt ref value gt Used to define a dynamic value for any of the value space parameter lt renderer gt lt required if gt Appendix B Specification Language Reference Description flow may be specified content flow _ based upon this specification language Many pronunciations may be included in a label dictionary _ Defines any processing that an appliance does to a content stream in a pass through content flow such as if a channel is blocked _ Defines a content flow sink that records the stream sent to it tags except the lt pointpos gt tag that depends on the value of a numeric state variable E g a ref value can be used to set the maximum of a numeric state variable to be the value of another state variable ref value elements may also be used with dependencies and in several _ other locations Defines a content flow sink that renders the stream sent to it _ Specifies the circumstances when a state variable s value is required If this element is omitted from a state variable s definition then a value _ is not required whenever the state variable is active Otherwise a value is required whenever the contents of this element are satisfied 219
326. map gt lt map index false gt lt labels gt Appendix A Sample VCR Specification 199 lt label gt TV lt label gt lt labels gt lt map gt lt value labels gt lt type gt lt labels gt lt label gt TV VCR lt label gt lt labels gt lt state gt lt group gt lt group name TimedRecordings gt lt labels gt lt label gt Timed Recordings lt label gt lt labels gt lt list group name List gt lt labels gt lt label gt Timed Recording lt label gt lt labels gt lt state name Channel gt lt apply type type name ChannelType gt lt labels gt lt label gt Channel lt label gt lt labels gt lt state gt lt group name StartTime is a date time gt lt labels gt lt label gt Start Time lt label gt lt labels gt lt state name Date is a date gt lt type gt lt string gt lt type gt lt labels gt lt label gt Date lt label gt lt labels gt lt state gt lt state name Time is a time gt lt type gt lt string gt lt type gt lt labels gt lt label gt Time lt label gt lt labels gt lt state gt lt group gt lt state name Duration is a time duration gt lt type gt lt integer gt lt type gt lt labels gt lt label gt Duration lt label gt lt labels gt lt state gt lt list group gt lt group name Commands is a list commands gt Appendix A Sample VCR Specification 200 lt command name
327. may have mutual exclusive groups and the planned move may cross the boundaries of these groups The easiest case is when an item that is not dependent on the mutually exclusive state is moved into one of the mutually ex clusive groups No extra work is needed because the moved item will always be available in its new location in the interface The opposite case when a dependent item is moved out of the mutually exclusive groups is trickier because the item may be disabled and it may not be clear why from the item s new location The most difficult case occurs when an item needs to move from one group to another that is mutually exclusive of the first In this case without any extra work the moved item would never be available because its dependencies would conflict with the organization that is automatically generated by the interface generator Rules that address this problem are discussed below The second difficulty arises when a mapping references more than one variable or command in a specification and these items are not adjacent I refer to this situation as a split mapping because the items are split across multiple locations in the specification Split mappings arise most often in two situations when the options of a single function in one specification are represented as two separate functions in another specification or when a single function in one specification is duplicated in a second specification because it is available in more th
328. me ReminderFrequency priority 5 gt cCVpes lt enumerated gt lt item count gt 3 lt item count gt lt enumerated gt lt value labels gt lt map index 1 gt lt labels gt lt label gt Weekly lt label gt lt labels gt lt map gt lt map index 2 gt lt labels gt lt label gt Daily lt label gt lt labels gt lt map gt lt map index 3 gt Appendix D Specification Authoring Study Instructions 299 lt labels gt lt label gt Hourly lt label gt lt labels gt lt map gt lt value labels gt lt type gt lt labels gt lt label gt Reminder Frequency lt label gt lt label gt Reminder Freq lt label gt lt label gt Reminders lt label gt lt labels gt lt state gt lt group name Categories priority 5 gt lt labels gt lt label gt Category Setup lt label gt lt labels gt lt list group name List gt lt state name Category gt Lype gt lt string gt lt min gt lt constant value 1 gt lt min gt lt average gt lt constant value 10 gt lt average gt lt max gt lt constant value 25 gt lt max gt lt string gt lt type gt lt labels gt lt label gt Category lt label gt lt labels gt lt state gt 7 list grou p gt lt group name Commands is a list commands gt lt command name Add is a list add gt lt labels gt lt label gt Add Category lt label gt lt label gt Add lt label gt lt labels gt lt command gt
329. me StaticOrReference gt lt xs choice gt lt xs element name constant type valueAttribNoContent gt lt xs element name ref value type stateAttribNoContent gt lt xs choice gt lt xs complexType gt lt xs complexType name valueAttribNoContent gt lt xs attribute name value type xs string use required gt lt xs complexType gt lt xs complexType name SelectionTypeType gt lt xs attribute name number type TypeNameType use required gt lt xs attribute name access type AccessType use optional default read write gt lt xs complexType gt lt xs complexType name ObjectApplyType gt lt xs attribute name type name type xs string use required gt lt xs attribute name name type xs string use optional gt lt xs attribute name priority type PriorityType use optional gt lt xs attribute name access type AccessType use optional gt lt xs complexType gt lt xs complexType name ModifiesStateType gt lt xs attribute name name type xs string use required gt lt xs complexType gt lt xs complexType name CommandType gt lt xS sequence gt lt xs element name appliance reference type ApplianceReference minOccurs 0 maxOccurs 1 gt lt xs element name modifies state type ModifiesStateType minOccurs 0 maxOccurs 1 gt lt xs element name labels type LabelDictionary gt Appendix B Specification Language Refere
330. mmand Play Command Stop Command Pause Command Rewind DVD Player lieder F suse State Mode ommand Recor Play Stop Pause Command Next Track Command Previous Track Edge Costs Infinite Zero Figure 6 2 An example mapping graph for the media control functions e g play stop and pause on four appliances The node counts indicate that the Panasonic VCR has been the basis for consistency three times for itself the answering machine and the DVD player and the Cheap VCR has been the basis for consistency once just for itself The answering machine and DVD player were generated to be consistent with the Pana sonic VCR and thus both have counts of zero Chapter 6 Consistency 96 6 4 2 Automatically Finding Mappings In collaboration with Brandon Rothrock I have explored automatically extracting mappings from a new specification and previous specifications that the user s controller device has al ready stored The challenge of automatic mapping is the lack of substantial semantic information about each function within the specification Our work in finding mappings between specifications is similar to previous work in schema and ontology matching from the database community Shvaiko 2005 Work in matching has tried numerous approaches including matching with the help of a thesauras Madhaven 2001 or the use of machine learning Doan 2001 Several matching techniques take advan tage of the availabi
331. mposing appliance interfaces for many different controlling devices It relies upon a centralized inter face manager to distribute interfaces to handheld devices sometimes automatically generating the interface and other times distributing a hand designed interface that is already available Crafter can even distribute speech interfaces described by the VoiceXML language to those controllers that support speech Support for the automatic generation of user inter faces is limited however and they also mention the difficulty of generating speech interfaces Perhaps the most interesting feature of Crafter is its ability to aggregate appliances together and provide a user interface To support composition Crafter relies on a set of service in terfaces that abstract the functionality of services and a set of interface aggregators that are each hand coded to build an interface for a particular pattern of service interfaces When a user requests an interface for multiple services Crafter looks for an aggregator that matches the pattern and if an aggregator is found returns a single interface generated by that aggre gator For example a camera might implement the DataProducer interface and a printer might implement the DataConsumer interface The generic aggregator for the DataPro Chapter 2 Related Work 24 ducer DataConsumer combination could then generate a combined interface for the camera and printer Crafter s approach h
332. mprove usability by hiding controls that are not active In some cases the specification author may not have in cluded structure for a mode and the interface generator also needs a means of detecting this and if possible inserting the appropriate structure Dependency information can be used to address both of these issues with the concept of mu tual exclusion If two sets of components are shown to never be active at the same time then the interface generator can decide that these sets of components should be placed in different groups and that those groups should be marked with as important to carry through in the concrete interface Another way to solve this problem might be to have the specification au thors place a marker on all groups that have mutually exclusive branches Relying on dependency information instead of a marker makes specification design easier Instead of finding all cases of mutual exclusion and making the tree fit the author can give the tree any hierarchy that seems appropriate and the dependency information can help the generator infer structure from it Structure can even be inferred from a flat tree with only one group Chapter 8 Interface Generation 121 Unfortunately the task of determining mutual exclusiveness for an arbitrary set of state vari ables and commands can be shown to be NP hard I reduce the difficulty of the problem by considering mutual exclusion with respect to a single given variable and its l
333. n However when building the generation rules for list interfaces I found that list interfaces were easier to gener ate well when the interface generator could identify the commands that operated upon it Furthermore commands that added items to a list could not automatically display a dialog box to enable editing of the newly added item The solution I chose was to create the list commands Smart Template which is a merged template that may contain multiple other templates These constituent templates include list add list remove list clear list move before and several others The list tem plates allow all operations to be grouped appropriately next to the list control add operations to perform the correct dialog box opening behavior and for move com mands to be handled directly inside the list control rather than being displayed as separate commands away from the list It is important to note that while the list templates allow for better list interfaces it is possible for PUC interface generators to produce an adequate list interface with no knowledge of these templates Discussion The unique feature of Smart Templates is their ability to integrate small snippets of hand designed user interfaces into interfaces that are otherwise automatically designed This allows the design conventions that users expect to be included in the appliance user interfaces while still allowing flexibility for those interfaces to be generated on mult
334. n Language 73 one selection then an implicit Selection variable is created which contains the index of the current selection undefined means no selection If multiple selections are created then an implicit list group named Selections is created This group contains a Length state as all list groups do and a Selection state which contains all of the selected indices We have also developed a special dependency operator for lists named apply over that can be true if all items any items or no items in the list match a dependency equation that references items in the list The second special group is the union group which is similar to specifying a union in a programming language like C Of the children within a union either groups or appliance objects only one may be active at a time An implicit state variable named ChildUsed is automatically created within a union group that contains the name of the currently active child 5 2 1 6 Dependency Information The two way communication feature of the PUC allows it to know when a particular state variable or command is unavailable This can make interfaces easier to use because the con trols representing those elements can be disabled The specification contains formulas that specify when a state or command will be disabled depending on the values of other state variables These formulas can be processed by the PUC to determine whether a control should be enabled when
335. n about layout This requirement enforces modality independ ence by limiting how detailed a designer can specify the functions of an appliance If it were possible to describe concrete interfaces within an appliance specification language designers would be tempted to include too many details about how each interface should be imple mented This has several disadvantages e Appliance specifications will get much longer because each one may turn into a complete description for several different types of concrete interfaces e Appliance specifications might lose their forward compatibility to PUC devices of the fu ture It seems likely that variety will increase in the devices of the future as non rectangular screens and different interaction styles become more common For example specific information for a dialog box style interface would probably not be useful for a new watch with a circular screen and several nested dials for interaction e Some of the other advantages of automatic generation might be lost For example a PUC can ensure interface consistency by making certain interactions the same across multiple appliances This is not possible if the PUC does not have the freedom to choose the interaction style and positioning for representing given functions 3 6 4 Hierarchical Grouping A fundamental requirement of any user interface is good organization because users must be able to intuitively find a particular function An app
336. n all in one printer fax setup functions are used in the Setup mode but in fax mode on the HP all in one printer and generating a consistent interface between these printers requires moving items into a group with conflicting dependencies In such cases this rule removes dependencies on the moved items and automatically handles any needed mode switches in the background when the user uses a moved item Move Split Moded Items Handles moves for a mapping in which a single item in _ one specification is mapped to multiple items in another specification and each of the multiple items is mutually exclusive of the others For example this rule handles the situation for the buttons that start copying faxing and scanning on the HP and Canon all in one printers Move One to Many Split Similar to the move split moded items rule this rule also _ handles moves for a single item in one specification that is mapped to multiple items in another specification _ None of the multiple items may be mutually exclusive of each other for this rule however Bring Together Split Works for group mappings that contain a state mapping Dependents and one or more task mappings This typically means that one appliance has a set of functionality that is _moded whereas at least some of that functionality is _ available simultaneously in the other specification If that functionality is split in either spec this rule hand
337. name Server side error correction minOccurs 0 maxOccurs 1 gt lt xs sequence gt lt xs attributeGroup ref ApplianceObjectAttribs gt lt xs attribute name access type AccessType use optional gt lt xs complexType gt lt xs complexType name ApplyPrimitiveType gt lt xs attribute name type name type xs string use required gt lt xs complexType gt lt xs complexType name PrimitiveType gt lt xsS sequence gt lt xs choice minOccurs 1 maxOccurs 1 gt lt xs element name binary type BinaryType gt lt xs element name boolean gt lt xs elemen name enumerated type EnumeratedType gt lt xs element name fixedpt type FixedPtType gt lt xs element name floatingpt type FloatingPtType gt lt xs element name integer type IntegerType gt lt xs element name list selection type ListSelectionType gt Ch sehr E Gk Yok lt Gks ocr 5 lt xs element name string type StringType gt lt xs choice gt lt xs element name value labels type ValueLabelsType minOccurs 0 maxOccurs 1 gt lt xs sequence gt lt xs attribute name type name type xs string use optional gt lt xs complexType gt lt xs complexType name ValueLabelsType gt lt xs sequence gt lt xs element name map type MapType minOccurs 1 maxOccurs unbounded gt lt xs sequence gt lt xs complexType gt lt xs complexType name MapType gt lt xs all
338. nce 210 lt xs element name active if type ActivelfType minOccurs 0 maxOccurs 1 gt lt xs sequence gt lt xs attributeGroup ref ApplianceObjectAttribs gt lt xs complexType gt lt xs complexType name ExplanationType gt lt xS sequence gt lt xs element name appliance reference type ApplianceReference minOccurs 0 maxOccurs 1 gt lt xs element name labels type LabelDictionary gt lt xs element name active if type ActivelfType minOccurs 0 maxOccurs 1 gt lt xs sequence gt lt xsr attributeGroup ref ApplianceObjectAttribs gt lt xs complexType gt lt xs complexType name StateType gt lt xSs sequence gt lt xs element name appliance reference type ApplianceReference minOccurs 0 maxOccurs 1 gt lt xs choice minOccurs 1 maxOccurs 1 gt lt xs element name type type PrimitiveType gt lt xs element name apply type type ApplyPrimitiveType gt lt xs choice gt lt xs element name labels type LabelDictionary minOccurs 0 maxOccurs 1 gt lt xs element name active if type ActivelfType minOccurs 0 maxOccurs 1 gt lt xs element name regquired if type DependencyContent minOccurs 0 maxOccurs 1 gt lt xs element name default value type StaticOrReference minOccurs 0 maxOccurs 1 gt Gt CE set och lt xs element name completions available minOccurs 0 maxOccurs 1 gt lt xs element
339. ncia E E ESA 173 LOG Discussion E E E T EE N 174 11 Conclusion 177 UED o o EA A E A E AEE T T E EAA ETA A TE 177 HS EAA VOE N TE ee ee 182 A E Be ches D a 0 a E E E E T E E A E T 184 EA Furore Work rora a T 185 M Tina RENAT kS ooe N aac seduanetwaetauee 195 XIV A Sample VCR Specification B Specification Language Reference Bale IVs SC hetrass iciasierca oo cteawnsdescsanooaenceaesseag noeetadonatecteneneecausrenass Be Blenienelridex sci ee Bed Element Descriptions ecinic AE C Other PUC XML Language Schemas C 1 Communication Protocol Schema eee ceeecccseecceeececcecneees CD WG Wied Ce Base Sc DENA osteitis E E E C 3 Multi Appliance Wiring Diagram Schema eee eeeeeeeeeseeeeeees D Specification Authoring Study Instructions E Usability Study Instructions F Gallery of PUC Interfaces BiDHOgtAP D Vrises i E EERE 197 205 263 303 309 XV XVI List of Figures Figure 1 1 Figure 1 2 Figure 1 3 Figure 1 4 Figure 1 5 Figure 1 6 Figure 1 7 Figure 1 8 Figure 2 1 Physical interfaces for three different appliances with copier functionality a a Canon NPG6035 office copy machine b a Canon PIXMA MP780 All In One Photo Printer and c a Hewlett Packard Photosmart 2610 All In One printer The latter two appliances also have fax and special photo printing capabilities 00 0 lee eeeeeeeeseeeeeeeeee Interfaces generated by the PUC on the PocketPC for several different applianc
340. nd stage filtering process for this aggregate takes only functions that are found within the Setup group This process typically finds functions that affect the output of the currently active flows but will be used infrequently such as the brightness and contrast controls for the television and the speaker level controls for the receiver This AUI will still not include any controls that cause the flow to stop being active however since those are best controlled us ing the FBI The Active Flow Setup AUI is organized by appliance because a desired setup function is typically easier to find with this organization See Figure 9 5 for two shots of the Active Flow Setup interface generated for the flow for the DVD to receiver and television We originally tried to organize this aggregate by content type which in the case of a home theater would give top level groups for audio and video but this created lower quality interfaces The ap proach worked reasonably for appliances whose functions could be classified by their place in the content flow such as the receiver and television which in some flows receive only audio AF PUC v4 0 700 UH lt P 42340 G Picture Quality ki 5 Preset Mone al Play Test Tone a Save Current Settings In Personal HE Front Balance M A Manual Picture Settings a 2 SEN Rear Balance ML MM contrast Tmf e i 10 35 Rear Level A _ __ A color a fs i 10
341. nd the bottom shows Smartphone The Smartphone control maintains consistency for the user by copying the layout for the Smartphone version of Windows Media Player the only media player application we could find on that platform This interface overloads pause and stop onto the play button c Different configurations of media playback controls automatically generated for several different Bp AIT COS E E A E T E A E ace cues E E Screenshots of Smart Templates rendered as part of the GMC Denali navigation system on the PocketPC platform a Demonstrates the time absolute Smart Template used for a clock function The 12 and 24 hour option of the template changes the way time is rendered throughout the interface as can be seen in the clock at the top of these screenshots b Demonstrates the list commands templates integrated with one of the PUC s list controls Several commands for adding and deleting items are located underneath the list control along with the edit button that is part of the list control Two move commands have also been integrated with the list control as the arrow buttons located on top of the selected list item c Demonstrates the address template s capability of integrating with the PocketPC s built in Outlook contact database The leftmost screen shows the interface before the user has pressed the Select Contact button Pressing this button shows the middle screen which allows the user to select a contact from the
342. ndling Domain Specific and Conventional Knowledge 106 7 2 1 Implementing a Smart Template for an Interface Generator Each implementation of a Smart Template must accommodate all of the required and op tional parameters of that template For many templates this puts a significant burden on the implementer to handle many different possible inputs and reflect all of these differences in the user interface The implementation challenge becomes even greater for merged templates which are Smart Templates that contain one or more other Smart Templates For example the date time template is a merged template because it must contain one instance of both the date and time absolute templates Merged templates are more complex to implement because the implementer must be able to handle the different input combinations for all of the constituent templates The implementation complexity of a Smart Template can be reduced by separating the im plementation task into two layers e The data translation layer converts data back and forth between the appliance format defined by the specification and the template data structure The template data struc ture contains all the possible data for a given template in one format along with flags that define the optional parameters used in an instance of the template If the data in the template data structure changes this layer converts the data back into the appli ance s format and sends the changes to the appl
343. ndon 41226874 Ring after 5 rings 3 Jeff 4122687239 Pe 4 Laura 4122683472 5 Polo 4122687566 Ring Pattern Paper Size Letter C Fax Reduction M Backup Incoming Fax To use the scan features iconnect your HP all in one to a Status computer PUC Servers Devices Generation Ej A PUC v4 0 1133 UH qx 8 24 amp To use the photo features insert a memory card or Status connect a PicBridge camera to zs your HP all in one LE UC v4 0 1133 UH x 8 Ao d Fi Presets Standard v 4 a z Exposure o Deby a Tr P 4 Lcttest Derkest Sen Photo seo TTT PUC Servers Devices Generation PUC v4 0 1133 UH 4x 8 24 amp Maintenance Ad PUC v4 0 1133 UH 4x 8 24 amp HP Photosmart 2610 All in One Figure F 3 The full interface for the HP All In One printer generated to be consistent with the Canon printer as seen by users in the usability study Appendix F Gallery of PUC Interfaces 312 PUC v4 0 1133 UH 4x 8 21 PUC v4 0 1133 UH 4x 8 21 amp UC v4 0 1133 UH 4x 8 21 amp UC v4 0 1133 UH 4x 8 21 amp
344. nerator assumes that the value of a state may be changed to any value within that type at any time that the state is enabled It is possible for state variables to be undefined i e without any value This commonly happens just after an inter face is generated before any values have been assigned but could occur for other reasons e Commands Any function of an appliance that cannot be described by variables alone They may be used in situations where invoking the command caused an un Appendix D Specification Authoring Study Instructions 265 known change to a known state variable such as the seek function on a radio or in situations where the state variable is not known manufacturer choice or other rea son the dialing buttons on a standard phone would all be commands e Explanations Descriptive information that is not appropriate to include as the label of a state or command and is more important than a simple group label This repre sents an early attempt at including help information within our specifications Explanations are used rarely in specifications Although there are differences between states commands and explanations they also share a common property of being enabled When an object is enabled or active the user interface widgets that correspond to that object can be manipulated by the user Knowing the circum stances in which an object will be enabled or disabled can provide a helpful hint for struct
345. ng minOccurs 0 maxOccurs 1 gt lt xs element name source type SourceType minOccurs 1 maxOccurs 1 gt lt xs sequence gt Appendix C Other PUC XML Language Schemas 258 lt xs attribute name specl type xs string use reguired gt lt xs attribute name spec2 type xs string use reguired gt lt xs complexType gt lt xs complexType name TemplateMappingType gt lt xs sequence gt lt xs element name label type xs string minOccurs 0 maxOccurs 1 gt lt xs element name source type SourceType minOccurs 1 maxOccurs 1 gt lt xs element name cost type MappingCostType minOccurs 1 maxOccurs 1 gt lt xs sequence gt lt xs attribute name specl type xs string use reguired gt lt xs attribute name spec2 type xs string use reguired gt lt xs complexType gt lt xs complexType name StateMappingType gt lt xs sequence gt lt xs element name label type xs string minOccurs 0 maxOccurs 1 gt lt xs element name source type SourceType minOccurs 1 maxOccurs 1 gt lt xs element name cost type MappingCostType minOccurs 1 maxOccurs 1 gt lt xs choice minOccurs 0 maxOccurs 1 gt lt xs element name all values equivalent gt lt xs sequence gt lt xs element name value type ValueType minOccurs 0 maxOccurs unbounded gt lt xs sequence gt lt xs choice gt lt xs sequence gt lt xs attribute name specl state typ
346. ng rule creates a new button that makes appropriate mode change and places this button in a similar location to where the unmoded functionality was in the previous specifi cation If there were any functions dependent on this mode they are also moved to the location of the new button see Figure 8 8 8 4 3 2 Re ordering The reordering sub phase moves functions within groups to ensure a consistent ordering For example Figure 8 9 shows that the parameters for a timed recording have a different order ing between the Mitsubishi DVCR and the Samsung DVD VCR This sub phase like the moving sub phase traverses the abstract user interface until it encounters a mapping Re ordering rules operate on the children of a node however so unlike the moving sub phase there is no need to apply these rules to leaf nodes in the group tree Before the reordering rules are executed on a group the sub phase determines the previous specification with which the group will be made consistent The sub phase then creates a block list data structure for the group in the new specification and its equivalent group in the previous specification The block list is important because it allows rules to analyze and manipulate functions as if the functions are in a list when the underlying representation of structure in the abstract user interface is a tree The tree structure can become problematic when a function s objects span multiple levels of tree hierarchy
347. no regular users of the Smartphone interface generator who can verify this Both methods also suffer from the small area available for the label on the soft button In many cases it is not possible to display a sufficient label in the space provided on the inter face particularly when both the name and value of a function need to be shown One solution might be to use icons but the PUC system currently does have any way for a speci fication author to include icons as a label for functions 8 6 Modifying the Concrete Interface for Consistency The preceding sections have discussed the first five phases of the PUC s interface generation process At this point the process has transformed an appliance specification into a abstract user interface modified the abstract interface to ensure consistency and transformed the ab stract interface into a concrete user interface that is specific to generation platform Before the final user interface is displayed to the user several rules check the concrete interface for inconsistencies with previous interfaces and provide fixes for any that are found Inconsistencies may occur for at least two reasons First for the PocketPC and desktop there is often a mismatch between the deep hierarchy of the abstract interface and the flat hierar chy of the concrete interface This mis match sometimes introduces inconsistencies in the ordering of controls within a panel which could not be discovered because of t
348. not affect subsequent tasks The last point was especially important to minimize the possibility that a subject might notice an element used in a future task while working on an earlier task This effect was also minimized somewhat by presenting the next task description only after sub jects had completed their previous task however this does not prevent subjects working on their second block from remembering the tasks from the first block The tasks used in the order they were always presented to subjects are listed below The or der of tasks was not varied for each subject so that whatever learning effects might exist between the tasks despite best efforts to eliminate such effects would be the same for each subject The task wording is paraphrased for brevity exact wording for the tasks can be found in Appendix E 1 Send a fax to the number stored in the third speed dial 2 Configure the fax function so that it will always redial a number that is busy 3 Configure the fax function so that any document received that is larger than the de fault paper size will be resized to fit the default 4 Configure the fax function so that it will only print out an error report when it has a problem receiving a fax Chapter 10 Usability Evaluation 167 5 Make two black and white copies of the document that has already been placed on the scanner of the all in one printer 6 Imagine you find the copies too dark Improve this by ch
349. nother application a future wiring technology or the user Furthermore this separation also pushes most of the work to the manu facturers because most of the the complexity in any content flow model is found in the internal flows within each appliance In order to support content flows two sections were added to the PUC specification lan guage to specify the physical ports of the appliance and the internal content flows that use those ports see section 5 2 2 The content flows within an appliance are represented using three different structures Chapter 9 Aggregating User Interfaces 149 e Sources represent content that originates within the appliance such as from a DVD player playing a DVD or a VCR playing a videotape Display devices that have inter nal tuners such as televisions receiving broadcast signals through antennas are not defined as sources however because the content does not originate inside of the tun ing device Instead broadcast signals are described as a special external source that must be routed through a tuner to be viewable by the user e Sinks represent locations where content may either be displayed to the user or stored for later retrieval For example the television screen receiver speakers and VCR tape for recording may all be sinks for content in our home theater scenario e Passthroughs represent an appliance s ability to take in some content as an input and redirect it through one or
350. ns for adding and removing items from both the to do list and the categories list In our description language we have chosen to use a Smart Template for defining these special functions and allowing the inter face generators to take special action when rendering them Object O State Variable O Read only Name Add Priority 0 10 Smart Template liskadd o O oo y Command O Explanation Object O State Variable O Read only Name olete Priority 7 0 10 x Command 0O Explanation Smart Template ise V wnyve Y Y Labels juletre Ts Do Tren Delere Ts D v Oclere Labels Add TeDe Tisi Add To Do Figure D 7 The object cards that describe the add and remove commands for the to do list These objects are contained within a group that is tagged with the list commands Smart Template lt group name Commands is a 1ist commands priority 10 gt lt command name Add lt labels gt lt label gt Add To Do Item lt label gt lt label gt Add To Do lt label gt lt label gt Add lt label gt lt labels gt lt command gt ip a list eca priority c gt lt command name Delete is a list remove priority 7 gt lt labels gt lt label gt Delete To Do Item lt label gt lt label gt Delete To Do lt label gt lt label gt Delete lt label gt lt labels gt lt active if gt lt greaterthan state ToDo List Length gt lt constant value 0
351. nt depth User interfaces designed for small screens would need every branch in the tree whereas large screen interfaces might ignore some deeper branches Chapter 3 Preliminary User Studies 48 It was also found that grouping is influenced by modes For example the Aiwa shelf stereo has a mode that determines which of its components is playing audio Only one component can play at a time In the stereo interfaces shown in Figure 3 3c d you will note that a tabbed interface is used to overlap the controls for the CD player tape player etc Other controls that are independent of mode such as volume are available in the sidebar Unlike regular grouping information information about modes gives explicit ideas about how the user in terface should be structured If two sets of controls cannot be available at the same time because of a mode they should probably be placed on overlapping panels We designed de pendency equations to describe appliance mode information in our language The prototype interfaces also showed that feedback is important One important way the interfaces provided feedback was by disabling or graying out a control so the user could tell when that function was not available Many of the errors that users made with the actual ap pliance interfaces occurred when they pressed buttons for functions that were not currently available This is another area where dependency information is helpful because it defines exactly wh
352. ny appliances The design of these elements of the language was driven primarily by the GM navigation system specification which contains many lists of complex data such as destinations Several iterations on this specification led to our current design which combined grouping with list and union features and is capable of Chapter 5 Specification Language 79 representing all forms of structured data The design has since been used without modifica tion on many other specifications including those for many of the home entertainment and office appliances Another question in our design was support for interactive list operations such as adding deleting and moving items Such operations are quite common on appliances such as ma nipulating a play list on an MP3 adding a timed recording on a VCR or changing the route list on a navigation system Standardizing these operations within the language did not seem appropriate because there are many such operations with many variations within a particu lar operation e g add before add after add at end etc Through experimentation with Table 5 1 Complete list of appliance specifications authored by the PUC research team eeueneaennusaenensesesesesaesesseaeseseeseseeseseseeeeseseeeeseseeeeseseeeeseseeeeeeseeeeseseeeeseseeeeeeseeeeeeee0eeee0eeeses0e0e00s0000S50000805050550505050050000005050500505055050500505050500505055050505505050550505058 SESESS000S00 00 SESESEESESESESSESESESSESESESSES
353. oal of this instruction was to make subjects as capable as possible with the first appliance before testing them on a second appli ance This simulated the scenario of a expert user using a new appliance with familiar functionality which is where we would expect the most impact from the PUC s consistency algorithms During both task blocks users were required to figure out how to perform each task on their own and were vot provided with a user manual or any other instruction on how to use the printer interfaces Users were allotted a maximum of 5 minutes to perform each task and were not allowed to move on to the next task until they succeeded or the maximum period was complete We chose 5 minutes based on several pilot studies that suggested that most subjects would finish within that window or else would never succeed We recorded the time that it took subjects to complete each task If a subject did not finish within the allotted pe riod we recorded his or her completion time as 5 minutes and marked the task as not being completed Our protocol has two independent variables the type of interfaces that a subject used and the order in which the subject used the two all in one appliances Three different configura tions of interface type were used in our studies e Built in One built in interface followed by the other built in interface e g HP fol lowed by Canon e AutoGen PUC interface without consistency for one appliance e
354. object or group Placement Inside the lt command gt lt explanation gt lt group gt lt list group gt lt union group gt lt state gt lt spec gt or lt map gt elements May Contain lt label gt lt ref value gt lt phonetic gt lt text to speech gt lt text to speech gt lt text to speech text lt whisper gt mute lt whisper gt recording mute au gt Defines a text to speech entry to be included within a label dictionary The text pa rameter may contain embedded SABLE markup tags Placement Inside the lt labels gt element Parameters e text required the text to be spoken May contain embedded SABLE tags e recording a recording of the text if available Appendix B Specification Language Reference 235 lt label gt lt label gt lt label gt Defines a label to be included within a label dictionary Placement Inside the lt labels gt element May Contain string lt phonetic gt lt phonetic gt lt phonetic gt Defines a pronunciation to be included within a label dictionary Placement Inside the lt labels gt element May Contain string of phonemes using the arpabet representation lt active if gt lt active if ignore all parent gt lt active if gt Contains dependency information for an appliance object or group of objects De fines an and relation with all dependencies that are contained within unless they are grouped within
355. oblem the new interface is searched for any layout sub problems that became apparent after the fix was applied This process is repeated until either no layout problems remain or the remaining problems cannot be fixed by any of the rules For example the interface for the GMC Yukon Denali driver information console before the layout fixing rules would appear as one long column of controls as shown in Figure 8 11 This column is much longer than the screen height so layout fixing rules are applied to fix this problem The first rule is the Fix with Tabs rule which finds that the long panel can be broken up into three labeled groups Because no tabs have been used to organ ize this interface yet the rule will break the long panel into three overlapping panels based on the labeled groups Tabs are used to allow the user to navigate between these panels The longer trip section can then be broken down further using the Fix with Combo Box Panel rule to create three overlapping panels for the Trip Information Personal Trip and Business Trip groups see the final interface in Figure 8 1b 8 5 2 Smartphone The Smartphone interface generator produces list based interfaces rather than the panel based interfaces of the PocketPC and desktop interface generators This different style of in terfaces leads to several unique design challenges for our Smartphone interface generator The most important challenge is to make the hierarchical list structure int
356. ocation within the specification group tree The algorithm starts by obtaining a list of all state variables that other commands and states depend upon In practice this is a small number of variables for most specifications The algorithm iterates through this list starting with the state that is de pended upon by the most other variables and commands Usually the states that are most depended upon represent higher level modes and the algorithm prefers to create high level structure earlier in the process For each state the algorithm finds its location in the group tree and gets a pointer to the state s parent group Dependencies on the state are collected from each of the parent group s children and are analyzed to find mutual exclusion If mutual exclusion is found the group tree is re arranged so that all children in each mutually exclusive set and any children not dependent on the state are each placed in their own group The parent group is then marked as containing mutual exclusion which will be taken into account when transforming the ab stract user interface into a concrete interface Figure 8 5 shows an example of how the tree is transformed to take mutual exclusive situations into account a Before b After Figure 8 5 A demonstration of changes made to the tree structure when mutual exclusion is found The cir cles represent nodes within the interface structure which could represent groups state variables or commands
357. of a Smart Template for handling a particular binary data type means that data cannot be presented in the interface Second the controls needed for displaying any binary data would need to be custom built for the particular platform which Smart Templates are already capable of adding to the generated interface Finally binary typed data requires extra communication to negotiate the particular sub types of data such as image formats that a particular platform supports This extra communication is likely to differ based on the type of binary data being managed requiring different Chapter 7 Handling Domain Specific and Conventional Knowledge 108 7 4 implementations for each type The data translation layer of the Smart Templates for different binary data types is an ideal place for this code List operations can also be handled by Smart Templates allowing for special integra tion with the controls for handling list data that would not otherwise be possible A challenge the PUC faced for handling lists was how to represent list operations There are several different kinds of list operations such as add delete move insert etc with so many different variations that did not seem practical to include as first class elements in the specification language e g add first add last add after add before and so on Ultimately I decided that most list operations could be specified as separate commands outside of the list they operated o
358. of the Black Color and Cancel BUON aara a E E E A rCn rg Pp wo Configuration of appliances in two multi appliance system scenarios a a home theater and b a presentation room ssssssssssssessssssessssesseseess Architecture of the aggregate interface generation features sesessesse An example of using the Flow Based Interface to configure a DVD player to play video through a television with the audio routed through the stereo s Speakers c ccccccccececcececcececcececceeceseccecescecescucesenceses g p Active flow control interfaces a playing a DVD movie with the video shown on the television and audio coming through the stereo s speakers b presenting Powerpoint slides through a projector c sed OD sel 140 14 144 Rea he sil DO XXIl Figure 9 5 Figure 9 6 Figure 9 7 Figure 10 1 Figure 10 2 Figure 10 3 Figure 10 4 Figure 11 1 Figure 11 2 Figure 11 3 Figure 11 4 Figure 11 5 watching broadcast television with audio playing through the television s speakers and d watching broadcast television with audio playing through the receiver s speakers Note that the volume control in c and d appear the same even though they are actually controlling different appliances sciscessttecgehitesaatstsdasctarteaee acne eet 156 Two shots of the Active Flow Setup AUI for the DVD player to receiver and television flow Note that the interface
359. oint slides through a projector c watching broadcast television with audio playing through the television s speakers and d recording a tape in one VCR from a tape playing in DOE VCR eaaa a a a aa a a ROR TOE aaa a The Philips Pronto TSU9600 remote control device eeeeeeeeeeeeeee errr 6 sists 8 bose 9 PR 10 TR 12 a 19 xvii Figure 2 2 Figure 2 3 Figure 2 4 Figure 3 1 Figure 3 2 Figure 3 3 Figure 3 4 Figure 3 5 Figure 4 1 Figure 4 2 Figure 4 3 Figure 4 4 Figure 4 5 Figure 5 1 Two Harmony remote control devices the 890 and 1000 models PROT CUE COO onie E striate E aieuacden 19 An interface generated for WinAmp on a PocketPC using the INCITS V2 framework IMTC 2006 Reproduced with permission 20 Examples interfaces for a classroom controller generated by Supple for different devices a a standard desktop computer with a mouse and b a touchscreen The classroom has three sets of lights with variable brightness an A C system and an LCD projector with a corresponding motorized screen Gajos 2004 Reproduced with PEEEMISS LOM PA E EEE EE AAE A E E 35 a The Aiwa CX NMT70 shelf stereo with its remote control and b the AT amp T 1825 office telephone digital answering machine used in OLEE TTO LT e EEA E E S E E A E ode 40 Paper prototypes of the phone a b and stereo c d interfaces for the Palm 40 Screenshots of the implemented phone a b and s
360. olean type HourMode This state optionally specifies whether the time is specified in 24 or 12 hour mode The state must have a boolean type with labels for each value One value must be la Appendix D Specification Authoring Study Instructions 292 beled as 12Hr and the other as 24Hr If this state exists it should be linked to the rendering for the time value if included Increments may be specified only for the least significant unit For example if a particular template defines an hours state and a minutes state then an increment may be specified only for the minutes state Another template that defines all four of the possible states may only have an increment specified for the fraction state time duration Overview This template describes data that stores a duration of time This could be used by media player devices to describe the length of a song or the current playback point in a song or by microwaves to display the amount of cooking time that remains This template supports resolutions in the fractions of a second seconds minutes and hours The fractions of a sec ond resolution is purposely left ambiguous and may be defined by the specification designer by defining a range for the state Milliseconds might be the fraction used for a timer applica tion whereas frame number might be used by a VCR or other video application Contents The contents of this Smart Template may be represented in several different ways
361. ommands are also useful for situations in which state is not available per haps by manufacturer choice or an inherent limitation of the appliance hardware 3 6 6 Dependency Information In most graphical interfaces there is a visual indicator when a control is disabled such as the typical grayed out appearance We have found that information about when a function is active can be specified concisely in terms of the values of state variables Not only does this allow graphical interfaces to display an indicator of whether the function is available but it can also be useful for inferring information about the panel structure and layout of the inter face Appliances with modes especially benefit from this approach because each mode is typically associated with several functions that are active only in that mode If the depend ency information is in a representation that can be analyzed the interface generator can search for sets of controls that are never enabled at the same time and then create a graphical interface that saves space and prevents user confusion by displaying only the controls for the active mode This knowledge can also be used by Universal Speech Interface applications to solve the problem of disambiguation Dependency information may also be useful for generating help information as in the UIDE system Sukaviriya 1990 UIDE used built in pre and post condition information to de termine why a particular function is n
362. ompleteness and usability Chapter 6 discusses consistency including previous work in the area and what consistency means for the PUC system This chapter also covers the infrastructure needed for generating consistent interfaces including the knowledge base that stores similarity mappings between specifications and maintains a history of interfaces that the user has seen Chapter 7 describes the Smart Templates technique that allows domain specific design con ventions to be included in automatically generated interfaces The PUC interface generation process is split into two chapters Chapter 8 describes the process for single appliance user interfaces on the PocketPC desktop and Smartphone plat Chapter 1 Introduction 15 forms Chapter 9 covers the generation of aggregate interfaces for multi appliance systems and the PUC s novel use of content flow information Chapter 10 describes a usability evaluation of the generated interfaces compared to the manufacturers interfaces for two all in one printers and an evaluation of the consistency al gorithms compared to generation without consistency Finally Chapter 11 discusses the overall PUC system reviews the contributions of the disser tation and outlines directions for future work Chapter 1 Introduction 16 CHAPTER 2 Related Work This chapter surveys previous work in three areas of research control of appliances user in terface generation and aggr
363. on then an implicit Selection variable is created which contains the index of the current selection undefined means no selection If multiple selections are created then an Appendix D Specification Authoring Study Instructions 268 implicit list group named Selections is created This group contains a Length state as all list groups do and a Selection state which contains all of the selected indices Unions A union group is similar to specifying a union in a programming language of the children within a union either groups or appliance objects only one may be active at a time An im plicit state variable named ChildUsed is automatically created within a union group that contains the name of the currently active child Dependency Information Dependency information is specified for each appliance object as a Boolean equation This information gives the interface generator some approximate a priori knowledge of when the object will be enabled see the Appliance Objects section above for more information Five kinds of dependencies can be specified Each of these dependencies specifies a state that is depended upon and a value or another state variable to compare with e Equals True when the specified state has the specified value e GreaterThan True when the specified state has a value greater than the specified value e LessThan True when the specified state has a value less than the specified value e
364. on html B 1 XML Schema lt xml version 1 0 encoding utf 8 gt lt xs schema targetNamespace http www cs cmu edu pebbles puc elementFormDefault qualified xmins http www cs cmu edu pebbles puc xmlins mstns http www cs cmu edu pebbles puc xmins xs http www w3 org 2001 XMLSchema gt lt Top Level Element gt lt xs element name spec gt lt xs complexType gt lt xs sequence gt lt xs element name labels type LabelDictionary gt lt xs element name aggregate spec type AggregateSpecTag minOccurs 0 maxOccurs 1 gt lt xs element name types type TypesGroup minOccurs 0 maxOccurs 1 gt lt xs element name groupings type MultipleGroups gt lt xs element name ports type PortsGroup minOccurs 0 maxOccurs 1 gt lt xs element name content flow type ContentFlowGroup minOccurs 0 maxOccurs 1 gt lt xs sequence gt lt xs attribute name name type xs string use required gt lt xs attribute name version type SpecVersionType use required gt lt xs attribute name guid type xs string use optional gt lt xs complexType gt lt xs element gt 205 lt Attribute Types gt lt xs simpleType name SpecVersionType gt lt xs restriction base xs string gt lt xs enumeration value PUC 2 3 gt lt xs restriction gt lt xs simpleType gt lt xs simpleType name IgnoreType gt lt xs restriction base xs string gt lt xs enumerat
365. on that is already familiar to the user These new interfaces are personally consistent because they are based on their user s previous experiences Different users may receive different generated interfaces for the same new appliance because their pre vious experiences are different My work is the first work to demonstrate that an automatic generation system can ensure consistency for its generated interfaces and that this consistency is valuable in practice 1 1 The Personal Universal Controller I have explored all of these ideas in a system that I call the Personal Universal Controller PUC which enables UI devices to be constructed and supports the automatic generation of user interfaces The generation of user interfaces is enabled by the PUC s specification language which al lows each appliance to describe all of its functions in an abstract way My goal is designing this language was to include enough information to generate a high quality user interface but not include any specific information about look or feel Decisions about look and feel are left up to each interface generator More than 30 specifications have been written for real appliances using the PUC language see the full list in Table 5 1 These specifications de scribe the complete functionality of their appliances unlike today s so called universal remote controls that typically support only the most common subset of an appliance s func tionality An autho
366. on the previous interface The third rule modifies the sidebar panels that the PUC interface generator sometimes creates around overlapping panels This rule checks the orientation of the side panel which may be either horizontal or vertical and ensures that the orientation is the same as in the previous interface 8 7 Results and Discussion After the concrete interface is generated and modified by the concrete consistency rules then the interface can be presented to the user The generation process is fast enough to run on actual PocketPC and Smartphone devices which differs from most other interface generation systems for handhelds which have offloaded generation to a remote server Ponnekanti 2001 Gajos 2005a The most complex appliance interface that the PUC has generated the GMC Yukon Denali navigation system takes approximately two minutes to generate on a Hewlett Packard iPAQ h4150 PocketPC device with a 400 MHz Intel XScale processor running the NET Compact Framework 2 0 Most appliances take less than one minute to generate All interfaces including the navigation system generate in less than 5 seconds on a current desk top computer The consistency rules do not noticeably affect generation time on either the PocketPC or Smartphone The biggest impact on performance time for the PocketPC is the layout fixing phase which must re layout the interface after every layout fix in order to evaluate whether new layout sub problems have
367. onal should have indicator lights that show when the button can be pressed Many problems such as poor la bels could be addressed with basic user centered iterative design The question remains whether it is economical for manufacturers to make these improve ments Screens and indicator lights for buttons could add substantial manufacturing cost to an appliance that already has a low profit margin Although usability is becoming more of a marketing point it is still not clear that consumers value it over price except in a few in stances e g the iPod We believe that the PUC could be an excellent solution for appliance manufacturers that currently find themselves in this situation Chapter 10 Usability Evaluation 176 CHAPTER 11 Conclusion The preceding chapters have shown how the PUC system is able to generated user interfaces that are more usable than the interfaces built into today s appliances The PUC system has also extended interface generation technology to support the design of interfaces that are per sonally consistent for each individual user and how information about content flow can be call used to produce aggregate interfaces for systems of multiple appliances This chapter dis cusses high level issues surrounding the PUC system describes the PUC systems impact to date reviews the contributions of this dissertation and describes some possible directions for future work 11 1 Discussion The PUC system w
368. ond study examined the PUC s algorithms for automatically generating interfaces that are consistent with the user s previous experience see Chapters 6 and 8 The hypothesis was that automatically generated interfaces can provide benefits beyond those shown in the first study through user customizations that would be impractical for human designers to provide In this study users were first trained on the same eight tasks from the first study using one interface After users could successfully perform these tasks they were asked to per form the same tasks on a second different interface with similar functionality The results showed that users were twice as fast when the second interface is generated by the PUC to be The work in this chapter was recently submitted for publication as Jeffrey Nichols Duen Horng Chau and Brad A Myers Demonstrating the Viability of Automatically Generated User Interfaces Submitted for Publication 161 consistent with the first interface as compared to when the second interface is generated with the consistency algorithms disabled Both user studies compare interfaces for two different all in one printer appliances a Hew lett Packard HP Photosmart 2610 with a high quality interface including a color LCD and a Canon PIXMA MP780 with a few more features and an interface that turned out to be harder to learn than the HP These two represented the top of the line consumer models from these man
369. ons on different appliances These mappings can be automatically generated by rules or manually specified by the user and in the future could made available for download over the Internet Chapter 6 Consistency 93 A mapping defines a one to one relationship between similar functions in two specifications This approach simplifies the authoring of mappings because the author of a mapping needs to only examine differences between two specifications without considering how these differ ences relate to other specifications The processing of mappings is also simplified because the consistency algorithms currently only need to understand the differences between two speci fications the specification being generated and the specification that is serving as the basis for consistency The drawback of this approach is that mappings are required between all pairs of appliances with similar functions which could be limiting if this technology be comes widespread and the similarities between many appliances must be included in the knowledge base The PUC supports six different types of mappings each of which was identified from the specifications written for the authoring studies Each mapping type is described in Table 6 1 Most of these types are generally applicable for creating mappings with any type of func tional specification language though one type is based on the PUC s unique Smart Templates construct see Chapter 7 In order to sp
370. ontrol pad as might be found on a DVD player _ or navigation system This template also includes optional support for an enter _ button that appears in the middle of the four directional buttons In general I _ hope that specification authors can avoid use of this control for navigation on _ screen menus that could be replicated on the PUC There are situations however such as for navigation of map content that cannot be displayed on the PUC where _ this control may be needed image Represents an image of any type Currently this template is used in the interfaces _ for the PowerPoint and photo browser app Images are transmitted to the PUC _ through state variables with binary types This template negotiates with the appli _ ance to get an image with a size and type compatible with the controller device Seeeeeeeeaeneeeeaeneeeeaeseeeeseseeeeseseeeeeseseeseseeeeseseseeseseseeseseseeseseseeseseeeeseseeeesepeeeeseseeseseseeseseseeeeseseeeeseseeeeseseeeeseseeeeseseeseeseseeeeseseeeeseseeseseseeeeseseeeeseseeeeseseseeseseeseseseeeeseseeeeseseeeeseseeseseseeeeseseeseseseseeseseeeeseseseeseseeeesesesseseseeeeseseeeeseseeseseseeeeseseeseseseeeeseseeseseseeeeseseeeeseseeeeseseeeeseseeeeseseeeeseseeeeseseeeeseseeeeseseeeeseseeeeseseeeeseseeeeseseeeeseees image list Handles a list of objects which must include one object that handles the image template The typical representation of this template in a user interface is two over _ lapping views
371. or Complex Appliances in Proceedings of the 15 annual ACM symposium on User Interface Software and Technology UIST Paris France October 27 30 2002 pp 161 170 Jeffrey Nichols Brandon Rothrock and Brad A Myers UNIFORM Automatically Generating Consistent Remote Control Interfaces in Proceedings of the Conference on Human Factors in Computing Systems CHI Montreal Quebec Canada April 22 27 2006 pp 611 620 and Jeffrey Nichols and Brad A Myers Controlling Home and Office Appliances Using Smartphones in ZEEE Pervasive Computing July September 2006 pp 60 67 113 8 1 Generation Platforms Interface generators have been implemented for graphical interfaces on three different plat forms PocketPC Smartphone and desktop and another generator was built for speech using Universal Speech Interface USI Rosenfeld 2001 techniques All of the graphical generators were built using the NET Compact Framework which allows code to be shared across platforms for common functions such as parsing specifications and communicating with appliances Interface generation rules are also shared many between the PocketPC and desktop and only a few between the PocketPC and Smartphone The speech interface generator was produced as a Masters project for a student in Carnegie Mellon University s Language Technologies Institute Harris 2004 This chapter will briefly summarize how this interface generator works and describe t
372. or example be moved to a special panel where all unique functions are collected Testing this premise and conducting detailed studies of the unique function issue are subjects for future work The heuristics are e Move unique functions only with their siblings e Unique functions in the middle of a group follow their preceding sibling e Unique functions at the beginning or end of a group are not reordered The central assumption behind these heuristics is that unique functions should be kept near their surrounding elements from the initial specification This is particularly important for moving when functions may end up far away from their initial location in the specification The re ordering heuristics only violate this assumption for functions that are located at the beginning of end of a group This was added because functions at the top are usually impor tant and should be left in place Functions at the bottom are usually the least important and any reorganization upward makes these items too prominent Chapter 8 Interface Generation 124 8 4 2 Functional Modifications The functional phase ensures functional consistency by inspecting each function in the new specification determining whether there is a previous function with which the new function should be consistent and then making changes to the abstract interface to ensure consis tency The previous function to be consistent with is found by traversing the mapping graph
373. or other reason the dialing buttons on a standard phone would all be commands In the VCR specification the Eject function is represented by a com mand see Figure 5 1b Commands in the PUC specification language cannot have explicit parameters as they may in other languages such as UPnP Where parameters are needed the author can use state variables and specify dependencies that require the user to specify those variables before the command can be invoked We could have allowed explicit parameters but this feature would have overlapped with state variables increased the complexity of the language and broken our only one way to specify principle Explanations Explanations are static labels that are important enough to be explic itly mentioned in the user interface but are not related to any existing state variable or command For example an explanation is used in one specification of a shelf ste reo to explain the Auxiliary audio mode to the user The mode has no user controls and the explanation explains this Explanations also represent an early attempt at in cluding help information within PUC specifications though they are rarely used Chapter 5 Specification Language 69 Although there are differences between states commands and explanations they also share a common property of being enabled When an object is enabled or active the user interface widgets that correspond to that object can be manipulated by the us
374. or the Samsung DVD VCR and in a new Status tab in the consistent interface Also note that Controls and Timed Recordings from the DVCR are located under the VCR tab on the Samsung DVD VCR Chapter 8 Interface Generation 129 exclusive groups As mentioned above there are two situations where such a move may cre ate an unusable situation when the moved items are disabled either because the interface generator may accidentally make it impossible to enable the items or the action required to enable the items is located far away from the moved items In both cases this rule causes any the moved item s dependencies to not be used in the generated user interface Thus the item is always enabled even if its actual dependencies indicate that it should not be If the user manipulates the item when it should be disabled the interface generator makes the appropri ate mode changes in the background confirms the users change and then returns any modes back to their original settings without displaying any of these changes in the user interface For example the HP all in one printer has its fax setup functions located in its fax mode whereas the Canon all in one printer has the same functionality in its setup mode When the HP interface is made consistent with the Canon the fax setup functions are moved to the setup tab where without the appropriate modifications they could never be enabled This rule allows these users to change the fax setup settings
375. oriented verti cally Any items not in groups are moved into one panel and the _ remainding items are placed in the other panel This allows one of 2 the other fixing rules to add more organization in the second panel Similar to the Fix with Vertical Split rule but creates panels that are _ oriented horizontally instead of vertically Looks for multiple consecutive buttons on a panel and compresses them into the fewest number of rows possible given the width of the panel and the buttons labels Looks for a labeled group of controls on the panel moves the group _ of controls to a dialog box and replaces them on the original panel _ with a button that opens the dialog box If the problem occurs on a one column row with no label the one _ column row is replaced with a full width row that only displays the _ control no space is left for a label If the problem occurs on a labeled two column row with no label _ the controls are moved to a row that does not save any space for dis playing a label If the problem occurs on a one column row with a label then the label and control are placed into separate rows with the label above the control 137 8 5 1 2 Fixing Layout Problems There are nine rules for fixing layout problems of insufficient height or width see Table 8 4 For each layout problem the rules are tested in order until a rule is found that can pro vide a fix to the problem After a rule fixes a pr
376. ose objects and pre conditions and post conditions for each action that describe what must be true for the action to be executed and conditions that are true once the action has been executed Pre conditions and post conditions are similar to the dependency information used in the PUC specification language The development of UIDE led to several advances in the automatic design and layout of dialog boxes It was shown that a decision tree could be constructed that performed well for choosing the interface element to use for a particular variable or action de Baar 1992 and the DON system Kim 1993 used metrics and heuristics to create pleasing layouts of interface elements The PUC interface generators use and extend these techniques Another interesting tool developed as a part of UIDE is Cartoonist Sukaviriya Chapter 2 Related Work 29 1990 a system for automatically generating animated help from pre and post condition information It may be possible to create a similar system using the PUC specification s de pendency information but that is a subject for future work 2 2 1 4 Humanoid HUMANOID Szekely 1992 is a tool for supporting the creation of the entire application going beyond the creation of menus and dialog boxes and focusing on the construction of interfaces with visualizations for complex data An important feature of HUMANOID is the designer s interface which integrates all design aspects of the system into a singl
377. ossibly passing through other appliances along the way For example the task of playing a DVD movie in a home theater can be viewed as configuring the appliances to allow content from the DVD player to flow to the television and speakers I have developed a scaleable modeling technique to make content flow information available to the UI device and the flow based interface concept to allow users to directly specify the high level task s they would like to perform An AI planner is used to automatically configure the system to perform the tasks that users specify and task based interfaces are generated based on the currently active content flows This work also advances interface generation as it is the first work to automatically generate interfaces that combine functionality from different sources Chapter 1 Introduction 5 Another problem for users is inconsistency between similar appliances from different manu facturers For example Figure 1 1 shows interfaces for three appliances that share similar photocopier capabilities Notice that while the appliances share a few of the same physical elements such as a number pad and screen the user interfaces for these appliances are quite different The methods of interaction are different as the office copier see Figure 1 1a uses a large touch screen one of the few appliances expensive enough to afford such a luxury while the other two use physical buttons for navigating menus displayed on sma
378. ot available and to generate instructions for making the function available In our comparison study with the graphical interfaces we observed that users most often sought help when they wanted to use a function that was currently inactive Chapter 3 Preliminary User Studies oy Dependency information is similar to pre and post condition information and could be used to generate the same kind of help as UIDE As mentioned above it is important that dependency information be in a form that can be analyzed by the interface generators The original version of the V2 standard included de pendency information but the dependencies were defined as arbitrary ECMAScript expressions which were difficult if not impossible to analyze This precluded the depend ency information from being used for graphical layout speech generation or command help The PUC avoids this problem by specifying dependency information as a concise set of rela tions joined by logical operations The PUC is the first system we are aware of that uses dependency information as an input to its automatic interface generator and the current ver sion of the V2 standard has adopted a similar approach 3 6 7 Sufficient Labels Our comparison study of the hand designed interfaces with the actual appliance interfaces showed that good labels are an important part of creating a high quality user interface Labels are an even more important part of speech interfaces because there a
379. ough type PassThroughType gt lt xs choice gt lt xs sequence gt lt xs complexType gt lt xs complexType name SourceType gt lt xs sequence gt lt xs element name active if type DependencyContent minOccurs 0 maxOccurs 1 gt lt xs element name output ports type ContentPortType minOccurs 1 maxOccurs 1 gt lt xs element name objects type ContentStatesType minOccurs 0 maxOccurs 1 gt lt xs sequence gt lt xs attribute name name type xs string use required gt lt xs attribute name content type type ContentTypeAttrib use required gt lt xs complexType gt lt xs complexType name SinkType gt lt xs sequence gt lt xs element name active if type DependencyContent minOccurs 0 maxOccurs 1 gt lt xs element name input ports type ContentPortType minOccurs 1 maxOccurs 1 gt lt xs element name objects type ContentStatesType minOccurs 0 maxOccurs 1 gt lt xs sequence gt lt xs attribute name name type xs string use required gt Appendix B Specification Language Reference 214 lt xs attribute name content type type ContentTypeAttrib use required gt lt xs complexType gt lt xs complexType name PassThroughType gt lt xs sequence gt lt xs element name active if type DependencyContent minOccurs 0 maxOccurs 1 gt lt xs element name input ports type ContentPortType minOccurs 1 maxOccurs 1 gt l
380. ough this feature could be added Unfortunately there is no obvious solution to the second problem besides educating consumers My hope is that manufacturers may find this issue less of a concern if consumers begin to demand more usable interfaces and the PUC can offer these interfaces at less cost than hand designing such interfaces in house After all most PUC devices can provide far better interaction hardware than is cost effective to put into each appliance I also hope that manufacturers will be more willing to adopt PUC technologies if it can be shown that interface generation can offer benefits not possible in the hand designed inter faces on the actual appliances In particular the ability of the PUC to provide improved interfaces automatically for systems of multiple appliances cannot be implemented with non automatic generation approaches There also seems to be considerable value in generating interfaces with personal consistency as shown by the evaluation Generation of consistent interfaces is another issue that likely to concern manufacturers how ever particularly where branding is concerned The current version of the PUC s consistency system is quite likely to make changes that would be inappropriate such as copying a branded label from one appliance interface into an interface for an appliance from a compet ing brand For example the Mitsubishi DVCR appliance has an automatic rewind function that is labeled as RentalE
381. our own description of this variable and then comparing it to my cards or XML shown in Ap pendices B amp C respectively Specifying List Data Lists are specified using the special list group feature which is specified on the group card see Figure or with the lt list group gt element in XML see Figure I have decided that the list does not have any bounds and there will be one selection and these decisions are re Appendix D Specification Authoring Study Instructions 279 flected in the descriptions that you see below Note that in the card based system a list pa rameter card must be attached to the group card to define the bounds and selections Group Information f List Parameters Name 1S t Size Priority 1d 0 10 P Unrestricted Smart Template O Fixed Number of Elements Labels O Bounded Min Max Selections Rl One Write additional labels on back o Many YX List Group O Union Group O Read only Figure D 3 The group and list parameters card that describe the to do list This list has four state variable members not shown lt list group name List priority 10 gt selections number one gt lt list group gt Figure D 4 The XML code for describing the to do list The four state variable are defined where the appears in the above code Each of the variables contained within a list group has multiple values one value for each item in the list If you are familiar with prog
382. pe MappingCostType minOccurs 1 maxOccurs 1 gt lt xs choice minOccurs 1 maxOccurs 1 gt lt xs sequence gt lt xs element name initial type SpecStepsType minOccurs 1 maxOccurs 1 gt lt xs element name step type SpecStepsType minOccurs 1 maxOccurs 1 gt Appendix C Other PUC XML Language Schemas 259 lt xs sequence gt lt xs sequence gt lt xs element name specl type OperationType minOccurs 1 maxOccurs 1 gt lt xs element name spec2 type OperationType minOccurs 1 maxOccurs 1 gt lt xs sequence gt lt xs choice gt lt xs sequence gt lt xs complexType gt lt xs complexType name SpecStepsType gt lt xs sequence gt lt xs element name specl type OperationType minOccurs 1 maxOccurs 1 gt lt xs element name spec2 type OperationType minOccurs 1 maxOccurs 1 gt lt xs sequence gt lt xs complexType gt lt xs complexType name OperationType gt lt xs sequence gt lt xs choice minOccurs 1 maxOccurs unbounded gt lt xs element name change type ChangeOperationType gt lt xs element name invoke type InvokeOperationType gt lt xs element name repeat type RepeatOperationType gt lt xs choice gt lt xs sequence gt lt xs complexType gt lt xs complexType name RepeatOperationType gt lt xs sequence gt lt xs element name count type CountType minOccurs 1 maxOccurs 1 gt lt xs choice minOccurs 1 maxOccurs unbounded g
383. pe StaticOrReference minOccurs 0 maxOccurs 1 gt lt xs element name incr type StaticOrReference minOccurs 0 maxOccurs 1 gt lt xs element name specific values important minOccurs 0 maxOccurs 1 gt lt xs sequence gt lt xs complexType gt lt xs complexType name FixedPtType gt lt xsS sequence gt lt xs element name pointpos type xs integer minOccurs 1 maxOccurs 1 gt lt xs element name min type StaticOrReference minOccurs 0 maxOccurs 1 gt lt xs element name max type StaticOrReference minOccurs 0 maxOccurs 1 gt lt xs element name incr type StaticOrReference minOccurs 0 maxOccurs 1 gt lt xs element name specific values important minOccurs 0 maxOccurs 1 gt lt xs sequence gt lt xs complexType gt lt xs complexType name EnumeratedType gt lt xsS sequence gt lt xs element name item count type xs integer minOccurs 1 maxOccurs 1 gt lt xs sequence gt lt xs complexType gt lt xs complexType name BinaryType gt lt xS sequence gt lt xs any minOccurs 0 maxOccurs unbounded gt lt xs sequence gt lt xs complexType gt lt xs complexType name ActivelfType gt lt xs choice minOccurs 1 maxOccurs unbounded gt lt xs element name or type DependencyContent gt name and type DependencyContent gt lt xs elemen lt xs element name not type NotType gt name apply over type ApplyOverType gt
384. pec xmins http www cs cmu edu pebbles puc name SpecNameHere wWersion PUC 2Z2 3 gt The name attribute may be filled in with an appropriate name of your choice The xmlns attribute is the most important as it defines to Visual Studio the schema that you want to use for validation To verify that Visual Studio has appropriately linked your file to the schema open the XML menu and select the Validate XML Data item If you see the following error then the schema has not been found Visual Studio could not locate a schema for this document Validation can only ensure this is a well formed XML document and cannot validate the data against a schema Double check that you have copied the schema file to the appropriate directory above If eve rything appears to be correct then try restarting the Visual Studio application If you still have problems please contact me Appendix D Specification Authoring Study Instructions 276 3 Example Specification Walkthrough Now we will walk through the design of a specification for a hypothetical to do list applica tion Note that while this example uses one process for building a specification you do not need to mimic this process when designing your own specifications We do not know what the best process is for writing specs and part of this study is exploring different methods and their outputs The to do list application has the following functionality e List of to
385. pecific consis tency guidelines have been proposed such as those described by Nielsen Nielsen 1993 These include maxims such as the same information should be presented in the same loca tion on all screens Another mechanism for maintaining consistency is called design languages Rheinfrank 1996 which are used by designers to ensure that common features and branding are shared across a family of products ITS SUPPLE and DiamondHelp are all systems that address the consistency of user inter faces with automatic design The ITS system Wiecha 1990 was successful in producing consistent interfaces across a family of applications such as all the displays for a World s Fair and for multiple versions of the same application Interfaces were generated using a rule based approach and consistency resulted because the rules applied the same interaction technique in every place where the same condition was found Note that although the PUC does achieve some consistency by using the same generation rules for each interface it cannot rely on the underlying appliance models being the same Appliances of the same type may have substantially different underlying models and one of my contributions is finding simi larities between underlying models and creating consistency based upon these similarities Chapter 6 Consistency 87 SUPPLE Gajos 2004 automatically generates layouts for user interfaces using an optimiza tion approach to
386. pensive and time consuming The trend has been that as appli ances get more computerized with more features their user interfaces get harder to use Brouwer Janse 1992 The Wall Street Journal reports that appliances TVs telephones cameras washing machines microwave ovens are getting harder to use The result is a new epidemic of man machine alienation Gomes 2003 One solution to these problems is to move the user interfaces from the appliances to some other intermediary UI device that is independent of the appliance manufacturer and fo cuses solely on the user interface A UI device could be a handheld computer like Microsoft s PocketPC a mobile phone or even a speech system built into the walls of a building One advantage of this approach is that people are increasingly carrying a mobile device that could be used as such a UI device Many of these devices already have the ability to communicate with other devices in the environment using wireless networking protocols like 802 11 Wi Fi or Bluetooth Furthermore these mobile devices are built with special ized interface hardware like color and or touch sensitive LCD screens which make the creation of high quality user interfaces easier A UI device could leverage its specialized hardware to provide a superior user interface as compared to what can be cost effectively built into an appliance There are several potential approaches for moving the user interface
387. player that was automatically generated by the Debug Server Chapter 4 System Implementation 58 4 3 Generating Interfaces on Controller Devices I have built graphical interface generators for three different platforms Microsoft s Pock etPC Smartphone and TabletPC and collaborated on the creation of a speech interface generator with researchers from Carnegie Mellon s Language Technologies Institute LTT All interface generators speak the PUC communication protocol and generate interfaces from the specifications written in the PUC s specification language The speech interface generator creates interfaces using Universal Speech Interface techniques Rosenfeld 2001 developed by my collaborators in LTI This generator was implemented using a variety of speech technologies developed at Carnegie Mellon including the Phoenix parser Ward 1990 and the Sphinx recognizer CMU 2006 Unlike the other interface gen erators the speech generator attempts to determine up front which appliances the user will want to control and then it automatically generates a grammar that supports control for all of those appliances A language model and a pronunciation dictionary are also automatically generated to assist the speech recognizer The graphical interface generators are all implemented in C using the NET Compact Framework 2 0 Even though the generators run on different platforms using NET has al lowed me to share a substantial amount of
388. pliance Parameters are specified in terms of the primitive elements of the specification language and consist of a list of the state variables and commands that the template may contain along with definitions of the names types values and other properties that these elements must have Some of the elements may be optional to support functions that would not be used in all instantiations of a template For example two representations of play controls are allowed by the media controls template a single state with an enumerated type or a set of commands If a single state is used then each item of the enumeration must be labeled Some labels must be used such as Play and Stop and others are optional such as Record If multiple commands are used then each command must represent a function such as Play and Stop Some functions must be represented by a command and others are optional This template also allows three commands for functions that are commonly included in the same group as the play controls including the previous and next track functions for CD and MP3 players and the play new function for answering machines Allowing many combinations of states and commands in a template definition allows a single Smart Template to be applied across multiple kinds of appliances see Figure 7 3c The challenge for the template registrar is to find the different combinations of states and commands that an appliance implementer is likely to use This m
389. plore might be wristwatches which have even more limited interaction capabilities than smart phones To support a platform like wristwatches it might be necessary to build a user interface that supports only the most common functionality of an appliance rather than creating interfaces for the full functionality of an appliance like my current generators do The challenge is to reliably determine the functions that users will want to use from the full appliance specifica tion The group tree and the priority information of the specification language already contain information that should help with making this decision though determining the proper threshold for functions to keep versus functions to omit may be difficult to find In particular it would unfortunate if the appliance has a set of items that are used together such as a list and several commands that operate on the list and not all of the set is included in the generated interface It might also be interesting to create an interface generator for web interfaces Such a genera tor might be able to integrate ideas from each of the existing generators to adapt the web page layout for the particular device that the web pages will be rendered upon New Web 2 0 technologies could also be used to create appliance interfaces on a web page with similar levels of interactivity as that of the interfaces currently generated by the PUC While interface generators can be built for many e
390. pped Play This label identifies the state when the media is playing Pause This label identifies the state when the media is paused Rewind This label identifies the state when the media is rewinding Fast Forward This label identifies the state when the media is fast forwarding Record This label identifies the state when media is being recorded If no state information is available from the appliance a command based description may be used instead Such a representation must include one or more of the following commands These commands should not be included if the Mode state is included in the description Al lowable commands are Play When this command is activated the appliance should begin playing Stop When this command is activated the appliance should stop playing Pause When this command is activated the appliance should pause playback Rewind When this command is activated the appliance should begin rewinding FastFwd When this command is activated the appliance should begin fast forwarding Record When this command is activated the appliance should begin recording Appendix D Specification Authoring Study Instructions 290 In addition to having either the state or a set of commands additional states and commands may be specified that will be integrated with the rest of the playback controls Commands NextTrack When this command is activated the next track should be selected PrevTrack When t
391. ppliance with func tionality that is similar to a previous appliance that the user has seen the interface generator uses special algorithms to ensure that the similar functions are represented the same way and placed in the same location The difficulty with generating consistent interfaces is that while Chapter 1 Introduction 8 PUC v2 0 3 Windows NX Yj 1 Play Control 2 Title Why Georgia 3 Time 4 Volume 52 5 Mute False 6 Media Player Info PUC v3 0 Simulated Elt T 1 Floor 1 Called 2 Floor 2 Off 3 Floor 3 Off 4 Floor 4 Off 5 Floor 5 Called 6 Security Code 7 Open Doors Holding 8 Close Doors Off 9 Exit fee gut PUC v3 0 Audiophase lt 1 Power On 2 Volume Up 3 Yolume Down 4 X Bass Off Figure 1 3 Interfaces generated by the PUC on the Smartphone for several different appliances a Windows Media Player b a simulated elevator c d a shelf stereo with CD radio and tape functionality Figure 1 4 The full PocketPC interface generated for the navigation system from a 2003 GMC Yukon Denali The arrows show buttons that cause a dialog box to open Chapter 1 Introduction 9 AF Puc v3 1525 Pf et e250 Q f FJ Puc v3 1 525 Pf oP d 249 amp of Copies l Copy Copy of Copies Light Dark Paper Select Paper Select Light Dark Content Quantity Quantity Content Halftone 2 Sided Copy
392. projects also suggests that the PUC technologies are also easy enough to learn and make use of that people can adopt them suc cessfully with little support from me 11 3 Contributions This dissertation describes a complete system that can improve the usability of today s com plex computerized appliances by moving the user interface to a handheld device that the user is already carrying The interfaces are automatically generated to be customized to the con troller device the user and the particular configuration of appliances that the user has The generation process is supported by a specification language that is capable of describing the complete functionality of a wide range of appliances The language was tested in an au thoring study that showed that authors with no previous experience could learn the language in about 1 5 hours and write a complete specification for a low cost VCR in about 6 hours The dependency information in the specification language can be used by the interface gen erators to infer some of the structure of the user interface When the dependencies for multiple sets of functions are found to be mutual exclusive the interface generator can take special action by placing the sets of functions on separate overlapping panels The Smart Templates technique allows domain specific design conventions to be referenced in appliance specifications and rendered appropriately in the generated interfaces Templates
393. pt a system like the PUC The PUC requires features like two way communication access to all of the func tions of the appliance and support for a more reliable physical communication layer than IR such as Bluetooth Wi Fi or Zigbee Some of these features add development cost to a new appliance such as changing the appliance s software or writing a functional specifica tion Other features add to the manufacturing cost for each unit such as adding new hardware to support a new wireless communication protocol An increase is manufacturing cost is thus substantially worse than an increase in development cost because any manufac turing cost increases are multiplied by the number of units produced Some increases in development cost to support the PUC may not be as substantial as they might seem For example most manufacturers already develop some form of functional specification as part of the product development process While these specifications may not include all of the same kinds of information as a PUC specification they should provide a good starting point My intuition is that writing a PUC specification for an appliance would not add substantially to the development cost of an appliance over current industry practices Many manufacturers are also working to add support for general communication infrastruc tures such as UPnP to their appliances These technologies have similar server side requirements as needed for the PUC such as
394. pulate the appliances so that they work properly with each other Troubleshooting problems when they occur can be difficult because the user must determine which appliances are configured correctly and which are not It is common for users to mistakenly believe that the problem lies in an appliance that is working correctly which lengthens troubleshooting time and frus trates users These interface problems occur for two reasons 1 the user interfaces for each appliance are designed separately with few if any cues about how one appliance s interface will affect the operation of other appliances and 2 for each task that might be performed with a system each appliance has many functions that are not relevant for performing that task These functions though important for other tasks may interfere with users progress on their current task A UI device that automatically generates interfaces can address this problem by taking into account how appliances are connected A UI device can also combine the user interfaces of multiple appliances into a single interface that is customized to the user s task given that the UI device knows what task the user is trying to perform and how the appliances work to gether to perform that task To obtain this information I have a developed a novel solution based on the observation that most tasks in a system of appliances involve the flow of content from a source appliance to a sink appliance p
395. r has been success fully programmed and beeps twice if programming failed UI devices are able to address these problems for several reasons UI devices can afford to have screens that are larger and higher resolution than the screens on most appliances These screens can render at least a few lines at a time which allows for better explanations of error conditions and feedback on the appliance s state A UI device also has built in infrastructure for supporting the development and rendering of complex user interfaces This typically in cludes a toolkit that supports the creation of on screen virtual controls such as buttons and sliders and a set of UI guidelines that govern how the interface should look and feel Through use of the built in infrastructure a UI device can ensure that every appliance func tion is represented by a separate control and prevent the functional overlap that makes so Chapter 1 Introduction 4 many appliances hard to use An automatic interface generator can also make its interfaces easier to use by following the UI guidelines of the device and allowing users to leverage their knowledge of conventions used by the UI device s other applications Another usability problem for appliances arises when multiple appliances are used together in a system such as in a home theater or presentation room In order to accomplish their tasks users must understand both how to use each appliance individually and how to mani
396. r whereas human designers are limited to designing interfaces for large user groups Although previous work has examined how interfaces might be gener ated for multiple platforms from the same model Szekely 1995 Eisenstein 2001 until now there had been no exploration of how automatic generation might be used to customize in terfaces to each individual user and their particular environment and situation An important question to ask then is in what specific ways can UI devices with automatic interface generation be used to improve the usability of appliances To answer this question I will discuss several usability problems and how my approach can address them One of the biggest problems for appliance user interfaces is the large number of functions present in most appliances coupled with the limited number of interactive physical elements such as buttons and screens which can be built into a computerized appliance To address this problem most appliances overlap two or more functions on each button For example on the AT amp T 1825 telephone answering machine pressing and releasing the play button will play all messages but pressing the play button and holding it down will play only the new messages The limited size of the display screens also means that most appliances rely on cryptic messages or audio cues to give feedback to the user For example the same AT amp T answering machine beeps once to notify the user that a speed dial numbe
397. ramming languages this is similar to defining an array of records or structs Each to do list item has four values description completion date category and completion flag so this list group will contain four different state variables The specification of the description completion date and completion flag variables is left as an exercise but the category variable is discussed in the next sub section There is also another list in the to do list application which contains each of the valid cate gories that a to do list item may have As an exercise I suggest that you attempt to specify this list group including its state variable s You can check your work by referring to the completed specifications in Appendices B amp C Using the List Selection Type Appendix D Specification Authoring Study Instructions 280 The categories variable in the to do list has a value that is one of the items in the user configurable categories list This means that we cannot use an enumerated type for this vari able because we do not know in advance what categories the user will choose Our language supports this problem with the list selection state variable type In this case we can create a category variable within the to do list group that has a list selection type The type will specify that it selects an item from the category list which is elsewhere in the speci fication When the user changes the category list the changes will autom
398. rates the crea tion of the user interface from the manufacturers of the appliances and the controller devices This gives UI devices a greater opportunity to improve upon the usability of current appliance interfaces than the other approaches The technical focus of this dissertation is on the automatic generation of user interfaces for end users Researchers have been exploring the automatic generation of user interfaces for nearly two decades but most work relies on an interaction designer to guide the generation process and or to edit the resulting user interfaces to fix any design flaws Szekely 1996 I do not believe that end users of any UI device will be willing to spend the time and effort to modify their automatically generated interfaces in this way and thus a UI device will need to generate high quality user interfaces on the first attempt Chapter 1 Introduction 3 Previous work in automatic interface generation has also focused in large part on building user interfaces for desktop applications a task at which trained human designers can produce higher quality artifacts than any current automated system While I would like interfaces generated by my system to approach the quality of human designers my focus is instead on applying automatic generation techniques to create interfaces that would not be practical for a human designer to produce For example an automated system can produce a custom in terface for every individual use
399. rcular screen and dials for interaction would be very difficult to sup port with this approach as it will not practical for appliance manufacturers to produce new interfaces for all of their appliances every time a new handheld device is released Most of today s web technologies can be seen as using this approach as many pages are carefully designed to support large resolution screens and do not ren der appropriately on small devices like mobile phones e The controller device downloads abstract specifications from appliances and uses those specifications to automatically generate an interface that is customized to the control lers particular design properties of the user and the user s environment It is important that the specification be abstract enough to support the creation of inter faces on different platforms and in different modalities but also contain enough information about the appliance in order to create a high quality user interface One advantage of this approach is that neither the controller devices nor the appliances need to know anything about the other in advance to ensure that the interfaces are usable Another advantage is that external factors can be taken into account in the design of the interface such as the user s previous experience or the functionality of other connected appliances Although the automatic generation of user interfaces is difficult I believe this approach is the most promising because it sepa
400. rder gt and lt renderer gt elements May Contain ot lt output ports gt lt OULDUL pOrtcs gt lt coutpUuL ports gt Describes the output ports that may be used by a content flow Placement Inside the lt pass through gt or lt source gt elements Appendix B Specification Language Reference 248 May Contain ports lt processing gt lt processing gt lt processing gt Defines processing that may be done to a content stream as it is being passed through an appliance Placement Inside the lt pass through gt element May Contain olock gt lt channel gt lt channe value 4 gt lt channel state Base Controls Channel gt Defines the channel of a multi channel content stream to be modified by the appli ance Must have one of the two parameter options Placement Inside the lt port gt element Parameters e value Defines a state value for the channel e state Defines a state variable or group that will contain the value of the channel to be operated on Appendix B Specification Language Reference 249 lt block gt lt blLock channel 4 gt lt block gt Defines the block operation that may be applied by an appliance to one or more channels of a multi channel content stream When this operation is active the chan nel s received from the input are not passed through the output Placement In the lt processing gt element Parameters
401. re no graphical hints to assist the user s understanding of the interface To give flexibility to the interface generator a label in an appliance specification should not be a single text string but instead a collection of text strings pronunciation keys and text to speech recordings Pronunciation keys and text to speech recordings help improve the quality of the speech interface Multiple text strings give a graphical interface generator the flexibility to select the label with the most informa tion that can be fit in the allotted space The PUC was the first system to provide more than just single string text labels and this approach has since been adopted by the V2 standard 3 6 8 Shared High Level Semantic Knowledge Despite all of the previous requirements we must recognize that it is impossible to encode all the information into an appliance specification that a human would use to design an inter face In addition to functional information about the appliance a human designer will also use his knowledge of conventions when creating an interface There are many such conven tions such as the arrangement of buttons on a telephone number pad or the country specific format for specifying dates Conventions tend to be used across different types of appliances but their usage will differ depending on the functionality of each appliance For example media controls like play stop and pause can be found on many appliances but while most
402. ready be available for interfaces built with a model based development environment such as Mobi D 2 2 2 2 ConcurTaskTrees Early options for specifying task models were the formal specification language LOTOS ISO 1988 or GOMS Card 1983 and many of the first model based systems to use task models created their own languages for specifying the models Recently ConcurTaskT rees Paterno 1997 has become popular as a language for representing tasks in several model based systems including TIDE Ali 2002 and TERESA Mori 2004 ConcurTaskT ees is a graphical language for modeling tasks that was designed based on an analysis of LOTOS and GOMS for task modeling ConcurTaskTrees extends the operators used by LOTOS and allows the specification of concurrent tasks which is not possible in GOMS Concur TaskTrees also allows the specification of who or what is performing the task whether it be the user the system or an interaction between the two A special development environment was built for creating task models using ConcurTaskTrees called the ConcurTaskTrees Envi ronment CTTE Mori 2002 Chapter 2 Related Work 32 2 2 2 3 XIML The eXtensible Interface Markup Language XIML Puerta 2002 is a general purpose lan guage for storing and manipulating interaction data based on Mobi D XIML is XML based and capable of storing most kinds of interaction data including the types of data stored in the application task and presentation
403. red gt lt xs attribute name physical type type xs string use required gt lt xs complexType gt lt xs complexType name PortGroupType gt lt xs choice minOccurs 1 maxOccurs unbounded gt lt xs element name port type PortType gt lt xs element name port group type PortGroupType gt lt xs choice gt lt xs attribute name name type xs string use required gt lt xs attribute name content type type ContentTypeAttrib use required gt lt xs complexType gt lt xs complexType name ContentFlowGroup gt lt xs choice minOccurs 1 maxOccurs unbounded gt lt xs element name content group type ContentGroupType gt lt xs element name source type SourceType gt name renderer type SinkType gt lt xs elemen lt xs element name recorder type SinkType gt Ce Gi eh Gk lt xs element name pass through type PassThroughType gt lt xs choice gt lt xs complexType gt lt xs complexType name ContentGroupType gt lt xs sequence gt lt xs element name active if type ActivelfType minOccurs 1 maxOccurs 1 gt lt xs choice minOccurs 1 maxOccurs unbounded gt lt xs element name content group type ContentGroupType gt lt xs element name source type SourceType gt name renderer type SinkType gt lt xs elemen lt xs element name recorder type SinkType gt Ch yer Cha ck lt xs element name pass thr
404. rface etc and the labels that its values take _ types that the re use throughout a specification pendency that a state variable must not have any value valid value specific values that the variable might have 220 B 3 Element Descriptions lt spec gt lt spec name Sample Specification version PUC 2 1 gt lt spec gt Every specification begins with this element The name specified in the name attrib ute is a machine readable name for the parser The contained lt labels gt element specifies human readable names for the appliance Placement First element of the spec after the required XML header E g lt xml version 1 0 encoding UTF 8 gt Parameters e name required The name of the appliance defined in this specification e version required The version of the specification language being used Cur rent valid values are PUC 2 0 PUC 2 1 PUC 2 2 and PUC 2 3 This document describes PUC 2 3 e guid required A global identifier for this specification Must Contain lt groupings gt lt labels gt lt types gt lt ports gt lt content flow gt lt types gt lt types gt lt types gt Contains snippets of specification that the author will reuse in the groupings section Placement Inside the lt spec gt element May Contain Sroup lt List qroup lt unLon gqroup gt lt statle gt lt command gt lt explanation gt lt type gt
405. rface that might not be visible on the actual appliance Graphical interfaces can also use the current state coupled with dependency information to disable components that are not currently active Knowledge of the current state is also very important for speech interfaces which must be able to respond to user queries about the current state even if the information is available visually on the appliance This is especially helpful for blind users or when the user is not near the appliance 3 6 2 Simultaneous Multiple Controllers It is also important that multiple controllers can communicate with the same appliance si multaneously Users will expect this feature and it has the added benefit of allowing different interface modalities to be freely mixed together by using several different controller devices in tandem For example a user might combine a handheld controller with a headset Chapter 3 Preliminary User Studies 50 to create a multi modal graphical and speech interface Most current systems seem to fulfill this requirement 3 6 3 No Specific Layout Information The appliance specification should include information about the functions of that appli ance but it should not include specific information about how controls should be positioned on the screen We share this philosophy with the V2 INCITS V2 2003 and Xweb Olsen Jr 2000 projects but not with UIML Abrams 1999 which can include concrete information in its descriptio
406. rfaces speech recognition Smart Templates user interface consistency personal consis tency familiarity il Automatically Generating High Quality User Interfaces for Appliances Jeffrey Nichols ill IV Abstract In this dissertation I show that many appliance usability problems can be addressed by mov ing the user interface from the appliance to a handheld device that the user is already carrying such as a Personal Digital Assistant PDA or mobile phone This approach called the Personal Universal Controller PUC takes advantage of the increasing pervasiveness of wireless communication technologies that will allow handheld devices to communicate di rectly with appliances Automatic generation of the appliance user interface allows the PUC to create interfaces that are customized to the platform of the controller device the user s previous experience and all the appliances that are present in the user s current environment This dissertation makes several contributions to the state of the art in automatic interface generation e Automatic generation that makes use of dependency information to determine better structure for the generated user interfaces e The general Smart Templates technique for incorporating domain specific design con ventions into an appliance specification and automatically rendering the conventions appropriately on different platforms and in different interface modalities e Algor
407. ring study of the specification language had subjects with no previous knowledge learn the language and then produce a specification of moderate size for a low cost VCR Subjects were able to learn the language in an average of 1 5 hours and produce the VCR specification in around 6 hours Four interface generators have been implemented that produce user interfaces from specifica tions written in the PUC s language including graphical interface generators on PocketPC Microsoft s Smartphone and desktop computers and a speech interface generator that uses the Universal Speech Interfaces USI framework Rosenfeld 2001 The graphical interface generators for the PocketPC and Smartphone cover two very different interface styles The Chapter 1 Introduction 7 AF PUC v3 1 438 4 10 59 amp Copy Qualityixd Curren t Location Jee Current Title The Old Apartment PORAN i PSDS i 1400 Pennsylvania Ave NW 7 E zen E Presets 00 00 03 2 Lighter Darker e o Volume ocation ress ca Es KA i L Mute i a Is Color Saturation 0 Status Stopped Add Destination m m Lesst vivid Most vivid Version 9 0 0 3008 Add Preferred Road EN EJ Le Enhancements avigation Configuration A PUC Servers Devices Ej te La a 27 a b d Figure 1 2 Interfaces generated by the PUC on the PocketPC for several different appliances a Windows Me dia Player 9 b the navigation system from a 2003 GMC Yukon Denali
408. rithm Work is needed to understand what the impact of errors in the knowledge base are on the current consistency rules and whether the rules can be changed to mitigate any problems that arise More work is also needed on the mapping algorithms themselves In particular it would be interesting to explore whether mapping can be improved by leveraging the existing content in the knowledge base Use of the knowledge base would provide more information from which to make a match It is also possible that the knowledge base could contain information about likely sources of false positives in the mapping algorithm which could be used to filter the generated mappings and reduce errors Chapter 11 Conclusion 189 The PUC s consistency algorithms ensure personal consistency which has an ordering effect new interfaces are generated based on previous interfaces If some aspect of the previous in terfaces was bad then that low quality element might be copied into the new interface One of the requirements of my consistency algorithms was to support user choice so that users could control how consistency is applied and choose the best possible interface The interface to allow users to make these choices has not been implemented yet however It is also unclear how users will know that a better interface is available if they never see it because of the consistency algorithms Additional work is needed to understand how users can be made aware of th
409. rk was conducted on command line interfaces however and may not apply to today s graphical user interfaces GUIs Reisner Reisner 1981 explored consistency through two drawing applications and found that users made fewer mistakes with the interface that was structurally consistent In Reis ner s case structural consistency is defined in terms of the production rules describing the interface An interface where all of the rules for a similar action have the same form would be structurally consistent whereas an interface that uses rules of several different forms would be inconsistent For example shape selection in the first application was inconsistent because most shapes were selected by pressing the select button flipping the switch for the desired shape type and pressing the GO button whereas no action was required to select a text shape because the keyboard was always available The second application used the same rule moving the cursor to the box specifying the desired shape type and was thus structurally consistent The consistency that Reisner explores is within an application however and she did not examine how users who learned the first drawing application subsequently performed on the second It is this latter form of consistency that the PUC system aims to provide Polson and his collaborators showed that the common elements theory of knowledge transfer from experimental psychology could account for the positi
410. roups show the dependencies of each group on the variable A The A node is not dependent at all on A b In the after tree a new mutual exclusion group has been added that contains the A state variable and child groups for each set of mutually exclusive groups The nodes not dependent on A are moved into their own group under the same parent but outside of the mutex group vies veers Meee teeeeeanees Containment stacks for the previous specification Mitsubishi DVCR the new specification Samsung DVD VCR and the results of two consecutive movements a Shows the movement of the clock group and b shows how the rule chains with the movement of the clock channel state cccccccccccccccccccccccececcececcsceccsecceccecescucseecesecceseecess User interfaces generated by the desktop PUC for a Mitsubishi DVCR and a Samsung DVD VCR without consistency and the Samsugn DVD VCR generated to be consistent with the Mitsubishi DVCR Note that the clock functions are located under the Status tab for the Mitsubishi DVCR under Setup for the Samsung DVD VCR and in a new Status tab in the consistent interface Also note that Controls and Timed Recordings from the DVCR are located under the VCR tab on the Samsung DVD VCR 0 eee eeeeeeeseeeeeneeeeees A demonstration of the Bring Together Split Dependents rule The top image shows the location of the resize mode function and the bottom shows the location of the repeat image function and i
411. rrect steps are not counted as additional missteps No missteps are counted for a task after the user has requested external help 3 3 Study 1 The first study compared the actual appliance interfaces to our hand designed paper proto types see Figure 3 2 For the handheld portion of our experimental procedure subjects were given a stylus and a piece of paper that showed a picture of a Palm V handheld device displaying the remote con trol interface Subjects were instructed to imagine that the picture was an actual handheld and to interact with it accordingly Whenever the subject tapped on an interface element on the screen a new picture was placed over the old one to show the result of the action If auditory feedback was required such as when the subject pressed play on the CD panel of the stereo see Figure 3 2c the test administrator would verbally tell the subject what hap pened Chapter 3 Preliminary User Studies 43 3 3 1 Participants Thirteen Carnegie Mellon graduate students volunteered to participate in this study five fe male and eight male All subjects were enrolled in the School of Computer Science and all had significant computer experience Seven owned Palm devices at the time of the study Only one subject had no Palm experience and the remaining five had exposure to Palm de vices in class or through friends Everyone in the group had some experience with stereo systems Only two did not have a stereo Four
412. rs 1 gt lt xs sequence gt lt xs attribute name name type xs string use required gt lt xs attribute name channel type xs string use optional gt lt xs complexType gt lt xs complexType name ChannelType gt lt xs attribute name State type xs string use optional gt lt xs attribute name value type xs string use optional gt lt xs complexType gt lt xs complexType name ProcessingType gt lt xs choice minOccurs 1 maxOccurs unbounded gt lt xs element name block type BlockProcessingType gt lt xs choice gt lt xs complexType gt lt xs complexType name BlockProcessingType gt lt xS sequence gt lt xs element name active if type DependencyContent minOccurs 0 maxOccurs 1 gt lt xs element name Channel type ChannelType minOccurs 0 maxOccurs 1 gt lt xs element name port type BlockPortType minOccurs 0 maxOccurs unbounded gt lt xs sequence gt lt xs attribute name Channel type xs string use optional gt lt xs attribute name port type xs string use optional gt lt xs complexType gt lt xs complexType name BlockPortType gt lt xs attribute name name type xs string gt lt xs complexType gt lt xs schema gt Appendix B Specification Language Reference 215 B 2 Element Name lt active if gt Contains dependency information for an appliance object or a group sand gt Defines an AND relation with the dependencies t
413. rtphone generator starts complex and simplifies but neither can guarantee an optimal result Both are also sensitive to the struc ture of the abstract interface and the order in which rules are applied Optimization algorithms are less predictable however and require far more computational resources to process A hybrid approach may be most successful but this is a subject for future research Chapter 8 Interface Generation 146 CHAPTER 9 Aggregating User Interfaces The last chapter discussed how the PUC automatically generates user interfaces for a single appliance This chapter describes how the PUC is able to generate a user interface that com bines functionality from multiple appliances that are connected in a system A key input to this generation process is a content flow model which describes the different routes that content take within the system from a source appliance possibly through one or more passthrough appliances to a sink appliance where that content is either displayed for the user or recorded for later use From the content flow model the PUC creates two types of inter faces a flow based interface that allows users to quickly specify their high level goals for the system of appliances and aggregate interfaces that provide low level control while a flow is active The interface aggregation features of the PUC are collectively known as Huddle The work in this chapter was originally published in Jeffrey Ni
414. rts gt Figure 5 6 The ports section of the example VCR specification their systems The specification language also supports port groups which allow the author to give a single name to collection of related ports typically ports that carry a piece of a larger content stream For example in Figure 5 6 the Output port group is a combination of the physical Video port with the physical Left and Right audio ports The port group con venience makes it easy for specification authors to define that a video stream uses the Output port without needing to specify each of its consistuent ports 5 2 2 2 Internal Flows Sources Sinks and Passthroughs Each of the internal flow types is specified with three basic pieces a dependency formula de fining when the flow is active a description of the ports associated with the flow and a list of state variables commands and groups that can be used to modify the behavior of the flow In the specification language sinks are divided into two sub types recorder and renderer that describe what the sink does with the content it receives Figure 5 7 shows an example of a source content flow from the example VCR specification The ports that may be associated with a flow depend on the type of flow Only output ports may be associated with a source only input ports with a sink and both are allowed for a pass through For each port another dependency formula may be specified t
415. ry and the location of the menu bar which is at the bottom of the screen instead of the top as on desktop computers 3 4 1 Participants Twelve students from Carnegie Mellon volunteered to participate in the study in response to an advertisement posted on a high traffic campus newsgroup The advertisement specifi cally requested people with little or no knowledge of handheld computers Subjects were paid US 7 for their participation in the study which took between thirty and forty five minutes to complete Eight men and four women participated with a median age of 22 and g g Chapter 3 Preliminary User Studies 45 an average of five years experience using computers Subjects self rated their skill at using computers for everyday tasks and their knowledge of handheld computers on a seven point Likart scale On average subjects rated their knowledge of handhelds three points less than their skill with everyday computers an average of 5 5 for everyday skill and 2 5 for handheld knowledge Half the group owned Aiwa brand stereos and two had AT amp T digital answer ing machines 3 4 2 Results The results of the study indicate that subjects performed significantly better p lt 0 05 for all using the PDA interfaces in all three metrics time to complete the tasks number of help re quests and number of missteps Note that time could be measured in this study because there was no longer overhead from shuffling papers Figure 3 5 show
416. s tem for lists to address problems that might arise if the user decided to interact with a list while the list s data was being updated In practice list data synchronization has not been a problem for PUC interfaces however so we have not yet implemented this feature The PUC protocol also handles binary data differently both to limit the number of large data transfers and to allow the controller device to specify a binary format that is compatible with its capabilities When a state variable with a binary type is changed on the appliance a state notification is sent to the appliance with the name of the state variable but without the new binary data The controller device must then explicitly request the binary data if it can handle the data appropriately The PUC specification of a binary typed state variable may indicate that the appliance is able to manipulate the binary data that it sends to the controller e g scaling an image If the specification indicates that such as appliance feature is avail able then the the controller may specify additional parameters in its request for the binary data such as the size of an image The appliance will send the binary data once the request is received from the controller device This process prevents extra transfers of potentially large binary data in two ways 1 controller devices that cannot process the binary data will not request it all and 2 if a binary state is updated rapidly then the con
417. s were difficult to represent cleanly in the PUC specification language and this may have car ried their complexity through to the generated interfaces There are fewer individual tasks on the HP printer for which the PUC interface was signifi cantly faster than the built in interface only tasks 2 4 7 and 8 I believe this is because the HP printer already has a well designed interface and seemed to perform well especially for the easier tasks The tasks where the PUC interfaces excel are generally the more difficult tasks like tasks 2 and 3 which require the users to find obscure settings deep in the interface Chapter 10 Usability Evaluation 170 The five minute maximum completion time was chosen with a goal of limiting failures to between 5 10 of the total tasks In this data there were 48 subjects performing 8 tasks each for 384 total tasks and 37 failures were recorded This gives a 9 6 failure rate which is high but still within our goal range Since the time measurements were cut off at 5 minutes one might worry that this biased the results However more than 70 of the failures are found in the Built In condition This suggests that our results which already show the Built In condition to be slower overall are likely to be correct since allowing more time would only have made that condition slower This study of usability at least for the first block of tasks compares the performance of nov ice users There is then
418. s Devices Generation E a b c d Figure 9 4 Active flow control interfaces a playing a DVD movie with the video shown on the television and audio coming through the stereo s speakers b presenting Powerpoint slides through a projector c watch ing broadcast television with audio playing through the television s speakers and d watching broadcast television with audio playing through the receiver s speakers Note that the volume control in c and d appear the same even though they are actually controlling different appliances controlling a slideshow in a presentation room copying a tape from one VCR to another and watching television with the audio coming from the television or receiver speakers The PUC identifies functions to be used in the Active Flow Controls AUI in two stages In the first stage functions are extracted from the appliance s specifications that are either men tioned in the flow dependencies for the currently active flows or are noted as being related in the appliance s content flow model In the second stage these functions are filtered to select only the most common functions that users will likely want to manipulate The PUC uses two heuristics in the filtering stage because no information is available in the specification to directly identify the commonly used functions of an appliance The first heuristic is to eliminate any functions associated with Setup Nearly all specifica tion
419. s a 1ist commands template which may only contain list add list remove list clear list move after or list move before templates One tricky aspect of the list commands template is linking each template instance to the list that it manipulates This is done by requiring that the template be at the same level and immediately precede the list group that it manipulates We suggest that each pair of com mands and list group be placed in their own group but this is not required Contents The list commands template only supports the multi state form The constituent tem plates list add list remove list clear list move after and list move Appendix D Specification Authoring Study Instructions 288 before must be single commands Constituent templates may be used outside of the list commands template As mentioned above any instance of the 1ist commands template must be placed at the same level of the specification tree immediately preceding the list group that the command modify This requirement is also true of any of the consituent templates when used outside of the List commands template The following are descriptions of the constituent templates list add Commands with this template should add a new item to the list The server upon re ceiving the command should insert an item into the list at the appropriate location in the list The template implementation can automatically display a dialog box in this case or i
420. s box plots comparing the handheld and actual interfaces for each metric on the stereo and phone respectively For both appliances users of the actual interfaces took about twice as long needed external help five times more often and made at least twice as many mistakes as users of the PDA interfaces note that this improvement is similar to that found when using the automatically generated interfaces compared with the actual interfaces as described in Chapter 10 stereo telephone Time H Time rT 5 La LS 35 aS ig 15 5 10 15 20 25 10 a5 Help m i Help __ Requests CH Requests CH Missteps Missteps actual appliance _ handheld Figure 3 5 Box plots of results from the second user study Chapter 3 Preliminary User Studies 46 3 4 3 Discussion The results of the second study are very similar to those of the first Most of our subjects did not need to use external help to complete tasks using the handheld and those that did use help only used it once This compares to each subject s average of 3 6 uses of help for the actual stereo and 4 3 uses for the actual phone Poor labeling insufficient feedback and the overloading of some buttons with multiple functions can account for this large difference on the actual appliances The worst examples of poorly labeled buttons and overloaded functions were found on the AT amp T phone This phone has several buttons that can be tapped quickly to activate one function and
421. s complexType gt lt xs schema gt C 3 Multi Appliance Wiring Diagram Schema lt xml version 1 0 encoding utf 8 gt lt xs schema targetNamespace http www cs cmu edu pebbles puc wiring elementFormDefault qualified xmlns http www cs cmu edu pebbles puc wiring xmlns mstns http www cs cmu edu pebbles puc wiring xmilns xs http www w3 org 2001 XMLSchema gt lt xs element name puc wiring description gt lt xs complexType gt lt xsS sequence gt lt xs element name appliances type AppliancesType minOccurs 1 maxOccurs 1 gt lt xs element name wires type WiresType minOccurs 1 maxOccurs 1 gt lt xs element name preferred flows type PreferredFlowsType minOccurs 0 maxOccurs 1 gt lt xs sequence gt lt xs attribute name name type xs string gt lt xs complexType gt lt xs element gt lt xs complexType name AppliancesType gt lt xs sequence gt lt xs element name appliance type ApplianceType minOccurs 1 maxOccurs unbounded gt lt xs element name external source type ExternalSourceType minOccurs 1 maxOc curs unbounded gt lt xs sequence gt lt xs complexType gt lt xs complexType name ApplianceType gt lt xs attribute name name type xs string use required gt lt xs attribute name spec guid type xs string use required gt Appendix C Other PUC XML Language Schemas 261 lt xs attribute name server type xs string use
422. s contain a high level group with the name Setup or something similar This group will often be identified in the knowledge base used for achieving consistency The PUC uses the knowledge base s mapping information to identify the Setup group in each appliance and then filters out any functions that are contained in these Setup groups The second heuristic eliminates any functions that if used or modified would always cause the flow to stop being active This eliminates all power functions which can be easily ac cessed elsewhere the input select variables from the stereo and television the VCR TV functions of the VCRs in some situations and a number of other variables that may be common but would overlap with the functioning of the FBI The exceptions to this rule are media control functions such as play stop and pause which are always included Although Chapter 9 Aggregating User Interfaces 156 the user may deactivate a flow by pressing stop or eject we feel that users would be annoyed if these functions were not easily available and that users can easily recover if they use these functions in a way that deactivates a flow 9 5 2 Active Flow Setup The Active Flow Setup AUI combines setup functions that are related to the currently active flows The PUC identifies the functions for this aggregate using the first stage of the process used for the Active Flow Controls AUI Unlike for that earlier aggregate however the seco
423. s may be used For the single variable case there are the following type restrictions String The state variable must have a string type The value of the state variable must be a parseable date string in one of the ISO 8601 international standard formats Appendix D Specification Authoring Study Instructions 286 If multiple states are used to represent the date they must have the following form Month This state may have an integer type with bounds from 1 12 or an enumerated type with 12 items Day This state must have an integer type ranging between 1 and 31 The constant maxi mum may also point to a state variable that gives the proper maximum for the given month If maximum is set to a constant value of 31 then the user interface will en force the proper maximum for the given month and year Lear This state must have an integer type Bounds on this variable are optional Negative values in this state variable will be interpreted as years BC and positive years as AD Increments may also be specified for this state date time Overview This template combines a date template with a time absolute template to form a complete representation for dates and times together Contents The contents of this Smart Template may be represented in several different ways One state with the string type may be used or the template may contain two groups that separately instantiate the date and time absolute smart templates If one
424. s ouseeseesausseeseeseesaeeaeeseeseeseusseeseeseeseesseesuesesesesaeeseeseeseuseussusseeseeseessueseeseeseessesseeseeseeseessusseeseeseessesseeseeseseusseeseeseeseesseeseeseeseeseesseeseeseeseesaeeseeseeseessussesseeseeseesausseeseeseuseussesseeseeseesaeeseeseeseeseusseeseeseeseeseeeseeseeseeseesseeseeseeseesseeseeseeseesaesseeseeseeseuseessesseesegsausaeeseeseeseusans _ Maps two state variables Particular values of the state may be mapped together and a _ shortcut is available to define that the two states have entirely equivalent values seaeneneeaececeeaececeeaeseeeesenenseceseseeseseeeeseseeseseseesesesessese seeeseseeseseseeseseeeeseseseeeeseseeeeseeeeseseseeeeseseeseseseeseseeeeseseeeeseseeeeseseseeseseeeeseseeeeseseseeseseseeeeseseeseseseeseseseeseseseeseseseeseseeeeseseeeeseseseeseseeeeseseeeeeeseeeeseseeeeseseeeeseseeeeseseeeeseseeeeseseeeeseseseeseseseeseseseeseseeeeseseeeeseseeeeseseeeeseseseeseseeeeseseeeeseseeeeseseeeeseeeseeseseeeeeeseeeeeesee0eees0e0005050582525888 _ Specifies that a node in the group tree from one specification is similar to a node in an _ other specification A node could represent a group command or state variable template Maps two Smart Templates Smart Templates are described in detail in Chapter 7 Media Controls Mapping Graph Example Answering Machine State Mode Panasonic VCR Play Stop Pause State Mode Command Play New Messages Play Stop Pause Rewind F Fwd Record Cheap VCR Co
425. s variable These parameters can affect the size of the text box that the generated interfaces will display for a state of this type It is important to note that complex types often seen in programming languages such as re cords lists and unions are not allowed to be specified as the type of a state variable Complex type structures are created using the group tree as discussed below 5 2 1 3 Label Information The interface generator must also have information about how to label appliance objects Providing this information is difficult because different form factors and interface modalities require different kinds of label information An interface for a mobile web enabled phone will probably require smaller labels than an interface for a PocketPC with a larger screen A speech interface may also need phonetic mappings and audio recordings of each label for text to speech output We have chosen to provide this information with a generic structure that we call the abel dictionary Each dictionary contains a set of labels most of which are plain text The dictionary may also contain phonetic representations using the ARPAbet the phoneme set used by CMUDICT CMU 1998 and text to speech labels that may contain text using SABLE mark up tags Sproat 1998 and a URL to an audio recording of the text The assumption underlying the label dictionary is that every label contained within whether it is phonetic information or plain text will
426. similar to debugging a program or iteratively testing a user interface and is necessary to ensure that no functions are forgotten understandable labels are used etc The advantage of the PUC system is that these improvements are only needed once and will migrate properly to interfaces generated on any platform Note also that both specifications included all of the features of their appliances even the features not tested Therefore the resulting generated user interfaces are complete in that they represent all of the features that could be accessed from the appliance s own user interfaces The specification for the HP consists of 1924 lines of XML containing 85 variables and commands and the specification for the Canon is 2949 lines of XML containing 134 vari ables and commands The PUC s consistency algorithms also need information about the similarities between specifications see Chapter 6 An automatic system was used to generate an initial set of mappings between the two all in one printer specifications I then revised the resulting map pings to produce the complete set used in the consistency study The two specifications and the mappings between them were then used by the PUC to pro duce the four different interfaces used in the studies PUC HP without consistency PUC Canon without consistency PUC HP generated to be consistent with the PUC Canon inter face and PUC Canon generated to be consistent with the HP see Figure
427. sity of Vienna Tracked PocketPC as a multi purpose inter action device left Tangible interface in a screenshot of the AR LEGO application right PDA screen capture of the LEGO robot s control GUI Barakonyi 2004 Reproduced with permission Chapter 11 Conclusion 183 The PUC specification language was also used as a component of a project conducted in the Spring 2005 semester CS 491 Generative Programming class at the University of Alabama Sandridge 2005 The goal of the project was to provide a framework that would allow a mobile application to run on any device regardless of the device s operating system or inter action technologies The students initially tried two languages XUL and Laszlo but found that both were not abstract enough to describe a user interface in a form that could be ren dered differently across multiple platforms They discovered the PUC language and found that it fit their requirements for producing interfaces The students produced a rudimentary code generator for J2ME based on the PUC language and work was continued through a summer REU undergraduate research project Collectively these projects show that the PUC system has some value for researchers outside of the PUC research team Many of these projects show that there is a need for distributing a user interface across different platforms and that the PUC technologies can be used to ac complish this goal Furthermore the existence of these
428. sk I have developed one solution to this problem called Smart Templates which augment the PUC specification language s primitive type information with high level semantic informa tion Interface generators are free to interpret the semantics of a Smart Template and if appropriate augment the automatically generated interface with the conventions expected by the user Smart Templates are specially designed to integrate hand designed user interface fragments that implement the conventions with an otherwise automatically generated inter face Templates are also designed to scale across different appliances without requiring help from the user after generation Interface generators are not required to understand every template and templates are designed such that the full functionality of every template is available to the user even if the interface generator does not understand that template 7 1 Roles The design implementation and use of a Smart Template involve people in four different roles shown visually in Figure 7 1 e Users own controller devices and make use of the interfaces containing Smart Tem plates produced by automatic interface generators e Specification Authors write specifications for appliances I expect that most authors would work for the appliance manufacturers though some specifications might be written by third parties Specification authors learn about available Smart Templates from the Template Registrar
429. street number of the address It must have an integer type but restrictions such as bounding are optional It must support the use of comple tions and server side error correction if either of those features are available CLEY This state represents the city of the address It must have a string or enumerated state It must support the use of completions and server side error correction if either of those features are available SLake This state represents the state or province in which the address is located It may have a string or enumerated type ZipCode This state represents a zip code in the address It may have a string integer or enu merated type It must support the use of completions and server side error correction if either of those features are available Counc ry This state represents the country of the address It may have a string or enumerated type It must support the use of completions or server side error correction if either of those features are available date Overview This template describes data that stores a date This might be used by an appliance to store and display the current date or to record a date in the future when some pre defined action will be taken This template currently supports both single state and multiple state instantia tions Contents The contents of this Smart Template may be represented in several different ways One state with the string type may be used or multiple state
430. structure then the final rule creates a default structure that fits any situation The most commonly used mutual exclusion rule creates overlapping panels that are controlled either by a tab control and a sidebar for any non dependent items see Figure 8 10b d This rule has two requirements Table 8 3 The row layouts supported by the PocketPC interface generator Row Name Example Description Labeled One Image Size 2 5 x 3 25 inch A single control is shown on the right and an optional Column _ label is shown on the left The space allowed for the label is fixed for a panel typically at 40 of the avail able width Labeled Two tep EE ae Two controls are shown on the right and an optional Column anit label is shown on the left The label space is again fixed per panel at the same value as the labeled one column row Full Width One control occupies the entire width of the panel Multiple controls occupy the entire width of the panel Controls are sized to fit their needs and if nec essary will be proportionally enlarged to fill the full panel width Overlapping See Figure 8 10c d This row allows an overlapping panel to be added Panels 2 2 inline with the rest of the controls This is used for the fourth mutual exclusion structure rule discussed _ below Chapter 8 Interface Generation 134 there must be a set of controls that are mutually exclusive for every value of the mutually ex
431. sts of functions which are the basis of the Smartphone interface and lists of data stored on the appliance Another interesting approach would be to explore mixing the optimization approach for generating interfaces such as that used by SUPPLE with the rule based approach that the PUC currently uses Numerical optimization algorithms would be useful for converting the abstract user interface into a concrete layout for example but might benefit from informa tion from the PUC s mutual exclusion rules for specifying particular organization that an interface must have Optimization might also be useful for achieving structural consistency between interfaces though rules may be needed to ensure that unique functions are handled appropriately and predictably The key for resource constrained platforms such as the hand held devices used by the PUC is to limit the use of optimization to areas where it can provide the most benefit while still providing reasonable performance There are several directions of future work to explore with the specification language Firstly the authoring study discussed in Chapters 5 and 6 was designed primarily to explore incon sistencies in specifications and not to understand any usability issues that the language might have More in depth authoring studies would be valuable to better understand the difficulties that users have when writing specifications and improve the language Secondly the current authoring
432. subelements as table Liesl pent ts No automatic tables for elements Autom append mandaton children to new elements p Append first choice of mandatory choice Elements Auto complete in Text View I recommend that you uncheck the Autom append mandatory children to new elements checkbox circled in red This feature of XMLSpy seems to cause some curious behavior when used with the PUC language schema Click OK to exit the dialog box Now you can create a new appliance specification by opening the File menu and selecting New The following dialog box will be displayed Appendix D Specification Authoring Study Instructions 272 Create new document Ze Select the type of document you wish to create Web Service Description Language ML Data Reduced Schema SHTML Document DocBook 4 2 Article DocBook 4 2 Book EJB 2 0 Deployment Descriptor IAS BRL Document J2EE 1 3 Application J2EE 1 3 Application Client JZEE Connector 1 0 Resource Adapter Servlet 2 3 Deployment Descriptor ML Document a A A SA aa AAEE EAE ee Select a StyleVision Stylesheet Be sure that XML Document is selected and then click OK You will then see this dialog box New file Leg IF You intend to create a new 4ML document instance based on 4 certain schema or DTO amp MLSpy can automatically add the proper assignment for pou in this case please select whether you Intend to base your document
433. supporting access to the state of an appliance and eventing to notify external entities when a state has changed It is possible that a manu Chapter 11 Conclusion 181 facturers existing implementations for these infrastructures may be easily adaptable to sup port technologies such as the PUC Some of the extra development and manufacturing costs to support the PUC may also be offset by a reduction in other costs In particular manufacturers may choose to reduce the complexity of their on appliance interfaces in favor of making some functions only available through the PUC interface Such a choice would reduce both the development cost of the physical interface since developing a simpler interface should cost less than a complex one and the manufacturing cost of each unit as fewer buttons and smaller screens are needed for a simpler interface on the physical appliance Unfortunately this cost trade off may not oc cur in appliances that are initially deployed with PUC technology because manufacturers may be unwilling to make certain functionality available only through an interface technol ogy that has not yet achieved broad acceptance Unfortunately it still seems that development and manufaucturing costs must increase in order to build PUC technologies into appliances Thus there will need to be other motivat ing factors for manufacturers justify the extra cost incurred in adding this functionality A few possible motivations coul
434. t lt xs element name change type ChangeOperationType gt lt xs element name invoke type InvokeOperationType gt lt xs element name repeat type RepeatOperationType gt lt xs choice gt lt xs sequence gt lt xs complexType gt lt xs complexType name CountType gt lt xs choice gt lt xs element name variable type VariableType gt lt xs element name constant type ConstantValueType gt lt xs choice gt lt xs complexType gt lt xs complexType name InvokeOperationType gt lt xs attribute name command type xs string use reguired gt lt xs complexType gt lt xs complexType name ChangeOperationType gt lt xs choice minOccurs 0 maxOccurs 1 gt lt xs element name constant type ConstantValueType gt lt xs element name variable type VariableType gt lt xs element name increase type IncreaseValueType gt lt xs element name decrease type DecreaseValueType gt lt xs choice gt lt xs attribute name State type xs string use required gt lt xs complexType gt lt xs complexType name VariableType gt lt xs attribute name name type xs string gt lt xs complexType gt lt xs complexType name IncreaseValueType gt lt xs attribute name value type xs string use optional gt lt xs attribute name wrap type WrapAttributeType use optional gt lt xs complexType gt lt xs complexType name DecreaseValueType gt Append
435. t 1 Video channel 4 gt lt port gt This element has two different uses The first use when within the lt ports gt block defines a physical port of the appliance The full name of this port is similar to the naming of appliance objects and has the form lt groupname gt lt groupname gt lt name gt where the port group names are from port groups that contain this port In this use the port element may not contain any other elements or content The second use when within the lt content flow gt block defines a port or port group across which content is being accepted as an input or produced as an output In this use the port element may contain several elements In this use the name must reference a complete unique port name defined in the ports block Placement First use inside the lt inputs gt lt input output gt lt outputs gt or lt port group gt elements Second use inside the lt input ports gt or lt output ports gt elements Parameters e name required The name for this port group Must be unique among other names for ports and port groups at this level e content type required The content type that is carried over the ports con tained in this group e physical type required The type of the physical port on the appliance E g RCA HDMI VGA etc e channel Only available in the second use of this element Defines the chan nel to be used from a multi channel stream M
436. t Name lt default value gt Specifies a default value for a state variable In an interface generator lt defined gt Used in conjunction with the lt active it gt element to define a de senumerated gt Enumerated type Define the number of items in the enumeration sequals gt Used in conjunction with the lt active if gt tag to define equals de lt explanation gt Defines an explanation appliance object lt false gt False for use in dependency expressions lt fixedpt gt Fixed Point type for variables that take the form of decimal values lt floatingpt gt Floating Point type for variables that take the form of decimal values lt greaterthan gt Used in conjunction with the lt active if gt tag to define greaterthan J Mee este g 8 lt group gt Used to define the nodes of the group tree lt groupings gt Defines the section that includes the group tree lt incr gt Defines the increment that an integer or fixed point variable must use lt inputs gt Defines the section that describes the input ports of an appliance lt input output gt Defines the section that describes the input output ports on an appli Appendix B Specification Language Reference Description these values would be used when the UI needs to prompt the user for a new value and could also be used for demo an offline interface _ pendency that a state variable must have some value using the lt item count gt tag Labels
437. t also makes it impossible for an author to influence the look and feel of user interfaces by writing their specification in a particular way Some examples of design choices influenced by this principle are shown later 5 2 Language Design The design of the specification language has evolved over more than six years Although new features such as complex data structure support and content flow information have been added since the initial version the basic elements of the language have remained the same There are two main categories of information that can be described with the language func tional and content flow which are discussed in the following two sections Chapter 5 Specification Language 67 5 2 1 Functional Language Elements The focus of the language is on the functional aspects of appliances which directly influence the design of interfaces for them The functional elements of the language allow a specifica tion author to describe the features that an appliance has how those features relate to each other and how those features should be presented to the user The main features of the specification language are The functions of an appliance can be represented by either state variables or state less commands State variables have specific type information that describes how they can be manipulated by the interface Commands and states are collectively called appli ance objects The structure of the hand d
438. t group gt lt min gt lt max gt lt union group gt lt union group name UnionGroupA access read only gt lt union group gt Defines a special node of the group tree that represents a union A union group has the all the same qualities of a regular group It does not contain any extra elements for describing the union The union group automatically creates one state named ChildUsed which defines the active child variable group The access parameter of this state is defined by the access attribute of the union group element Placement Inside the lt groupin s gt lt group gt lt list group gt lt types gt or lt union group gt d ping g P l Vas g P elements Parameters e name required The local name of this group e type name required if within types section optional otherwise defines a type with the given name e access Defines how users interact with the ChildUsed variable Possible val ues are read only and read write e is a The Smart Template that represents this group and its children e priority The priority this group should be assigned relative to other objects in its parent group May Contain lt labels gt lt active if gt lt state gt lt command gt lt explanation gt lt group gt lt xlist group gt lt UnLon group Appendix B Specification Language Reference 224 lt selections gt lt selections number one access read only gt Defines the num
439. t issue for systems like the PUC How do users keep people from driving by on the street and maliciously controlling their stereos or kitchen appliances Again a lot of interesting work could be done in this area but I have chosen not to address this in my thesis work 1 3 Contributions My thesis is A system can automatically generate user interfaces on multiple platforms for remotely controlling appliances where the user s performance is better than with the manufacturer s interfaces for the appliances I have conducted evaluations of my interface generators that demonstrate that this thesis holds as discussed in detail in Chapter 10 This dissertation also makes a number of other contributions e An abstract appliance specification language for describing the complete functionality of a wide range of appliances e The use of dependency formulas in appliance specifications to help determine the structure of generated user interfaces e The general Smart Templates technique for incorporating domain specific design conventions into an appliance specification and rendering the conventions appropri ately on different platforms and in different interface modalities e A language for describing semantic similarities between appliance specifications e Algorithms that apply knowledge of appliance similarities and user history to gener ate new interfaces that are consistent with previous interfaces the user has seen Chap
440. t it gt cjg 5 lial BL E e 8 ia s a IF 5 al fe El ik HI i 23 3HR B a ak A Oo f o Bx BS 5 Be Be Gx p BG g HHH ae gE ee 2a Eo go 255 5 28 amp lt 3 3 gt J gt amp 2s g 3 3 3era 23 22 EF ek ba m i E e i Figure 8 11 The interface generated for the GMC Yukon Denali driver information console without the use of any layout fixing rules rotated to fit better on the page The high level structure from the abstract user inter face underlying this panel is shown above the interface Table 8 4 The layout fixing rules used by the PocketPC interface generator Problem Name Insufficient Fix with Tabs Fix with Combo Box Fix with Row of Buttons Fix with Dialog Box Insufficient Fix One Column with No Label Fix Two Column with No Label Fix Too Wide One Column Chapter 8 Interface Generation Description Breaks a long panel into multiple overlapping panels that are con trolled by tabs This requires that the panel only contains labeled groups so that all controls can be placed in a tab and each tab has a Breaks a long panel into multiple overlapping panels that are con trolled by a combo box at the top of the panel group Like the Fix with Tabs rule this requires that the panel only contains labeled groups so that all controls can be placed on a panel and each panel has a name in the combo box Breaks a long panel into two separate panels that are
441. t may rely on the required if infrastructure to determine whether an editing dialog box should be displayed list remove Commands with this template should delete an item from the list preferably the cur rently selected item s list clea r Commands with this template should remove all items from the list list move after Commands with this template should move the currently selected item s to the next higher index list move before Commands with this template should move the currently selected item s to the next lower index media controls Overview This template represents the interactions that control the playback of any audio visual me dia such as a CD MP3 or VHS tape This template supports either state or command based representations of the controls and also handles related functions such as Next Track and Previous Track Contents Appendix D Specification Authoring Study Instructions 289 This Smart Template may represent the controls as a single state variable and or several commands The state based representation must have an enumerated type with standardized labels mapped to each value in the type in the valueLabels section The state must have the name Mode The labels are described below Other labels may be included in addition to the standardized ones for the benefit of interface generators that don t recognize this Smart Template See S This label identifies the state when the media is sto
442. t name VHF UHF Antenna gt lt output ports gt lt pass through gt lt content group gt lt active if gt lt equals state Base Power gt lt constant value true gt lt equals gt lt active if gt lt content group gt Appendix A Sample VCR Specification 201 lt active 1if gt lt eguals state Base PoweredItems Controls TV VCR gt lt constant value true gt lt equals gt lt active if gt lt source name Tape content type av gt lt active if gt lt equals state Base Powereditems Status TapeiIn gt lt constant value true gt lt equals gt lt not gt lt or gt lt equals state Base PoweredItems Controls PlayControls Mode gt lt constant value 10 gt lt equals gt lt equals state Base PoweredItems Controls PlayControls Mode gt lt constant value 11 gt lt equals gt lt or gt lt not gt lt active if gt lt output ports gt lt port group name Output gt lt port name VHF UHF Antenna channel 3 gt lt output ports gt lt objects gt lt group name Base PowerediItems Controls gt lt objects gt lt source gt lt pass through content type av gt lt input ports gt lt port name VHF UHF Antenna gt lt input ports gt lt processing gt lt block gt lt channel value 3 gt lt block gt lt processing gt lt output ports gt lt port name VHF UHF Antenna gt lt output ports gt lt pass through gt lt cont
443. t ports gt element May Contain SPOT por l group lt outputs gt lt outputs gt lt outputs gt Defines the physical output ports of an appliance Placement Inside the lt ports gt element May Contain DOLL lt porl oroup Appendix B Specification Language Reference 243 lt input output gt lt i nput output gt lt input output gt Defines any physical ports of an appliance which can simultaneously be both inputs and outputs Placement Inside the lt ports gt element May Contain DOLL lt Portl gqroup lt port group gt lt port group name Output content type av gt lt port group gt Defines a group of physical ports This construct is useful for grouping ports that are activated simultaneously Referencing the name given to a port group in a content flow is shorthand for referencing all of its contained ports Placement Inside the lt inputs gt lt input output gt lt outputs gt or lt port group gt elements Parameters e name required The name for this port group Must be unique among other names for ports and port groups at this level content type required The content type that is carried over the ports con tained in this group May Contain Kt es SL COO Appendix B Specification Language Reference 244 lt port gt lt port name Right content type component audio right physical type RCA gt lt port name Inpu
444. t s control GUI Barakonyi 2004 Reproduced with permission 06 183 The cards that specify the reminder frequency setting All of these cards would be attached together with a paper Clip eeseeeeseeeeneeeeeeeees 278 The XML code for the reminder frequency state variable eeseeeeeeeeeeee 279 The group and list parameters card that describe the to do list This list has four state variable members not shown sesesseseseesssersssesessrsessrsse 280 The XML code for describing the to do list The four state variable are defined where the appears in the above code ssssssssssssssssssesessee 280 XXII Figure 11 6 Figure 11 7 Figure 11 8 Figure 11 9 The object and state variable type cards for the Category state variable that is contained in the to do list group Note the use of the list selection type on the state variable type Card ixcided sissies adios The XML code for the Category state variable that is contained in the to do list group Note the use of the list selection type in this 110 ee EN TEE ATEO NE CA ea a ae EA EN ASTEN ETATON The object cards that describe the add and remove commands for the to do list These objects are contained within a group that is tagged with the list commands Smart Template eesecesssceceeeneeeeeeneeeeeeees The XML code for describing the add and remove commands for the to do list Note the group that contains these items
445. t user interface underlying this panel is shown above the interface ccccccsceccecccccecccceccuccecescuccucecceceececceseucuecescesceseesesceseusescescuseccesensens Example screens from automatically generated Smartphone interfaces a The opening screen for controlling a shelf stereo Our dependency information rule created the separate lists for CD Radio etc b c Two screens from a simulated elevator interface The particular screen shown to the user depends on whether the user is b outside or c inside the elevator car d A Smartphone rendering of the media controls Smart Template from an interface for controlling the Windows Media Player application on a desktop computer The template s design is based on the Smartphone Windows Media Player application and is operated using the right left and select buttons of TINS VO TIS stnim Dy Se eon e E E O Diagrams showing how the first a and second b rules for optimizing the list structure behave Black solid arrows indicate how the screens are connected and red dashed lines indicate changes made by the rules Note that in a some items were list only and thus were promoted to the top level list while the others were placed on a panel The items all happen to be labels so this panel is a summary pane Interfaces generated for the HP and Canon all in one printers demonstrating the effects of the concrete interface re ordering rule Note the difference in the order
446. t xs element name processing type ProcessingType minOccurs 0 maxOccurs 1 gt Gh E EET Gr lt xs element name output ports type ContentPortType minOccurs 1 maxOccurs 1 gt lt xs element name objects type ContentStatesType minOccurs 0 maxOccurs 1 gt lt xs sequence gt lt xs attribute name name type xs string use optional gt lt xs attribute name content type type ContentTypeAttrib use required gt lt xs complexType gt lt xs complexType name ContentStatesType gt lt xs choice minOccurs 1 maxOccurs unbounded gt lt xs element name object type ContentObjectType gt lt xs element name group type ContentObjectType gt lt xs choice gt lt xs complexType gt lt xs complexType name ContentObjectType gt lt xs attribute name name type xs string gt lt xs complexType gt lt xs complexType name ContentPortType gt lt xs choice minOccurs 1 maxOccurs unbounded gt lt xs element name port type ContentPortParamType gt lt xs element name port group type ContentPortParamType gt lt xs choice gt lt xs complexType gt lt xs complexType name ContentPortParamType gt lt xs sequence gt lt xs element name active if type DependencyContent minOccurs 0 maxOccurs 1 gt lt xs element name Channel type ChannelType minOccurs 0 maxOccurs 1 gt lt xs element name objects type ContentStatesType minOccurs 0 maxOccu
447. tation room control panel The developers then showed that SuprLe could generate similar versions of each of these interfaces by varying the task information provided to the interface generator The interface used in this study had only a few simple functions however and users performance on the Supple interfaces was not measured or compared with any other interfaces The following sections highlight many of the model based systems that have been produced over the years The discussion is broken down into a discussion of the early systems ending in the early to mid 90 s and more recent systems y y 2 2 1 Early Model Based Systems The initial research in model based systems was conducted from approximately the mid 80 s to the early 90 s An excellent review of early model based user interface research can be found in Szekely 1996 2 2 1 1 Mickey One of these early systems was Mickey Olsen Jr 1989 which automatically generated menus and dialog boxes from function signatures and strategically placed comments in the Chapter 2 Related Work 28 code implementing the application logic This simplified the construction of user interfaces for programmers who could now implement the logic add a few special comments and immediately have a limited user interface for their application While the generated user in terface was rarely sufficient for the entire application the techniques demonstrated by Mickey showed promis
448. ted Computing Systems Workshops June 2005 pp 514 519 Rosenfeld 2001 Roni Rosenfeld Jr Olsen Dan and Alex Rudnicky Universal Speech Interfaces znteractions New Visions of Human Computer Interaction 2001 VIII 6 pp 34 44 Sandridge 2005 Nick Sandridge Brian LaShomb Katrina Roan and Terri Chapman Code Generation for J2ME and C based Mobile Devices using Domain Specific XML Based Source Languages 2006 2005 http unix eng ua edu blashomb reu 491paper doc Satzinger 1998 John W Satzinger and Lorne Olfman User Interface Consistency Across End User Applications The Effects on Mental Models Journal of Information Man agement Systems 1998 14 4 pp 167 193 Shriver 2001 S Shriver Toth A Zhu X Rudnikcy A Rosenfeld R A Unified Design for Human Machine Voice Interaction Extended Abstracts of CHI 2007 Seattle WA March31 April 5 2001 pp 247 248 Shvaiko 2005 Pavel Shvaiko and Jerome Euzenat A Survey of Schema based Matching Approaches Journal on Data Semantics 2005 Sproat 1998 R Sproat Hunt A Ostendorf P Taylor P Black A Lenzo K Edging ton M SABLE A Standard for TTS Markup International Conference on Spoken Language Processing Sydney Australia 1998 Bibliography 320 Srivastava 2003 Biplav Srivastava and Java Koehler Web Service Composition Current Solutions and Open Problems ICAPS Workshop on Planning and Scheduling for
449. ted States of America C Figure 7 4 Screenshots of Smart Templates rendered as part of the GMC Denali navigation system on the PocketPC platform a Demonstrates the time absolute Smart Template used for a clock function The 12 and 24 hour option of the template changes the way time is rendered throughout the interface as can be seen in the clock at the top of these screenshots b Demonstrates the list commands templates integrated with one of the PUC s list controls Several commands for adding and deleting items are located underneath the list control along with the edit button that is part of the list control Two move commands have also been integrated with the list control as the arrow buttons located on top of the selected list item c Demonstrates the address template s capability of integrating with the PocketPC s built in Outlook contact database The leftmost screen shows the interface before the user has pressed the Select Contact button Pressing this button shows the middle screen which allows the user to select a contact from their database Pressing OK from this dialog causes the selected information to be filled appropriately into the fields of the template rightmost screen pliance if appropriate The current PocketPC implementations of the media controls tem plate allow the left and right directional buttons to be used for next track and previous track if those functions are available Chapter 7 Ha
450. ted with the PUC In particular it would seem best to use a location based discovery system to ensure that users are controlling the appliances they intend to control and not for example their neighbors appliances Appliances are increasingly being built with communication protocols that allow two way communication but unfortunately these protocols are often proprietary and not compatible with the PUC protocol To enable control of real appliances we build adaptors hardware and or software that translate the appliance s proprietary protocol to the PUC protocol In some cases it is not possible to control the actual appliance In these cases we are forced to write adaptors that simulate the behavior of appliances APPLIANCES Stereo Alarm Clock etc CONTROLLER DEVICES automatic interface generation ADAPTOR publishes description appliance state controls appliance PROTOCOL PROTOCOL two way communication two way communication of specification amp state device specification amp of specification amp state state feedback COMMUNICATION COMMUNICATION 802 11 Bluetooth Zigbee etc cc 802 11 Bluetooth Zigbee etc control Figure 4 1 Diagram of the PUC system architecture showing the communication between each of the differ ent components Chapter 4 System Implementation 56 4 2 Controlling Appliances We have built adaptors for nine different existing appliances see Table 4 1 and sim
451. ter 1 Introduction 14 e The Flow Based Interface concept which allows users to quickly and easily specify high level goals for a multi appliance system e Algorithms use a model of content flow in a multi appliance system to generate task based interfaces that combine functionality from multiple appliances e Interface generation software on multiple platforms PocketPC Microsoft s Smart phone and desktop computers which make use of the above contributions to produce appliance interfaces that have been demonstrated via user testing to be more usable than manufacturers interfaces for the same appliances 1 4 Dissertation Overview The next chapter Chapter 2 surveys related work The remainder of the dissertation de scribes the different components of the PUC system building towards the description of the interface generation process in Chapters 8 and 9 and the usability evaluation of the generated interfaces in Chapter 10 Chapter 3 describes preliminary studies of appliance user interfaces that set the groundwork for the design of the specification language and the interface generator Of particular impor tance in this chapter is the list of requirements for any system that intends to control appliances Chapter 4 discusses the general architecture of the PUC system and the infrastructure needed for communicating with and controlling real appliances Chapter 5 describes the specification language and discusses its c
452. ter 7 Handling Domain Specific and Conventional Knowledge 110 Table 7 1 Description of all implemented Smart Templates in the PUC system Template Name Description address Represents all aspects of a street address including street name number city state _ and zip code Primarily used by the navigation system but also supports entry of zip codes on appliances that need location information to configure program guides etc On the PocketPC platform this template also integrates with the built _ in Outlook database to allow users to automatically insert contact information into the fields of a PUC interface channel Represents the channel on a television station on a radio or the tuning parameter of some other appliance This template is needed to support the automatic ma _nipulation of channels by the multi appliance interfaces discussed in Chapter 9 but also offers better support for channel manipulation than is currently afforded 2 by the automatic interface generation For example up and down buttons can be provided for manipulating channels along with the arbitrary numeric entry that the PUC normally supports Bee eee eee naw awecussecsecsssnsnocccccccccccccccccccccccccccccccccccccccccccscccccccccccccwcswseees SpescscsenensnsnanscsnsnsenenenaacscsnsnsesenenanananscsesnsenenenanansssesesnsesenesaUaUaeseseseSeUeUeUGUSUSESCS0S00 0000 004020S0SCSESESEEAUAUAUSESESESESSSAUAUAUSSESESSSESSSSUAUAUSESESESESESESSUAUSUSESESE
453. tereo Our dependency information rule created the separate lists for CD Radio etc b c Two screens from a simulated elevator interface The particular screen shown to the user depends on whether the user is b outside or c inside the elevator car d A Smartphone rendering of the media controls Smart Template from an interface for controlling the Windows Media Player application on a desktop computer The template s design is based on the Smartphone Windows Media Player application and is operated using the right left and select buttons of the phone s thumb stick Chapter 8 Interface Generation 140 The first of these rules searches for situations where the number of empty slots in the parent list is greater than the number of list only items If this is found then all of the list only items are promoted into the parent list The current list is then replaced with an editing pane and the remaining items are placed on that pane see Figure 8 13a Note that this causes any panel or list items to be displayed on the editing pane This has the side effect of occasion ally creating summary panes when all of the list or panel items are labels see Figure 8 2c e and Figure 8 13a The second rule searches for situations where there is more than one panel only item If this is found then the generator looks for sets of panel only items that have labels with a com mon prefix or suffix For each set that is found an editing pane is created
454. tereo c d interfaces for the PocketPC s nsssnsesesssenssseneserssserssssrsssrosssrsssersssersssresseree 40 Box plots showing the range of missteps and help requests uses of external help for each appliance and interface type esscccessseeeeeeneeeeeeneeeees 44 Box plots of results from the second user study eessceeeseeceeseeeeeneeeeeeeeeees 46 Diagram of the PUC system architecture showing the communication between each of the different components ceeeseeeeeees 56 The interface for the PUC Debug Server a The main window showing the appliances currently being simulated by this server b The interface for simulating a Philips DVD player that was automatically generated by the Debug Server acesiescvceesteqeceseseateeoreeeastoeveeesepart 58 Screenshots of the menu interface for the PocketPC PUC interface generator The desktop interface generator has a similar menu structure The backgrounds of all these screenshots show the logging panel where messages from the interface generator are displayed 11 60 Screenshots of interfaces for the Smartphone interface generator showing the menu based interface a c and custom controls built for Ele TAR ONS E E ET N A N E N E at EA 61 Message format for the PUC communication protocol 0 eeeseeeseeeeereeees 63 Examples of a a state variable representing the current channel tuned by the VCR and b a command for ejecting the tape currently in the VER ceciisc
455. termine the organization of application windows determine navigation between windows determine abstractly the behavior of each presentation unit map abstract presentation unit behaviors into the target toolkit and de termine the window layout At each step the interface designer could ask the system to perform the step using one of several techniques or do the work themselves For example TRIDENT determined layout using a bottom right method that for each element would ask should this element be placed to the right or below the previous element A set of heu ristics were used to automate the decision or the interface designer could explicitly decide often resulting in interfaces with a pleasing appearance TRIDENT also used its task models specified in a format called an Activity Chaining Graph ACG to automatically determine the number of windows needed for an application 2 2 2 Model Based Systems for Very Large Interfaces and Platform Independence Later model based systems concentrated on two features of earlier systems that were particu larly successful the use of task models to describe users goals with an interface and the combination of multiple interface models to produce a final interface With the advent of XML another recent trend has been the development of user interface description languages UIDLs in an attempt to standardize the model formats used by different model based sys tems Chapter 2 Related Work
456. the V2 standard to adopt many of the design decisions already included in the PUC system The current version of the standard is much improved though the PUC and V2 still share some differences More information is available about the similarities and differences between V2 and the PUC system elsewhere Nichols 2004a The PUC system has also been used by two research projects A group at the Technical Uni versity of Vienna led by Dieter Schmalstieg has used the PUC in their research into augmented reality applications In one project the PUC was used to provide handheld con trol of augmented reality characters resulting in a publication at a leading augmented reality conference Barakonyi 2004 Figure 11 1 shows an example of the PUC in action with this project A group of Italian researchers have also integrated the PUC system into their middleware system called DomoNet Miori 2006 This system allows communication between several of the common device communication protocols including UPnP and X10 The PUC pro vides the user interface capability for this middleware allowing users to control UPnP X10 and other devices 4x 21 06 X 1 on Direction A U D power a ET A Engine B off X _ Direction 8 U D power 8 IT A Engine C free Direction C U D Figure 11 1 The PUC being used to control a character in an augmented reality application as part of work performed with the PUC at the Technical Univer
457. the appliance state changes Five kinds of dependencies can be speci fied each of which specifies a state that is depended upon and a value or another state variable to compare with e Equals True when the specified state has the specified value e GreaterThan True when the specified state has a value greater than the specified value e LessThan True when the specified state has a value less than the specified value e Defined True when the specified state has any value e Undefined True when the specified state does not have any value These dependencies can be composed into Boolean formulas using AND and OR NOT may also be used Chapter 5 Specification Language 74 lt active if gt lt equals state Base Power gt lt constant value true gt lt equals gt lt active if gt Figure 5 5 An example of a common type of dependency equation specifying that a variable or command is not available if the appliance s power is turned off Specifying dependencies on list state variables must be done using the special apply over operation This element applies the dependencies it contains over some items in a list and returns a value depending on the value of its true if property The dependencies are ap plied over the set of items based on the items property which may be set to all or selected The true if property may be set to all or any If true if is all then the dependency for mula contained in the apply over el
458. the appliances themselves and thus is not subject to these limitations 2 2 Automatic amp Guided User Interface Design Research in interface generation has a long history dating back to some of the earliest User Interface Management Systems UIMSs developed in the mid 80 s such as COUSIN Hayes 1985 The original goal of these systems was to automate the design of the user in terface so that programmers who were typically not trained in interface design could produce applications with high quality user interfaces This work led to creation of systems in the late 80 s and early 90 s such as UIDE Sukaviriya 1993 ITS Wiecha 1990 Jade Vander Zanden 1990 and Humanoid Szekely 1992 which required designers to specify models of their applications that could then be used to automatically generate a user inter face The generated interfaces could generally be modified by a trained interface designer to produce a final user interface These interfaces were sometimes called model based user inter faces because of the models underlying their creation These early model based systems had several drawbacks Most notably creating the models needed for generating an interface was a very abstract and time consuming process The modeling languages had a steep learning curve and often the time needed to create the mod els exceeded the time needed to manually program a user interface by hand Finally automatic generation of the user int
459. tion type xs string minOccurs 0 maxOccurs 1 gt lt xs elemen lt xs element name object count type xs integer minOccurs 1 maxOccurs 1 gt Ch uch Gh ch lt xs element name Similar specs table type SimilarSpecsTableType minOccurs 0 maxOccurs 1 gt lt xs element name mappings table type MappingsTableType minOccurs 0 maxOccurs 1 gt lt xs sequence gt lt xs attribute name name type xs string use optional gt lt xs attribute name guid type xs string use required gt lt xs complexType gt lt xs complexType name MappingsTableType gt lt xs sequence gt lt xs element name mapped type MappedType minOccurs 1 maxOccurs unbounded gt lt xs sequence gt lt xs complexType gt lt xs complexType name MappedType gt lt xs attribute name path type xs string use required gt lt xs attribute name type type MapTypeAttributeType use optional gt lt xs attribute name spec name type xs string use optional gt lt xs attribute name spec guid type xs string use required gt lt xs complexType gt lt xs complexType name SimilarSpecsTableType gt lt xS sequence gt Appendix C Other PUC XML Language Schemas 257 lt xs element name spec entry type SpecEntryType minOccurs 1 maxOccurs unbounded gt lt xs sequence gt lt xs complexType gt lt xs complexType name SpeckEntryType gt lt xSs sequence gt lt xs
460. tionType gt lt xs element name alert information type xs string gt lt xs element name register device type DeviceType gt Cr er Gr et th Ache Gr yeh lt i ace ery ack lt xs element name unregister device type UnregisterDeviceType gt lt xs choice gt lt xs complexType gt lt xs element gt lt xs complexType name BinaryStateChangeNotifyType gt lt xsS sequence gt lt xs element name state type StateWithContentAttribType minOccurs 1 maxOccurs 1 gt lt xs sequence gt lt xs complexType gt lt xs complexType name StateWithContentAttribType gt lt xs simpleContent gt lt xs extension base xs string gt lt xs attribute name content type type xs string gt lt xs extension gt lt xs simpleContent gt lt xs complexType gt lt xs complexType name StateValueRequestType gt lt xs sequence gt lt xs element name state type xs string minOccurs 1 maxOccurs 1 gt lt xs any minOccurs 0 maxOccurs unbounded gt lt xs sequence gt lt xs complexType gt lt xs complexType name CommandinvokeRequestType gt lt xs sequence gt lt xs element name command type xs string minOccurs 1 maxOccurs 1 gt lt xs sequence gt lt xs complexType gt lt xs complexType name DeviceSpecType gt lt xs sequence gt lt xs element name spec type xs string minOccurs 1 maxOccurs 1 gt lt xs sequence gt lt
461. tional variables that define how time should be rendered such an enumeration specifying whether to use a 12 hour or 24 hour time format see Figure 7 4a When these variables are encountered the PUC PocketPC generator saves references to them but passes them back to interface generator for normal rendering The references allow the template to know when the user the changes the variables so that it can take these changes into account when rendering absolute time values elsewhere in the interface In other situations an interface generator may explicitly decide not to implement an entire Smart Template because the interface generated by the normal interface generator is already sufficient For example a speech interface generator might not implement the media controls template because the best interaction is speaking words like Play and Stop and this is the interface that would already be produced The code for a Smart Template on a controller device can also access special features or data that are specific to that controller device For example the address and phone number Smart Templates in the PUC PocketPC interface generator were implemented to take advantage of the built in Outlook contacts database that is present on every PocketPC Whenever an ap pliance requires the entry of an address or phone number the template provides a special button that opens a dialog box that allows users to pick one of their contacts When they r
462. together in multi appliance systems The specification language allows authors to describe the input and output ports that an ap pliance possesses and the internal content flows that use those ports Internal flows include the following e Sources represent content that originates within the appliance such as from a DVD player playing a DVD or a VCR playing a videotape Display devices that have inter nal tuners such as televisions receiving broadcast signals through antennas are not defined as sources however because the content does not originate inside of the tun ing device Instead broadcast signals are described as a special external source that must be routed through a tuner to be viewable by the user e Sinks represent locations where content may either be displayed to the user or stored for later retrieval For example the television screen receiver speakers and VCR tape for recording may all be sinks for content in our home theater scenario e Pass throughs represent an appliance s ability to take in some content as an input and redirect it through one or more of its outputs For example some televisions have the capability of taking the audio received as an input and making it available as an output for other appliances Tuning appliances such as cable television set top boxes are also represented as pass throughs which usually take a multi channel in put from an antenna and output single channel data Wir
463. tributeGroup name ApplianceObjectAttribs gt lt xs attribute name name type xs string use required gt lt xs attribute name type name type xs string use optional gt lt xs attribute name is a type xs string use optional gt lt xs attribute name priority type PriorityType use optional gt lt xs attributeGroup gt lt xs complexType name UnionGroupType gt lt xS sequence gt gt gt lt xs element name appliance reference type ApplianceReference minOccurs 0 maxOccurs 1 gt lt xs element name labels type LabelDictionary minOccurs 0 maxOccurs 1 lt xs element name active if type ActivelfType minOccurs 0 maxOccurs 1 lt xs choice minOccurs 1 maxOccurs unbounded gt lt xs element name group type GroupType gt lt xs element name list group type ListGroupType gt lt xs element name union group type UnionGroupType gt lt xs elemen 1ame State type StateType gt lt xs element name command type CommandType gt Cee Yer lt Ghs a o2Gr 2 lt xs element name explanation type ExplanationType gt Appendix B Specification Language Reference gt gt 209 lt xs element name apply type type ObjectApplyType gt lt xs choice gt lt xs sequence gt lt xs attributeGroup ref ApplianceObjectAttribs gt lt xs attribute name access type AccessType gt lt xs complexType gt lt xs compl
464. troller device can condense these updates and request new data at the speed with which the controller device can receive it Chapter 4 System Implementation 64 CHAPTER 5 Specification Language The PUC specification language was carefully designed to include just the right amount of information to generate high quality user interfaces while also being easy to use and concise An important goal of the language was to allow experienced specification authors to create a specification in the same amount or less time as an experienced user interface designer would need to create a user interface for one platform Accomplishing this goal would address a problem with many previous automatic and guided generation systems whose specifications required much more time to create than simply building the user interface by hand Infor mally our experience with the specification language suggests that we have come close to accomplishing this goal while providing enough detail for interface generators to produce high quality interfaces This chapter starts with a brief discussion of the design principles that we used in the crea tion of the specification language building on the requirements and analysis of the hand designed interfaces discussed in Chapter 3 It then describes the design of the language in The work in this chapter was originally published in Jeffrey Nichols Brad A Myers Kevin Litwack Michael Higgins Joseph Hughes and
465. ts parameters In column c the dialog box shown on the bottom is opened by pressing the Repeat Image button shown in the top image EEE E PE EEI EE EIEEE AE PAIE EESTE eat TAE AA AEEA AETA has Recah ETTA Block lists created for the timed recordings groups of the Mitsubishi DVCR and Samsung DVD VCR VCR and Type are unmapped blocks in the block lists ssosssseeseseenessesessesessssesseresseressesesss Screenshots of the HP all in one printer interface demonstrating the three mutual exclusion rules a The special power off screen generated by the second rule The remaining shots are of the power on view b The fax mode of the all in one printer This mode is accessed through tabs at the bottom of the screen created by the first rule Another tab can be seen in screen shots c and d c The copy eee s L29 slal gelo XXI Figure 8 11 Figure 8 12 Figure 8 13 Figure 8 14 Figure 9 1 Figure 9 2 Figure 9 3 Figure 9 4 mode of the all in one printer with the resize options set to zoom 25 400 The resize options state is a mode with several different options created by the third rule d Another view of the copy mode With the resize options Set to POSter SIZE ansia vient cee E eee ete The interface generated for the GMC Yukon Denali driver information console without the use of any layout fixing rules rotated to fit better on the page The high level structure from the abstrac
466. u edu pebbles puc puc protocol xmilns xs http www w3 org 2001 XMLSchema gt lt l The PUC protocol consists of an eight byte header XML content and optional format independent binary content The header is divided into two four byte chunks The first chunk is an integer giving the full length of the message not including the header The second chunk is an integer giving the length of the XML portion of the message Both of these integers are sent using the network byte order The XML content is described by this Schema The binary content may be in any arbitrary format The format is usually defined within an element of the accompanying XML message gt 253 lt xs element name message gt lt xs complexType gt lt xs choice minOccurs 1 maxOccurs 1 gt lt xs element name state change notification type PUCDataType gt lt xs element name state change request type PUCDataType gt lt xs element name binary state change notification type BinaryStateChangeNotifyType gt lt xs element name state value request type StateValueRequestType gt lt xs element name command invoke request type CommandInvokeRequestType gt lt xs element name spec request gt lt xs element name device spec type DeviceSpecType gt lt xs element name full state request gt lt xs element name Server information request gt lt xs element name Server information type ServerInforma
467. ufacturers and the most complex all in printers available for home use at the time of their purchase All in one printers were chose as the appliances in these studies for two reasons e Complex appliances are typically more difficult to use than trivial ones and I wanted to test the PUC with appliances that would be challenging for its generation algo rithms All in printers seem to be at least as complicated if not more so than many of the other appliance types that have been explored containing 85 variables and commands for the HP and 134 for the Canon The two chosen for these studies have several different main functions including copying faxing scanning and photo manipulation that all must be represented in the user interface They also have many special configuration options for each of the main functions which make the initial setup process difficult and time consuming e It was not possible for the PUC to actually control the all in one printers but simu lating this control was easy to achieve by configuring a computer to print documents on the printers with the correct appearance based on the task the user was currently performing This resulted in a realistic setting for users of the PUC interfaces which allows for better comparisons of the PUC interfaces with the existing manufacturers interfaces The existing manufacturers interfaces from both printers were used for the comparisons con ducted in the studies The
468. uilt In Canon AutoGen Consistent Built In Table 10 2 Average completion time and total failure data for the second block of tasks N 8 for all condi tions indicates a significant difference between that row s condition and the Consistent AutoGen condition for that appliance p lt 0 05 and f indicates a marginally significant difference p lt 0 1 Completion times and total failures were compared with a one way analysis variance and failures per task were compared with a one tailed Fisher s Exact Test Chapter 10 Usability Evaluation 172 pletion time for the consistent PUC interface was again more than twice as fast as the normal PUC interface M 3 39 vs M 7 45 and more than four times faster on average than the built in interface M 3 39 vs M 15 44 If the appliance used by subjects is not considered it is possible to compare the total comple tion times of the two blocks of tasks for each of the three conditions Neither the Built In F130 3 24 p lt 0 09 or AutoGen F130 2 46 p lt 0 14 conditions were significantly dif ferent from the first block to the second although the Built In condition is marginally significant and the AutoGen condition may be trending in that direction The Consistent AutoGen condition is significantly different from the first block to the second Fi 30 10 45 p lt 0 004 10 5 2 Discussion of Consistency The results show that users perform faster over all eight tasks usin
469. uitive to the user so that functions can be found quickly while at the same time minimizing the number of different screens that make up each generated interface The number of editing panes also must be minimized especially to prevent situations in which only one control is on an edit ing pane A part of minimizing editing panes is deciding whether a particular variable should be manipulated through a list item or a control on an editing pane A challenge to all of this is to make navigation quicker without significantly violating the structure described in the appliance specification A final challenge is deciding which function to assign to the left soft button which is supposed to invoke the most commonly used function on the current screen Creating an intuitive list hierarchy is one of the most important challenges for our Smart phone interface generator because users will be unable to interact with an appliance if they Chapter 8 Interface Generation 138 cannot find the functions they want to use The list hierarchy is built starting with the top most group in the abstract user interface A list is constructed by making each child group that is labeled into a child list Every AIO that is encountered is added to the list as an item Groups are not required to have labels so not all groups in the abstract interface will have corresponding child lists in the concrete interface This may mean that lists are created that are larger than
470. uity Knowledge continuity refers to properties of the interface such as the labels and layout and task continuity refers to the ability of users to change devices in the middle of their workflow They suggest a set of de sign principles for dealing with these problems such as using similar labels and organization across all interfaces Florins Florins 2004 expands on these ideas by relating them to exist ing theory and breaking the idea of knowledge continuity into three parts perceptual cognitive and functional Perceptual continuity is found when interfaces have a similar ap pearance Cognitive continuity occurs when users understand the underlying concepts behind the interface Functional continuity occurs when the same set of functions are avail able on both platforms Although both Denis amp Karsenty and Florins suggest some design guidelines to address continuity none of these guidelines are sufficiently concrete to apply in the PUC s automatic interface generation g The work thus far identifies benefits of consistency but other researchers have found that there may be downsides if consistency is carried too far In particular Grudin Grudin 1989 shows five examples where the consistent design decision is not beneficial for ease of learn ing ease of expert use or both Grudin s conclusion is that the most important factor in an interface design should be matching the user interface to the user s tasks not on consistency
471. ulators for seven others see Table 4 2 There is also a generic Debug Server that can read any PUC specification create a simulator based that specification and provide a user interface that allows the developer to adjust the state of the appliance as though that appliance was actually functioning The user interface generated by the Debug Server uses a subset of the same algorithms as our other graphical interface generators though it allows read only state variables to be modified and does not show commands see Figure 4 2 Most of the adaptors we have built were created entirely in software and require the appli ance to be attached to a PC in order to function For example the Sony Camcorder is controlled through the AV C protocol running over an IEEE 1394 Firewire cable Our adaptors for desktop applications typically use the Add In capability provided by the devel oper while our other adaptors communicate through the serial port or with general protocols such as UPnP The one adaptor not written entirely in software is for the Audiophase Shelf Stereo which did not originally have any mechanism for communicating its internal state to a third party This appliance was modified by our collaborators at MAYA Design a local Pittsburgh design firm with custom hardware that electronically watched the LCD screen of the stereo and determined its state from the set of lights that were currently displayed Control of the shelf stereo w
472. unction split on the Mitsubishi The control for brightness could be placed in a matching location for the brightness control on Mitsubishi and the same could be done with the control for content Chapter 8 Interface Generation 130 The Bring Together Split Dependents rule handles a very particular situation that we have found in several specifications A specific example appears in the HP and Canon all in one printers The HP all in printer has several options for resizing a copied image including zooming creating a poster across multiple pages and repeating the same image multiple times on a single page Only one of these options can be chosen at a time on the HP The Canon all in one printer also has these options but several of these features are independent of each other In particular the image repeat function can be used with any of the other re sizing options Speaking more generally this rule addresses the situation where two appliances share the same functionality but some of that functionality is moded on one ap pliance but not on the other This situation is described in the knowledge base with a group mapping that contains a state mapping and multiple task mappings involving the states from the state mapping A moving rule is needed in this situation because the unmoded functionality may not be lo cated in the same place within the specification as the mode state variable If the new PUC 4 0 1103 Y 4 12
473. ur rently address helping with the initial wiring of the system and trouble shooting problems when they occur Both features could be added to the PUC using Roadie s Lieberman 2006 approach which relies on a planning system similar to the PUC s Some of the wiring prob lems could also be addressed in a tool that helps users specify the diagram needed to build the PUC s system wide content flow model This tool could also help users determine how to best wire their system to support all the flows that they expect to use It is worth noting that Roadie takes a different approach to configuration by including wiring instructions in the plans that it generates for each user task as the user is using the system This ensures that Chapter 11 Conclusion 190 users are always able to perform a task if it is possible with some configuration of their sys tem but it seems better to me to perform this kind of analysis at setup time since at least in my experience it is unlikely that users will want to rewire their system on a regular basis The PUC currently generates four different kinds of aggregate user interfaces and it would be interesting to explore both improving the existing set and building new kinds of aggregate interfaces A promising direction is a usage based aggregate interface perhaps based on the ideas of Omojokun et al Omojokun 2006 A context sensitive aggregate interface would also be interesting especially if it could o
474. uring the interface because items that are active in similar situations can be grouped and items can be placed on panels such that the widgets are not visible when the object would not be active Specifying the prior knowledge of the enabled property is discussed in more detail later in the Dependency Information sub section Label Information Another common property of appliance objects is the need to specify rich labeling informa tion for flexibility when generating interfaces in different form factors To support specifying labeling information we use the concept of a label dictionary At any place in the specification where a label can be entered more than one label may be provided It is expected that all these labels contain the same general information but vary in terms of length and detail The interface generator would choose the longest label that fits within the space allocated on the screen Labels can be specified for any appliance object and also be linked with particular values of an appliance state s type State Variable Types Every appliance state has a type object associated with it The type information is used to determine what kinds of widgets can be used to manipulate the state and is one of the pa rameters that are used to recognize Smart Templates Appendix D Specification Authoring Study Instructions 266 There are seven different kinds of types that can be used in the specification e Boolean
475. used for repre senting the fractions of a second If multiple states are used to represent the time absolute template then they must have the following form Time This state specifies the time It may have any of the types listed above for the single state format Either this state or some combination of the hour minute etc states below may be included but not both Hours This state must have an integer type The minimum and maximum must be 0 23 This state is required if the Time state is not specified and may not be included if the Time state is defined Minutes This state must have an integer type ranging from 0 to 59 This state must be speci fied if the Time state is specified and may not be included if the Time state is defined Seconds This state must have an integer type ranging from 0 to 59 This state must not be in cluded if the Time state but is otherwise optional TimeZone This state optionally specified the time zone It must have an enumerated type where each value has at least one label of the form GMT X or GMT X where X specifies the number of hours from Greenwich Mean Time The labels should also include common names for each time zone such as Eastern for GMT 5 to ensure that generators not equipped with this Smart Template can still render an understandable interface DaylightSavings This state optionally specifies whether it is currently daylight savings time This state must have a bo
476. ut information should be included in the specification language fol lowing the system requirement described in section 3 6 3 Support generation for different devices and modalities especially for small devices and both the graphical and speech modalities It is important to note that although the previous principle helps to address this one this principle also suggests that specifications may need to contain extra information to enhance support for particular devices or mo dalities For example specifications may need to include labels with pronunciation or text to speech information to support the generation of speech interfaces Short and concise are very important principles for the design of our language Appli ance specifications must be sent over wireless networks and processed by computing devices that lack the power of today s desktop machines To ensure performance is ade quate the specification language must be concise Why then choose a verbose format like XML as the basis for our language We chose XML because it was easy to parse and there were several available parsers XML is also a very compressible format which can reduce the cost of sending specifications over the network though the PUC system does not use any compression Only one way to specify any feature of the appliance is allowed in our specification lan guage This principle makes our language easy to author and easy to process by the interface generator I
477. uthoring specifications should see the complete language refer ence which is included in Appendix B or can be downloaded from the PUC web site at http www pebbles hcii cmu edu puc specification html 5 1 Design Principles Before and during the design of the specification language we developed a set of principles on which to base our design The principles are Descriptive enough for any appliance but not necessarily able to describe a full desk top application We found that we were able to specify the functions of an appliance without including some types of information that earlier model based systems included such as task models and presentation models This is possible because appliance inter faces almost always have fewer functions than a typical application and rarely use direct manipulation techniques in their interfaces Sufficient detail to generate a high quality interface as based on the hand designed user interfaces discussed in Chapter 3 Note that this principle is different than the first It would have been possible to completely describe the appliance without the readable la bels or adequate grouping information that are needed for generating a good user interface For example the Universal Plug and Play UPnP standard UPnP 2005 in Chapter 5 Specification Language 66 cludes an appliance description language that does not include sufficient detail for gener ating good interfaces No specific layo
478. ve found that a common result of this requirement is that our consistent user interfaces do not always have a similar visual appearance However there may be some benefit to hav ing a different visual appearance work by Satzinger Satzinger 1998 found that users were able to learn the user interface for a conceptually similar application more easily when the interface used the same labels but had a different visual appearance The first five requirements apply to the user interfaces that the PUC generates and illustrate the actions that the PUC will take to ensure consistent interfaces A pre requisite for all of these requirements however is R 6 Interface generators must provide a method to find similar functions across multi ple appliances Chapter 6 Consistency 92 Although this is a general requirement for any system that wants to create consistent inter faces the implementation of this method is likely to be specific to the particular type of input that the system receives In the case of the PUC the input is written in the specifica tion language which provides a functional model of each appliance The PUC s method for finding similarities may be applicable to other systems that use functional models but may not apply to systems that use other types of input such as task models The final requirement applies to the PUC s design The PUC makes consistency decisions based on the previous interfaces that users have exper
479. ve transfer effects observed from the usage of consistent user interfaces Polson 1986 In this case the common ele ments were the shared steps needed to accomplish the same task in two different Chapter 6 Consistency 84 applications Through several experiments with word processing applications Polson showed that the effects of transfer due to a consistent interface were substantial in several cases within an application between different versions of the same application and between dif ferent applications Polson 1988 In all cases where transfer occurred there were a large number of external cues to indicate to users that previous knowledge might apply This sug gests that the PUC must not only create new interfaces with steps that are similar to previous interfaces but it must also make the new interfaces appear visually similar to the old inter faces so that users will realize that their previous knowledge can be reused There has also been work exploring how user interfaces for the same application can be made consistent across different platforms Denis amp Karsenty Denis 2003 describe some of the common problems that arise when creating multi platform applications such as differing sets of functions different partitioning of functions and data and the ability to recover context when changing platforms They identify that these are all problems of continuity which they break down into knowledge continuity and task contin
480. vigationally consistent if they used different controls to navigate between different areas of the interface There is also visual consistency which means that interfaces have roughly the same appearance and there are likely other dimensions as well Because of differences between appliances it is of ten not possible to achieve consistency along all of these dimensions simultaneously Work is needed to examine the trade offs between different dimensions and to understand which di mensions are most important for users productivity Lexical consistency for example would seem to be a constant requirement but perhaps structural consistency is less important than Chapter 11 Conclusion 188 visual consistency and the PUC s consistency rules should be adapted to put more emphasis on the visual aspects of consistent interfaces It would also be valuable to explore the value of consistency versus the amount of similarity between appliances My work in this dissertation focuses on achieving consistency for appli ances that share many similar functions but not for appliances that share only a few functions For appliances with few similarities it seems that the need for consistency rules changes Functional consistency rules are nearly always helpful but it seems that structural consistency rules are just as likely to be harmful as helpful This is because similar functions across mostly different appliances may have a different context of use
481. w of the aggregation architecture is shown in Figure 9 2 The PUC s interface aggregation requires three types of input in order to function First it requires a wiring diagram that describes how the multi appliance system is wired together Chapter 9 Aggregating User Interfaces 150 which is currently is specified by hand in XML see the schema in Appendix C 3 The wir ing diagram contains a number of wire lt begin end gt pairs corresponding to the physical wires that connect the appliances The second input required is the set of all PUC appliance specifications from each of the appliances in the multi appliance system By combining this information together the PUC creates a complete model of the possible content flows through the entire system see the center portion of Figure 9 2 which is then used to gener ate user interfaces The aggregate interface generation also makes use of the knowledge base that is primarily used to generate consistent interfaces see section 6 4 1 In the context of aggregation this information allows the PUC to create interfaces that organize functions from multiple appli ances in a meaningful way The PUC produces two kinds of interfaces to help users interact with their multi appliance systems The Flow Based Interface FBI allows the user to quickly create and activate con tent flows between appliances by tapping or dragging the icons for desired sources and sinks G oug R a Du
482. wd Sonya Allin Anupriya Ankolekar Ryan Baker Gaetano Borriello Anind Dey Jodi Forlizzi Krzysztof Gajos Gary Hsieh Pedram Keyani Queenie Kravitz Johnny Lee Joonhwan Lee Ian Li Sue O Connor Trevor Pering Kai Richter Fleming Seay Irina Shklovski Rachel Steigerwalt Joe Tullio and Roy Want I would like to thank my parents Mick and Sally and my sister Amy for all their support love and guidance I have dedicated this thesis to my parents for kindling my interest in computers at a young age first buying a Commodore 64 on which I learned BASIC signing me up to take classes on programming and later paying for Internet service in the early 90 s when very few people knew what that was Without their encouragement of my interests and my education in general I would not be where I am today Finally I must thank my wonderful girlfriend Naomi I am very happy that we have been able to remain close even though our degree programs have forced us to live apart on oppo site coasts for many years and I am thankful that our time apart is almost over There is no one who has supported me more and I could not have finished this without her XXVIII XXIX XXX CHAPTER 1 Introduction Every day users interact with many computerized devices at both their home and office On an average day I use my microwave oven television DVD player alarm clock and stereo at home a computer for tracking speed and mileage on my bike
483. x B Specification Language Reference 238 lt greaterthan gt lt greaterthan state SomeState gt value lt greaterthan gt Used in conjunction with the lt active if gt tag to define greater than dependency in formation for this state variable Placement Inside the lt active if gt lt and gt lt apply over gt lt not gt or lt or gt elements Parameters e state required The name of the state that is depended upon May Contain lt ref value gt lt constant gt lt lessthan gt lt lessthan Sstate Somestate gt value lt lessthan gt Used in conjunction with the lt active if gt tag to define less than dependency informa tion for this state variable Placement Inside the lt active if gt lt and gt lt apply over gt lt not gt or lt or gt elements Parameters state required The name of the state that is depended upon May Contain lt ref value gt lt constant gt Appendix B Specification Language Reference 239 lt and gt lt and gt lt and gt Defines an and relation with the dependencies that are contained within Placement Inside the lt active if gt lt and gt lt apply over gt lt not gt or lt or gt elements May Contain lt and gt lt apply over gt lt defined gt lt equals gt lt false gt lt greaterthan gt lt lessthan gt lt not gt lt or gt lt true gt lt undefined gt lt or gt lt Or gt lt Or gt
484. xisting platforms it would also be interest ing to build a new device that is specifically designed to support the PUC system Most of today s high end universal remote controls with touchscreens also have some number of physical buttons for controlling common functions such as volume and channel see the lat est Philips Pronto in Figure 2 1 but the PUC is designed to control a broader range of appliances than just home A V equipment What physical buttons would be appropriate for a PUC to have If soft buttons are used what is the trade off between assigning functionality to them versus showing the functions in an organized hierarchy as is done by the PUC s cur rent generated interfaces Chapter 11 Conclusion 186 More generation rules could be implemented for any of the PUC interface generators The current rules are sufficient for generating interfaces for the appliances that have been speci fied and are likely to work for many others but there is still room for improvement Rules to improve the generation of interfaces for lists could be especially helpful as the PUC cur rently has only a few rules for dealing with the most common types of list structures A rule that creates a grid for certain two dimensional list structures is lacking List data is not sup ported at all in the current Smartphone interface generator so there is also a need to port current rules from the PocketPC and develop appropriate controls to differentiate li
485. xpress Some appliances also have branded interactions such as Sony CD players which have different behavior for the next track button than most other manufacturers pressing the next track button on a Sony immediately after starting the player Chapter 11 Conclusion 180 advances to track 1 instead of track 2 as most others do If the difference in these interac tions can be described in the specification language then these differences are also likely to be copied inappropriately to a new interface by the consistency algorithms My philosophy with branding is that manufacturers should not use branded interactions where they may be detrimental to the usability of the appliance These situations are exactly those described in the previous section It is very appropriate for the manufacturers logos and brand marks to be maintained in a consistent user interface but using different names or behaviors for functions that also appear on other appliances just confuses users Of course manufacturers are unlikely to agree with me and the PUC system should probably be altered to allow certain branded interactions to be preserved by the consistency algorithms This may be possible simply by allowing markers to be added in the specification that define labels and configurations of state variables and commands that should not be changed by the consis tency algorithms Cost is another important factor that will affect whether manufacturers ado
486. xs complexType gt lt xs complexType name ServerInformationType gt lt xs sequence gt lt xs element name server name type xs string minOccurs 1 maxOccurs 1 gt lt xs element name device type DeviceType minOccurs 0 maxOccurs unbounded gt lt xs sequence gt lt xs complexType gt lt xs complexType name DeviceType gt lt xs sequence gt lt xs element name name type xs string minOccurs 1 maxOccurs 1 gt lt xs element name port type xs integer minOccurs 1 maxOccurs 1 gt lt xs sequence gt lt xs complexType gt Appendix C Other PUC XML Language Schemas 254 lt xs complexType name UnregisterDeviceType gt lt xsS sequence gt lt xs element name port type xs integer minOccurs 1 maxOccurs 1 gt lt xs sequence gt lt xs complexType gt lt xs complexType name PUCDataType gt lt xs choice minOccurs 1 maxOccurs 1 gt lt xSs sequence gt lt xs element name State type xs string gt lt xs element name value type OldValueType gt lt xs sequence gt lt xs element name data type ListDataType gt lt xs element name change type ChangeDataType gt lt xs element name insert type InsertType gt lt xs element name delete type DeleteType gt lt xs element name replace type ReplaceType gt lt xs choice gt lt xs complexType gt lt xs complexType name OldValueType mixed
487. y used function for a given interface This button is currently PUC v3 0 DIC CX Yi 1 Trip Information 2 Fuel Information 3 Personalization PUC v3 0 DIC cX Yil 1 General 2 Start PUC v3 0 DIC Y Alarm Warning Settings 10 5 Headlamps and Horn gt Average Fuel Economy 13 5 Instant Fuel Economy Drive without Refuel PUC v2 03 Windows NEX i 3 Stop 1 Play Control 4 Personal Trip 2 Title Why Georgia 5 Reset Personal Trip 3 Time 6 Business Trip 4 Volume 52 7 Reset Business Trip 5 Mute False Automatic Locking z Lock Doors Out of Park 4 gt Automatic Unlocking Unlock All in Park gt 6 Media Player Info Seat Position Recall PUC v3 0 DIC CY B PUC v3 0 DIC abc Y PUC v3 0 DIC Total Distance Driven Personal Trip Total hours engine run 39 Annual Mileage Current Distance Current Distance 62 7 81 4 Amount of fuel used Amount of fuel used 5 0 10 0 Average Economy Average Economy 20 5 21 0 Figure 8 2 A Smartphone displaying a PUC interface for Windows Media Player and the automatically gen erated interface for the Driver Information Center in a 2003 GMC Yukon Denali SUV The user navigates through list panes a b to get to summary c e and editing panes d f g Chapter 8 Interface Generation 116 assigned to always perform the back function though I have experimented with other ap proaches 8 1

Download Pdf Manuals

Related Search

Related Contents

INSTRUCTION MANUAL - Institut de statistique de l`Unesco  このたびは当商品をお買い上げ頂き誠にありがとうございます。ご使用に  flexControl 3.0 User Manual  「エコチェッカView」取扱説明書  Samsung HMX-F80BP Kullanıcı Klavuzu  Samsung Diva S5150 User Manual  Samsung DIGIMAX UCA4 User Manual  Samsung SM-G3502L Manual de Usuario(open)    

Copyright © All rights reserved. Failed to retrieve file