Home

Computer user interface architecture that saves a user`s non

1. 241 01496 jenuew jenuew u I 7 ue1S 0 2 196 yaiym efed Aue JO au je e s aui peurejuo2 91 185 siy asn ued noA 5 St jo jurod Guyyeys 10 doy eu abed ues s d 5uo5 21529 US 7 496 830 B2 Sheet 9 of 11 Feb 24 2009 U S Patent SNINE sy jo 3SH aed quand eu 20 1881 9 ejdsip nim siu 9 631 4 sBuiddjp pue ued ojpne pue usia senpn ed e pappe ueeq eu suopeouue Aue i Sujdde 1 SNOLLYLONNY AAOHS ITIH atO 4009 30900 Jano Supeoy e se 3j Aejdsjp pue eBed yuewno yo doo epu098s e ez alj M uo Budde 4 S HL T uaou mata uj Aqua uno si ze 3 uei 31ON deL 310 LYISNI o 3523102 ao Baipz0g q paaip2qo iid 3 q0 DAD spaoara o 21 ajout 348 3094000 Jo 8042 uadeu Sn0JA81d eat se ooq uj Auowwea 3 Tuose Jaquinu ay uo EWN 3514 a
2. 01 0142 eq IA ue amp 1OjA e3euec paureye Lony S RpOpA eq jueso1d ye sn 39 puooes 194 spued Q00 ST Jo no o1d e eeorunururoo MOY St em urajqo4d aq s Qoiapues jo mj ujnour sty iors ein pondet suet Te g onsenb et oj 3oe1 p eui o pue LNeod Jo ayy yMb sn yoy eueorqreg pres MOU Jo e3nq e o3 jesu podde pue umop yes euIooJeAQ n JOpessequre erns 8 07 onp mA any In peareoe1eq jou st a im nue3 j u eS pucoes e sopra t Jo Aypides oy q n youney oye me Burpedoxe peyrugur uot etm eq 8 spared 10 ses1oq jsesjius etn JO rey OF seum peipunq req uouueo ed jo sseujtus oq peiueAureAeQ punos pue pura jo sjeuejd oy pue sjeuico eu saes ay Jo jo pue Arousa sseu r s eq ure o 3er eoue15nxe ojur eq uouueo etn sey ueur syoue d ot pue ses etn peyee1o sey eouepi amp o1djI Jomod jo jueorjruSeur your s puru s o1 ueujenue8 eq uouueo bo p gr neg Jo eul A 19deu5 0 ET
3. 2 58000 eu uy 0 8 5 tn 40 pue 32exe ue ayea ji eed 3eu jo Agen foc uay pue 1xa uoj8a e 3upejas 9NIddI D Bed zey 04 xooq ay uedo jp xieurjooq e Budde uj j eded zep uo reunjooq 8901 jo saddn uj Suidde 00 8th jo fya sno aa4d2 M JXBU et 04 ejdsip ayy jo si8u103 sedda uj suofing MOUE pue 381 BY 39Vd E fai TEANA 2 q0 t0A0 q 0130 6 jenuew bip 05 sil Kivoja tmurmeui eaor I She fpuroiqseg perder w SEL peurejqo pue Suon s riv puooos Jed spares oog Jo e qr xis Jo ouetsip uo Bey BuryBiem joys P MAIN pPrqum o eq pucces e Jo Ayoojea entur ue e noefo1d noy pesseudurt spred ooo S Jo eoueysip poneo sues8 yeq Jepunod oot eq eq Aes T Jo eq Jo 1equieui SEM T eq Bump eq pardar spsee puy sun
4. 09 456 127 entitled Bookmarking and Placemarking a Displayed Document in a Computer System filed X which is incorporated herein by reference Example bookmarks 800 and 802 are depicted in FIG 8 Ink annotations are described further in commonly assigned and currently pending U S patent application Ser No 09 455 805 entitled System And Method For Annotating An Electronic Document Indepen dently Of Its Content filed X which is incorporated herein by reference These annotations may be performed with not only different input methods but adapted to the most convenient or natural one such as a finger for highlighting a stylus for writing or doodling and speech for lengthier commentary Storing Annotations The annotations exist apart from the content as files with links to the content therefore annotations can be displayed not only layered on the content in appropriate positions within the content such as for instance note 1000 in FIG 10 but in other locations and visualizations That is each of these annotations is available to the user not only in situ in the content where it was created but cross referenced in auto matically created indices which can be manipulated much as described in the Viewing History section above by time by appearance by site document section page etc Likewise annotations could be shared selectively shared or kept pri vate with the kind of functionality described in the Collabo ration sec
5. Feb 24 2009 U S Patent ua seq samun Que L o soydey ued sity 6z 191deuo JIS MON V 02 dey peram noq 4z 92 1 speq peurg Sz AYOoy eq jo doos l 1 eu Pz 1 2 e orqeA e noefoid eu z saydeyo SPIS pantun em Jo u zntO M N eu zz 1 1deuo Jregjy uy soseueyy ueurqouo14 v eysodny pueyoeny oz 1 44 auon V 19 deuo ezuepy eu jo JeBuesseg ayy gr 19 deuo yoqedsiq v Lt saydey PeIqUINJOD 9 2 Sunse Jo eq Sr 1 1e1deuo TH suos sadeg 49203 1920 zt z deuo Sex L pue eptzojq 11 uaydeyy Jo suomi Sz A Aweug 1949 em jo uonsend 6 om jo A1ONIH 8 ideya peg uouueg y jo 1e1deuo jeneg pue eouesouBy sun yL 9 edeg atp jo aouewoy 5 te1deu5 jo 10 19540 2 P 1oYdeu5 sjuepiseigeuijoioegH ejdeq s eueomregjuaprsaid z uoo IN oy O Yq P H ouy d 9 Senn Fra ar ia ig a N EINE SLNE LNOO dO TIV L 318
6. US 7 496 830 B2 Sheet 10 of 11 Feb 24 2009 U S Patent aed b he ie E 545 d hi a I SC OL V pies ur uo uey e JOYLI p noA T PS 94 01 SJEY M ol ymy P anu ep 803ON mot 8007 upumsnq perde z yunt v pres Wu on eM don qnp ang p 1594 Kur 3uepiseid em teueotqreq ieu peuure poom em o8 e u os nod PIPMOPTEI axe q p a usaq peq ym Sjo3ej Sui jes perdn2oo Sun 10A ueurqsnq tre SEM eeu T Inoq we 19p4oq om possed asnu eueotqeg j eQ pue sinoy apun 9 jo 651 weuruarj V TZ 191deuo 91 yore JOU p noo Ley spjerjeor pue sxee1o pey pue mop sure d eq pesso1o peq aq pueuepiy PYON em 3nq Spsepour petid fae K amp 1oue yonu wy pey uoraeduioo sty uepiV PYON jere suourep PYM sInoq 104 0188 45 dn pue sZop reu 44 perueduroooe SINUL A 51015200095 U
7. being a built in document these are just documents that a PV UI happens to contain by default The list can be customized as desired Possible standard built ins are Sign in Page This document provides the form for sign ing in if and when a PV is set in a secure mode Start Page An example Start Page is depicted in FIG 5 This is the central starting point for the PV UI From here a user can link to any content including UI accessible to the user for instance documents on the PV itself as well as in the user s desktop workspace and the Internet The Start Page may take the form of a personal newsletter My Documents 520 This is a folder listing of all the documents a user has read authored subscribed to or purchased It is essentially a catalog of a user s per sonal library It supports various views to let a user get an overview of what content the user has and to find the content that the user is looking for My Documents may have links to other PV content like the Address Book and Guide Book for instance Guide Book This is both the PV s help document and the PV s user interface Example pages of a Guide Book are shown in FIG 7 The UI is embedded in the Guide Book as links that perform UI functions according to the principles of this invention The Guide Book includes the Quick Help page of fre quently used UI commands FIG 8 depicts an example Quick Help page Map book A book of maps including the current s
8. ways such as an underlined blue label a button an icon ora graphic image They can be copied from one place to another moved and deleted Their properties can be manipulated such as to change their appearance or to preset some or all of their parameters Commands can also be created the same way that any kind of link is created via a Link command which establishes a link between a source and destination that the user specifies in the case of creating a command link the destination is an executable object such as a command script or binary Everything users can do to a link or to objects in general since a link is an object users can do to commands Guide Book Users get at commands by navigating to a page where the desired command is found The documents with links UI is organized to make frequently used commands a single navi gation step away or through customizations no steps away Less commonly used commands may take more steps to get to The documents with links UI includes a self explanatory document the Guide Book This is literally a readable manual that users can go through in a logical order a page at a time like any current user manual The difference is that each command mention is an active command instance that can be invoked in place A variety of Quick Help pages and indices US 7 496 830 B2 7 make it easy to get quick access to sets of commands that are commonly used together that are logically related
9. 12 United States Patent Rubin et al US007496830B2 US 7 496 830 B2 Feb 24 2009 10 Patent No 45 Date of Patent 54 75 73 21 22 65 63 51 52 58 56 COMPUTER USER INTERFACE ARCHITECTURE THAT SAVES A USER S NON LINEAR NAVIGATION HISTORY AND INTELLIGENTLY MAINTAINS THAT HISTORY Inventors Darryl E Rubin Redmond WA US Andrew C Baird Kirkland WA US John L Beezer Redmond WA US Jonathan C Cluts Redmond WA US Susan D Woolf Seattle WA US Assignee Microsoft Corporation Redmond WA US Notice Subject to any disclaimer the term of this patent is extended or adjusted under 35 U S C 154 b by 420 days Appl No 10 875 174 Filed Jun 25 2004 Prior Publication Data US 2004 0233235 A1 Nov 25 2004 Related U S Application Data Continuation of application No 09 456 975 filed on Dec 7 1999 now Pat No 6 820 111 Int Cl G06F 17 00 2006 01 USCh eun 715 206 715 205 715 208 715 240 715 853 Field of Classification Search 715 500 1 715 501 1 206 205 208 240 853 See application file for complete search history References Cited U S PATENT DOCUMENTS 5 146 552 A 9 1992 Cassorla et al 5 237 648 A 8 1993 Mills et al Continued FOREIGN PATENT DOCUMENTS EP 0342838 A 11 1989 Continued OTHER PUBLICATIONS Hirotsu et al Cmew U A Multimedia Web Annotation Sharing System NTT Network In
10. 369 811 6 370 497 6 377 983 6 389 434 6 393 422 6 397 264 6 405 221 6 418 421 6 421 065 6 425 525 6 437 793 6 437 807 6 446 110 6 457 013 6 460 058 6 484 156 6 486 895 6 490 603 6 529 920 gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt 1 2000 1 2000 2 2000 3 2000 3 2000 3 2000 3 2000 4 2000 4 2000 4 2000 4 2000 5 2000 6 2000 6 2000 6 2000 7 2000 7 2000 8 2000 9 2000 10 2000 10 2000 11 2000 11 2000 11 2000 11 2000 12 2000 12 2000 1 2001 2 2001 2 2001 2 2001 3 2001 3 2001 3 2001 4 2001 5 2001 5 2001 6 2001 6 2001 7 2001 8 2001 8 2001 8 2001 8 2001 9 2001 9 2001 10 2001 11 2001 11 2001 11 2001 12 2001 12 2001 1 2002 3 2002 4 2002 4 2002 4 2002 5 2002 5 2002 5 2002 6 2002 7 2002 7 2002 7 2002 8 2002 8 2002 9 2002 9 2002 10 2002 11 2002 11 2002 12 2002 3 2003 Harada et al Herbert III Finkelstein et al White et al Astiz et al Himuel et al Danneels Bertram et al Gill et al Tran Kessenich et al Ho Bates et al Wen Sidana Mortimer et al Buxton et al DeRose et al Kanerva et al Jaremko et al Lebling et al Jain et al England Fraenkel et al Rangan et al Bates et al Fogg et al Tarpenning et al Bates et al Bauersfeld et al Chen et al Ferrel et al Fiedler Umen et al Himmel et al Borman et al 715 501 1 Pacifici et al Shwart
11. Al 2002 0194260 Al 2005 0060138 Al 7 2002 Gottfurcht et al 7 2002 Rubin et al 12 2002 Headley et al 3 2005 Wang et al FOREIGN PATENT DOCUMENTS EP 0 822 501 Al 2 1998 EP 0 890 926 A1 1 1999 EP 0 902 379 A2 3 1999 WO WO 87 01481 3 1987 WO WO 96 20908 7 1996 WO WO 97 22109 A 6 1997 WO WO 98 06054 2 1998 WO WO 98 09446 3 1998 WO WO 99 49383 9 1999 OTHER PUBLICATIONS Kuo et al A Synchronization Scheme for Multimedia Annotation Multimedia Information Networking Laboratory Tamkang University Taiwan IEEE 1997 pp 594 598 Hucklefine Defined Apr 1990 http www ylem org artists mmosher huck html pp 1 3 Tour of Korea Hypercard Stack 1996 Grose Educational Media http www entrenet com groedmed kor2 html pp 1 2 Bizweb2000 com screenshots of an e book published on May 1999 1 4 http web archive org web http www bizweb2000 com eshots htm Schilit and Price Digital Library Information Appliances ACM Conference on Digital Libraties Jun 23 1992 pp 217 226 Graefe et al Designing the muse A Digital Music Stand for the Symphony Musician Proc of Conf on Human Factors In Comput ing Systems Apr 13 18 1996 pp 436 441 Crespo Chang Bier Computer Networks and ISDN Systems Responsive interaction for a large Web application the meteror shower architecture in the WebWriter II Editor 1997 pp 1508 1517 SOFTBOOK Press The Leader in Internet Enabled D
12. FIG 1 include a local area network LAN 112 and a wide area network WAN 113 Such networking environments are commonplace in offices enterprise wide computer networks intranets and the Internet When used in a LAN networking environment the com puter 100 is connected to the local network 112 through a network interface or adapter 114 When used in a WAN net working environment the personal computer 100 typically includes a modem 115 or other means for establishing a communications over the wide area network 113 such as the Internet The modem 115 which may be internal or external is connected to the system bus 130 via the serial port interface 106 In a networked environment program modules depicted relative to the personal computer 100 or portions thereof may be stored in the remote memory storage device It will be appreciated that the network connections shown are example and other means of establishing a communica tions link between the computers can be used The existence of any of various well known protocols such as TCP IP Eth ernet FTP HTTP and the like is presumed and the system can be operated in a client server configuration to permit a user to retrieve web pages from a web based server Any of various conventional web browsers can be used to display and manipulate data on web pages FIG 2 shows a tablet and stylus computer that can be used in accordance with various aspects of the present invention Any or all
13. and Simple The Ul architecture of this invention also referred to herein as a documents with links UI supports creation of UI s that have essentially zero clutter and few concepts to master In this way it is a major departure from prior art UI s using a desktop metaphor The user of a documents with links UI according to this invention focuses primarily on content and not on UI appurtenances Starting with just knowledge of how to page through a document and to follow links a user can learn how to do any other UI operation Significantly the documents with links UI works without drop down menus toolbars windows or other cluttering UI elements although some of these elements may optionally be made available where they are desired Content and UI are Unified In the UI architecture of this invention there is essentially no distinction between UI pages and content pages and content are the same thing and exist in the same navigation space As described in more detail below smart next previous logic and intelligent management of the navigation chain solve technical problems caused by treating and content as the same thing in a unified navigational context Because there is no seam between UI and content no notion of dual spaces the documents with links UI is con ceptually simpler for the user than a model that has separate UI and content webs The one space model is also more powerful and cust
14. and move mouse Jotting is done via the keyboard i e the user gets text instead of ink The mouse supports a few other idioms as follows These and other idioms can be customized Right click brings up a context menu the same one that a hold operation at this location would Mouse wheel does page forward and back operations Clicking the mouse wheel third button links to the Quick Help page Navigating Hyperlinking One way to navigate is by hyperlinking The PV UI renders textual hyperlinks using a visual emphasis similar to the way a prior art browser does Tapping a link will navigate the user to the linked to place The user can use the Previous and Next operations to move along the link chain The PV UI Start Page can exist as the most previous document page in the chain A user can also hold a link Doing this will perform a link specific action The default behavior for hyperlinks is to present preview information about where the link will take the user 1 document name and document position infor mation and or a thumbnail view of the linked to document Continuing to hold the link could cause the preview informa tion to expand into a navigational map of the link context emanating from the link the user is holding Popup menu choices could also be presented allowing the user to manipu late the link such as to change its properties When the user releases a held link no navigation occurs The user can tap the link t
15. could also be included What appears on the Start Page can be both user configured as well as configured from profile information the PV UI has learned by observing a user s browsing and e mail reading patterns and the like User can Modify Start Page Advanced users can perform the same kinds of operations on the interface itself as those performed on content If the default components on their Start Page are not to their liking they can delete or alter even system offered components such as Mail Messages Or for instance if the automatic page number used by the bookmark header is insufficient for rec ognition they can alter its text color or any other property Audio Recording and Annotation The PV UI supports the creation of audio clips that can be used for annotation of any displayed document The clips are basedonatimeline model in which audio or video recording 0 jak 5 40 45 24 is a data stream parallel to and synchronized with the material in a document If the user has changed pages while recording then the clip when played back will also change the page when the appropriate place in the audio clip is reached The interface supports both document and page specific audio notes as well as global recording The interface is presented with cassette like controls including index forward back and editing Each clip is stored as an individual document and can be sorted and filtered to present the clips in mult
16. display and or a dual display Some dual display UI actions will cause only the rightmost display image to be replaced When this happens the original image may or may not be shifted to the left display when this happens A rule that can be used is that when the UI action was invoked from a link or menu action initiated on the right display the image is shifted to the left display and the new page opens on the right If the action was initiated on the left display the new page simply opens on the right and no shift occurs The result is that the new page opens on the right and the page from which it was initiated appears on the left For a single display PV the current display image is simply replaced by the new page A user can use the Previous and Next functions to flip between the original and new pages A PV can be held horizontally or vertically Further dual display PV s can treat the two displays as separate pages or as halves of the same large page The PV can adapt its assignment of case touch zones and display areas to present a consistent geometry to the user regardless of its orientation and display mode That is the PV re maps its definition of things like upper left corner and right margin to be con sistent with its current orientation Gestures for Performing PV UI Actions While other suitable input equipment could also be used PV Ul actions will be described as occurring via touch opera tions using a fin
17. heuristics are described further in the context of navigating to a particular portion of the Guide Book in commonly assigned and cur rently pending U S patent application Ser No 09 456 952 entitled Method and Apparatus for Providing Help and Set tings Control to Users of an Electronic Book filed X which is incorporated herein by reference Frequently Used Links The PV UI could also provide commands actually links embedded in the page that are related to the kind of material on the page In a mail message for example reply reply all and forward are links so frequently used as to warrant their inclusion directly on the page There may be other com mands or links used so often that they are dynamically bubbled up or in the case of linear reading used so infre quently that none appear The second level of visibility hold on the page for more information offers a more extensive list If none ofthese satisfy the user s need the user can summon the Guide Book Start Page The top level ofthe PV UI provides the user with a Personal Newsletter or Start Page which is the launch point for many activities An example Start Page is depicted in FIG 5 High lights might include urgent mail messages projects PIM items documents news which could be divided into system offered default components such as Messages Notes Clip pings and the like Favorite links that the user wants to keep readily available
18. issued the New command Physi cally the object may be stored in the user s sea of free space in a hidden system folder not part of any folder the user is aware of unless and until the user chooses to file it some where The user does not need to put documents into a filing hierarchy or save them This is because the documents with links UI stores all navigational history Accordingly users can find the documents they create by viewing or searching their history map A user could file a document into a folder as an optional step using a Save As command or by manually creating a link in the folder that leads to the document the Save As command could be simply a shortcut for creating such a link Also the user could use Save Save As to update or create versions of a document in the filing hierarchy as desired EXAMPLE HARDWARE PLATFORMS FOR IMPLEMENTING ASPECTS OF THE INVENTION FIG 1 is a schematic diagram of a conventional general purpose digital computing environment that can be used to implement various aspects of the invention Computer 100 includes a processing unit 110 a system memory 120 and a system bus 130 that couples various system components including the system memory to the processing unit 110 The system bus 130 may be any of several types of bus structures including a memory bus or memory controller a peripheral bus and a local bus using any of a variety of bus architectures The system memory includes re
19. known input devices EXAMPLE STEPS FOR IMPLEMENTING ASPECTS OF THE INVENTION FIG 3 is a flowchart showing simplified steps at a high level of abstraction for implementing a UI architecture according to the principles of this invention upon start up Processing begins at start block 300 In step 302 the system checks to see whether there is a prior existing state If there is a prior existing state then that state is used as the current state Ifthere is no prior existing state then in step 304 the system retrieves a previously stored default prior state and assigns the value of that default state which could be for instance a default start page to the current state In a preferred embodi ment like other content and UI the current state is stored as a document that contains links In step 308 the system waits for input from a user Upon user input the system determines the nature ofthe user input in step 310 Gestures for performing UI actions are discussed below The nature of the user input could include a combina tion of the input device such as a pen and the gesture per formed by the user The system then determines the region of the user input in step 312 and based upon the region of the user input determines an object associated with the region in step 314 The system then determines an action to be per formed based upon both the nature of the user input and the object associated with the region ofthe input in step 31
20. margin of rightmost display 518 clipping tabs Clippings or other objects placed here are associated with the containing page Inner margin to right of spine 600 see FIG 6 dual display PV UI no default use Inner margin to left of spine 602 no default use In the above list examples of other objects that may be present in any of the margins are user annotations A user can reassign which functions are associated with which margin Accordingly the margins may be referred to as the bookmark document clip and command margins respectively Of course other suitable default margin arrangements could also be used For instance there could be a zone around the entire document that is outside the docu ment called the edge zone or zones The edge zones could contain command shortcuts and other kinds of links Com mand and shortcuts and toolbar tabs could prefer the lower edge zone and links representing the thumbnail view of documents could prefer the left and upper edge zones Because a page margin would not be needed for carrying command links clipping tabs could by default use the left margin of the left page for clippings on the left page and the right margin of the right page for clippings on the right page The inner margins could remain unused except if the user puts annotations or other links there The difference between the margins only embodiment and the edge zones embodiment is that in the margins only embodiment the u
21. of claim 9 wherein the history view map depicts the stored navigational history non linearly 11 The computer readable medium of claim 9 wherein the history view map depicts the stored navigational history restricted to a particular time frame 12 The computer readable medium of claim 6 wherein the second navigational path includes one of the plurality of user interface documents and one of the plurality of user content documents 13 The computer readable medium of claim 12 wherein at least one page of the first and second navigational paths is the same page 14 The computer readable medium of claim 13 wherein the same page is one of the plurality of user interface docu ments 15 The computer readable medium of claim 13 wherein the same page is one of the plurality of user content docu ments 16 The computer readable medium of claim 1 wherein the user interface includes only one set of navigation controls to navigate the generated user interface and wherein the single frame including at least one user content document and user command document is displayed in only one navigation space navigated by the only one set of navigation controls 17 The computer readable medium of claim 1 wherein the user command documents include information describing the user interface commands the information describing the user interface commands including active command instances of the user interface commands 18 The computer readable
22. or that are typically used as part of a given scenario Such Quick Help pages could be assembled dynamically based upon the con text of the document or documents being viewed by the user Users can also use search to find commands This could call into play an intelligent user assistant or other conventional help mechanisms when appropriate Even if all commands were only one navigational hop away however the documents with links UI would not be ideal because users want the most common commands to be zero hops away Users also want context sensitivity to com mand presentation like that provided by conventional context menus The documents with links UI therefore accommo dates things like toolbars context menus and other UI short cuts that the user can customize Like everything else in the documents with links UI shortcuts are implemented as documents with links The UI shortcuts can be conceptual ized as being layered on top of the base documents with links UI yet shortcuts are actually constructed out of the same components that comprise the documents with links UI documents with links As a user looks up commands from the Guide Book the user will learn short cuts so that the more a user interacts with the UI the less often the user will typically need to navigate to the Guide Book Therefore shortcuts will be discussed below with reference to a preferred embodiment of this UI architecture implemented on a personal viewer as
23. recently selected area All commands that are configured to operate on selections will operate on that most recently selected area US 7 496 830 B2 9 Having executed they will trim the navigation context at the branch point that leads from the current selection to the com mand itself The benefit is that users are free to link into the web of UI pages exploring them as necessary to find the desired com mand and then to invoke it The act of doing so will end up trimming all the UI navigation from the context leaving the user back where the user was before navigating to UI pages Note that depending on the length and content of the naviga tional path between the command and recent selection the UI may show the user the target and prompt the user to confirm before proceeding Note that if no selection exists when a command is executed then the next selection the user establishes will be considered to be the most recent selection for purposes of determining the command target In this case after the user makes the selection command buttons will be presented in context with the selection by which the user can confirm or cancel execution of the previously selected command Of course other suitable methods of determining what selection to operate on are also possible For instance one such method is to allow only a single selection to be extant at a time That is any time a new selection is initiated any prior selection is cancell
24. screen that give the user direct access to UI help information can also be provided Because the Guide Book is simply content operations that may be performed on other types of content such as the Start Page the UI and content in general may also be performed on the Guide Book Unlike prior art user interfaces the way a user gets help for performing a function and the way the user US 7 496 830 B2 25 performs the function are the same To perform a function and to get help for a function the user simply activates a link CONCLUDING REMARKS The foregoing has described a user interface architecture based on documents with links that facilitates creation of user interfaces that allow users to read annotate collaborate and perform other tasks typical of knowledge work as well as alter the interface to best suit their work patterns It will be appreciated that many modifications and variations of the invention are possible and the specific examples and descrip tions herein do not limit the scope of the invention We claim 1 A computer readable medium having computer execut able instructions that when executed by a computer cause the computer to perform steps of generating a computer user interface comprising a plurality of user interface documents having links and a plurality of user content documents having links the user inter face documents including user interface commands wherein the generated user interface inclu
25. several steps and then navigates through a different set of links the user is still able to go back and then retrace the original path of links they navigated With prior art brows er s this is not possible recollection of the first set of docu ments that were backed over is lost Further the PV UI maintains a nonlinear navigation con text that records every place a user has ever navigated to when in time the user was there and where the user went to from there This in turn allows a user to navigate from general content pages into user interface pages perform inter 20 25 30 35 40 45 50 55 60 65 22 face functionality and then return to the user s documents without losing what the user s navigational context was before navigating to the user interface pages As described in more detail above in the Rich Navigation And Maintaining The User s Navigational History section this invention auto matically removes navigation from the beginning of naviga tion into UI pages that lead up to performance of some UI functionality so the user s navigational history does not get cluttered with navigation within UI pages Saving the user s navigation history may also help a user retrace navigational steps that the user would not otherwise be able to remember For instance suppose a user does not remember where they were when they viewed some content they liked If the user remembers where they had been before
26. their linear navi gation chain the documents with links UI doesn t forget all the twists and turns of where the user has been just because the user backs up and proceeds in a different direction The documents with links UI stores not only where the user has been but also the path s the user took to get there The user can use the history map and or the Next Previous commands to get back there again The Next function works in this network style navigational context by using heuristics to pick which path forward the user most likely intends The most basic rule is to pick the forward path along which the user backed to the current node Other rules provide additional intelligence to account for a 0 an 5 40 45 60 65 8 user s known navigational patterns such as whether the user got to the current node by navigating back by individual pages or by groupings of pages such as by site or by linking from a parent to a child The Next function could include options to present a user with a list of forward choices textually and or as a map in which the user could zoom into desired areas In this way the user can control which branch to take ifthe user is not satisfied with the documents with links UI s selection Part of what makes the navigation and context trimming heuristics possible is built in knowledge of logical levels of information grouping For example the documents with links UI can include knowled
27. they viewed that content then the user can navigate to this prior place and query navigational history for everywhere they had navigated to from that particular location This is very unlike prior art multi windowing UIs in which each application has its own navigation history that can not be integrated with the history of other applications a user is concurrently running For example suppose a user runs a word processor and a browser concurrently on a desktop computer The user can switch between them in a way well known in the art The word processor and the web browser will both have their own state information In other words the word processor and the web browser will each be separately nested in their own experience of history or navigation As a result if the user switches from the word processor to the browser visits a few web sites and wants to return to the word processor the user will not be able to get back to the word processor by hitting the browser s back button The browser and the word processor essentially exist in separate contexts with each context having its own state information and unique history Annotations Making Annotations The PV UIallows a user to interact with content and the UI to make either or both more memorable via bookmarks clippings highlights overlaid and embedded ink and audio notes Bookmarks are described further in commonly assigned and currently pending U S patent application Ser No
28. user s To field fill in choices The user has the full power of the normal UI for navigating viewing and searching the address book and can navigate to other documents containing addresses where the user can make other choices Note that because the selection state is associated with the path the user took to reach the address book the user will see the TO items highlighted in the address book only ifthe user links to the address book or successor nodes via the TO field in question Thus the existence of an active To field does not interfere with other uses of the address book or with other active To fields This is a significant principle of the documents with links UI the path a user takes to reach a document typically affects the behavior and presentation of the document This is a way to achieve state like behavior without requiring special modes or UI mechanisms like dialogs The implementation of chooser fields is one of the more important uses of this con cept Modeless UI Unlike prior art UI s the documents with links UI is essentially modeless For example a user could be involved in filling out a form for carrying out a UI operation such as creating a formatting style for a table In the middle of doing this the user could navigate away from this UI form and get engaged in some other UI operation such as filling out another form for a different purpose and then at any time navigate back to the original incomplet
29. 3 ayy US 7 496 830 B2 Sheet 7 of 11 Feb 24 2009 U S Patent Sjnayous e aui 0 Apoq eu uy ereyaiue eaae uuroyed ues noA I 18quinu abed ayy 0 dunf uea nod e u3ns sapeay uo Bas jeu jo uoiyod Aue Inoqe 196 0j Japso pjoy 0j nu luoo pue e u 1x luo2 Moyoup 2 1no pue pej2ojes 4x9 jo 5 yeyy Burddij3 ajeaia jo ay uay eBed pa uoruod e yong 1eul yey o 196 0j x2eq asn ues nod yoy eBed yey uo pasu jwa jo seddn ay dey nod ji sy4eurjoog S3do2u02 21528 jenuew Buyjesado jenuew 31004383 Jo awos ji 2004 djey e dn sje siyy 2984S 51001 S US 7 496 830 B2 Sheet 8 of 11 Feb 24 2009 U S Patent ewnqume 40 351 e eBed 3uaun3 10 4 eig Aedsip yit siu Sujdde 16
30. 311414084 1 suddo pue SHEURIOOY Bunyan pue Supmesp uad sazou pue esed SA Aous uo Su dde 1 SNOLLV LOMNY AOHS 3 IH d se 41 Aejdsip pue eSed yaum eu jo Adoo 5 aed 9300 ohne 40 310N 1 35 deL 310 LU3SNI Pe ak Tr MEE ee hap fo 3u2 800 2g B jpjoy Aq ypu qp ibAb aso Spana aod Saymojzof 941 4 1238 0 1 snojAeud tansyon se yooq zep uj juowwoo JO FSEQuasaud p aaquunu uo 3ujpjoH YIEVINN 39Vd E lt 5870 uon2ajes pue 32exe ue jj yo S ET pue j unie e BUHIGAS ONIddITD BREZAT 3ed yep 0 Uado yreunooq e Budde Uy esja j aBed uo aged e saddi u Buddel ONDUYWHOOa es your 004 ely 40 eBed Ae1dsIp eu Jo episyno MOLE 3jaj 1 3974 5 had BIGM 240
31. 6 The system then performs the action in step 318 The system then loops back to step 308 to wait for more user input FIG 4 is a flow chart showing simplified steps for imple menting the display state step step 306 in FIG 3 Refer ring to FIG 4 processing begins at block 400 In step 402 the system retrieves a link from the previously saved display state which may include a list of active or visible links The retrieved link s display properties are examined in step 404 In step 406 a frame is then displayed based on the link s properties In step 408 a check is performed to see whether based upon the link s properties linked to content or other information about the linked to document is to be displayed in the link s display frame If such content or information is to be displayed then the link s target is examined as depicted in step 410 Otherwise there is typically no need to examine the link s target Then in step 412 the link with or without content or information or both from the linked to document is displayed in the display frame Steps 402 412 are repeated 20 25 30 35 40 45 50 60 65 14 until the check for more links in the current display state fails in step 414 Display state processing then ends in step 416 Preferred Embodiments of the UI Architecture for a Personal Viewer Various preferred embodiments of the documents with links UIofthis invention are described herein as
32. 61 683 5 784 058 5 786 814 5 790 818 5 801 685 5 801 687 5 802 516 5 819 301 5 821 925 5 822 720 5 826 025 5 832 263 5 835 092 5 838 313 5 838 914 5 845 262 5 854 630 5 860 074 5 862 395 5 877 757 5 893 126 5 893 132 5 918 236 5 920 694 5 923 326 5 924 104 5 931 912 5 933 139 5 933 140 5 937 416 5 940 080 5 948 040 5 950 214 5 956 034 5 956 048 5 978 818 5 982 370 5 983 248 5 986 665 6 011 537 6 012 055 6 018 334 6 018 342 U S PATENT DOCUMENTS 8 1993 12 1993 9 1994 2 1995 7 1995 7 1995 11 1995 9 1996 11 1996 4 1997 4 1997 5 1997 5 1997 7 1997 9 1997 10 1997 10 1997 11 1997 11 1997 2 1998 2 1998 2 1998 3 1998 3 1998 4 1998 4 1998 6 1998 6 1998 6 1998 7 1998 7 1998 8 1998 9 1998 9 1998 9 1998 10 1998 10 1998 10 1998 10 1998 11 1998 11 1998 11 1998 11 1998 12 1998 12 1998 1 1999 1 1999 3 1999 4 1999 4 1999 6 1999 7 1999 7 1999 7 1999 8 1999 8 1999 8 1999 8 1999 8 1999 9 1999 9 1999 9 1999 9 1999 11 1999 11 1999 11 1999 11 1999 1 2000 1 2000 1 2000 1 2000 Morgan et al Norwood Agulnick et al Milne et al Beernink et al Matheny et al Webb et al Gough et al Judson Rivette et al Levine et al Zellweger Warren et al Aihara et al Logan Levine et al Beernink et al Volk et al Pritt Graber et al Moran et al Hoddie et al Barrett et al Rothrock et al Rowe et al Pisutha Arnold Berman et al Munyan Logan et al LaStrange et
33. 7 496 830 B2 Sheet 2 of 11 Feb 24 2009 U S Patent Z ld U S Patent Feb 24 2009 Sheet 3 of 11 US 7 496 830 B2 300 302 prior existing No 304 state get default state Yes display stat aoe 308 310 determine nature of user input determine region of user input 314 determine object associated with region 316 determine action based on input and object 312 318 FIG U S Patent Feb 24 2009 Sheet 4 of 11 US 7 496 830 B2 get the next link in the current display state examine the link s display properties display a frame based on the link s properties 408 400 402 404 406 display linked Yes to content es examine link target 412 display link in frame 414 more links in the current display State US 7 496 830 B2 Sheet 5 of 11 Feb 24 2009 U S Patent ubiz2p IN s22u9421u0 5 53004 21004226 3 S LYHM MEI WBA anog preg SLSINWNIOD 1 aaow 53219 1405 5 P A 9118 WO 53564434 359191 ASINO SOUPI UA 1 spuni TAT 008 WIINUNIS 7 532045 SN 10 5 S3NT1GUd4H S AYAOL 4 SLNAWNIOTD A US 7 496 830 B2 Sheet 6 of 11
34. IG 8 UI being content takes advantage of the large format of pages and the ability to turn navigate pages versus the smaller size of menus dialog boxes in prior art UIs and their limited or nonexistent provisions for navigation At the other end of the spectrum the UI scales to small screens because of the flexible selection model that allows command selection regions to be initiated either before or after the desired command is chosen This means for example that on a small screen a user can call up a page of command choices that completely obscure the original docu ment due to screen size limitations choose the command then return to the user document and then select the region for the command to operate on Suppose a user wanted to refor mat some text but had not selected the text to be reformatted yet The user could press a link that indicates that it will reformat text The UI of this invention will display an indica tor on the screen prompting the user to select the text to be reformatted After selecting the text the user can finish the reformatting operation by clicking on another link such as a link that indicates that the selected text will be reformatted Alternatively the user could select the region first and then go find and invoke the command Being able to select the object upon which a command will operate after selecting the com mand is unlike the selection model of prior art UIs where object selection must prece
35. O esneo days es ej e s 3ueurour e eyejsrur Buls y eq Jo 3ueos ayy Suruuno aey goua8rjayut yomb samed o jo suerpug ey Jo s ne doud jnyjepuow esotp 16400 aq ru 511 ueq L SjSeeq JOJO OM spenp eyeatud wet SunpoN ue ed 194 y niioddo J0j peusia 3uo eq Kao sem Surjeeut ay e u os om erp pue pue seed KI eAH e qduris esoze yey 1 s pey ureydeo y pue yuepis 1d eq spuary sque Aym pue eyep PIO eueorq reg Aso em jo cor eq p o H seo eu Jo eyes er Wepiy Uoyse yy ETD sr Burp Seprns UMOL edure jo sqingns 10 oM em seynuru req sse pue seqyo sty uo ind peq undnuejut uepay sem I RM u UEPIY St yey pue tury dojs eueorqeg 1 ugnou sey uo e ouo pue sty aM pue ue St mo uei sr eq JI eueo
36. PIN THIS PAGE the description states that Tapping on this command will freeze a secondary copy of the current page and display it as a floating page over the book FIG 9 shows an example of a pinned page In addition to describing this command the text PIN THIS PAGE is a link that when activated will execute the PIN THIS PAGE command on either the most recently selected page or on a page to be selected after activating the PIN THIS PAGE command link Obtain Help and Perform Functions in the Same Way Unlike prior UIs the PV UI is constructed as a set of document pages just like a user document or a web site is constructed as set of document pages This invention puts all of the user s content pages and user interface pages into a single grouping of information that the user can access This invention then leverages well known browser like naviga tional capabilities to allow a user to navigate back and forth between various pages and to put links to pages in favored lists and the like All of this is leveraged to provide the means by which a user navigates to user interface pages The PV UI can provide context specific dynamically syn thesized views of links to give the user direct access to rel evant UI help pages The user can search for user interface functionality in the same way the user can search other docu ment pages Of course pop up menus and other devices such as permanent links placed on the
37. UI display may be divided into regions in which different content may be displayed Unlike prior art U s each region may be individually navigated without changing focus or window state Each region may include concurrently active links that may be activated without changing focus This aspect of the current invention is referred to as page pinning Page pinning provides access to and interaction with multiple sources of content concurrently while reducing the complexity associated with both the design and use of the interface In prior art user interfaces each window runs a separate application that has state information associated with it For instance the state information for a word processor might be that it is currently in insert mode or replace mode and what the current font is The user often has to keep in mind which application is running in which window and what the state of that application is to avoid unexpected results In the PV UI unlike prior art user interfaces there is no notion of a selected one of multiple windows having the current focus Any display region may include active links The user can touch anywhere on the screen and will get an appropriate action based on what the user touches For instance ifa user has a page pinned as depicted in FIG 9 and both pages being displayed contain links the user can simply touch a link in either page and the link will be activated Superficially page pinning rese
38. ad only memory ROM 140 and random access memory RAM 150 A basic input output system 160 BIOS containing the basic routines that help to transfer information between ele ments within the computer 100 such as during start up is stored in ROM 140 Computer 100 also includes a hard disk drive 170 for reading from and writing to a hard disk not shown a magnetic disk drive 180 for reading from or writing to a removable magnetic disk 190 and an optical disk drive 191 for reading from or writing to a removable optical disk 192 such as a CD ROM or other optical media The hard disk drive 170 magnetic disk drive 180 and optical disk drive 191 are connected to the system bus 130 by a hard disk drive interface 171 a magnetic disk drive interface 193 and an optical disk drive interface 194 respectively The drives and their associated computer readable media provide nonvola tile storage of computer readable instructions data structures program modules and other data for the personal computer 100 It will be appreciated by those skilled in the art that other 20 25 30 35 40 45 50 55 60 65 12 types of computer readable media which can store data that is accessible by a computer such as magnetic cassettes flash memory cards digital video disks Bernoulli cartridges ran dom access memories RAMs read only memories ROMs and the like may also be used in the example operating environment A number of p
39. al Moran et al Martin Miller et al Peterson et al Shwarts et al Rowe et al Carey et al Bookman et al Gramlich Hansen et al Boudreau et al Hou et al Carleton et al Nozue et al Nielsen Rowe et al Bier Baldwin et al Drews et al Huffman et al Wical Carleton et al Bittinger et al Earl Wu et al Feigner et al Strahorn et al Menzel Ruehle et al DeLorme et al Rivette et al Sachs et al Gaston Lin Kamper DeRose et al Wrey et al Slotznick Campbell et al Eckerberg et al Bristor gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt t gt 6 018 344 6 018 742 6 025 841 6 034 689 6 035 330 6 037 934 6 038 598 6 049 812 6 052 514 6 054 990 6 055 538 6 064 384 6 072 490 6 076 917 6 081 829 6 091 930 6 094 197 6 105 044 6 122 649 6 133 925 6 141 007 6 144 375 6 144 991 6 151 622 6 154 771 6 157 381 6 163 778 6 181 344 6 184 886 6 195 679 6 195 694 6 199 082 6 205 419 6 205 455 6 211 871 6 226 655 6 230 171 6 243 071 6 243 091 6 266 772 6 271 840 6 272 484 6 279 005 6 279 014 6 289 126 6 289 362 6 301 590 6 320 169 6 320 577 6 321 244 6 331 866 6 331 867 6 340 980 6 356 287 6
40. buttons and the like The links may look like buttons but the links are simply links Even the desktop metaphor of files and folders is expressed as lists of links on a page Additionally links have properties that govern their appearance and behavior For example a link s properties may dictate that it appear as a blue under scored text string as in prior art browsers or as a 3D button as a graphic icon as a thumbnail image of the content being linked to or even as an embedded frame that is open on the content being linked to As described in more detail below a user can modify how a link manifests by modifying the link s properties The abil ity to control a link s appearance and behavioral properties makes possible rich authoring and customization of both content and the user interface Actions in a UI according to the principles of this invention occur by clicking on commands that exist on pages To the user a command looks like a link and in fact it is a link Specifically a command is a link whose source anchor is the command hotspot whose destination anchor is the script or code that implements the command and whose properties are the command parameters Some commands may run with canned parameters while others may present users with a form for entering the parameters for example the Properties command Because a command is a link it has all the functionality of links For example commands can visualize in multiple
41. cifications the visual extent and appearance of the hotspot and the link behavior options Each page of the link property sheet is bookmarked so a user can jump directly to it if desired Together the set of link property pages encompasses a lot of information including general properties source and des tination anchors hotspot characteristics and link behavior General properties These include the type and file system properties of the link Link type information describes the semantic nature of the link and how it relates the things being linked Typically only authoring and view ing software would ever access the type properties as they pertain to the organization of the material contain ing the links Type indicates whether the link expresses a parent child or peer relationship and whether the destination represents A document component a figure table footnote or other cross reference The next or previous page in the web sense of the current document topic The next or previous document topic in an authored web of documents topics Something not part of the current document topic a comment or hyperlink Source anchor Specifies whether the link is anchored to a character word paragraph image part of an image table row cell or column or an arbitrary range of docu ment positions This parameter is automatically set according to the source selection the user makes when creating the link this propert
42. de command selection The selec tion model of the documents with links UI is discussed in more detail below General Concepts Underlying the Documents with Links UI Architecture No Distinction Between Content Documents and UI Docu ments The basic principle of the documents with links UI is that everything the user sees and interacts with is a document This applies equally to both content and UI These documents could be implemented in HTML XML and the like There is essentially no difference between content and UI documents In fact the same document can mix content and UI Even though some UI elements like context menus and toolbars may be presented in ways that do not look document like they are in fact implemented as documents and can be manipulated as such The uniform treatment of content and UI pages is impor tant for several reasons Users only need to deal with one set of navigation controls and conventions and only one navigation space Users never need to think about whether they are in content space or UI space Users can use the full power ofthe UI to manipulate the UI itself For example users can search annotate custom ize and edit UI pages the same as any content subject to permissions Users can select from multiple views for the page being viewed The model naturally accommodates dynamic content including downloaded content that mixes content and UT on the same page or as part of a network of relat
43. des a unified navigational context for allowing a user to interface with at least one of the plurality of user interface documents and at least one of the plurality of user content docu ments within a single frame by activating the links of the user interface documents and the links of the user con tent documents in response to activation by a user of a first set of one or more links of the plurality of user interface documents having links and the user content documents having links navigating forward from a first document page along a first navigational path of at least one document page the first navigational path including one of the plurality of user interface documents and one of the plurality of user content documents then in response to activation of a previous link at least once by the user navigating back along the first naviga tional path to the first document page then in response to activation by the user of a second set of one or more links of the plurality of user interface documents having links and the user content documents having links navigating forward from the first document page along a second navigational path of at least one document page the second navigational path being dif ferent than the first navigational path then in response to activation of the previous link at least once by the user navigating back along the second navi gational path to the first document page and then in response
44. e document pages are displayed in a single view ing frame This unified approach simplifies the user s inter action with both user content and user interface documents by US 7 496 830 B2 3 reducing the number of concepts a user must learn in order to access interact with and modify both user content docu ments and the user interface A non linear navigation history is maintained such that a user can navigate along a first path back up using a previous link an appropriate number of times navigate along a second path back up along the second path using the previous link an appropriate number of times and re navigate along the first path again using a next link Every document page to which a user navigates is saved in the user s navigation history Users can query their navigation histories and view their navigation history in various ways such as by time by appearance by site document section page and the like User can also view their navigation history as nodes with side tracking branches as a linear list or as a combination of most recently viewed pages and last few task categories According to a preferred embodiment navigation from user content pages through user interface pages that results command being executed is automatically removed from the user s view of the naviga tional history in order to keep the navigational history view uncluttered A flexible selection model is supported allowing users to se
45. e form There is no restriction on the number of such incomplete operations that can be in progress simultaneously Nor is there any limitation on switching away from such incomplete operations This is unlike the prior art where complex operations typically per formed via dialogs must generally be complete or cancelled before the user switches to another activity Unlike prior UI s US 7 496 830 B2 11 the user of a documents with links UI is typically not restricted from interrupting one operation to initiate another or several others nor would such an interruption cause the user to lose work already performed in partially completing the first operation Creating and Saving Information Another architectural topic central to the documents with links Ulis the model for creating and saving information Any time a user creates something new the user is creating a new document and linking it into a context For an object inserted into an existing document such as an embedded annotation the user is linking it into the document that is to contain it with link properties implicitly set to make the object visualize in place OLE style embedding Physically the object is stored as a child within the parent document s container For new standalone objects like a new word processing document the object is instead added to the current naviga tion context as if the user had done a Next to it from wherever the user was when the user
46. e to navigate or search Accordingly in addition to the need for a simpler more uni fied experience for the user of an application there is also a need to facilitate the uncluttered presentation of user inter faces for applications providing very rich functionality Usability data for prior art UIs show that users of multi windowed systems don t always know which actions will produce results in which window Therefore there is a need to reduce the complexity and confusion sometimes caused by multi windowed user interfaces Prior art UIs typically offer limited capabilities for custom izing the UI Accordingly there is a need for a UI architecture that provides greater flexibility to users content developers and third party software developers by providing broader capabilities for easily customizing the UI For example dif 20 25 30 35 40 45 50 55 60 65 2 ferent groups of users may be of different levels of computer skill and have need of different sets of features and the UI can be customized to better suit their needs Users of prior art UIs sometimes become extremely frus trated when their work is lost because their work was not properly saved Accordingly there is a need to provide a save less model so that users do not need to explicitly save their work Users of prior art UIs typically do not have a convenient and seamless way to record notes verbally and to associate notes with particular par
47. ecisions are made the link will typically con tinue to be displayed in the same way Depending upon a link s properties when the link is dis played content of the linked to document or other informa tion about the linked to document may be displayed in a frame in the linked from document For instance two example previous links previous link 700 in FIG 7 and previous link 1102 in FIG 11 depict how examining a link s target allows the PV UI to display information about the destination of a link For instance the destination of link 700 is Earth to the Moon while the destination of link 1102 is the Start Page Link Property Sheet A UI link property sheet like user content and other UI pages is implemented as a document It has several pages The first page is the subset of properties that users would most commonly change presented in a way that less expert users can understand Subsequent pages provide the full set of advanced properties and options Typically only the most expert user a content author would ever modify these For example the first page may contain just the property comments a notes field information about where the link leads to and a chooser that lets a user choose from a few US 7 496 830 B2 19 options on how the hotspot should appear e g emphasized text button icon or thumbnail The subsequent pages give full access to properties includ ing the source and destination anchor spe
48. ed un selected Another method is to choose what selection to operate on by doing a backwards scan in time order through the navigation context until an active selection is found In most cases envisioned by the inventors this latter solution produces the same result as the first one discussed 1 the most recent selection will be found The effect of the algorithm for trimming the navigational context is typically to isolate and remove the branch whose purpose was to find the command that was just executed Trimming the navigational context of navigation to UI pages does not always occur however For example an Apply com mand for property setting could leave the property form active and not trim the navigation context Of course different com mands may choose to operate on the navigation context in different ways Nevertheless a couple standard ways will typically cover most of the cases Flexible Selection Model The documents with links UI does not require users to make a selection before invoking a command Users are free to select first and then click acommand or click the command first and then make a selection If an appropriate selection does not exist when a command is invoked the user is prompted to make a selection at that time The selection mechanism and command verbs are designed to give users a lot of latitude about the order in which they do things when carrying out commands Besides fitting better to users per s
49. ed pages Links as Commands and Link Properties A link can be an association between two different spots in a collection of document pages The spots could be two spots on the same page The spots could be a spot on one page and aspotonanother page Links can have properties that indicate certain things about the link beyond simply the fact that it relates two different places What a link relates need not necessarily always be displayable pages A link can relate a 20 25 30 35 40 45 50 55 60 65 6 spot in a document and an active runable object such that when a user activates that link or touches that spot in the document the associated object is run Links therefore can act as commands Links may be to any kind of command code Scripts are one example Binary code objects are another example As a result pages that have links replace the drop down menus and dialog boxes of prior art UI s A link may manifest in various ways based on the link s properties Links can look like not only clickable spots but also fill in fields and other kinds of well known and later developed user interface elements A document page could havethese other kinds ofactive elements that are really a form of link Accordingly activating links goes beyond merely clicking on colored underlined text and can include filling in fields pressing radio buttons and the like Again pages with links replace prior art dialog boxes having
50. ented presentation is required The user may touch the scroll bars to perform the scrolling actions This looks and works much like conven tional scroll bars Users can Create Links PV UI users are not limited to the links that are authored into content they can create their own The PV UI imple ments a rich linking model in which links can have source and destination anchor ranges as well as their own properties Despite the richness of available features users can create links as easily as they can copy text and all with a fingertip the pen is not required Links Displaying Links How a link is displayed is controlled by its properties Conventions and heuristics may be used to assign values to these properties at the time the link is created A simple example is that when the user taps inside the content of a document a note is created This entails creating a note docu ment and then creating a link leading from the spot that was tapped to the new note document In this case the link prop erties are set to visualize the link as a note icon As another example if the user highlights some text in a user document and then chooses the Make link command followed by the steps to complete that command the link is made to visualize as a hotspot over the originally selected text with the selected text changed for example to a blue underlined font Unless a user changes a link s display properties once these prop erty setting d
51. from any given document By restricting filtering this categorization to a particular time frame the navigation network as it existed at a particular point in time can be shown Of course other kinds of views are also pos sible For example the history can be categorized by higher level groupings like web sites or user tasks with the results presented in alphabetical rather than time order thus letting users return to a previous location according to the kind of information or activity as opposed to the time during which the location was last visited To make certain views such as the network view more efficient to recreate additional prop erty information may be stored on each navigation record Accordingly maintaining the navigation context as a data base of navigation records allows support for a rich variety of queries and views Applies to UI Because the UI is built out of content features used to search text change viewing options on lists and tables anno tate and the like all work for UI as well as for user content As anexample the contents of any view such as a view ofa filing or command hierarchy become searchable sortable and annotatable Non Linear Navigation History A key difference between the navigation mechanism of the PV UI and prior art browsers is that the PV UI unlike prior art browsers maintains a non linear navigation history For example if a user navigates through a set of links then goes back
52. ge of collections of pages mak ing up sections and chapters of collections of sections and chapters making up a document of collections of documents making up web sites and so on The same is true for the layers of command finding and invocation Such knowledge of semantic clustering helps guide decisions about popping con texts and for presenting the user with reasonable choices about points to jump to along the Next Previous chain A problem with treating content and UI as part of the same navigation space is that the user s navigation chain gets clut tered with Ul related pages The documents with links UI solves that through intelligent management of the navigation context and by making that context a true network not just a linear chain Specifically when a user navigates from one place to another a new branch in the navigation chain is started So if a user s context is currently B in the existing chain of docu ment pages A B C and the user navigates to D then the new context is D D might be a UI page the user navigated to from document B When the user clicks a command on page D the command executes and removes D from the navigation con text Thus after finding and executing the command the user s navigation context is restored to document page B To find a needed command the user might have to navigate along a chain from D to several other pages in the Guide Book resulting in a chain of several steps branching off f
53. gertip or pen Users can use a fingertip or the pen also referred to as a stylus interchangeably The pen could have three tips an erasing tip at one end a highlighting tip at the other and by twisting the pen barrel to extend a small shaft a writing tip The fingertip may be recognized as a highlighting tip with a gesture allowing it to erase The basic touch idioms are Tap touch and release after a brief time The action taken depends on where a user tapped It does not depend on what tip the user is using The tap would typically last a certain minimum configurable time so that an acci dental or glancing touch will be ignored Hold touch and hold contact The action taken depends on what the user held usually it opens an object or its context menu The action does not depend on what tip is being used The hold time can preference parameter Swipe A swipe is any continuous motion ofa tip in contact with the display surface Swiping will write or erase ink or highlighting depending on which tip is being used Swipes are also used to make and extend current selec tions Tap swipe tap immediately followed by a swipe starting at the same spot This causes the pen or fingertip to start a current selection region Either the writing or high lighting tip can be used Swiping starting at the current US 7 496 830 B2 17 edit cursor location i e the tap is optional when starting on top of the edit cursor can a
54. implemented on a personal viewer hereinafter the term PV refers to a personal viewer upon which various aspects of preferred embodiments of a documents with links UI according to the principles of this invention are implemented Personal Viewer UI Elements Referring now to FIG 5 an example Start Page is dis played on a PV UI 500 that includes the following elements The display surface 502 This is touch sensitive for both fingertip and pen operations The touch zones at the four corners 504 506 508 and 510 are used to initiate navi gation operations such as turning pages and traversing the user s navigational history chain Labels may be applied to these areas to cue completely naive users how to turn pages the first time they interact with a PV UI The display margins Documents that present on a PV UI typically show as print like pages including margins The margin space is available for jotting notes and dis playing UI generated features like bookmark tabs and UI shortcuts A default margin arrangement could be Top margin 512 bookmark tabs Bookmarks or other objects placed here are associated with the containing document Bottom margin 514 document and command shortcuts and toolbar tabs Document shortcuts or other objects placed here are associated with the display surface Left margin ofleftmost display 516 clippings tabs Clip pings or other objects placed here are associated with the containing page Right
55. iple ways all audio on a given page all audio for a given book audio notes in the order they were recorded and the like Additionally the audio clip recorder supports intelligent VOX for hands off note taking A further feature is the ability to embed audio notes at specific points in the content of a document in a manner similar to creating footnotes These audio notes can be created and played back by a single tap on the screen at the point where the audio note was or is to be inserted The presence of embedded audio notes is signified by a small icon in the content that is laid out in a manner similar to a footnote symbol Guide Book Because the commands in the PV UI are simply links on a document page there is no difference between documenta tion help wizards and the other ways the PV UI helps a user perform an action Asking for help by clicking on a help link summons an appropriate help page composed on the fly from the content of the Guide Book The principles that underlie this composition are contextual that is if the user is currently in the list or books of Mail Messages the help system infers that requests for assistance are about how to perform work with Mail Messages For instance FIG 8 shows two pages of an example Guide Book The page on the right page 9 of 10 includes descrip tions of commands such as the PIN THIS PAGE command 800 The Guide Book describes what the command does For instance for
56. ite map history map topic browsing map net neigh borhood physical vicinity nearby machines and local machine Annotations Folder This is a folder of the notes clip pings bookmarks jottings and highlights a user has entered into the user s various documents A user will typically be able to see each of these forms of anno tations in the context of the original document to which they relate These items are stored as external annotations in the Annotations Folder A user does not usually view this folder but instead views one of the following persistent views derived from the Annota tions Folder Clippings Folder A persistent view of the Annota tions Folder showing only clippings The default 20 25 30 35 40 45 50 55 60 65 16 view categorizes by document with more recently read documents ahead of less recently read docu ments Bookmarks Folder Same as the Clippings Folder but showing only bookmarks The Notes Folder Same as the Clippings Folder but showing only notes Notebook This is a blank document where a user can write notes that are not associated with a particular document The notebook is provided by default because most users would like to have a notebook to write in The mailbox calendar and address book In a preferred embodiment these will synchronize with a user s desktop or network based counterparts Personal Viewer Display Modes A PV Ul caninclude a single
57. lect the object of a command either before or after the command itself is selected This flexible selection model allows UIs built according to the principles of this invention to scale to small display areas UIs built according to the principles of this invention also scale well to applications having voluminous and or complicated user interfaces by facilitating an organized and uncluttered view of the user interface command hierarchy and or user interface help infor mation for such applications Users get at commands by navigating to a page where the desired command is found In a preferred embodiment the documents with links UI according to the principles of this invention is organized to make frequently used commands a single navigation step away or through customizations no steps away A self explanatory document the Guide Book is provided The Guide Book is a readable manual that users can go through in a logical order a page at a time like any conventional user manual Each command mention however is an active command instance that can be invoked in place Other features and advantages ofthe invention will become apparent through the following description the figures and the appended claims BRIEF DESCRIPTION OF THE DRAWINGS FIG 1 is a schematic diagram of a conventional general purpose digital computing environment that can be used to implement various aspects of the invention FIG 2 shows a conventional tablet and stylu
58. lso draw a selection Erase a jiggling swipe When done with the fingertip this is treated as a swipe with the erasing tip The motion is the same as erasing with a real pencil eraser With the pen a user can just swipe with the erasing tip A user can automatically extend the range of a highlight current selection or erasure by holding the pen or fingertip after swiping a part of the range that is swipe part of the range and then hold for auto complete to take over The range auto extends to end of word sentence or line paragraph page section or chapter and document in that order the longer a user holds The UI provides feedback on the extent of the selection In tables the selection auto extends by cells to the end of a row or column and then by rows or columns depending on whether the swipe was across or down Arbi trary rectangular blocks can be highlighted selected or erased by swiping a box shape around the desired area Selecting the page number of a page selects the whole page Typically whatever a user can do with a pen the user can do with a fingertip except write ink While other mapping are of course possible an example set of keyboard mouse idioms for the pen idioms and pen tips is Tap is left click Hold is hold left button Swipe is hold left or right button and move mouse Swiping with left button down selects with right down high lights There is no need for tap swipe Erase is hold both buttons
59. mbles a multi windowing user interface The same may be said for dialog boxes that may be included in a documents with links UI according to the principles of this invention Nevertheless unlike this invention multi windowing UIs contain more than one UI Each window in a multi windowing UI gets its own UI with its own state information such as history information and its own UI features such as menu or tool bars According to the principles of this invention pinned pages like all other con tent the Ul included share a single UI having only one set of state information and having a unified navigational history There are no separate UI controls such as menu bars or tool bars for the separate concurrently displayed pages The uni fied navigational history is discussed in more detail below When the user has pinned a page and is viewing two pages side by side the user essentially has two different view ports into a single navigation context A set of documents exists in a navigation context The navigation context is essentially a record of every place a user has ever navigated to and when the user navigated there The navigation context can be thought of as a map or history of the user s navigation A US 7 496 830 B2 21 visible frame such as a pinned page is just a view port into a single shared history The only state associated with a dis played page pinning frame is an indication of the document page that is currently bei
60. medium of claim 1 wherein the single frame includes at least one margin including at least one of a document or command shortcut a bookmark asso ciated with the single frame a user annotation and a clipping wherein each of the at least one the document or command shortcut the bookmark associated with the single frame the user annotation and the clipping having been customized by the user
61. navigational preview information as explained above Run Causes the destination content to be executed with specified run parameters The target is typically a command or script Additionally for links set to appear as thumbnails or in place renderings other behavior properties could indicate un 0 a 5 25 40 45 55 60 65 20 latency periods for updating the display of the linked to con tent relative to a change in the linked to content by for instance specifying a link as hot display updated often warm display updated less often than hot or cold display not updated Further behavior properties could set the pre fetch refresh and caching parameters for hot and warm links Customizable UI The user may completely customize the PV UI unless authored in content controls prevent editing Because there is no distinction between the interface and the content the kinds of normal editing commands the user might use to alter content can be used to customize the interface Users can Place Links in UI For instance links are not restricted to existing only in the content area of the display They can exist anywhere A user could for example place a link over a bookmark the link would take precedence for tap operations meaning that the bookmark would act as a document associated shortcut to some other document whereas a bookmark is normally a link within the current document Page Pinning The PV
62. ng displayed Page pinning is described further in commonly assigned and currently pending U S patent application Ser No 09 556 865 entitled Methods And Apparatus For Displaying Multiple Contexts In Electronic Documents filed XXX which is incorporated herein by reference Navigating Documents Link Properties The PV UI contains methods for paging within a document previous next and for traversing chronology history through back forward Attaching properties to links enables a rich model for navigation For example holding on a link offers additional information about where that link will lead Link preview information could be displayed in thumbnail form and further holding could result in a graphical map ofthe links attached to that prospective destination Back and for ward buttons by default display thumbnail views of the pages that tapping that button will lead to Viewing History History which could include all the pages the user has viewed may be viewed in a number of ways by time by appearance by site document section page and the like Since a user s navigation history is saved the users sequence may be viewed as nodes with side tracking branches as a linear list or as a combination of most recently viewed pages and last few task categories For example a query could perform a multilevel categorization by document id resulting in a hierarchy that represents all the forward navigation paths a user has taken
63. nnotation from paper books to the digital library ACM International Conf on Digital Libraries 1997 pp 131 140 D Munyan Everybook Inc Developing the E Book in Relation to Publishing Standards Sep 1998 http www futureprint kent edu articles munyanO I htm pp 1 7 PhotoImpact 3 0 Ulead Systems Inc 1996 Landay et al Making Sharing Pervasive Ubiquitous Computing for Shared Note Taking IBM Systems Journal vol 38 No 4 1999 pp 531 550 Landay J A Using Note Taking Appliances for Student to Student Collaboration 29th Annual Frontiers in Education Conference IEEE Computer Society Nov 1999 p Session 12C4 15 12C4 20 Adobe Acrobat 3 0 Review downloaded from http www hmung org Reviews reviewed Mar 1997 pp 1 18 cited by examiner US 7 496 830 B2 Sheet 1 of 11 Feb 24 2009 U S Patent L DI 861 16 961 S61 n Nouvonddv 5 vava Rudo W31SAS ASNOW NOI LVOITddV ONILVH3dO A ti31n dWOO 310N3H viva INVHOOHd 3OV3U34NI 1 vigas 3O0VJ3H3 INI WOlLdO 3O9V3U3ANI 3OVAYSLNI OLL3NSVN X3HOMAL3N V3HV 1V907 261 WYH OHd H3ldvav LINN O3GIA SNISS32O0Ud W31SAS AYOWAW WJLSAS HOLINOIN US
64. novation Laboratories Japan IEEE 1999 pp 356 359 Continued Primary Examiner Stephen Hong Assistant Examiner Matthew J Ludwig 74 Attorney Agent or Firm Birch Stewart Kolasch amp Birch LLP 57 ABSTRACT A user interface architecture wherein user content and user interface are composed of documents with links is described User content documents and user interface documents are both displayed in a single viewing frame A non linear navi gation history is maintained such that a user can navigate along a first path back up using a previous link navigate along a second path back up along the second path using the previous link and re navigate along the first path again using a next link Every document page to which a user navigates is saved in the user s navigation history Users can query their navigation histories and view their navigation histories in various ways Users can access annotate and customize the user interface in the same manner in which users access annotate and modify user content documents 18 Claims 11 Drawing Sheets US 7 496 830 B2 Page 2 5 239 466 RE34 476 5 347 295 5 390 138 5 434 929 5 434 965 5 471 568 5 559 942 5 572 643 5 623 679 5 625 833 5 630 125 5 632 022 5 644 674 5 666 113 5 680 636 5 682 439 5 687 331 5 689 717 5 717 860 5 717 879 5 719 595 5 727 129 5 729 687 5 737 599 5 745 116 5 760 773 5 761 485 5 7
65. nt the user needs to choose from and captures anything the user selects After selecting the addresses the user wants the user can simply navigate back to the send form where the TO field is Alternatively the user could close the address book or use an OK command either of which would return the user automatically There is noth ing to save because all choice state is captured as the user proceeds If there is a change in plan the user could simply cancel the current selection or clear the TO field when the user returns If the user later wants to change the TO field choices the user may click the TO field link again and the user is taken back to the address book with all the current choices still highlighted The current choices are highlighted because the command that takes the user to the address book picks up the TO state and paints the required selection regions To facilitate making selections in scenarios like this address book example an additional facility is provided The command which presents the document to choose from can also cause checkboxes to appear next to each entry in the document Rather than manually selecting items in the docu ment the user can check or uncheck the checkboxes This causes the associated entry to be selected or unselected respectively In this address book example the user is accessing a stan dard document the address book and the normal multiple selection idiom to make and change the
66. o make it navigate If holding the link opened a navigation map the map will persist for a short time after the user lets go This gives the user a chance to tap a spot on the map to go there or to hold a spot in order to preview and then to expand the map context around that point Next and Previous Commands Another way to navigate is via browser like Next and Pre vious commands performed in the default assignment by tapping the PV s lower right and left corners 508 and 510 20 25 30 35 40 45 50 55 60 65 18 respectively Specifically these functions navigate a user along the chain of visitations caused by link operations Page turning operations are not part of the next previous chain For example if a user opens a book reads the first ten pages and then does previous the user will return to wher ever the user was before the book was opened The user will not go back to the previous page of the current document for that the user can use the previous page operation 504 The distinct page level and navigation level functions exist because the page forward back functions 504 and 506 take the place of the traditional scrollbar for scanning within a single document Scrolling is Possible Even though the PV UI is page oriented there are times when a scroll oriented presentation is the only reasonable display choice The PV UI supports this by displaying tradi tional scroll bars when stream ori
67. ocument Dis tribution and Reading Systems http www softbook com con sumer reader asp Rocket eBook Using the Rocket eBook http www rocketbook com Products Faq using html Explore the Features of the Rocket eBook wysiwyg 212http www rocketbook com Products Tour index html Adobe Acrobat 3 0 Reader Online Guide Adobe Systems Inc pp 1 110 Kunikazu T Patent Abstracts of Japan Publication No 11327789 for Color Display and Electronic Blackboard System published Nov 30 1999 Open eBook Forum Open eBook Publication Structure 1 0 http www ebxwg org oebps oebps1 0 download oeb1 oebps htm Kristensen A Formsheets and XML Forms Language Mar 1999 Dublin Core Resource Types Structurality Draft Jul 24 1997 Feb 27 1998 Dublin Core Metadata Element Set Version 1 1 Reference Descrip tion 1999 Dublin Core Metadata Initiative User Guide Working Draft Jul 13 1998 1999 Greenberg Saul A Fisheye Text Editor for Relaxed WYSIWIS Groupware Chi 1996 Proceedings pp 1 to 5 Marshall C Annotation from paper books to the digital library ACM International Conf on Digital Libraries 1997 pp 131 140 Joke eBook Jul 1998 PrimaSoft PC Inc Electronic Book Series Version 1 0 pp 1 5 Munyan D Everybook Inc Developing the E Book in Relation to Publishing Standards Sep 1998 http www futureprint kent edu articles munyan01 htm pp 1 7 C Marshall A
68. of the features subsystems and functions in the system of FIG 1 can be included in the computer of FIG 2 Computer 201 includes a large display surface 202 e g a flat panel display on which a plurality of windows 203 is dis played Using stylus 204 a user can select highlight and write on the display area Computer 201 interprets marks made using stylus 204 in order to manipulate data enter text and execute conventional computer application tasks such as spreadsheets word processing programs and the like One US 7 496 830 B2 13 commercially available tablet and stylus computer incorpo rating many of these features is the Stylistic 2300 computer sold by Fujitsu Personal Systems Inc of Santa Clara Calif Astylus could be equipped with buttons or other features to augment its selection capabilities In one embodiment a sty lus could be implemented as a pencil or pen in which one end constitutes a writing portion and the other end constitutes an eraser end which when moved across the display indi cates that portions of the display are to be erased Other types of input devices such as a mouse trackball or the like could be used Additionally a user s own finger could be used to select or indicate portions ofthe displayed image on a touch sensitive or proximity sensitive display Consequently the term user input device is intended to have a broad definition and encompasses many variations on well
69. omizable as described in more detail below Shortcuts Numerous UI shortcuts and direct manipulations may exist as aconfigurable layer on top ofthe documents with links UI so more experienced users can do the most common opera tions in context without navigating to UI pages The user gets the best of terse command access plus the richness of the full browser and answer system for exploring the command set The documents with links UI uses a web architecture with UI shortcuts layered on top A naive user will typically start by using the documents with links UI without the short cuts that is by using the Guide Book to access UI functions An advanced user will typically be able to perform all com mon operations via the shortcuts and without resorting to the documents with links UI Guide Book as often Scalability to Various Display Sizes and Types Various preferred embodiments of the documents with links UI will be explained below in the context of a portable personal viewer platform Nevertheless the documents with links UI is scalable across a wide range of device and display types from desktop computers to laptops to hand held devices Accordingly the documents with links UI is intended to be implemented on any type of computing plat US 7 496 830 B2 5 form The documents with links UI exploits a large screen by being able to show content in a book like way two full side by side pages as depicted for instance in F
70. onal habits this makes it harder for users to do something wrong such as inadvertently applying formatting to text other than the text the user wants to reformat Page Display Depends Upon the Link Used to Navigate to the Page The documents with links UI may display the same page of a document differently depending upon which link navi gated a user to the page Suppose the user wants to fill in the TO field of an e mail message In this situation the user wants to open the address book and make possibly several choices To accomplish this within the document navigation metaphor forms could have special purpose chooser controls where this provides a good shortcut for the most common choices But the chooser UI should also make it possible to leverage the full power of the navigation browsing viewing and search UI that is available in list oriented documents like file folders and the address book Making a choice from such 40 45 55 10 a list should be a matter of just navigating to that list as a user would in any other context and making the choice For the address book example the documents with links UI provides a link to the address book that is associated with the input field The documents with links UI makes the link from an input field to its choice document the address book in this example be a command with navigation behavior as opposed to being an ordinary link This command navigates the user to the docume
71. qe pue eae paesu pegs Buiuuni dn JN 109 SSA pue Givy uo dn wea 106 quA eunjdeN pue pega yum Bunaaw pejnpeuas yaloid uo penuiuo pe o1dde uBisep sno 106 pue jue ui pega arog 5586014 99A US 7 496 830 B2 1 COMPUTER USER INTERFACE ARCHITECTURE THAT SAVES A USER S NON LINEAR NAVIGATION HISTORY AND INTELLIGENTLY MAINTAINS THAT HISTORY This application is a continuation of and claims priority from allowed application Ser No 09 456 975 filed Dec 7 1999 FIELD OF THE INVENTION This invention relates generally to computer user interface architectures More particularly the invention provides a user interface architecture in which both user content and user interface are composed of document pages with links BACKGROUND OF THE INVENTION Many personal computer users find the desktop metaphor of prior art computer user interfaces UIs confusing and difficult to learn Accordingly there is aneed fora system that simplifies the user s interaction with the computer by using fewer kinds of user interface controls in a more general way Further the ways in which users interact with information about prior UIs is different than the way the user interacts with content such as documents presentations and the like For example in prio
72. r art UIs content and UI information are displayed entirely differently Content is typically displayed in a particular region or frame of the display User interface information is never displayed there Instead user interface information is displayed in dialog boxes drop down menus and tool bars User content never shows up in dialog boxes drop down menus and tool bars Similarly users find user content documents and UI help information differently Accordingly there is a need for a UI architecture in which concepts and actions the user must learn are the same for interacting with both content and the UI Such a unification makes computer software easier and more efficient to use Prior art UIs for desktop computers typically require a keyboard and mouse in order for a user to interact with them and most pen enabled palmtop computers have cumbersome means of interaction Therefore there is a need for more natural styles of interacting with a computer by using a minimum number common gestures such as touch hold erase draw or write Prior art UI desktop metaphors applied to small form factor devices are typically cluttered and difficult to use In addition applications that provide rich functionality are sometimes constrained by the limited ability of a user to navigate menus and dialogs of prior art UIs For instance for such applica tions the menus and tool bars may get too big and the help system may get too cumbersom
73. rogram modules can be stored on the hard disk magnetic disk 190 optical disk 192 ROM 140 or RAM 150 including an operating system 195 one or more appli cation programs 196 other program modules 197 and pro gram data 198 A user can enter commands and information into the computer 100 through input devices such as a key board 101 and pointing device 102 Other input devices not shown may include a microphone joystick game pad sat ellite dish scanner or the like These and other input devices are often connected to the processing unit 110 through a serial port interface 106 that is coupled to the system bus but may be connected by other interfaces such as a parallel port game port or a universal serial bus USB A monitor 107 or other type of display device is also connected to the system bus 130 viaan interface such as a video adapter 108 In addition to the monitor personal computers typically include other periph eral output devices not shown such as speakers and print ers The computer 100 can operate in a networked environment using logical connections to one or more remote computers such as a remote computer 109 Remote computer 109 can be a server a router a network PC a peer device or other com mon network node and typically includes many or all of the elements described above relative to computer 100 although only a memory storage device 111 has been illustrated in FIG 1 The logical connections depicted in
74. rom B When the user finally picks a command the documents with links UI knows what to act on and what to remove from the navigation context as follows Commands operate on the current selection and in general remove the navigation nodes that lie on the branch leading from the current selection to the command Further additional heuristics may be used for unusual cases Current selections in the documents with links UI are similar to current selections in prior art desktop UIs but there are differences because the documents with links UI deals with a network of active documents that are different than a desktop of active windows The two schemes are similar in that every document can have a single possibly disjoint selected area The schemes differ in that the documents with links UI can tuse the idea ofthe current focus to decide what selection a command should operate on In prior art desktop Uls the document the user wants to operate on typically has the current focus and all UI elements implicitly reference this focus In the documents with links UI world the user may have navigated several hops away from the document the user wants to operate on as the user looked for the desired com mand so in the documents with links UI focus doesn t dis ambiguate anything Therefore in accordance with a preferred embodiment instead of the current selection being the one whose docu ment has the focus the current selection is the most
75. rqeg Woop 0 ay 9E PjogSr US 7 496 830 B2 Sheet 11 of 11 Feb 24 2009 U S Patent STAT EE A TIUT aT be e Te ag ely ap E 3 26 2 1e ea 734 54 snyes 54 573835 132229 uuoc 1 839 vS53H 1 1 SDVSSEN 4 gt LX N 29pao0 ui 9 10 4 IP MBIA W R5VSSSM BDOIASWd 9 AXIN siu IIRO Mous SDVSSEN SNOIARYd Wa5Y6GSRM LXUN 2103 5103 jo uorssnosip eu WISALVW Q31V 134 43H10 8uou 4 auou tu qoid 9 1 1 18 1848 ued 6u106 uo sioafoid sjaeup 10 Indui pue 2 11 ssed 181 s6nq seque pue ojoud pega 20 jesodojd yesp 45 maias epyneg 20 ZW 10 ssaiGoid g 11 pee s inle j ZIN puana 104 adeh Sseguopd ay Aq paonpoid ised pue juano uo pajeuq 105 Buyeuq jeba Bureau awoy Buiuuojsureiq seg 0 u ndu ABalens aiseq e Butu2ie joafoud wea jno
76. s based com puter that can be used to implement various aspects of the invention FIG 3 is a flowchart showing simplified steps at a high level of abstraction for implementing a UI architecture according to the principles of this invention upon start up FIG 4 is a flow chart showing simplified steps for imple menting the display state step of the flow chart in FIG 3 FIG 5 depicts an example Start Page displayed on a per sonal viewer according to a preferred embodiment of the invention FIG 6 depicts an example book cover page and table of contents displayed on a personal viewer according to a pre ferred embodiment of the invention FIG 7 depicts example Guide Book pages displayed on a personal viewer according to a preferred embodiment of the invention 0 kak 5 20 35 40 45 50 60 4 FIG 8 depicts example Quick Help pages one of which is bookmarked displayed on a personal viewer according to a preferred embodiment of the invention FIG 9 depicts an example of a pinned user content page displayed on a personal viewer according to a preferred embodiment of the invention FIG 10 depicts an example of inked annotation displayed onapersonal viewer according to a preferred embodiment of the invention FIG 11 depicts an example End Page displayed on a per sonal viewer according to a preferred embodiment of the invention DETAILED DESCRIPTION OF THE INVENTION Introduction Clutter Free
77. s et al Berstis Suzuki Finseth et al Martin et al Zellweger Schilit et al Ishisaka Van Der Meer Siow et al Clothier Alexander Liu et al Eisenberg Eberhard et al Ho Ruberry et al Graham et al Knowles Cohen et al Rivette et al Wone Stasnick et al Levine et al Hurtado et al Walden et al Swaminathan et al Kaasila Berquist et al Lection et al Saxton et al Koppolu et al Gupta et al Robertson et al Keenan et al Arons et al US 7 496 830 B2 Page 3 6 535 294 Bl 3 2003 Arledge Jr et al 6 539 370 Bl 3 2003 Chang et al 6 549 220 Bl 4 2003 Hsu et al 6 549 878 Bl 4 2003 Lowry et al 6 560 621 B2 5 2003 Barile 6 571 211 Bl 5 2003 Dwyer et al 6 571 295 Bl 5 2003 Sidana 6 573 907 Bl 6 2003 Madrane 6 585 776 Bl 7 2003 Bates et al 6 631 496 B1 10 2003 Lietal 715 501 1 6 636 238 10 2003 Amir et al 6 647 534 11 2003 Graham 6 662 310 B2 12 2003 Lopez et al 6 687 878 Bl 2 2004 Eintracht et al 6 697 997 2 2004 Fujimura 6 710 790 3 2004 Fagioli 6 714 214 Bl 3 2004 DeMello 6 760 884 Bl 7 2004 Vertelney et al 6 766 494 B1 7 2004 Priceetal 715 203 6 772 139 B1 8 2004 Smith 6 788 316 Bl 9 2004 Ma et al 6 871 318 Bl 3 2005 Wynblatt et al 6 904 450 B1 6 2005 Kingetal 709 203 2001 0031128 Al 10 2001 Manbeck 2002 0013834 Al 1 2002 Esakov etal 709 223 2002 0097261 Al 2002 0099552
78. ser sees no surrounding frame and a tap action in the margins has the defaults specified in the list above In the edge zones embodiment the user sees a sur rounding frame that is wide enough to provide an edge zone US 7 496 830 B2 15 and taps in the corresponding edge zones have the effects specified in the list above In this latter embodiment the default action for taps in each of the document s margins is to insert an embedded annotation i e a user note In the mar gins only embodiment the user would insert annotations into a margin not by tapping there but by explicitly invoking a command such as an Insert Note command available on the margin s popup menu The display content This is the area inside the page mar gins where content is displayed Content may include links that the user can navigate by touching them as well as user created highlighting jottings and embedded notes and recordings The note form This is a document template that is used to implement several important PV UI features including bookmarks notes and clippings The note form is just a blank document that has predefined fields like an entry field for a title and option boxes to control the note s presentation and behavior e g its color type source back link and anchor spec The note form can also have a Send button The built in documents The PV has a few built in docu ments as listed below There is nothing special about
79. ses the display to navigate to that other document A link can also relate a spot or region in a document and an active runable object such that when a user activates that link or touches that spot in the document the associated object is run Parameters for the execution of the object may be supplied by properties associated with the link Links therefore can act as com mands Links may be to any kind of command code A link may manifest in various ways based on the link s properties Links can look like not only clickable spots but also fill in fields and other kinds of well known and later developed user interface elements Alternatively links can manifest in the containing document a frame displaying the contents of part or all of the linked to document When links are displayed in addition to basing the display format of the link on the link s display properties the link display format may depend upon the characteristics of the linked to docu ment The path a user takes to reach a document typically affects the behavior and presentation of the document State like information for displaying a linked to document page is stored separately from the linked from and linked to docu ments as part of the link s properties Users access interact with and navigate among both user content documents and user interface documents in a unified way namely by acti vating links Further both user content document pages and user interfac
80. the short cuts would appear to a user because eventually shortcuts are what most users would use in their day to day activities with the documents with links UI Rich Navigation and Maintaining the User s Navigational History When a user performs a navigation action a record is created of where the user navigated to and at what date and time this occurred It is thus possible to query this set of records to derive many views ofa user s navigational history including for instance a network view also referred to as a history map view Ways in which a user s navigational history may be viewed are discussed in more detail below in the Viewing History section The history map view is a generalization of the linear history provided by prior art browsers The history map view makes it easy for a user to revisit a place the user visited previously with important cues about the context in terms of other places the user was visiting at the time The Next com mand works with the branching history too A user can explore a chain of links back up explore a different chain back up and Next the user s way down the original chain to get back to where the user was This is much easier than having to manually re follow the original chain of links which would be impossible if the user has forgotten the chain of links they had previously followed This is an important extension of the prior art browsing metaphor Unlike prior art browser UIs with
81. tion below US 7 496 830 B2 23 Interface Shortcuts and Smarts Most Likely to Use Links and Other Heuristics The PV Ul can include heuristics which allow it to offer up most likely to use links to additional material One case of these is the context menu that appears when the user holds the Next button in addition to the most recent documents the user has visited subsequent to the current one the Next menu includes heuristically determined choices of other places the user may wish to visit for example documents on a topic related to the current one Another case is a feature called an End Page such as for instance End Page 1100 depicted in FIG 11 Such an End Page is essentially a summary page at the end of a document or book that offers the user related topics such as additional works by this author on this topic written in the same time period commented on by these crit ics etc In the case of directories such as email where each message is technically a document the end page offers threads based on that message other messages by same author other text on the same topic and the like Ofcourse other heuristics could also be used depending on the context For example when offering up a set of command choices to a user the heuristics could be based on analysis of the user s current document context recently used com mands and commands the user has used most frequently in this and similar contexts in the past Such
82. to activation of a next link by the user at least once navigating forward along the first naviga tional path 2 The computer readable medium of claim 1 wherein the second navigational path includes one of the plurality of user interface documents and one of the plurality of user content documents 3 The computer readable medium of claim 1 wherein at least one page of the first and second navigational paths is the same page 20 25 30 35 40 45 50 55 26 4 The computer readable medium of claim 3 wherein the same page is one of the plurality of user interface documents 5 The computer readable medium of claim 3 wherein the same page is one of the plurality of user content documents 6 The computer readable medium of claim 1 further com prising a step of storing a navigational history of the first navigational path and the second navigational path 7 The computer readable medium of claim 6 wherein the stored navigational history includes information about when the user navigated to each document page in the first naviga tional path 8 The computer readable medium of claim 6 wherein the stored navigational history includes information about when the user navigated to each document page in the second navigational path 9 The computer readable medium of claim 6 further com prising a step of generating a history view map for displaying the stored navigational history 10 The computer readable medium
83. ts of a document Accordingly there is a need to provide rich support for audio note taking with the ability to correlate and synchronize audio and textual material and to review and retrieve audio notes Prior art device to device and device to PC synchroniza tion schemes typically are not seamless and require a great deal of configuration and attention from the user Accord ingly there is a need to provide automatic and transparent synchronization between a user s computers such as a hand held computer and a desktop computer In prior art UIs methods for getting help are currently separate from the content and often require completely dif ferent interactions than interacting with content Accordingly there is a need to make the process of getting help about a function the same as the process for carrying out the function Prior art UIs typically have a single user model at the heart of their interface metaphors which makes sharing con tent and annotations with other users difficult and non intui tive Accordingly there is a need to make sharing and col laborating on documents easier and more automatic SUMMARY OF THE INVENTION According to various preferred embodiments the inven tion includes a user interface architecture in which user con tent and user interface are composed of documents with links A link can relate a spot or region in a document with a spot or region in another document so that touching the link cau
84. y provides a way to change the anchor Hotspot Specifies the physical extent and appearance of the hotspot By default the extent matches that of the anchor but it can be made bigger or smaller and of arbitrary shape noncontiguous hotspots are also pos sible For hotspots on text the default appearance is blue underlined text but the foreground and background col ors and text attributes can be changed Other appearance options include manifesting the link as an icon button thumbnail of the link target or as an in place active rendering of the link target Hotspots can also be invis ible which is appropriate for links over GIF images for instance Another hotspot option specifies how the link is previewed any or all of the name of the target a thumbnail of it and or specific balloon text Destination anchor Specifies the target of the link and its range The basic link creation UI results in destination anchors that are a single document position a point ratherthana range Destination ranges that are not points are a very advanced feature mainly used in implement ing certain viewing and collaboration features For example ifthe destination anchor 15 a range the viewing software can automatically synthesize appropriate link preview information based on the content of the range Behavior Specifies the action to take on tap and hold operations Choices include Navigate Goes to the link destination Preview Pops up

Computer user interface architecture that saves a user`s non

Contents

Download Pdf Manuals

Related Search

Related Contents