(12) United States Patent (10) Patent N0.: US 7,864,996 B2
Contents
1. 12 United States Patent Hemmer et al US007864996B2 10 Patent No US 7 864 996 B2 45 Date of Patent Jan 4 2011 54 75 73 E Q1 Q2 65 51 52 58 56 SYSTEM FOR MACROSCOPIC AND CONFOCAL IMAGING OF TISSUE Inventors Paul Hemmer Rochester NY US Zachary M Eastman Pittsford NY US Jay M Eastman Pittsford NY US William J Fox Rochester NY US James M Zavislan Pittsford NY US Assignee Lucid Inc Rochester NY US Notice Subject to any disclaimer the term of this patent is extended or adjusted under 35 U S C 154 b by 1263 days Appl No 11 357 234 Filed Feb 17 2006 Prior Publication Data US 2007 0206275 A1 Sep 6 2007 Int Cl G06K 9 00 2006 01 p e scart nha hin ta deacon 382 128 Field of Classification Search 382 133 382 134 171 128 240 345 611 600 407 See application file for complete search history References Cited U S PATENT DOCUMENTS 5 034 613 A 7 1991 Denk et al RE34 214 E 4 1993 Carlsson et al 5 297 034 A 3 1994 Weinstein 5 381 224 A 1 1995 Dixon et al 5 473 706 A 12 1995 Bacus et al 5 532 873 A 7 1996 Dixon 5 532 874 A 7 1996 Stein 5 548 661 A 8 1996 Price et al 5 602 674 A 2 1997 Weissman et al A 5 655 029 8 1997 Rutenberg et al Continued FOREIGN PATENT DOCUMENTS WO WO 00 49392 8 2000 Continued OTHER PUBLICATIONS Rajadhyaksha M et al Confocal Laser Microscope Images Tissue I2. positions with respect to the actual tissue surface as well as to the earlier captured picture ofthe tissue surface of the macroscopic image 57 in window 56 The Z motor controls the depth of the confocal image in tissue which is not controllable from the macroscopic imaging user interface Selection of VivaScope 1500 tab 54b provides live con focal images on the display 16 from confocal microscope 12 During operation of the VivaScope 1500 confocal image s may be selected for storage in memory of the computer sys tem in one or more formats described latter The position of selected image s stored is recorded in a two dimensional map in memory of the computer system which is the same size as region 58 Such map 64 is shown as an overlay on region 58 in which graphical icons 66 are used to represent the position of stored confocal images as shown in FIG 7 Each ofthe graphical icons 66 represents a different type of image captured For example each of icons 66a represents the posi tion with respect to the tissue surface of a video of multiple frames of confocal images Icon 665 represents the capture of still confocal image Icon 66c represents the position of a VivaStack which are multiple confocal images at succes sive depths in the tissue ata common X Y frame location The darken path 67 shows the track of the confocal imager 18 taken in which the box 62 shows the current imaging position of the confocal imager Other types o
3. should be taken as illustrative and not in a limiting sense The invention claimed is 1 A system for imaging tissue comprising a macroscopic imager for capturing a macroscopic image a microscopic imager for capturing one or more optically formed sectional microscopic images on or within tis sue a tissue attachment device in which said macroscopic imager and microscopic imager are each individually presentable to said tissue wherein said tissue is not histologically prepared tissue that is mounted on a slide utilizing said tissue attachment device in a predefined alignment with said tissue attachment device wherein said tissue attachment device is adapted to attach to tissuethat extends beyond said tissue attachment device and a computer system coupled to said macroscopic imager and microscopic imager capable of receiving at least one macroscopic image from said macroscopic imager and one or more microscopic images from said microscopic imager 2 The system according to claim 1 wherein said computer system further comprises a display and memory for storing said macroscopic and microscopic images and said system further comprises a user interface operable on said computer system enabling display of said macroscopic image of said area of tissue when captured by said macroscopic imager on said display outlining of a region within the macroscopic image associated with a field of view of the tissue imagable by said microscopic i
4. 1 8 2003 Zeineh 5 719 700 A 2 1998 Corcuff et al 6 608 628 B1 8 2003 Ross et al 5 737 121 A 4 1998 Dixon 6 684 092 B2 1 2004 Zavislan et al 5 788 639 A 8 1998 Zavislan et al 6 707 461 B1 3 2004 Tomacket al 345 611 5 793 969 A 8 1998 Kamentsky et al 6 728 567 B2 4 2004 Rather etal oc 600 407 5 796 861 A 8 1998 Vogt et al 6 745 067 B1 6 2004 Zavislan et al 5 836 872 A 11 1998 Kenet et al D492 997 S 7 2004 DiStasio 5 836 877 A 11 1998 Zavislan et al D523 883 S 6 2006 Distasio et al 5 838 815 A 11 1998 Gur etal 7 155 049 B2 12 2006 Wetzel etal 382 133 5 848 177 A 12 1998 Bauer et al 5 851 181 A 12 1998 Talmor FOREIGN PATENT DOCUMENTS 5 880 880 A 3 1999 Anderson et al 5 891 619 A 4 1999 Zakim et al WO WO 01 35325 Al 5 2001 5 995 867 A 11 1999 Zavislan wo WO 02 073246 A2 9 2002 6 014 451 A 1 2000 Berry et al WO WO 2004 104645 A2 12 2004 6 031 929 A 2 2000 Maitz et al 6 031 930 A 2 2000 Bacus et al OTHER PUBLICATIONS 6 049 622 A 4 2000 Robb et al Rajadhyaksha M et al In Vivo Confocal Scanning Laser Micros 6 072 624 A 6 2000 Dixon et al copy of Human Skin Melanin Provides Strong Contrast The Journal 6 078 681 A 6 2000 Silver of Investigative Dermatology Jun 1995 vol 104 No 6 pp 946 6 101 265 A 8 2000 Bacus et al no s 382 433 955 6 151 127 A 11 2000 Kempe Schmitt Joseph M et al Optical Characterization of Dense Tissues 6 208 374 Bl 3 2001 Clinch Using Low coheren
5. 2a such as centered cross hair or an arrow on such tab to assist the user in alignment Optionally automatic rather than manual alignment may be provided where the ring 30 has features which key into features of ring mount 48 The macroscopic image provides a 10x10 mm macro scopic image to computer system 14 which may be for example 1000x1000 pixels This is in contrast with the 4x4 mm imagable area ofthe confocal imager Power to the board 36 in imager 32 may be supplied via the USB cable 31 or by a battery in housing 43 The housing members 34a and 345 mount 40 tube 38 barrel 44 and tissue ring mount 48 may be of molded plastic material and preferably tissue ring mount 48 is of clear molded plastic The macroscopic imager of the present invention also may represent a conventional digital camera adapted to have mount 48 which can be interfaced to computer system 12 so as to received digital images from the camera The macroscopic imager 32 may optionally be used to assist the user in the selection of the tissue to be confocally imaged while the tissue ring 30 and window 52 attached thereto is attached to imager 32 but before the tissue ring and window assembly is adhesively attached to tissue Thus the macroscopic imager 32 sends to macroscopic image to com puter system 14 for display as live macroscopic images on the user interface 33 while the user moves the imager 32 along and slightly above the tissue surface to scan for an
6. 31 to computer system 14 Internal electronic compo nents for enabling operation of the imager 32 may be manu factured by Lumenera Corp of Canada under model no Lu100 Communication between the macroscopic imager 32 and computer system 14 for transmitting signals and data between them may be specified by camera electronics manu facturer However conventional electronics of a digital cam era may also be used 20 25 40 45 50 55 60 65 6 A nose tube 38 is attached to housing 34 over an opening 37 in front housing member 345 Within the nose tube 38 is an assembly of components including a mounting 40 and an IR filter 39 also available from Lumenera Corp received in the backside of mounting 40 Mounting 40 has a threaded hole 42 for receiving a threaded optics barrel 44 Optics 44 may represent one or more lenses for focusing an image onto CCD array 38 For example two lens doublets may be used An LED board 46 has a ring of spaced LEDs 45 around an opening 46a LEDs 45 provide white light illumination and receive power by a cable from circuit boards 36 Optionally either in addition to or instead of the ring of white light LEDs one or more rings of different color LEDs may be provided on board 45 to allow selection of illumina tion of tissue with different wavelengths or wavelength ranges in which macroscopic imager s CCD array 38 is sensitive to light is such wavelength or wavelength ranges When differen
7. 996 B2 74 Ela 83 74 tle eb go q4 Ela aln pads US 7 864 996 B2 1 SYSTEM FOR MACROSCOPIC AND CONFOCAL IMAGING OF TISSUE FIELD OF THE INVENTION The present invention relates to a system and method for macroscopic and confocal imaging of tissue and relates par ticularly to a system for macroscopic and confocal imaging of tissue having a macroscopic imager and a confocal imager each individually presentable to tissue via a tissue attach ment device in a predefined alignment which spatially cor relates images from the confocal imager within an image from the macroscopic image Also the present invention relates to a user interface operable on a computer coupled to such macroscopic and confocal imagers to enable display of the macroscopic image and confocal images from the mac roscopic and confocal imagers respectively and graphical tracking and targeting of imaging locations of the confocal imager in the macroscopic image and marking on the mac roscopic image of the locations of selected confocal images with respect to the tissue surface BACKGROUND OF THE INVENTION Confocal microscopes optically section tissue to produce sectional microscopic images of tissue referred to herein as confocal images An example ofa confocal microscope is the VivaScope manufactured by Lucid Inc of Rochester N Y Other examples of confocal microscopes are described in U S Pat Nos 5 788 639 5 880 880 and 5 995 867 and in artic
8. A2 Such confocal microscope may be a VivaScope 1500 manu factured by Lucid Inc of Rochester N Y Confocal micro scope 12 has a computer system 14 such as a personal com puter PC coupled to a display 16 The computer system 14 receives confocal images representing optically formed microscopic sectional images such as cells or other tissue structures from a confocal imager imaging head 18 that is gimble mounted on a multi axis arm mechanism 20 having front and rear arms 21 and 22 The confocal imager 18 has a nose tube 24 preferably made of clear plastic which is attached to a conical hub 26 on the front of the confocal imager 18 Handles 28 are manually grasped and moved to permit multi axis movement of the confocal imager Cables extend along arms 21 and 22 to supply power and enable communication between confocal imager 18 and computer system 14 There are three stepper motor drivers provided in confocal imager 18 which drive an X direction stage drive motor a Y direction stage drive motor and a motor which moves the objective lens of the confocal optics in the confocal imager in the Z direction respectively where X Y Z are orthogonal dimensions In imaging tissue X and Y dimen sional are substantially parallel to the surface of tissue being imaged on or through and Z is substantially perpendicular to such surface to control depth of confocal imaging The motors are driven either by the user changing the imaging position v
9. a said device tracking on said display the two dimensional position of said microscopic imager with respect to the surface of said tissue in said macroscopic image and marking on said display of said macroscopic image the location of one or more of said microscopic images captured by said microscopic imager wherein said dis playing outlining tracking and marking steps are car ried out with the aid of a computer system coupled to said display 17 The method according to claim 16 further comprising the step of targeting in said macroscopic image of said tissue on said display the two dimensional position of microscopic images captured by the microscopic imager with respect to said tissue wherein said targeting step is carried out with the aid of said computer system 18 The method according to claim 16 further comprising the steps of US 7 864 996 B2 15 morphing on said display two macroscopic images of said same area captured at different times into a morphed image and selecting the contribution to said morphed image of each of said two macroscopic images to enable viewing of changes in said same area of tissue over the time period associated with said two of said different ones of said macroscopic images wherein said morphing and select ing steps are carried out with the aid of said computer system 19 A user interface operable on a computer system coupled to a macroscopic imager and a microscopic imager said
10. area of interest such as a lesion in the tissue When such area is viewed it is centered in the field of view of the macroscopic image the user than applies sufficient pressure on the mac roscopic imager in the direction of the tissue such that adhe sive on the lower surface of the window 52 facing the tissue adheres and retains the tissue ring and window assembly on the tissue When the tissue ring is so applied alignment ofthe macroscopic imager 32 is verified and a macroscopic image is then captured The macroscopic imager is then detached from the tissue ring 30 and confocal imager 18 attached and aligned to the tissue ring for capture of confocal images as described earlier Referring to FIGS 5 9 the user interface 33 for the mac roscopic imager 32 will now be described The user interface represents a program or application operating in memory of the computer system 14 and preferably is running when the macroscopic imager is connected to the computer system The user interface 33 is a graphical user interface in that a typical pointing mechanism such as a mouse 17 FIG 1 is coupled to computer system 14 to enable the user to move select click and or drag a displayed cursor or graphical element as typical of mouse functionality Other pointing mechanisms may also be used such as a touch screen track ball or the like coupled to the computer system 14 The user interface has various tabs 54 along the side of a window 56 Selec
11. cal images captured by the confocal imager that were selected by the user for storage in memory of the computer system Such markings may be capable of indicating different types of confocal image capture The computer system may be capable of storing different macroscopic images of the same area of tissue captured at different times and the user interface morphs or overlays two of the macroscopic images of the same area of tissue to produce a morphed or overlaid image and user control of the contribution of the two macroscopic images in the mor phed image so as to enable viewing of changes in the same area of tissue over the time period associated with the two macroscopic images The invention further embodies a method having the steps of displaying a macroscopic image of the surface of tissue captured by a macroscopic imager indicating a region within the displayed macroscopic image associated with a field of view imagable by a confocal imager tracking the two dimen sional position of the confocal imager with respect to the surface of the tissue in the displayed macroscopic image targeting in the displayed macroscopic image the two dimen sional position of confocal images captured by the micro scopic imager with respect to the tissue and marking on the displayed macroscopic image the location ofone or more user selected confocal images The user interface represents a graphical user interface operable on a computer system to provi
12. ce Interferometry 1993 SPIE vol 1889 pp 6 263 233 Bl 7 2001 Zavislan et al 197 211 6 272 235 Bl 8 2001 Bacus et al VivaScope 1500 Brochure Lucid Inc 6 330 106 Bl 12 2001 Greenwald et al VivaScope 2100 Brochure Lucid Inc 6 370 422 Bl 4 2002 Richards Kartum et al VivaScope 2500 Brochure Lucid Inc 6 411 434 Bl 6 2002 Eastman et al 6 424 852 BL 7 2002 Zavislan cited by examiner U S Patent Jan 4 2011 Sheet 1 of 9 US 7 864 996 B2 10 2 ae 26 a TONES lia U S Patent Jan 4 2011 Sheet 2 of 9 US 7 864 996 B2 U S Patent Jan 4 2011 Sheet 3 of 9 US 7 864 996 B2 US 7 864 996 B2 Sheet 4 of 9 Jan 4 2011 U S Patent tissue cone o gt d o a lt FIG 3 Y c Q o o a D P4 FIG 4 U S Patent Jan 4 2011 Sheet 5 of 9 US 7 864 996 B2 US 7 864 996 B2 Sheet 6 of 9 Jan 4 2011 U S Patent 23 9 S6 o Q SJ FIG 7 U S Patent Jan 4 2011 Sheet 7 of 9 US 7 864 996 B2 70e 70a U S Patent Jan 4 2011 Sheet 8 of 9 US 7 864 996 B2 LEEPRRERDAISEGBUNE Macro Identifier Lesioni Macro003 ERA BEEN RSEN EXER Color Correction D DT Brightness 0 000 Date Captured 01 26 05 12 55 39 Contrast A 1 000 i ME s Exposure a 45 ms Gain 3875 Gain Bue 1211 Gain Red AE 1 000 D A bon M Gamma 4 000 FIG 9 ix U S Patent Jan 4 2011 Sheet 9 of 9 US 7 864
13. claim 6 wherein said tissue is associated with a patient and said stored macroscopic image stored microscopic images and said image overlay are asso ciated in said computer system with said patient 8 The system according to claim 2 wherein said computer system is capable of storing different ones of said macro scopic image of said same area of tissue captured at different 5 un 0 20 25 30 55 60 65 14 times and said user interface further enables morphing two of said different ones of said macroscopic images to produce a morphed image 9 The system according to claim 8 wherein said user interface has a graphic enabling a user to select different contributions of said two of said different ones of said mac roscopic images in said morphed image so as to enable view ing of changes in said same area of tissue over the time period associated with said two of said different ones of said mac roscopic images 10 The system according to claim 2 wherein said the area of said displayed macroscopic image outside said outlined region is reduced in at least one of contrast or brightness 11 The system according to claim 1 wherein said computer system has a display and memory and said computer system is capable of storing different ones of said macroscopic image of said same area of tissue captured at different times and said computer system further has a user interface capable of mor phing two of said different one
14. computer system having memory for storing at least one macroscopic image when received from said macroscopic imager and said microscopic images when received from said microscopic imager said user interface comprising a first window displaying said macroscopic image and a region within the macroscopic image associated with a field of view of tissue wherein said tissue is not histo logically prepared tissue that is mounted on a slide imagable by said microscopic imager a second window selectable by the user displaying a plu rality of macroscopic images of said same area captured at different times for enabling selection of two of said plurality of macroscopic images wherein at least one of said plurality of macroscopic images of said same area was captured utilizing a device providing a field of view of the tissue in which the tissue extends outside of said device and a slide in which said first window has an morphed image of said two selected macroscopic images in which the con tribution of each of said two image in said morphed image is adjustable by the user utilizing said slide 20 A user interface operable on a computer system coupled to a macroscopic imager and a microscopic imager said computer system having memory for storing at least one macroscopic image of the surface of tissue when received from said macroscopic imager and one or more microscopic images when received from said microscopic imager said user interface comp
15. d in opening 30a by the user Optionally cross hair indicia may be printed or otherwise provided on one of the flat surfaces of window 52 outside of view of opening 30a and inside the rings of adhesive on such each of said flat surfaces to assist the user in centering the area of tissue interest The macroscopic imager 32 is then presented to the tissue ring 30 which are magnetically attached to mount 48 of imager 32 as described earlier An arrow 48a on the mount 48 is aligned with the center or window tab 525 by the user rotating the macroscopic imager housing 34 to couple such rotation to tissue mount 48 as shown for example in FIG 3 Switch 41 is operated by a user to capture a macroscopic image of the tissue surface 53 below and pressed against window 52 through window 52 and tissue ring opening 30a The macroscopic imager 32 is then removed from the tissue ring 30 with the assistance of moving levers 50 backward without removal of tissue ring 30 from tissue 53 and replaced 0 jak 5 40 45 50 55 8 with the confocal imager 18 having mount 48 which is then similarly aligned with its arrow 48a centered on window tab 52a as shown for example in FIG 4 An immersion fluid may be placed inside the tissue ring 30 before connecting the confocal imager 18 The confocal imager 18 remains so located on the tissue ring 30 throughout the confocal imaging session Other indicia than arrow 48a may be used to align the tab 5
16. de a window display ing a macroscopic image in which a region is indicated within the macroscopic image that is associated with the tissue imagable by a confocal imager One or more first graphical elements are provided for enabling tracking and targeting in the region of a two dimensional imaging location ofthe confocal imager with respect to the surface ofthe tissue and one or more second graphical elements are provided for enabling marking in the region ofthe window the location of one or more of the confocal images after being captured by the confocal imager and stored in memory by the computer system The macroscopic image in the window represents a 20 25 30 35 40 45 50 55 60 65 4 current macroscopic image The user interface may also have another window selectable by the user displaying one or more previously captured macroscopic images of the same area of tissue When one of such previously captured macroscopic images is selected a graphical slide is provided in which the window has an overlaid image of the selected previously captured macroscopic image on the current macroscopic image in which the contribution of the current and previous macroscopic images in the first window is adjustable by the user utilizing the graphical slide such that the images morph or blend into each other The user interface of the present invention may be a com bined user interface for displaying image from the macro scop
17. es a system for imaging tissue having a macroscopic imager for capturing a macroscopic image and a confocal imager for capturing one or more optically formed sectional microscopic confo cal images on or within tissue a tissue attachment device such as a tissue ring in which the macroscopic imager and confocal imager are each individually presented to the tissue utilizing the tissue attachment device in a predefined align ment with the device thereby imaging locations of the con focal imager with respect to the tissue surface spatially cor relate with the macroscopic image computer system is coupled to the macroscopic imager and microscopic imager and has a display and memory for storing at least one mac roscopic image received from the macroscopic imager and confocal images when received from the confocal imager A user interface operable on the computer system enables display of the macroscopic image on a display coupled to the computer system and then indicates a region within the mac roscopic image associated with a field of view of the tissue imagable by the confocal imager The user interface enables graphical tracking of the imaging location of the confocal imager in the macroscopic image and targeting the confocal imager to capture confocal images at one or more imaging locations selected in the macroscopic image The user inter face also enables marking on the displayed macroscopic image of one or more locations of confo
18. f images may also be captured stored and marked For example a VivaBlock image represents tissue sections arranged to map a region of tissue at a common depth in the tissue The map 64 may be cleared or hidden from view by selection of buttons 605 and 60c respectively Othertabs selectable by the user are patient records tab 54c for editing or viewing patient information archive tab 54d for viewing images of previous sessions stored with the same or a different patient system settings tab 54c to view or change parameters of imaging such as dimensions and or depth of VivaBlock or VivaStack map coarse or fine X Y and Z motor step settings freeze and hold time for a static image capture and typical print preferences For example button 61a may be click by the user to capture a confocal image by the confocal microscope at a location in the tissue selected by box 62 Other buttons on the screen may be provided to adjust color 615 or reset 61c the user interface to clear the displayed macroscopic image 57 and its overlaid map 64 thereby enabling capture of a new macroscopic image if desired Adjust color 616 allows the user to open a window to adjust imaging properties described earlier which may be desirable for different skin types 20 25 30 35 40 45 50 55 60 65 10 Each time the Add Lesion button 604 is selected a new record is generated in which images stored in the session from thereon after a
19. for assisting a physician in future examinations of the same tissue to Observe possible changes in the condition of the tissue when treatment of the lesion is deferred or is non invasive SUMMARY OF THE INVENTION Accordingly it is one object of the present invention to provide a system for macroscopic and confocal imaging of tissue having a macroscopic imager and a confocal imaging head of a confocal microscope where the macroscopic imager and confocal image head are each individually aligned to the same tissue attachment device for imaging tissue Itis another object of the present invention to a system for macroscopic and confocal imaging of tissue having a com puter system coupled to a macroscopic imager and confocal microscope for receiving macroscopic and confocal images respectively having a user interface facilitating the tracking targeting and marking of confocal images in a macroscopic image captured by such macroscopic imager It is a further would object of the present invention to a system for macroscopic and confocal imaging of tissue to having a computer system coupled to a macroscopic imager and confocal microscope for receiving macroscopic and con focal images respectively having a user interface which US 7 864 996 B2 3 enables morphing of macroscopic images of the same tissue captured at different times to observe possible changes in the condition of the tissue Briefly described the present invention embodi
20. he settings and archive buttons 89a operate similar to tabs 54c and 544 described earlier The Quit button 895 exits the user interface 774 Icons 96 may also be active when any image is presented in window 83 such that both confocal and macroscopic images may be annotated by the user Other VivaScope confocal microscope available from Lucid Inc such as the VivaScope 2100 or VivaScope 2500 may also be used in system 10 by adapting their imaging heads with tissue mount 48 so that they can be positioned to engage a tissue ring when mounted to in vivo patient or ex vivo tissue sample Also other optical microscopes may be also adapted with such tissue mount 48 Such as for example microscopes operating in accordance with optical coherence tomography or interferometry such as described in Schmitt et al Optical characterization of disease tissues using low coherence interferometry Proc Of SPIE Volume 1889 1993 or two photon microscopy such as described in U S Pat No 5 034 613 to Denk et al issued Jul 23 1991 From the foregoing description it will be apparent that a system method and user interface for macroscopic and con focal imaging of tissue has been provided Variations and modifications in the herein described system method and user interface in accordance with the invention will undoubt US 7 864 996 B2 13 edly suggest themselves to those skilled in the art Accord ingly the foregoing description
21. ia the computer system sending signals such motor s or via buttons on a control panel 29 As confocal micro scope 12 is described in U S patent Ser No 10 557 461 and International Patent Application No PCT US04 16255 which are herein incorporated by reference a detailed discus sion of the confocal microscope 12 and its confocal imager 18 are not provided herein System 10 further has a macroscopic imager or camera 32 which is connected for data communication via a cable 31 to aport suchas a USB port of computer system 14 such that the computer system can receive macroscopic images from imager 32 Conventional hardware and software at the imager 32 and computer system 14 may be provided for interfacing and communicating digital images Referring to FIGS 2A and 2B macroscopic imager 32 is shown in more detail Imager 32 has a housing 34 which is formed by the mating rear housing member 34a and front housing member 34b Housing members 34a and 345 may be of molded plastic A pair of printed circuit boards 36 supports the imaging electronics and includes a CCD array 38 Such CCD array 38 may be a color CCD array similar to that used in a conventional digital camera The imaging electronics may have a microcontroller or microprocessor programmed to operatethe camera for providing live images from the CCD array 38 via cable 31 to computer system 14 and sending a high resolution still image when a switch 41 is pressed via a cable
22. ic imager and confocal microscope on a single screen or different screens may be used to provide macroscopic image and confocal images Although the application describes confocal imagers for imaging optical formed microscopic sections utilizing con focal microscopy other imager may be used to provide opti cal formed microscopic sections operating in accordance with two photon microscopy or optical coherence tomogra phy BRIEF DESCRIPTION OF THE DRAWINGS The foregoing objects features and advantages of the invention will become more apparent from a reading of the following description in connection with the accompanying drawings in which FIG 1 is a block diagram of the system of the present invention having a confocal microscope with a confocal imager or imaging head and a computer system and a macroscopic imager coupled to the same computer system FIG 1A is a perspective view of tissue attachment device or tissue ring attachable to the macroscopic imager and the confocal imager of FIG 1 FIG 2A isa perspective view ofthe macroscopic imager of FIG 1 and the tissue attachment device of FIG 1A FIG 2B is an exploded view of the macroscopic imager of FIG 1 FIG 2C is an exploded view ofthe tissue mount ring ofthe macroscopic imager of FIG 2B FIG 3 is a partial view of the front of the macroscopic imager of FIG 1 when connected and aligned to the tissue attachment device of FIG 1A which is adhesively coupled to
23. irst means and a different one of said second means being mounted to fourth means for imaging tissue microscopi cally below the surface of the tissue in which when said different one of said second means is mounted to said first means enables said fourth means to image tissue in said field of view of said first means 22 The apparatus according to claim 21 wherein said first means enables said third means and fourth means to be spa tially correlated with each other in their respective imaging
24. le imaging through openings 30a and window 52 Preferably the macroscopic imager 32 is calibrated by the user with the attached tissue ring and window assembly to assure that imaging is proper This can be performed by placing the tissue ring 30 over a white sheet of paper and providing a window to set image properties Such image properties may includes fields for adjusting CCD exposure time e g max 65 ms gain and white point balance red green and blue gain If available the user can enable expo sure time gain and white point balance to be set automati cally by the computer system 14 The computer system 14 may send signals to the macroscopic imager 32 to communi cate to macroscopic imager electronics such imaging param eters and thereby control macroscopic imager 32 operation Color and white point balance correction may be performed by the computer system on each macroscopic image received In operation the tissue ring 30 with attached window 52 is located on the tissue 53 to be imaged by the macroscopic imager and then the confocal imager 18 Surface 526 of window 52 has a ring of adhesive material to facilitate retain ing the tissue ring and its attached window 52 on the tissue surface Alternatively the window 52 may be adhesively coupled to tissue and then the tissue ring 30 located over and adhesively coupled to window 52 Either way the area of the tissue of interest is viewable through opening 30a and thus is centere
25. les by Milind Rajadhyaksha et al In vivo Confocal Scanning Laser Microscopy of Human Skin Melanin pro vides strong contrast The Journal of Investigative Derma tology Volume 104 No 6 June 1995 and Milind Rajadhy aksha and James M Zavislan Confocal laser microscope images tissue in vivo Laser Focus World February 1997 pages 119 127 The confocal microscope may image natu rally or surgically exposed in vivo tissue which is useful to evaluate a lesion in tissue without needing a biopsy and patho logical evaluation on slides of histologically prepared mechanically sectioned tissue specimens from such biopsy Also confocal microscopes are useful for pathological examination of ex vivo tissue i e tissue removed from a patient without requiring that such tissue be mechanically sectioned and histologically prepared for viewing on slides with a traditional microscope Systems have been developed for obtaining a macroscopic image of tissue for use in locating where sectional micro scopic images were imaged in such tissue by a confocal microscope U S Pat Nos 6 684 092 and 5 836 877 describe atelepathology system having a camera and confocal imager in a fixed spatial relationship in which both are presented over the tissue such that the camera captures a macroscopic image and the confocal imager captures one or more confocal images Instead ofthe camera the confocal imager described in these patents may utilize a differe
26. mager and tracking on said displayed mac roscopic image an imaging position of said microscopic imager with respect to the tissue displayed in said macro scopic image 3 The system according to claim 2 wherein said computer system further comprises a pointing device for controlling the location of a graphic on said display and said user interface further enables user targeting of two dimensional position of the microscopic images captured by said microscopic imager with respect to the surface of said tissue by utilizing said pointing device to move said graphic in said outlined region of said displayed macroscopic image in which the position of said graphic is spatially correlated in two dimensions with the microscopic images captured by said microscopic imager 4 The system according to claim 2 wherein said user interface further enables marking on said displayed macro scopic image the location of one or more of said microscopic images after being captured by said microscopic imager and stored in memory by said computer system 5 The system according to claim 4 wherein said user interface further enables user selection of one of said marked locations enabling viewing on said display of said one or more stored microscopic images associated with said one of said marked locations 6 The system according to claim 4 wherein said user interface provides said marking in an image overlay on said macroscopic image 7 The system according to
27. n Vivo Laser Focus World Feb 1997 pp 119 127 Continued Primary Examiner Anand Bhatnagar Assistant Examiner Claire Wang 74 Attorney Agent or Firm Kenneth J Lukacher 57 ABSTRACT A system for macroscopic and confocal imaging of tissue having a macroscopic imager for capturing a macroscopic image of the tissue s surface a confocal imager for capturing one or more optically formed sectional microscopic images on or within tissue a computer for receiving images from such imagers and a tissue attachment device in which the macroscopic imager and confocal imager are each individu ally presented to the tissue utilizing the tissue attachment device in a predefined alignment such that imaging locations of the confocal imager with respect to the tissue surface spatially correlate with macroscopic image A user interface is operable on the computer to enable display of the macro scopic image on a display coupled to the computer and to indicate a region within the macroscopic image associated with the field of view of the tissue imagable by the confocal imager The user interface enables graphical tracking and targeting of imaging locations of the confocal imager in mac roscopic image and marking on the macroscopic image of the locations of confocal images with respect to the tissue sur face 22 Claims 9 Drawing Sheets US 7 864 996 B2 Page2 U S PATENT DOCUMENTS 6 493 460 B1 12 2002 MacAulay et al 6 606 413 B
28. nt objective lens to obtain the macroscopic image The locations for imaging by the confocal imager may be selected automatically or manually by the user using the macroscopic image The macroscopic image and confocal images taken are then viewable on the display of the computer in which the location of confocal images may be referenced in the macroscopic image of the tissue USS Pat No 6 411 434 describes a system for imaging ex vivo tissue in a cassette where a camera views one side of the tissue to provide a macroscopic image and on the other side of the tissue is imaged by a confocal microscope A 0 a 5 40 45 65 2 display can provide an image of both the macroscopic image and confocal images in which the relative location of the microscopic image is indicated by an outlined region in the displayed macroscopic image U S patent application Ser No 10 471 332 filed Feb 23 2004 which has priority to International Patent Application No PCT US02 07173 published on Sep 19 2002 under International Publication No WO02 073246A2 describes a confocal microscope having a macroscope and a turret hav ing different objective lens to enable selection of an objective lens for macroscopic imaging or microscopic imaging in which light from the tissue through the selected objective lens is provided to optics of the macroscope or confocal micro scope respectively Sites may be marked such as on a print out of
29. o the computer system 13 and stored in associated with the image Other buttons 72a on screen of FIG 9 provide saving such comments in field 71 for associated with the image deleting the stored image from memory of the computer system 13 or canceling edit or entered comments in field 71