Home

A Workingperson`s Guide to Deconvolution in Light Microscopy

1. Alternatively if enough RAM is installed then the whole image can be processed as a single volume Noise Amplification or Excessive Smoothing Noise amplification has been allud ed to already as an artifact caused by deblurring and inversion algorithms It also occurs with iterative algorithms as repeated convolution operations intro duce high frequency noise The artifact appears as a distinctive mottling of the image that often is constant in every plane and particularly noticeable in background areas It is usually sup pressed by a smoothing filter or regu larization or roughness filter 53 If excessive noise is observed in a decon 1092 BioTechniques Raw 250 300 Figure 4 Examples of artifacts A D Images of bovine pulmonary endothelial cells from a prepared slide Fluocells Molecular Probes These cells were fixed permeabilized and stained with BODIPY FL phallicidin emission max 512 nm Images were acquired as described in previous figures vary ing the RI of the immersion oil A and B RI 1 514 C and D RI 1 524 A Raw image maximum intensity projection without appreciable spherical aberration B The same image after deconvolution C Raw image of a similar cell with significant spherical aberration note diffraction rings around points surrounding the cell D The same image as in panel C after restoration by iterative constrained decon volution Note that a horizontal line is i
2. The optical axis is parallel to the vertical axis of the image Left shows minimal spherical aberration Right shows significant spherical aberration Note the axial asymmetry and widening of the central node along the optical axis in the right hand image This leads to degraded axial resolution and blurring of signal In theory the size of the PSF is infinite and the total summed in tensity of light in planes far from focus is equal to the summed intensity at focus However light inten sity falls off quickly and eventually becomes indistinguishable from noise In an unaberrated PSF recorded with an NA 1 4 oil lens a typical high resolution lens light occupying 0 2 um at the plane of focus is spread out over 90 times that area at 1 um above or below focus In these images the object is a 0 1 um subresolution fluorescent bead Molecular Probes Eugene OR USA mounted in glycerol RI 1 47 with immersion oil of RI 1 5140 left and 1 5220 right The images are sections from 3 D image stacks collected using a DeltaVision wide field imaging system Applied Precision Issaquah WA USA Objective lens 100x NA 1 4 Excitation wavelength 580 nm emission wavelength 630 nm Scale bar 1 um B A population of 0 1 um beads adhering to the surface of a cover slip in focus 0 um and at 1 um defocus 1 um Imaging parameters are as in panel A The two images are scaled dif ferently to emphasize the out of focus rings in the de
3. 108 55 Tikhonov A N and V Y Arsenin 1977 So lutions to ill posed problems Winston Wash ington D C 56 Young IT 1989 Image fidelity characteriz ing the imaging transfer function Methods Cell Biol 30 1 45 Address correspondence to Dr Jason Swedlow Division of Gene Regulation and Expression MSI WTB Complex University of Dundee Dow Street Dundee DD1 5EH Scotland e mail j swedlow dundee ac uk BioTechniques 1097
4. Nearly any image acquired 1076 BioTechniques on a digital fluorescence microscope can be deconvolved In addition new applications to transmitted light images are now available 29 Three dimen sional images made up of a series of op tical sections are particularly well suited for improvement by deconvolution Deconvolution is often described as an alternative to confocal microscopy This is not strictly true since confocal images can themselves be deconvolved However most users apply deconvolu tion to images recorded on a standard wide field fluorescence microscope This approach yields images of compa rable resolution to a confocal micro scope 49 In fact confocal micro scopy and _ wide field deconvolution microscopy both work by removing blur but they do so by opposite means Confocal microscopy prevents out of focus blur from ever being detected by placing a pinhole between the objective lens and the detector through which only in focus light can pass 30 Wide field microscopy allows blurred light to reach the detector deconvolu tion then attempts to either subtract blurred light from the image or reassign it back to its source 1 37 53 Confocal microscopy is especially well suited for thick specimens such as embryos or tis sues while wide field deconvolution microscopy has proven to be a power ful method for imaging samples requir ing low light levels such as living cells bearing fluorescently
5. Refer ence 48 for an estimate However they may restore to a slightly higher degree of resolution than the classical algo rithms They also have the advantage that they impose constraints on the ex pected noise statistic i e a Poisson or a Gaussian distribution As a result these algorithms have a more subtle noise policy than simply regularization and they may give better results on noisy images However the choice of an ap propriate noise statistic may depend on the imaging condition and some com mercial software packages are more flexible than others in this regard Blind deconvolution Blind decon volution is a relatively new method that greatly simplifies the use of deconvolu tion for the non specialist Currently only Autoquant implements this method It was developed by altering the MLE algorithm so that not only the object but also the PSF is estimated 27 In this approach an estimate of the object is made This estimate is convolved with a theoretical PSF calcu lated from optical parameters of the imaging system The resulting blurred estimate is compared with the raw im age a correction is computed and this correction is used to generate a new es timate as explained above This same correction is also applied to the PSF generating a PSF estimate In further it erations the PSF estimate and the ob ject estimate are updated together Blind deconvolution works well not only on high quality images
6. To oversample sufficiently we routinely sample at 0 07 um in x y and 0 2 um in z under these conditions Of course these guidelines repre sent optimal settings and should be bal anced by considering the specific preparation in question For example the fluorescence signal may be so low e g in a live cell experiment that bin ning of CCD pixels is required or events may occur so quickly that there is not time to finely sample in z In such cases a suboptimal sampling interval will be required Fortunately restora tion algorithms still work quite well un der these conditions 9 54 although some changes in regularization may be necessary in some algorithms 47 Ringing and Edge Artifacts Ringing is an artifact chiefly found in deblurring or inversion methods but it does sometimes occur with iterative methods It has the appearance of dark and light ripples around bright features of an image Figure 4 E and F It can occur in z and in x y in z it looks like a shadow in a deeper z section outlining a fluorescent structure Ringing is generally caused by the conversion of a discontinuous signal into or out of Fourier space 38 A number of related issues can produce signal discontinuities and therefore cause ringing Discontinuities can occur at the edges of the image or of subvol umes of the image Figure 4D or even BioTechniques 1091 BioFeature at the edges of bright features Figure 4 E and F Disco
7. can be helpful in se lecting a lens Aberrations that are hard to detect when looking at complicated objects are very clear when examining the PSF from a single bead Therefore before purchasing a costly new objec tive lens we recommend that you ac quire PSFs from several lenses and choose the one with the most ideal PSF Third an empirical PSF allows you to measure the performance of your imag ing system Many potential problems that may occur during an experiment e g stage drift lamp flicker camera noise refractive index RI mismatches temperature changes due to heavy ven tilation etc will also occur during PSF acquisition and be more easily dis cernible Therefore aberrations in the empirical PSF suggest ways to improve your microscopy When acquiring an empirical PSF care must be taken to match the aberra tions of the raw image Ideally both the raw image and the PSF should be free of aberration but this is not always pos sible If major aberrations are present in the raw image then they should if pos sible be matched by aberrations in the PSF 45 Otherwise the deconvolved image may contain errors or be poorly restored In addition if the PSF is noisy then substantial noise will appear in the deconvolved image To reduce noise and eliminate minor aberrations many packages radially average the PSF or average the images of several beads to create a smoother PSF In addition most commercial packages a
8. does not eliminate structures recorded by the microscope However since there is a potential for loss of detail software implementations of inverse filters typically include an adjustable parameter that allows the user to con trol the tradeoff between smoothing and noise amplification 37 Constrained Iterative Algorithms To improve the performance of in verse filters a number of other 3 D al gorithms can be used to restore images 48 50 53 These methods are called constrained iterative algorithms They work in successive cycles and are therefore called iterative They also usually apply constraints on possible solutions These constraints not only help to minimize noise or other distor tion but also increase the power to re store blurred signal A typical constrained iterative algo rithm works as follows An estimate of Vol 31 No 5 2001 BioFeature the object is made this is usually the raw image itself The estimate is con volved with the PSF and the resulting blurred estimate is compared with the raw image This comparison is used to compute an error criterion that repre sents how similar the blurred estimate is to the raw image This error criterion or figure of merit is then used to al ter the estimate in such a way that the error is reduced A new iteration then takes place the new estimate is con volved with the PSF a new error crite rion is computed etc The best est
9. objects such as the bead in Figure 3 or even computer generated images of theoretical objects The relationship between performance with such test objects and performance with real biological specimens is not straightforward Furthermore unless the comparison is done quantitatively with objects of known size it is hard to know whether a more pleasing result is really more accurate For instance the algorithm might eat away the edges of features making them look sharper but confounding measurements In addition algorithm comparisons are usually published by biased parties with an interest in the result of the com parison Frequently these parties com pare an algorithm whose implementa tion they have developed and optimized over many years with a non optimized algorithm they have implemented straight out of the book However as noted above big differences in speed stability and resolution improvement can be attributed to implementation and optimization of the algorithm There fore the only fair comparison is be tween realized software packages rather than between algorithms Vol 31 No 5 2001 BioFeature We recommend that you the poten tial buyer of deconvolution software should compare the performance of various software packages on your own data Unfortunately this may require a certain amount of determination To compare images restored by different packages you will need to make sure the sales repre
10. real PSF of the raw image is larger than expected This might occur because of refractive index mismatch which caus es spherical aberration and z axis scal ing Both phenomena widen the real PSF in z making it larger and thereby lowering the effective NA of the lens If this is suspected then try subtracting a small increment e g 0 05 from the NA of the theoretical PSF In some soft ware a similar result is obtained by set ting the z step of the PSF to be smaller than the z step of the raw image Normally the z step of the PSF should always be identical to the z step of the raw image this ensures that the scaling of the PSF is appropriate to the imaging conditions With an empirical PSF however it may be possible to ac quire a PSF at finer z resolution than the raw image This makes the Fourier transform of the PSF more detailed and can benefit restoration However this trick works only if the software can in terpolate the image s sampling interval and make it correspond to the PSF Consult your user manual to determine how your software handles the PSF Spatial Variation of the PSF Currently most commercially avail able deconvolution packages assume that the PSF is constant for all points in the object a property known as spatial invariance Microscope optics general ly meet this assumption however oth er issues such as refractive index gradi ents in the specimen material or mismatch of immersion and m
11. x y and z where x and y are parallel to the focal plane of the specimen and z is parallel to the optical axis of the microscope In this case the PSF looks like a set of concentric rings in x y and it looks like an hourglass in x z or y z Figure 1A An x y image through the center of the wide field PSF looks like a set of con centric rings this is the Airy disk of classical light microscopy How does the PSF affect image for mation in a microscope In the theoreti cal model of image formation the PSF is considered the basic unit of any im age The PSF is to the image what the brick is to the house The best an image can ever be is an assembly of PSFs and increasing magnification will not change this A noted theoretical optics textbook explains It is impossible to bring out detail not present in the prima ry image by increasing the power of the eyepiece for each element of the prima ry image is a small diffraction pattern and the actual image as seen by the eyepiece is only the ensemble of the magnified images of these patterns 7 As an example consider a popula tion of tiny fluorescent beads mounted under a cover slip An in focus image of this specimen will show a cloud of dots each of which when examined at high resolution is actually a disk sur rounded by a tiny set of rings i e an Airy disk Figure 1B 0 um If this specimen is brought out of focus slight ly a large set of concentric
12. 1095 BioFeature equal to the radius of one Airy disk measured from its point of maximum intensity to its first ring of minimum in tensity For monochromatic images of a given fluorescence wavelength the Rayleigh criterion can be estimated us ing a standard formula that is found in many textbooks of optics including In ou and Spring 31 d 0 61A NA Eq 1 where d is the Rayleigh criterion A is the wavelength of emitted light and NA is the numerical aperture of the ob jective lens Note that the smaller the value of d the higher the resolution This formula can be used to assess resolution in the image plane but not along the optical axis z axis Howev er an adequate formula for the axial Rayleigh criterion can be deduced us ing similar reasoning The minimum resolvable axial distance between two point sources will occur when their axi al diffraction patterns are distinct However the axial diffraction pattern of a point source is not disk shaped rather it has the hourglass shape or flare of the PSF image in x z or y z planes Nonetheless this hourglass shape has a central bright region as does the Airy disk Therefore to define the axial Rayleigh criterion we can take the distance from the point of max imum intensity to the first point of min imum intensity of the central bright re gion along the z axis This can be estimated using the following formula d 2An NA 2 Eq 2 This formul
13. 20 Gibson S F and F Lanni 1991 Experimen tal test of an analytical model of aberration in an oil immersion objective lens used in three dimensional light microscopy J Op Soc Am 8 1601 1613 21 Gold R 1964 An Iterative Unfolding Method for Response Matrices Report no ANL 6984 Argonne National Laboratory Chicago 22 Goodman J W 1996 Introduction to Fourier Optics McGraw Hill New York 23 Hayat M A Ed 2000 Principles and Tech niques of Electron Microscopy Biological Ap plications 4th ed Cambridge University Press Cambridge UK 24 He X S Asthana and P K Sorger 2000 Transient sister chromatid separation and elas tic deformation of chromosomes during mito sis in budding yeast Cell 01 763 775 Vol 31 No 5 2001 25 Hell S G Reiner C Cremer and E H K Stelzer 1993 Aberrations in confocal fluores cence microscopy induced by mismatches in refractive index J Microsc 169 391 405 26 Hiraoka Y J W Sedat and D A Agard 1990 Determination of three dimensional imaging properties of a light microscope sys tem Biophysical J 57 325 333 27 Holmes T J 1992 Blind deconvolution of quantum limited imagery maximum likeli hood approach J Opt Soc Am A 9 1052 1061 28 Holmes T J S Bhattacharya J A Cooper D Hanzel V Krishnamurthi W c Lin B Roysam D H Szarowski and J N Turner 1995 Light microscopic images reconstructed by maximum likelihood deconvoluti
14. BioFeature A Workingperson s Guide to Deconvolution in Light Microscopy BioTechniques 31 1076 1097 November 2001 Wes Wallace Lutz H Schaefer2 and Jason R Swedlow University of Dundee Dundee Scotland Brown University Providence RI USA and 2Ad vanced Imaging Methodology Consultation Kitchener On tario Canada ABSTRACT The fluorescence microscope is routinely used to study cellular structure in many bio medical research laboratories and is in creasingly used as a quantitative assay sys tem for cellular dynamics One of the major causes of image degradation in the fluores cence microscope is blurring Deconvolu tion algorithms use a model of the micro scope imaging process to either subtract or reassign out of focus blur A variety of al gorithms are now commercially available each with its own characteristic advantages and disadvantages In this article we re view the imaging process in the fluores cence microscope and then discuss how the various deconvolution methods work Final ly we provide a summary of practical tips for using deconvolution and discuss imag ing artifacts and how to minimize them INTRODUCTION Deconvolution is a computational technique for improving the contrast and resolution of digital images It in cludes a suite of methods that seek to remove or reverse the blurring present in microscopes images caused by the limited aperture of the microscope ob jective lens
15. a was obtained by com paring the theoretical distribution of light intensity near focus given by Born and Wolf 7 with formulae given by Inou 30 and Keller 36 Notice that this formula includes n the RI of the mounting immersion me dia in the numerator The mounting and immersion media are assumed to have the same refractive index other wise spherical aberration degrades the resolution Note that all of these crite ria assume aberration free imaging conditions It may be tempting to be lieve based on this formula that reduc ing the RI of the immersion medium can improve z resolution However this is a fallacy because the lens NA is also reduced if n is reduced and since 1096 BioTechniques z resolution varies with the square of NA the reduction in NA outweighs the reduction in n and resolution is worse Notice also that x y resolution varies only with the first power of NA where as z resolution varies with the square of NA This means that x y resolution and z resolution both improve with increas ing NA but z resolution improves more dramatically Z resolution is closely related but not identical to depth of field The depth of field is the thickness of the slab of specimen which appears fo cused in the image When looking at a 2 D image we see a slab of the speci men with a certain thickness focused onto a single flat image Features that seem equally well in focus in the image may reside at dif
16. algorithm may be suspected In that case com pare the effects of different algorithms e g inverse vs iterative classical vs statistical on the appearance of the ar tifact This will verify whether a partic ular specimen is not restored well by a given algorithm If an apparent artifact is visible in both deconvolved and raw images then some aspect of specimen preparation or optical aberration is implicated In par ticular immunofluorescence staining is often discontinuous along cytoskeletal filaments actin filaments microtubules and intermediate filaments Several causes can contribute to this problem One is that cytoskeletal filaments may be masked by cytoskeleton associated proteins so antibody accessibility is variable along the filament Another cause is that the fixatives used in im munocytochemistry often do not faith fully preserve filamentous structures For example microtubules may frag ment during formaldehyde or methanol fixation A more faithful fixative such as glutaraldehyde can be tried but at the 1094 BioTechniques expense of antigenicity Optimal fixation methods are described and discussed in the following sources 4 14 23 44 Another type of specimen dependent artifact derives from RI gradients in the specimen These cause lensing effects in the specimen before light reaches the objective lens and therefore may dis tort the image Some specimens e g embryos or thick tissues have yol
17. and quality is dramatically affected by how a given deconvolution algorithm is implemented in software The algo rithm can be implemented in ways that reduce the number of iterations and ac celerate convergence to a stable esti mate For example the unoptimized JVC algorithm usually requires 50 100 iterations to converge to an optimal es timate 2 48 50 By prefiltering the raw image to suppress noise and cor Vol 31 No 5 2001 BioFeature recting with an additional error criteri on on the first two iterations the algo rithm converges in only 5 10 iterations In addition a smoothing filter is usual ly introduced every five iterations to curtail noise amplification When using an empirical PSF it is critical to use a high quality PSF with minimal noise No deconvolution pack age we know of uses the raw PSF recorded from the microscope Instead the packages contain preprocessing rou tines that reduce noise and enforce radi al symmetry by averaging the Fourier transform of the PSF Many packages also enforce axial symmetry in the PSF and thus assume the absence of spheri cal aberration These steps reduce noise and aberrations and make a large differ ence in the quality of restoration Another aspect of implementation is preprocessing of the raw image via routines such as background subtrac tion flatfield correction bleaching cor rection lamp jitter correction etc These operations can improve the
18. ar architecture in three dimen sions Ann Rev Biophys Bioeng 3 191 219 2 Agard D A Y Hiraoka P Shaw and J W Sedat 1989 Fluorescence microscopy in three dimensions Methods Cell Biol 30 353 377 3 Art J J and M B Goodman 1993 Rapid scanning confocal microscopy Methods Cell Biol 38 62 64 4 Bacallao R K Kiai and L Jesaitis 1995 Guiding principles of specimen preservation for confocal fluorescence microscopy p 311 325 InJ Pawley Ed Handbook of Biologi cal Confocal Microscopy 2nd ed Plenum Press New York 5 Baxes G A 1994 Digital Image Processing Principles and Applications John Wiley and Vol 31 No 5 2001 Sons New York 6 Bohren C F 1995 Scattering of Particles In M Bass Ed Handbook of Optics vol 1 McGraw Hill New York 7 Born M and E Wolf 1980 Principles of Op tics Pergamon Press New York 8 Carrington W 1990 Image restoration in 3D microscopy with limited data Proc Int Soc Optical Eng 1205 72 83 9 Carrington W A R M Lynch E D Moore G Isenberg K E Fogarty and F S Fay 1995 Superresolution three dimensional im ages of fluorescence in cells with minimal light exposure Science 268 1483 1487 10 Castleman K R 1979 Digital image process ing Prentice Hall Englewood Cliffs NJ 11 Chen H J R Swedlow M Grote J W Se dat and D A Agard 1995 The collection processing and display of digital three dimen sional images of
19. assessing the system s resolu tion limit Optical theory includes so phisticated models of blur and with modern computer power we can apply such a model to digital images this is the basis for deconvolution Because of its importance in deconvolution the theoretical model of blur will be intro duced in greater detail below However we emphasize here that all imaging systems cause blur independently of whatever other forms of image degra dation are caused by the specimen or the electronics It is precisely this inde pendence of blur from other types of degradation that makes it possible to re move blur by deconvolution Bohren 6 points out that scatter glare and blur have the same physical cause namely the interaction of light and matter However the composition and arrangement of molecules in a giv en material whether glass water or protein gives each material its particu lar optical properties For our purposes what distinguish scatter glare and blur are the location where they occur and the possibility of generating a mathe matical model for them Because scat ter is a local irregular phenomenon oc curring in the specimen it is difficult to model although see Kam et al 33 for an elegant treatment of this problem By contrast because blur is a function of the microscope system and principal ly the objective lens it can be modeled with relative simplicity Such a model makes it possible to reverse
20. biological specimens p 197 210 In J Pawley Ed Handbook of Biologi cal Confocal Microscopy 2nd ed Plenum Press New York 12 Conchello J A 1998 Superresolution and convergence properties of the expectation maximization algorithm for maximum likeli hood deconvolution of incoherent images J Opt Soc Am A 15 2609 2619 13 Cordeiro J M K W Spitzer W R Giles P E Ershler M B Cannell and J H Bridge 2001 Location of the initiation site of calcium transients and sparks in rabbit heart Purkinje cells J Physiol 53 301 314 14 Cramer L and A Desai Immunofluores cence of the Cytoskeleton http iccbweb med harvard edu mitchisonlab Pages gen 1 html 15 Dernburg A F K W Broman J C Fung W F Marshall J Philips D A Agard and J W Sedat 1996 Perturbation of nuclear ar chitecture by long distance chromosome inter actions Cell 85 745 759 16 Edwards A W F 1972 Likelihood Cam bridge University Press Cambridge UK 17 Fay F S W Carrington and K E Fogarty 1989 Three dimensional molecular distribu tion in single cells analysed using the digital imaging microscope J Microsc 153 133 149 18 Femino A M F S Fay K Fogarty and R H Singer 1998 Visualization of single RNA transcripts in situ Science 280 585 590 19 Freiden B R 1975 p 177 248 In T S Huang Ed Topics in Applied Physics Pic ture Processing and Digital Filtering vol 6 Springer Verlag New York
21. but also on noisy or spherically aberrated images It begins with a theoretical PSF but adapts it to the specific data being de convolved In this regard it spares the user from the difficult process of ac quiring a high quality empirical PSF 26 37 Also because it adjusts the PSF to the data it can partially correct for spherical aberration However this computational correction should be a last resort it is preferable to minimize spherical aberration during image ac quisition see Aberrations and Arti facts section Deconvolution of confocal and multi photon images As one might ex pect it is also possible to restore confo cal or multi photon microscope images The combination of confocal micro scopy and deconvolution improves reso lution beyond what is attainable with ei ther technique alone 49 However the major benefit of deconvolving a confo cal image is not so much the reassign ment as the averaging of out of focus light This results in decreased noise e g see Reference 13 Deconvolution of multiphoton images has also been used to successfully remove image arti facts and improve contrast 52 In all of these cases care must be taken to use the appropriate PSF especially if the confocal pinhole is adjustable The in terested reader is referred to a previous discussion on the implementation of de convolution algorithms for laser scan ning microscopes 28 Implementation Processing speed
22. calculation is rapid about as fast as the 2 D deblurring methods discussed above However the utility of this method is limited by noise amplifica tion During division in Fourier space small noise variations in the Fourier transform are amplified by the division operation The result is that blur re moval is traded against a gain in noise Also an artifact known as ringing can be introduced see Aberrations and Artifacts section Noise amplification and ringing can be reduced by making some assump tions about the structure of the object that gave rise to the image For in stance if we assume that the object was relatively smooth we can eliminate noisy solutions with rough edges This approach is called regularization A regularized inverse filter can be thought of as a statistical estimator that applies a certain kind of constraint on possi ble estimates given some assumption about the object in this case smooth ness A constraint on smoothness en ables the algorithm to select a reason able estimate out of the large number of possible estimates that can arise be cause of noise variability Regularization can be applied in one step within an inverse filter 42 or it can be applied iteratively The result is usually smoothed i e stripped of higher Fourier frequencies Much of the roughness being removed here occurs at Fourier frequencies well be yond the resolution limit and therefore
23. epends to a great extent on image contrast 1 e the ability to distinguish signal from background Think of a picture of London on a foggy day even with the best high resolution optics a gray pic ture cannot be distinguished from a gray background Image contrast in the biological context depends mostly on specimen preparation fixation quality antibody penetration evenness of stain ing background fluorescence etc Optimizing specimen preparation can improve resolution much more dramat ically and at cheaper cost than optics or computers However assuming a high quality preparation the limit of resolu tion for any application is always de pendent on the PSF and the Rayleigh criterion gives us at least a basic handle on the size of the PSF ACKNOWLEDGMENTS We are indebted to many colleagues for helpful discussions including Tim Holmes Ernst Keller Steve Paddock Paul Johannes Helm Beat Ludin Michael Wussow Tom Donnelly David Carter Joachim Walter Mario Moronne and Chuo Lung Wang We also ac knowledge the collective assistance of the Confocal Microscopy Listserver http listserv buffalo edu archives confocal html and we thank all partici pants for their generous advice and dis cussions W W is funded in part by grants from the National Institutes of Health J R S is a Wellcome Trust Ca reer Development Fellow 054333 REFERENCES 1 Agard D A 1984 Optical sectioning mi croscopy cellul
24. epths will show different amounts of spherical aberration and z scaling Z scaling does not affect resolution or signal intensity it is simply a linear scaling of z axis distance measure ments by the ratio of the RIs of the mis matched media 25 To correct for this distortion z distances can be multi plied by a scalar compensation factor and some software packages offer this feature On the other hand spherical aberration is difficult to correct and therefore may be tempting to ignore However a little attention to this issue can bring big improvements in image quality especially under low light con ditions such as in living cells There has been interest in digitally correcting for spherical aberration by deconvolving with a spherically aber rated PSF 45 This requires deconvo lution software that does not automati cally preprocess the PSF to make it axially symmetric If this is the case then one can precisely match the aber ration in the image and PSF by having on hand a family of empirical PSFs with different degrees of spherical aberration and selecting the most ap propriate PSF for a given imaging con dition This type of computational cor rection may restore lost resolution to some extent but it cannot restore lost signal Therefore a better way to cor rect for spherical aberration is to elimi Vol 31 No 5 2001 nate it beforehand by optical means The following are optical methods to cor
25. fact is visible in the raw image then it must be caused by factors upstream from deconvolution i e by specimen preparation optics or electronics By adjusting the con trast and brightness of the raw image you can sometimes detect an artifact you would not have noticed initially If the artifact is not detectable in the raw image then some aspect of deconvolu tion is implicated In this case it may be useful to compare the results of decon volution by different kinds of algo rithms e g an inverse filter versus a constrained iterative algorithm 35 The PSF The quality of the PSF is critical to the performance of a deconvolution algorithm A noisy aberrated or im properly scaled PSF will have a dispro portionate effect on the results of de convolution This is especially true for the iterative methods because the PSF is repeatedly applied In all cases the distribution and extent of blurred light in the raw image must match the PSF If a mismatched PSF is used then arti facts may result or restoration quality may diminish In the next sections we discuss common problems with the PSF and ways to correct them Theoretical versus Empirical PSFs In many deconvolution packages the user can choose either a theoretical or an empirical PSF for image restora tion In general results are better if an empirical PSF is used Procedures for acquiring an empirical PSF can be found in 26 37 There are several rea
26. ferent depths in the specimen The definition of focus is somewhat subjective but a standard depth of field unit is usually defined as half the axial Rayleigh unit 31 d An NA 2 Eq 3 Another criterion that is sometimes used instead of the axial Rayleigh crite rion is the full width half maximum FWHM of the central bright region of the PSF Formulae for estimating the FWHM in confocal microscopy are given by Art and Goodman 3 These formulae are identical to those we have produced above for the Rayleigh crite rion in wide field microscopy We em phasize that these are rough expressions that give practical estimations They are not exact analytical formulae which would require vector wave theory We would also caution that any res olution criterion is not an absolute indi cator of resolution but rather an arbi trary criterion that is useful for comparing different imaging condi tions The Rayleigh criterion applies specifically to the case when we want to distinguish two self luminous ob jects In other contexts such as differ ential interference contrast DIC bright field or dark field microscopy other criteria will apply 31 In some applications such as localization of a moving object resolution below the Rayleigh limit is possible e g Refer ence 43 This highlights the fact that resolution is task dependent and cannot be defined arbitrarily for all situations In addition resolution also d
27. focused image Images are used by permission of Macmillan Press 1998 Vol 31 No 5 2001 BioFeature bor or unsharp masking are funda mentally 2 D We refer to them here as deblurring algorithms These algo rithms apply an operation plane by plane to each 2 D plane of a 3 D image stack For example the nearest neighbor algorithm operates on a plane z by blur ring its neighboring planes z using a digital blurring filter then subtracting the blurred planes from z 1 10 48 50 Multi neighbor methods extend this concept to a user selectable number of planes A 3 D stack is processed by ap plying the algorithm to every plane in the stack In this way an estimate of the blur is removed from each plane The deblurring algorithms are com putationally economical because they involve relatively simple calculations on single image planes However there are major disadvantages to these approach es First noise from several planes is added together Second deblurring al gorithms remove blurred signal and thus reduce signal levels Third features whose PSFs overlap in z may be sharp ened in planes where they do not really belong i e the apparent position of features may be altered This problem is particularly severe when deblurring single 2 D images because they can contain diffraction rings or light from other structures that will then be sharp ened as if they were in that focal plane Taken toge
28. hello 1999 Three dimensional imaging by deconvolution microscopy Meth ods 19 373 385 38 Oppenheim A V and R W Schafer 1975 Digital Signal Processing Prentice Hall En glewood Cliffs NJ 39 Oshiro M 1998 Cooled CCD versus intensi fied cameras for low light video applications and relative advantages Methods Cell Biol 56 45 62 40 Pawley J 1995 Fundamental limits in confo cal microscopy p 19 37 In J Pawley Ed Handbook of Biological Confocal Microscopy 2nd ed Plenum Press New York 41 Pawley J B 1994 The sources of noise in three dimensional data sets p 47 94 In J Stevens Ed Three Dimensional Confocal Microscopy Academic Press New York 42 Preza C M I Miller J Thomas and J G McNally 1992 Regularized linear method for reconstruction of three dimensional micro scopic objects from optical sections J Opt Soc Am A 9 219 228 43 Qian H M P Sheetz and E L Elson 1991 Single particle tracking Analysis of diffusion and flow in two dimensional systems Bio phys J 60 910 921 44 Salmon E D Protocols Immunofluores cence __http www unc edu depts salmlab protocolsimmunofluorescence html 45 Scalettar B A J R Swedlow J W Sedat and D A Agard 1996 Dispersion aberra tion and deconvolution in multi wavelength fluorescence images J Microsc 182 50 60 46 Schaefer B C M F Ware P Marrack G R Fanger J W Kappler G L Johnson and C R Monks 1999 L
29. imate will be the one that minimizes the error criterion therefore as long as the error criterion has not been minimized each new estimate is blurred again an error criterion is computed etc This process is repeated until the error criterion is minimized or reaches a defined thresh old The final restored image is the ob ject estimate at the last iteration Most algorithms incorporate con straints on the range of allowable esti mates One commonly used constraint is smoothing or regularization as dis cussed above As iterations proceed the algorithm will tend to amplify noise so most implementations suppress this with a smoothing or regularization fil ter Another common constraint is nonnegativity 1 This means that any pixel value in the estimate that be comes negative during the course of an iteration is automatically set to zero Pixel values can become negative ei ther because of Fourier transformation or a subtraction operation in the algo rithm The nonnegativity constraint is realistic because an object cannot have negative fluorescence It is essentially a constraint on possible estimates given our knowledge of the object s structure Other types of constraints include boundary constraints on pixel satura tion constraints on noise statistics and other statistical constraints Classical algorithms The first ap plications of constrained iterative de convolution algorithms to microscope i
30. in the il lumination of the specimen These events will be accentuated by deconvo lution but can be seen in the raw image Both problems can be minimized by re placing the arc lamp if it is old If lamp fluctuations continue to be a problem even with a new lamp then a direct measure of arc lamp power fluc tuation combined with a polynomial fit of summed pixel intensities plane by plane can be used to apply a correction to the raw image 11 Some form of this correction is included in most de convolution packages The second problem changes in the illumination pattern due to arc wandering is hard to correct by image processing but can be eliminated by installing a fiber optic scrambler between the arc lamp and the microscope 34 SUMMARY Deconvolution algorithms represent a very powerful tool for the biological microscopist Deblurring algorithms are 2 D methods that remove blur from images They produce results quickly but at the expense of signal strength and quantitative accuracy Image Vol 31 No 5 2001 restoration algorithms are iterative 3 D methods that are somewhat slower but preserve quantitative relationships in image data Like any method deconvo lution can produce artifacts if not used carefully but close attention to speci men preparation and image acquisition will usually eliminate them APPENDIX RESOLUTION CRITERIA Resolution in fluorescence mi croscopy is often assessed by means of an
31. intensity is normalized to the minimum and maximum values of its own image stack Pixel intensity is plotted as a function of distance along the z axis from the center of the bead 0 um The FWHM of the z axis intensity profile is 0 7 um in the raw image and 0 45 um in the restored image The actual object measures 0 1 um This modest increase in resolution in the restored image will only rarely reveal a biological structure that was not visible in the raw image B To quantify the improvement in contrast we summed all pixel intensities in each focal plane of the raw and restored image stacks The summed intensity of each plane is normalized to the min imum and maximum values of its image stack for comparison Summed intensity is plotted as a function of focal z axis distance from the center of the bead 0 um Restoration causes a movement of signal intensity from out of focus volume to in focus resulting in a major improvement in contrast and signal to noise ratio However the integrated intensity of the whole image i e the sum of the summed intensi ties of each plane is the same in the raw and restored images Vol 31 No 5 2001 BioFeature numerical aperture NA 1 2 1 4 where small manufacturing variations in the lens can cause minor aberrations in the symmetry of the PSF 36 45 A theoretical PSF does not reflect these lens specific variations and yields infe rior deconvolution results Furthermore an empirical PSF
32. ive cell fluorescence imaging of T cell MEKK2 redistribution and activation in response to antigen stimulation of the T cell receptor Immunity 411 421 47 Schaefer L H D Schuster and H Herz Generalized approach for accelerated maxi mum likelihood based image restoration ap plied to three dimensional fluorescence mi croscopy J Microsc In Press 48 Shaw P J 1993 Computer reconstruction in three dimensional fluorescence microscopy In D Shotton Ed Electronic Light Microscopy Wiley Liss New York 49 Shaw P J and D J Rawlins 1991 The point spread of a confocal microscope its measure ment and use in deconvolution of 3 D data J Microsc 163 151 165 50 Shaw P J and D J Rawlins 1991 Three di mensional fluorescence microscopy Prog Biophys Mol Biol 56 187 213 51 Sluder G and D Wolf Eds 1998 Methods in Cell Biology vol 56 Academic Press New York 52 Straub M P Lodemann P Holroyd R Jahn and S W Hell 2000 Live cell imaging by multifocal multiphoton microscopy Eur J Cell Biol 79 726 734 53 Swedlow J R J W Sedat and D A Agard 1997 Deconvolution in optical microscopy p 284 309 In P A Jansson Ed Deconvolution of Images and Spectra 2nd ed Academic Press New York 54 Swedlow J R J W Sedat and D A Agard 1993 Multiple chromosomal populations of topoisomerase II detected in vivo by time lapse three dimensional wide field mi croscopy Cell 73 97
33. k granules or other organelles whose RI is significantly different from their sur roundings In addition there may be RI gradients in the mounted specimen caused by heterogeneous mixing of the mounting medium This is particularly likely if the specimen is mounted in a medium that it was not previously im mersed in Computational methods for correcting such heterogeneities may be available in the future 33 However it will always be best to minimize these artifacts by thoroughly immersing the specimen in a mounting medium that is well matched to the specimen s own RI and by adjusting the RI of the immer sion medium to compensate for that of the mounting medium Horizontal and Vertical Lines Probably the most common artifacts of deconvolution microscopy are hori zontal and vertical lines in x y and z Usually these lines can be seen in the raw image when examined carefully meaning that they are not caused by de convolution but are enhanced by it However lines or bands can also be due to edge artifacts at the borders between subvolumes a form of ringing dis cussed above In any case such arti facts cannot be mistaken for biological structures and are easily removed Horizontal or vertical lines in the x y plane are often due to column de fects in the CCD chip used to record the image 31 51 If one pixel in the read register of the chip has a defect or if transfer to that pixel is less efficient then
34. labeled proteins and nucleic acids 9 18 24 35 46 54 Our goal in this article is to intro duce deconvolution to the working bi ologist at a level that is more practical than theoretical but more rigorous than a user s manual Because of space con straints we focus on the application of deconvolution to 3 D wide field images of fluorescent biological specimens Causes of Image Degradation Image degradation can be divided into four independent phenomena noise scatter glare and blur 7 56 The principal task that deconvolution sets for itself is to remove blur Decon volution algorithms can and do remove noise but this is a relatively simple as pect of what they do Noise is a quasi random disarrange ment of detail in the image which in its most severe form has the appearance of white noise or salt and pepper noise the kind of signal degradation seen in broadcast television during bad reception We call it quasi random because the statistical distribution of noise can be predicted if the mechanics of its source are known In digital mi croscopy the source is either the signal itself so called photon shot noise or the digital imaging system The me chanics of both sources are understood therefore the statistical distribution of noise is known Signal dependent noise is characterized by a Poisson distribu tion while imaging system noise usual ly follows a Gaussian distribution Thus
35. mages were based on the Jansson Van Cittert JVC algorithm a procedure developed by Van Cittert 19 for use in spectroscopy and adapted by Jansson 32 This algorithm was modified by Agard for application to digital micro scope images 1 21 Various imple mentations of Agard s modified algo 1082 BioTechniques rithm are currently marketed by Vaytek Intelligent Imaging Innova tions Applied Precision Carl Zeiss and Bitplane In addition Carrington and co workers developed a regular ized least squares minimization method 8 17 that has been marketed by Vaytek and Scanalytics These algo rithms use an additive or multiplicative error criterion to update the estimate at each iteration 37 48 50 53 Statistical algorithms Another family of iterative algorithms uses probabilistic error criteria taken from statistical theory 12 27 48 Likeli hood a kind of reverse of probability 16 is used in the maximum likelihood estimation MLE and expectation maximization EM algorithms imple mented by SVI Bitplane ImproVision Carl Zeiss and Autoquant MLE is a popular statistical tool with applications in many branches of science A related statistical measure maximum entropy ME not to be confused with EM has been implemented in image deconvolu tion by Carl Zeiss Statistical algorithms are more com putationally intensive than the classical methods and can take significantly longer to reach a solution see
36. mmetric above and BioTechniques 1077 BioFeature below the x y plane axial symmetry and rotationally about the z axis radial symmetry An empirical PSF can de viate significantly from perfect symme try Figure 1A This deviation or aber ration is caused by irregularities or misalignments in any component of the imaging system light path especially the objective lens but also other lenses mirrors filters apertures etc The higher the quality of the optical compo nents and the better the alignment the closer the empirical PSF comes to its ideal symmetrical shape Both confocal and deconvolution microscopy depend on the PSF being as close to the ideal case as possible Keller 36 provides a survey of all known types of PSF aberration The most common type of aberration well known to any professional microscopist is spherical aberration This is an axial asymmetry in the shape of the PSF with a corresponding increase in size partic ularly along the z axis Figure 1A The result is a considerable loss of resolution and signal intensity 25 In practice the most common cause of spherical aberra tion is a mismatch between the refrac tive indices of the lens immersion medi um and the mounting medium in which the specimen rests 25 26 We empha size the importance of minimizing this omnipresent aberration While deconvo lution can partially restore lost resolu tion 45 no amount of image process ing can
37. noise in the image can be re moved by appropriate filters and most deconvolution software includes pre processing routines that accomplish this The topic of noise in digital mi croscopy is discussed elsewhere 31 39 41 Scatter is a random disturbance of light caused by its passage through re gions of heterogeneous refractive index within a specimen The effect of scatter is a truly random disarrangement of the image detail No completely satis factory method exists yet to predict scatter in a given specimen However we can say that the degree of scattering depends on the thickness of the speci Vol 31 No 5 2001 men and on the optical properties of the specimen material The thicker the specimen the more scatter there is and the more heterogeneous the refractive index of the specimen material the more scatter there is Glare like scatter is a random dis turbance of light but occurring in the lenses or filters of the imaging system rather than within the specimen The level of glare in the modern microscope is minimized by the use of lenses and filters with antireflective coatings Blur is a nonrandom spreading of light caused by its passage through the imaging system and lenses The cause of blur is diffraction and an image whose resolution is limited only by blur is considered diffraction limited 7 31 36 This is an intrinsic limit of any imaging system and is the determining factor in
38. ntinuities can also arise if the spatial sampling of the raw image or PSF is too coarse if the image or PSF are noisy or if the PSF size or shape is inappropriate to the image The surest way to avoid ringing is by proper windowing of the Fourier trans form This is an implementation issue that some software packages have not incorporated but is increasingly the norm Ringing can also be avoided by using a finer sampling interval in the image or PSF by smoothing the image or PSF by careful matching of aberra tions in the PSF with those of the image or by adjustment of PSF parameters Ringing at edges of the image can be suppressed by simply cutting out the edges Many algorithms cut out a guard band of 8 10 pixels or planes around the entire image This operation requires blank space to be left above and below the fluorescent structures in the image Blank space is advisable anyway be cause blurred light from above and be low an object can be reassigned and contribute to the signal for that object However the blank space can be artifi cially created without loss of fidelity by adding interpolated planes to the top and bottom of the image stack The re sulting cost in memory and processing time can be minimized by performing the interpolation in Fourier space 2 Ringing at the edge of subvolumes is a similar issue Figure 4D Usually the amount of subvolume overlap can be increased so as to remove the arti fact
39. ntroduced at the edge of a subvolume arrowhead This artifact is not seen in the raw image C and is therefore not caused by the imaging system E and F Pyramid cell dendrites from sections of rat visual cortex fixed and then injected with Lucifer Yellow emission max 533 nm Each image is a single representative optical section from a 3 D stack E Raw image F Af ter iterative deconvolution In this case significant ringing occurs dark outline around fluorescent den drite arrowheads probably because of spherical aberration in the raw image which is not matched in the PSF Note also noise amplification seen as a mottling of background in the deconvolved image G Detail of a similar sample to panel E before deconvolution Raw and after deconvolution by increasing numbers of iterations 250 and 300 A small feature disappears black arrow This type of artifact can be avoided by correct use of PSF as described in the text Vol 31 No 5 2001 volved image then the first step is to make sure the PSF is as noise free as possible If this is the case the next step is to look at the filters in the algorithm If the filters are adjustable by the user then try to set the parameters such that image features larger than the resolu tion limit are maintained while smaller variations are suppressed Do not in crease smoothing too much because ex cessive smoothing will degrade resolu tion and contrast Disappearing and Ex
40. o noise ratio These properties are shown in Figure 2 Restoration improves image contrast and subsequently allows better resolu tion of objects without the introduction of noise that occurs in deblurring meth ods Figure 2A Perhaps more impor tantly for image analysis and quantita tion the sum of the fluorescence signal in the raw image is identical to that in the deconvolved image When properly implemented image restoration meth ods preserve total signal intensity but improve contrast by adjustment of sig nal position Figure 2B Therefore quantitative analysis of restored images is possible and because of the improved contrast often desirable When used in conjunction with wide field microscopy iterative restora tion methods are light efficient This is most valuable in light limited applica tions such as high resolution fluores cence imaging where objects are typically small and contain few fluoro phores 15 18 or in live cell fluores cence imaging where exposure times are limited by the extreme sensitivity of live cells to phototoxicity 9 24 46 54 PERFORMANCE ISSUES Now that we have explained the principles on which deconvolution al gorithms are based we can offer some technical insights on how they perform and how best to compare them Resolution and Contrast Improvement What kind of quantitative improve ment in image quality can be expected from iterative deconvolution We have attempted
41. on Jn J Pawley Ed Handboook of Biological Con focal Microscopy 2nd ed Plenum Press New York 29 Holmes T J and N J O Connor 2000 Blind deconvolution of 3D transmitted light bright field micrographs J Microsc 200 114 127 30 Inou S 1995 Foundations of confocal scanned imaging in light microscopy p 1 17 In J Pawley Ed Handbook of Biological Confocal Microscopy 2nd ed Plenum Press New York 31 Inou S and K R Spring 1997 Video Mi croscopy 2nd ed Plenum Press New York 32 Jansson P A Ed 1997 Deconvolution of Images and Spectra 2nd ed Academic Press New York 33 Kam Z B Hanser M G L Gustafsson D A Agard and J W Sedat 2001 Computa tional adaptive optics for live three dimension al biological imaging Proc Natl Acad Sci USA 98 3790 3795 34 Kam Z M O Jones H Chen D A Agard and J W Sedat 1993 Design and construc tion of an optimal illumination system for quantitative wide field multi dimensional mi croscopy Bioimaging 1 71 81 35 Karpova T S J G McNally S L Moltz and J A Cooper 1998 Assembly and func tion of the actin cytoskeleton of yeast relation ships between cables and patches J Cell Biol 142 1501 1517 36 Keller H E 1995 Objective lenses for con focal microscopy p 111 126 In J Pawley Ed Handbook of Biological Confocal Mi croscopy 2nd ed Plenum Press New York 37 McNally J G T Karpova J Cooper and J A Conc
42. on in image jum Figure 2 Comparison of deblurring and restoration methods Data from a 3 D image stack contain ing 70 optical sections each separated by 0 2 um recorded using same apparatus as in Figure 1 The ob ject is a XLK2 cell fixed with 3 7 formaldehyde stained with mouse anti tubulin and Cy5 conjugated donkey anti mouse IgG A A single focal plane from the 3 D stack is shown before any processing Original Data after deblurring by a nearest neighbor algorithm Nearest Neighbor and after restora tion by constrained iterative deconvolution Restored using the DeltaVision softWoRx Applied Pre cision deconvolution software Both deblurring and restoration improve contrast but the signal to noise ratio is significantly lower in the deblurred image than in the restored image Scale bar 2 um Arrow shows the position of the line plot presented in panel B B Plot of pixel brightness values along a hori zontal line shown by the arrow in panel A Original data gray line deblurred thin black line restored thick black line Deblurring or any other 2 D filter causes a significant loss of pixel intensity all across the image whereas restoration causes a gain of intensity in areas of detail Vol 31 No 5 2001 BioFeature tology optics or electronics When try ing to diagnose the cause of an artifact the first step to take is a careful compar ison of the raw image with the decon volved image If the arti
43. optical unit called the Rayleigh cri terion This criterion was originally for Resolved Not Resolved Figure 5 The Rayleigh criterion Mathematically generated light intensity profiles from two point mulated for assessing the resolution of 2 D telescope images but it has spread into many other areas of optics It is de fined in terms of the minimum resolv able distance between two point sources of light In a 2 D image two point sources are resolvable if their Airy disks are distinct According to the Rayleigh criterion two Airy disks are distinct if they are farther apart than the distance at which the principal maximum of one Airy disk coincides with the first minimum of the other Airy disk Figure 5 If the point sources are of equal wavelength then their Airy disks have the same diame ter and the Rayleigh criterion is then sources of light The profiles can be imagined to represent pixel intensities along a line through the Airy disc i e the x y image of the PSF at focus The Rayleigh criterion occurs when the maximum intensity of one PSF overlaps with the first minimum of the other For two points emitting at the same wavelength this corresponds to the radius or peak to minimum distance of the central bright region of a single PSF Adapted from image by Rod Nave Department of Physics and Astronomy Georgia State University Atlanta GA USA Used by permission Vol 31 No 5 2001 BioTechniques
44. ors The size of an image file is usually reported by the operating system How ever if in doubt it can be calculated by multiplying the total number of pixels in the image by the number of bytes pixel bit depth The bit depth is originally set by the camera which may produce 8 10 12 or 16 bits pixel Once the image is acquired the bit depth is determined by your software 1086 BioTechniques and computer system almost always it will be 8 or 16 bit 8 bits 1 byte Ina multicolor image each color must be stored and deconvolved separately so one must be careful to get the bit depth for the whole image not just for one color channel An example a 3 D stack where each plane is 512 x 512 pixels containing 64 optical planes with three colors at 8 bits pixel 1 byte pixel measures 512 x 512 x 64 x 3 x 1 50 MB The image file header may add slightly to this size ARTIFACTS AND ABERRATIONS After deconvolution the restored image may include apparent artifacts e g striping ringing or discontinuous cytoskeletal staining Sometimes these problems are related to data representa tion and will not occur with a different algorithm or software package They can also occur when processing para meters are not set appropriately for the raw image Finally artifacts are often not caused by computation but by his wee ETA Original Data Nearest Neighbor Restored 4 a ad g a 3 ps 1o Positi
45. ounting media cause spatial variations in the PSF especially in thick specimens At present all commercial software pack ages assume spatial invariance Howev er increasing computer power may make it feasible to vary the PSF through the image in the near future It may also become possible to correct spatial variations in the PSF by using a transmitted light image to map refrac tive index gradients in the specimen and adjust the PSF accordingly 33 Spherical Aberration A notorious kind of PSF aberration and one of the most difficult to combat is spherical aberration It involves an axial asymmetry in the shape of the PSF which both increases the flare of the PSF and decreases its brightness This is one of the primary causes of de graded resolution and signal loss in both confocal and wide field mi croscopy 25 26 Spherical aberration can be detected by focusing up and down through the specimen looking for asymmetry in the out of focus rings above and below a brightly fluorescent point like detail Figure 4 A D Alter natively it can be detected in an ac quired image stack when viewing the stack in x z or y z projection look for axial asymmetry in the flare of blurred light around a fluorescent structure A third way to detect spherical aberration is to acquire a PSF image from a fluo rescent bead mounted under similar op tical conditions as the specimen and to look for axial asymmetry in the flare of
46. owing the logic that a good estimate of the object is one that when convolved with the PSF gives back the raw image An advantage of this formulation is that convolution operations on large matrices such as a 3 D image stack can be computed very simply using the mathematical technique of Fourier transformation If the image and PSF are transformed into Fourier space the convolution of the image by the PSF can be computed simply by multi plying their Fourier transforms The re sulting Fourier image can then be back transformed into real 3 D coordinates For an introduction to Fourier trans forms in optics see Reference 22 Inverse Filters The first image deconvolution algo rithms to be developed were inverse filters Such filters along with their cousins the regularized inverse filters have been used in electronic signal pro cessing since the 1960s and were ap plied to images in the 1970s 10 In image processing software these algo rithms go by a variety of names includ ing Wiener deconvolution Regular ized Least Squares Linear Least Squares and Tikhonov Miller regu larization 10 37 48 55 An inverse filter takes the Fourier transform of an image and divides it by the Fourier transform of the PSF Since division in Fourier space is equivalent to deconvolution in real space this is the simplest way to reverse the convo lution that produced the blurry image The
47. ploding Features Perhaps the most annoying artifacts of deconvolution are an apparent loss of dim features or the blowing up of very bright ones These artifacts differ from excessive smoothing in that they affect specific features as opposed to the image as a whole Figure 4G In such a case try to compare the effects of different algorithms on the appear ance of the artifact e g inverse filter vs iterative algorithms classical vs sta tistical If you see the artifact only with one algorithm then the cause may be al gorithm instability or a specific effect of the algorithm on a particular specimen type If you see the artifact with several different algorithms then there are a va riety of possible causes If small dim features of the image are disappearing then the cause can be a combination of nonnegativity and smoothing filters This is likely if there is excessive noise in the image or the PSF or if conditions are favorable to ringing Noise and ring ing can cause negative pixel values so that the nonnegativity constraint is in voked and features are broken up If this happens then the smoothing filter may smooth away the remaining fragments causing the feature to disappear This can be avoided by reducing noise in the image and the PSF e g with longer ex posure times or by averaging by elimi nating PSF mismatches as described above and by properly adjusting the smoothing filter If bright featu
48. rect for spherical aberration i Us ing a dipping objective 1 e one that goes directly into the mounting medium without a cover slip This way the im mersion medium and the mounting medium are one and no RI mismatch is possible Distortion can still occur however because of RI mismatch be tween the specimen and the immersion medium or RI gradients within the spec imen itself ii Adjusting the RI of the immersion medium to compensate for the RI of the mounting medium 26 If the specimen is mounted in a medium of lower RI than glass e g any glyc erol or water based media then you should increase the RI of the immersion medium This method allows a relative ly aberration free imaging condition up to a limited focal depth 10 15 um from the cover slip for a NA 1 4 lens observ ing a specimen mounted in glycerol The RIs of a few common mounting media are listed by Bacallao et al 4 Cargille Laboratories Cedar Grove NJ USA supplies a line of Laser Liquid immersion media with specified RIs These can be mixed to any desired inter mediate value iii Using a specially corrected objective lens such as the high NA water immersion lenses now available for use with cover slipped specimens These lenses have a correc tion collar that compensates for RI variations between the lens and the specimen These lenses are expensive currently about 10000 However they may provide the only way to elimi nate
49. res are expanding then the cause is probably pixel satura tion When the raw image contains bright features restoration will tend to increase their brightness even further However in very bright features this can cause pixel values to exceed the Vol 31 No 5 2001 BioTechniques 1093 BioFeature highest value representable at a given bit depth i e these pixels will satu rate they will be assigned the maxi mum value The feature will then be come very bright will appear to flatten out and may expand in size This can be avoided by using a camera and soft ware that allow greater bit depth and an algorithm that internally stores data as floating point numbers This increases memory demands but prevents the oc currence of such artifacts Specimen Dependent Artifacts There have been reports both infor mal and published of deconvolution ar tifacts seen with some kinds of speci mens but not with others In particular Karpova et al have reported that fila mentous structures are broken up with one class of deconvolution algorithms but not with another 35 37 If a struc tural change is observed then it is al ways advisable to compare the decon volved and raw images If close inspection reveals that a structure evi dent after deconvolution does not occur in the raw image then make sure that the PSF parameters are set correctly and that the PSF is not noisy If the problem persists then the
50. restore lost signal See the Artifacts and Aberrations section for further discussion of this problem VARIETIES OF DECONVOLU TION ALGORITHMS Now that we have surveyed a small amount of optical theory we proceed to the business of this article namely de convolution algorithms what they are and how they work We will not review these algorithms in detail since pub lished works already do so 32 37 48 50 53 but we will explain their basic principles We divide the available deconvolution algorithms into two classes deblurring and image restoration Deblurring algorithms are fundamentally 2 D because they apply an operation plane by plane to each 2 D plane of a 3 D image stack In con 1078 BioTechniques trast image restoration algorithms are properly 3 D because they operate si multaneously on every pixel in a 3 D image stack A few more technical terms must be defined The object refers to the 3 D processed digital image or image stack acquired from the microscope Particu lar regions within the image are re ferred to as features Deblurring Algorithms pattern of light emitted by fluorescent structures in the microscope s field of view The raw image refers to an un The algorithms called nearest neighbor multi neighbor no neigh Figure 1 Effect of spherical aberration on the PSF A x z projections of two PSFs showing different degrees of spherical aberration
51. rings will appear where each dot was in the fo cused image Figure 1B 1 um If you collect a 3 D image of this specimen then you will record a PSF at each bead The PSF describes what happens to each point source of light after it passes through the imaging system The blurring process described above is mathematically modeled as a convolution The convolution operation describes the application of the PSF to every point in the object light emitted from each point in the object is con volved with the PSF to produce the im age This convolution causes points in the object to become blurred regions in the image The brightness of every point in the image is linearly related by the convolution operation to the fluo rescence of each point in the object 5 22 Since the PSF is 3 D blurring from the PSF is an inherently 3 D phe nomenon The image from any focal plane contains blurred light from points located in that plane mixed with blurred light from points in other focal planes The situation can be summarized by saying that the image is formed by a convolution of the object with the PSF Deconvolution reverses this process and attempts to reconstruct the object Aberrations in the PSF The PSF can be defined either theo retically using a mathematical model of diffraction 20 or empirically by ac quiring a 3 D image of a fluorescent bead Figure 1A A theoretical PSF generally has axial and radial symme try i e it is sy
52. sentative saves your im ages in a consensus file format because otherwise each software company will save to its own proprietary file format which often cannot be opened by com petitors software Currently the format with the widest circulation is a stack of sequentially numbered TIFF files each representing a focal plane not a projec tion of the 3 D image You will also need a few example image files and PSF files that you can run through dif ferent packages If you do not have ex perience collecting PSF files you may want to gain some before testing decon volution software that requires an em pirical PSF It is better to test with your own images and PSFs because the preparation you use along with the lens magnification noise level signal intensity spherical aberration etc will affect the quality of deconvolution tremendously Speed and Memory Usage Increasing processor speed RAM and bus speed all increase the speed of deconvolution During deconvolution a number of large arrays representing dif ferent forms of the image are stored si multaneously in RAM and are moved around inside the computer via the bus As a result RAM is critical for the rapid processing of 3 D images as is bus speed As arule of thumb your comput er should have at least three times as much RAM as the size of the image you wish to deconvolve Also computers with fast buses perform much better even with nominally slower process
53. sig nal to noise ratio and remove certain kinds of artifacts Most available pack ages include such operations and the users manuals of the packages should explain them We will return to them below in our discussion of artifacts Other implementation issues concern data representation Images can be di vided into subvolumes or represented as whole data chunks Individual pixel val ues can be represented as integers or as floating point numbers Fourier trans forms can be represented as floating point numbers or as complex numbers In general the more faithful the data representation the more memory and processor time it requires Thus there is a tradeoff between the speed of compu tation and the quality of restoration These issues are discussed below in the Artifacts and Aberrations section Summary Iterative restoration algo rithms differ from both deblurring algo rithms and confocal microscopy in that they do not remove out of focus blur but instead reassign it In this way out of focus signal is used rather than thrown away After restoration pixel in tensities within fluorescent structures increase However the total summed in tensity of each image stack stays the same as intensities in formerly blurred areas diminish Blur surrounding details 1084 BioTechniques of the object is moved back into focus resulting in sharper definition of object from background better contrast and improved signal t
54. sons not to preferentially use a theoreti cal PSF First although good theoretical models for the PSF exist 20 they are not perfect models and an empirical PSF contains information not available in theoretical models Second the theo retical PSF available in commercial software packages generally assumes perfect axial and rotational symmetry this means it may misfit the distribution of blur in the raw image This problem is most serious at high resolution e g 1088 BioTechniques Focus ym 8 4 3 fc amp E z 2 Focus pm Figure 3 Restoration significantly improves contrast and modestly improves resolution Data from image stacks of a subresolution fluorescent bead sample and data collection as in Figure 1A left Dot ted line raw image Solid line image restored using constrained iterative deconvolution as in Figure 2 For graphical purposes each intensity value has been normalized to the maximum value of its own image stack Without such normalization data from the raw image would barely be visible on the graph because pixel intensities near focus are so much brighter in the restored image than in the raw image Nonethe less the total integrated intensity of each image stack is the same in both cases A To quantify the im provement in resolution we measured the pixel intensities on a line parallel to the optical axis through the middle of the bead before and after restoration Each pixel
55. spherical aberration when imaging deeper than 15 um into a thick speci men The correction collar on such lens es should not be confused with the vari able aperture correction collars found on less expensive dark field lenses The Sampling Interval The proper sampling interval in x y and z is important for good deconvolu tion results The standard practice is to sample twice per resolvable element this conforms to the Nyquist sampling theorem which states that two samples per resolvable element are required for accurate detection of a signal 22 How ever the Nyquist sampling frequency is really just the minimum necessary for a reasonable approximation of the real signal by discrete sampling A higher sampling frequency gives better restora tion especially when using 3 D algo rithms In contrast 2 D deblurring algo rithms work best when the sampling rate is lower in z i e when the spacing be tween optical sections is equal to or greater than the resolvable element For fluorescence microscopy the re solvable element is often defined using the Rayleigh criterion see Appendix For example with the dye FITC emit ting at 520 nm an NA 1 4 oil lens and a mounting immersion medium with an RI of 1 51 the resolvable element is 227 nm in x y and 801 nm in z accord ing to the Rayleigh criterion To sam ple at the Nyquist frequency the sam pling interval should be twice this or 0 114 um in x y and 0 4 um in z
56. the PSF If the specimen is comparatively thick gt 10 um spherical aberration may be induced gradually as you image deeper into the specimen Therefore a bead mounted directly under the cover slip surface may not reveal spherical aberration For this reason some peo ple recommend acquiring the PSF im age from a bead located within a piece of tissue e g by soaking the tissue in a solution of fluorescent beads Howev er this PSF should not be used for de convolution because scatter from the Vol 31 No 5 2001 tissue will make the PSF noisier Spherical aberration is caused by an imperfection in the light path of the imaging system This can be due to de fects in the objective lens but more fre quently by mismatches of refractive in dex RJ in the optical media in front of the lens Objective lenses are usually corrected to minimize spherical aberra tion but only if they are used with the proper type of cover slip glass the proper cover slip thickness and the proper immersion and mounting media The optical properties of these materi als are essential for the proper focusing of light by the objective lens Quite commonly in biology the RI of the immersion and mounting media are not the same In these cases two distortions may occur spherical aberra tion and a scaling of z axis distances Hell et al 25 describe both phenome na in detail Both phenomena depend on focal depth so features at different d
57. the blurring Vol 31 No 5 2001 process and deconvolution uses this model to reverse or remove blur The Point Spread Function The model of blur that has evolved in theoretical optics is based on the concept of a point spread function PSF This concept is very important to deconvolution and should be clearly understood to avoid imaging artifacts Inou and Spring 31 and Keller 36 provide good introductions to the con cept of the PSF that are recommended for further detail Several Web sites also provide tutorial information on the PSF deconvolution and 3 D micro scopy generally Two sites we recom mend are http www microscopy fsu edu primer and http 3Dmicroscopy wustl edu josec tutorials To understand the PSF consider an infinitely small point source of light Because the imaging system collects only a fraction of the light emitted by this point it cannot focus this light into a perfect 3 D image of the point In stead the point appears widened and spread into a 3 D diffraction pattern The 3 D diffraction pattern of an ideal point source of light is the PSF Depending on the imaging modality being used wide field confocal trans mitted light the PSF has a different shape In a wide field fluorescence mi croscope the shape of the PSF is an ob long football of light surrounded by a flare of widening rings To describe it in three dimensions we apply a coordi nate system of three axes
58. ther these findings mean that deblurring algorithms improve contrast but they do so at the expense of decreas ing signal to noise ratio and may also introduce structural artifacts Two dimensional deblurring algo rithms may be useful in situations when a quick deblurring is needed or when computer power is limited They work best on specimens that have fluorescent structures distributed discretely espe cially in the z axis However these al gorithms cause artifactual changes in the relative intensities of pixels there fore morphometric measurements quantitative fluorescence intensity mea surements and intensity ratio calcula tions should never be performed after applying a 2 D deblurring algorithm Image Restoration Algorithms Image restoration algorithms deal with blur as a 3 D problem Instead of subtracting blur they attempt to re 1080 BioTechniques assign blurred light to an in focus lo cation This is done by reversing the convolution operation inherent in the imaging system If the imaging system is modeled as a convolution of the ob ject with the PSF then a deconvolution of the raw image should restore the ob ject However the object cannot be re stored perfectly because of the funda mental limitations inherent in the imaging system and the image forma tion model 1 The best we can do is to estimate the object given these limita tions Restoration algorithms estimate the object foll
59. this will appear as a line perpen dicular to the read register This line is then enhanced by deconvolution This type of problem can be corrected with a flat fielding correction also called shading or background correction 11 and is included in most packages Vertical lines seen in x z or y z views called z lines are due to vari ation in the response characteristics of pixels Each pixel has a slightly differ ent gain and offset from its neighbors In extreme cases there are bad pixels whose photon response deviates signif icantly from their neighbors In such cases the same pixel systematically de viates from its neighbors throughout a stack of images leaving a z line This problem can be corrected by flat field ing or if not then by special bad pixel routines that search out bad pixels and replace them with the mean of their neighbors criteria for these are dis cussed in Reference 11 In contrast horizontal lines in x z or y z views represent whole planes in the image stack that are uniformly brighter than their neighbors 11 This problem is due to fluctuations in the illumination system if the arc lamp power output changes during data collection then this variation will be recorded as a sys tematic difference in fluorescence in tensity between planes In addition the arc in a lamp can wander over the sur face of the electrodes causing time de pendent spatial heterogeneities
60. to answer this by measuring the size and brightness before and after deconvolution of a test object of known size Figure 3 The object is a 0 1 um subresolution fluorescent bead This is a nearly ideal specimen there is no out of focus signal coming from any other object and all aberra tions were carefully minimized before data collection In the raw image the bead measures 0 7 um along the z axis in the deconvolved image it measures 0 45 um Figure 3A This is a modest improvement in resolution which may reveal a biologically interesting struc ture in only a limited number of cases However the major change in the image is shown in Figure 3B This plot shows the integrated pixel intensity the sum of all pixel values in each focal plane as a function of focal depth There is significant out of focus inten sity at 2 um away from the bead before processing Restoration by iterative de convolution moves the majority of the out of focus intensity back to its focal plane of origin The result is a signifi cant improvement in image contrast making it easier to resolve and distin guish features in the image Comparisons between Algorithms Which iterative deconvolution algo rithm gives the best restoration A number of Web sites compare the re sults of different algorithms but these comparisons can be misleading for a variety of reasons First algorithms are often compared using images of syn thetic spherical
61. utomati cally interpolate the PSF sampling in terval to match the sampling interval of the raw image If this is not the case the PSF and raw image must be acquired at the same sampling interval When using a theoretical PSF the PSF parameters must be set appropri ately The PSF parameters are imaging modality NA emitted light wavelength or A pixel size and z step These para meters affect the size and shape of the PSF In general PSF size increases with increasing wavelength and with 1090 BioTechniques decreasing NA See Appendix on res olution criteria The pixel size and z step parameters are used for scaling the PSF with respect to the raw image If the size and shape of the theoreti cal PSF are not appropriate to the raw image then artifacts can result for sev eral reasons First the algorithm inter prets the sampling interval of the image in terms of the PSF size Second the PSF determines the size and shape of the volume from which blurred light is reassigned If this volume does not cor respond to the true distribution of blur in the image then artifacts result This can happen if there is mis scaling or aberration of the PSF It can also occur if the real PSF of the image has an aberration that is not matched by the al gorithm s PSF When using a theoretical PSF there are cases where paradoxically a better result may be obtained if the PSF is too large A possible explanation is that the

A Workingperson`s Guide to Deconvolution in Light Microscopy

Contents

Download Pdf Manuals

Related Search

Related Contents