Phase 3 data release description
Contents
1. The possible reasons for the existence of such datasets are manifold e Aborted templates then the data are likely to be useless e Extended sources sometimes observations of extended sources are made with the OFF SET template in a mapping mode without OBJECT SKY pairs unfortunately the acquisi tion template then does not encode properly the information of the extended nature of the source in the headers nor of the intended strategy of the observations The proper reduc tion of these data would be possible with knowledge of the science case which however is beyond the scope of this project Furthermore there are occasionally processing failures A typical case is a high number of nodding input frames 20 or more for which the pipeline sometimes after long iterations did not con verge No major attempts were made to understand and fix the situation unless it was obvious Note that only in very rare cases we have rejected a product that was successfully created by the pipeline Even in case of heavy saturation we have decided to deliver the IDP to the archive since the saturated pixels are marked in the QUAL column Table 5 and the unsaturated regions might still be useful Also products with large parts of negative flux have not been rejected Instead the QCFLAGs have been set accordingly see Section Data Quality Table 4 Only in the following cases we have rejected a product e the template has been aborted AND another issue2. NIR chromatic energy distribution Slit losses In some cases PIs have taken their spectra with both the wide 5 slit for photome try and a narrow slit for resolution In these cases it is possible to measure the slit losses for the prevailing seeing and transparency conditions Because of their photometric quality at least with no slit losses we have marked the wide slit data with the quality bit 0 for good photometry while in all other cases they are flagged as 1 for unknown photometric quality depending on seeing slit losses could still be negligible The flux calibrated spectra are provided in units of erg cm2 s A Feige110 comparison XSHOOTER catalogue blue XSHOOTER 2012 06 03 green flx_std catalogue 0 04 iain Figure 4 The XSHOOTER Feige 110 spectrum from 2012 06 03 all three arms processed with the appropriate master response curves and plotted in blue compared to the tabulated energy distribution in green together with the 10 confidence range in purple which appears in the log plot as 0 04 lambda nm Feige110 comparison XSHOOTER UVES red UVES blue XSHOOTER 2012 06 03 green flx_std catalogue 0 04 Wis lambda nm Figure 5 The same spectrum as above compared to a UVES spectrum of Feige 110 overplotted in red the UVES spectrum comes in several chunks and extends to about
3. Observatory All master calibrations used here were certi fied meaning checked for quality and proper registration of instrument effects For XSHOOTER the selection pattern for master calibrations was complete from the begin of operations on cover ing both Service Mode SM and Visitor Mode VM science data and all settings Note that the fact that a certain set of XSHOOTER raw files does not have a product in this release does not necessarily imply a quality issue with the raw data There are sometimes acquisition pat terns for which the processing jobs can be predicted to fail e g NODDING with one raw file see Rejected or failed processing below In some of those cases the raw data would probably process fine with a fine tuned strategy In general the master calibrations were processed with different earlier pipeline versions than the science data in this stream The most significant change in the XSHOOTER pipeline was related to version xshoo 2 0 and higher reflecting the results of a major science grade review in 2012 and 2013 This change was implemented in 2013 01 All science data acquired later than that date have been processed with essentially the same pipeline version as their master calibrations All science data acquired earlier have still been processed with the current pipeline version but their master calibrations were processed with earlier versions See Master calibrations below for a discussion of the main dif
4. and flagged blue in the lowest panel of the QC plot Features Issue with pick noise At certain times e g during most of 2010 03 the VIS arm detector showed a strong pick noise pattern that the pipeline cannot remove from the data The impact is a high frequency noise that is easily spotted in the spectral windows plot 2 See an example in Fig ure 17 ASHOOTER QC Report SCIENCE_VIS Date 2010 03 07 mode SLIT STARE_VIS slit 04x11 bin 1x1 RuniD 064 C 0052 A product r SHOOT 2010 03 08T09 18 09 502_0000 fits B 367046 target HD 130437 Q fluxed spectrum reduced S N qual wavelength in nm spectral windows FLUX 10 16 erg cm 2 A sec tsbastvtnacctwstnceres pag ot Nah i del EPI PIT MAT Airs i wat kek area a Aidi ee ee AM aie al 9 Eliladldidai lali lililililklililililkililililililililidl i red S N Figure 17 Pick noise in VIS data which stands out prominently in the close up plots Steps at begin of order Occasionally there are a few steps at the begin of an order apparently induced by an imperfect flat fielding Figure 18 19 3 i S 3 ry i al a i Figure 18 Steps at begin of an order In this example the feature at 360nm is rather pronounced Such features appear at the begin of an order the orders are best visible in the S N panel Find more documentation about features and issues on line http www eso or c PHOENIX processing html Data Format
5. extension of the 2D ancillary spectra these regions can be easily spotted the saturated pixels have the bit code 4096 for saturated pixels In the extracted main output file the QUAL column has the bit code 2 19 524288 missing data see Figure 15 Outside of these gaps the data might still be useful b fluxed spectrum reduced 5 N qual Q wavelength in nm full lnfo rebin 0 45nm FLUX 10 16 erg cm 2 A sec Lr elle Figure 15 Spectral gaps due to saturation They are marked by the quality flag 2 19 blue in lowest panel of the QC plot In the NIR saturated raw data pixels might have negative pixel values Affected spectral pixels are flagged by bit 2 21 2097152 see Figure 16 and the impact on the spectrum are troughs embedded between peaks unsaturated regions Negative flux See Localization and extraction section for possible origins of negative fluxes in the spectra In the NIR negative fluxes might appear in raw data and are the result of saturation Figure 16 Also it might happen in rare cases that the reset anomaly causes negative pixels which then are perfectly valid except for the wrong sign 18 fluxed spectrum reduced S N qual wavelength in nm b full Info rebin 2 nm a n o oH pag Anec FLUX 10 16 er Figure 16 Spectral regions affected by negative saturated raw pixels in NIR spectra The affected regions are visible as artificial depressions
6. showed up e g saturation or negative fluxes e the product column FLUX is entirely 0 Flux Calibration One of the main improvements of the XSHOOTER pipeline over time was the flux calibration In 12 Remember that XSHOOTER STARE products are not stacked but come as one product per raw file 13 Such data could probably be processed interactively e g with Reflex itially before pipeline version 2 0 the response curves as derived from almost nightly flux stan dard stars had a poor algorithmic quality e g uncontrolled splines and were almost useless With the review and improvements of version xshoo 2 x these issues have been solved and the re sponse curves are of good quality and stability Nevertheless we decided against using nightly response curves The main reason is that we are aiming at a flux calibration but not at a photometric calibration XSHOOTER data have been taken under conditions which were not photometrically controlled and depending on the slit width used they suffer from slit losses Without the goal of a photometric calibration the use of daily close in time response curves would only be reasonable if instrument components were observed to vary on short timescales This is however not the case We followed therefore the same scheme as for UVES and derived a set of master response curves These are based on carefully compiled individual response curves The selection criteria were e good transparency in or
7. the master response curves were constructed Wavelength scale The XSHOOTER Echelle products are wavelength calibrated The wavelength scale is topocentric no corrections for barycentric or heliocentric motion have been applied The corresponding values have been calculated by the pipeline and are stored in the header HIE RARCH ESO QC VRAD BARYCOR and HELICOR in km s Telluric absorption No correction for telluric absorption lines has been applied although for almost all products one or more telluric standard stars have been measured within 2 hours of the observation and within 0 1 in airmass The telluric correction is not provided by the pipeline because it requires some assumptions on the science spectrum The user may want to download the telluric standard stars from the ESO archive through the calSelector service process them with the Reflex tools and apply the products to the science spectra Alternatively users may want to visit the ESO skytools web page for appropriate tools Bad pixel map We used static bad pixel maps for the processing selected per arm and binning 10 http www eso org sci software pipelines skytools 11 The pipeline also supports dynamic bad pixel maps derived from flats their advantage being a closer time These are from 2013 and have been carefully reviewed for the pipeline versions xshoo 2 x Normal daytime calibrations and AFC frames automatic flexure compensation During the lifetime of XSHOO
8. 1000nm The green line is again the tabulated energy distribution 11 Data Quality Master calibrations All used master calibrations have been quality reviewed and certified at the time of acquisition as part of the closed QC loop with the Observatory which also includes trend ing The most important parameters for the quality of the IDPs are the SNR of the master flats and the rms of the dispersion solution The SNR of the master flats was always high enough to be dominated by the fixed pattern gain noise which is important to not compromise the SNR of the science data The rms of the wavelength dispersion was lower in the first few years with pipeline versions xsh 1 x than later due to a lower number of lines found SNR There is a column SNR in the product table that is calculated from the signal and the corresponding error It has no independent information but is provided for convenience Its mean value is written as header key SNR There are also the header keys HIERARCH ESO QC FLUXn SN n 1 2 or 3 that describe the SNR in various spectral windows defined in the comment of the key XSHOOTER VIS ARC SLIT parameters FULL range 1638 days QC data range 2009 10 01 2014 03 25 2011 61 01 2010 01 01 2010 07 01 2011 07 01 2012 01 01 2013 01 01 2012 07 01 2013 01 01 2014 01 01 2010 01 01 2010 07 01 2011 07 01 2012 07 01 vt 2013 0
9. 4 05 08T10_27_01 077 fits Completed 100 1 3MB of 1 3MB 752 3KB s ADP 2014 05 08T10_27_00 563 fits Completed 100 19 0MB of 19 0MB 7 0MB s ADP 2014 05 08T10_27_00 890 png Completed 100 139 6KB of 139 6KB 152 5KB s ADP 2014 05 08T10_27_00 897 txt Completed 100 370 0bytes of 370 0bytes 244 0bytes s README_111717 txt Completed 100 3 4KB of 3 4KB 4 0KB s 09 22 08 ADP 2014 05 08T10 27_00 890 png Downloading gt 09 22 09 Completed 139 6KB 152 5KB s 09 22 09 README 111717 txt Downloading gt 09 22 10 Completed 3 4KB 4 0KB s 09 22 09 ADP 2014 05 08T10 27_00 897 txt Downloading gt 09 22 10 Completed 370 0bytes 244 0bytes s Speed 0 0bytes s Completed 27 of 27 files failed 0 115 5MB of 115 5MB Conc Downloads 1 5 26 Figure 1 List of downloaded files on the ESO Download Manager The first useful step to do is the organization of the data on the local disk The README file coming with every download contains the information necessary to properly rename the files and estab lish their association The fundamental unit of the renaming process is to capture column 2 in the README file technical name starting with ADP and column 3 original name starting either with XS_ for the fits and txt files or with rXSHOO r SHOO for the graphics and use something like mv 2 3 The file type is in column 4 find the possible values listed in Table 6 The IDP is of type SCIENCE SPECTRUM while the other files are
10. 7 01 t 2014 01 01 2011 01 01 2012 01 01 2013 07 01 resolution oe i ee a A O T 8 amp o pa 3 O wi U 56000 mjd_obs mjd_obs diffy_med diffy_rms pep ers Beas a EE T EE TEAT r i a aa ncaa Aa E s one q b ta diffy_med oO m 56000 mjd_obs mjd_obs powered by QC www eso org HC created by trendPlotter v3 3 on 2014 03 26T17 49 42 Figure 6 Evolution of the two QC parameters number of lines used for the dispersion solution 1 and rms 4 for the VIS arm 8 SPEC_RES The header key SPEC_RES contains the nominal resolving power as derived from the arclamp calibrations of the same slit width The actual resolution of the science data has not been measured but could actually be higher depending on seeing conditions and exposure time A better estimate would be available from the half width of the telluric absorption lines QC flag The header key QCFLAG in the XSHOOTER products describes automatically assigned quality flags It is composed of seven binary bits For each bit the value 0 means no concern 17 Check out for more under the XSHOOTER link of http www eso org qc ALL daily gc1 html 18 See http www eso org qc XSHOOTER reports FULL trend report ARC SLIT time VIS FULL html for the original plot and replace VIS by UVB or NIR to check the other arms 19 For a point like source the actual resolution is between the valu
11. ESO Phase 3 Data Release Description for Internal Data Products Data Collection Release Number Program Type Data Provider Document Date Document version Document Author On line version XSHOOTER_ECHELLE 1 Internal ESO Quality Control Group lt 09 05 2014 gt 1 0 Reinhard Hanuschik http www eso org qc PHOENIX XSHOOTER processing html Abstract This is the release of reduced 1D spectra from the XSHOOTER spectrograph ECHELLE SLIT mode as opposed to the IFU mode All spectra have been reduced under the assumption of point like sources This release is an open stream release it includes so far archived XSHOOTER data and will be continued into the future The processing scheme is as homogeneous as possible The selected data cover the vast majority of the entire XSHOOTER data archive from the begin of operations in October 2009 until present The data have been reduced with the XSHOOTER pipe line version xshoo 2 3 and higher All data have their instrument signature removed they have been de biased flat fielded wavelength calibrated order merged extracted sky subtracted and finally flux calibrated Telluric absorption has not been corrected for The pipeline output products come in the ESO 1D standard binary table along with some ancillary files The processing is performed by the Quality Control Group in an automated process The pipeline processing uses the archived closest in time quality controlled and cert
12. Files Types The primary XSHOOTER Echelle product is the flux calibrated spectrum in binary spectroscopic data format ORIGFILE product category Description names Starting HIERARCH ESO PRO CATG with XS_SFLX SCI_SLIT_FLUX_IDP_ lt arm gt flux calibrated product arm UVB VIS or NIR The primary product has an ancillary FITS file delivered with it useful for quality assessment in 2D image format XS_SRE2 SCI_SLIT MERGE2D_ lt arm gt Order merged 2D product before extraction and flux calibration arm UVB VIS or NIR The following naming convention applies to the ORIGFILE product name the ancillary FITS file has the same name with XS_SRE2 replacing XS_SFLX e g XS SFLX 922346 2013 08 09T23 33 54 611 1 5x1ll1 1x1 VIS NOD fits has the components ORIGFILE XS SFLX 922346 2013 08 S1 5x11_1x1_VIS_NOD fits component 09T23 33 54 611 refers to XSHO product type timestamp of setup string OTER SFLX where first raw file S 1 5x11 for the slit S stands for archival 1x1 for the binning dropped Science and in NIR FLX for VIS for the arm fluxed or NOD for the technique 20 ORIGFILE XS SFLX 922346 2013 08 S1 5x11_1x1_VIS_NOD fits component 09T23 33 54 611 RE2 for reduced 2D The ancillary files always delivered with the primary IDP have the following ORIGFILE names Table 6 Naming conventions of ANCILLARY files example ANCILLARY 2ZDSPECTRUM XS_SRE2_998129_2013 09 Same as primary ID
13. IT 0 4x11 ares mean_S N 197 rms_r 1124 9 SEEING 1 07 created 2014 03 24T17 17 29 by xshooter_ph on muc06 Figure 8 Main preview plot featuring 1 from top to bottom the flux calibrated IDP column FLUX both at original resolution in black and as smoothed version in red the unfluxed spectrum column FLUX_REDUCED the SNR log Q where Q is the pixel quality flags 2 small spectral windows at full resolution 3 a set of related QC parameters bottom 4 labels for identification top XSHOOTER QC Report ALL arms Date 2010 03 11 technique SLIT_NOD RuniD 084 C 0952 A products r SHOOT 2010 03 12T09 09 22 296_tpl_O019 fits UVB NIR OB 387834 target PDS 399 gt fluxed spectra all 3 arms rebinned to 0 5nm NIR 2nm Slit collapsed 2D UVB VIS NIR Figure 9 Second overview plot featuring the smoothed signal in all three arms 1 and the collapsed signal in cross dispersion direction as derived from the 2D product 2 This plot can be used to check the flux scale effect of slit losses and the cross dispersion profile It also has the slit widths labelled 15 XSHOOTER IDPs mean SNR vs mean product counts 5214 data points 600 NOD VIS arm 500 lt low gain mode 400 tx i high gain mode 300 high gain 0 9x11 c zZ 73 c v E 10000 20000 30000 40000 50000 mean reduced Figure 10 Mean SNR vs mean reduced spectrum for all NODDING products of the VIS ar
14. P 26T08 49 39 394_S5 0x11_1x1_UVB_ST starting with XS_SRE2 for A fits science reduced 2D ANCILLARY PREVIEW rXSHOO 2013 09 Technical filename of 26T07 56 43 243_tpl_0000 fits_1 png primary IDP with _1 png appended the VIS products have also a second preview file called _2 png ANCILLARY README XS_SFLX_998167_2013 09 Same as primary IDP with 261T07 56 46 364_S0 6x11_NIR_NOD txt fits replaced by txt The user may want to read the ORIGFILE header key and rename the archive delivered FITS files File structure The primary XSHOOTER product XS_SFLX comes as binary FITS table in multi column format The columns are labeled as follows Table 7 Internal structure of the XSHOOTER IDP 1 WAVE wavelengthinmnm O O O S O alll nna extracted wavelength calibrated sky subtracted flux calibrated SCIENCE signal in physical units erg s t cm2 A 1 3 ERR_____ Errorof FLUX same units O O Z O Z O O 4 QUAL Quliyfag O S O 5 SNR_______ Signabtomnoise ratio FLUX ERR _ _ O Z O O O FLUX_REDUCED extracted wavelength calibrated sky subtracted but not fluxed SCIENCE signal in counts Lan ERR_REDUCED corresponding error not fluxed The difference between FLUX and FLUX_REDUCED is the flux calibration The FLUX_REDUCED data might become useful if the user wishes to apply his own response curve instead of the mas ter response curve The SNR column is provided for convenience The QUAL column contains the
15. TER three different association schemes were used for ORDER_EDGE DISP_TAB and XSH_MOD_CFG_OPT master calibrations e 2009 10 01 2011 03 08 the products from the daytime calibrations have been taken i e no AFC data cf Table 1 e 2011 03 09 2012 06 30 only the products from the AFC frames XS_POTA XS_PDTA XS_PMCA have been used e 2012 06 30 now products from both daytime calibrations and AFC frames are associated A careful investigation of the shifts in X and Y direction to be compensated has shown that an effective difference in quality was not noticeable Hence we have decided to keep the historical association scheme and process the data accordingly Master calibration names and recipe parameters used for reduction The product header contains a list of all used master calibrations look for keys HIERARCH ESO PRO REC1 CAL lt n gt NAME and CATG with the index n The pipeline parameters and their values are listed as HIERARCH ESO PRO REC1 PARAM lt n gt NAME and VALUE Products The XSHOOTER pipeline creates a large number of intermediate and final product files The final IDP in the spectroscopic data format combines information from the following products e extracted spectrum de biassed flat fielded sky removed extracted wavelength calibrated rebinned but not fluxed including errors and quality flags flux calibrated spectrum same as above but also fluxed including er
16. ap always provided MRESPONSE_MERGE1D_SLIT_ XS_MRSP optional but Static response curve used for flux lt arm gt always calibration derived from selected provided sets of standard star measurements removes lamp response and remaining instrument signature ATMOS_EXT_ lt arm gt XS_GEXT optional but Static table with atmospheric always extinction provided SKY_LINE_LIST_ARM XS_GSLL optional for Static table with sky line positions STARE mode arm1 UVB or VIS arm UVB VIS NIR if missing pipeline would fail Flux calibration all three modes For the flux calibration there were initially two options for almost every night used for XSHOOTER science there exists a flux standard measurement On the other hand XSHOOTER is neither designed for nor operated under strictly photometric conditions and for that reason the flux calibration cannot aim at photometric accuracy but instead at remov ing the remaining instrumental signature mainly from the spectral energy distribution of the flat lamp For this purpose master response curves are sufficient which are carefully compiled from individual nightly response curves giving an accuracy of the flux scale in the science products of about 10 differentially meaning the instrumental effects are removed with that accuracy Still the photometry could be off by larger amounts due to uncontrolled transparency variations and slit losses See below for a discussion of the way
17. aster calibrations used for data reduction Type pro catg Mandatory content optional MASTER_BIAS_ lt arm1 gt XS_MBIA_ optional but master bias created from 5 raw always bias frames removes bias level provided and bias structure MASTER_DARK_NIR XS_MDRK optional master dark created from 1 or 3 mandatory raw dark frames removes dark for STARE level and dark structure ORDER_TAB_EDGES_SLIT_ lt arm gt XS_POES mandatory table with traces of order edges ORDER_TAB_AFC_SLIT_ lt arm gt XS_POTA optional in same corrected for flexure addition MASTER_FLAT_SLIT_ lt arm gt XS_MFSL_ mandatory master flat created from 5 VIS or 10 UVB NIR raw flats used for defining order edges removing gain noise blaze 9 Please check Sect 7 of the Pipeline User Manual for the description of calibration data Type pro catg Mandatory content optional function slit noise introducing lamp energy distribution In UVB two lamps are used a D2 anda quartz lamp DISP_TAB_ lt arm gt XS_PDT2 optional dispersion table DISP_TAB_AFC_ lt arm gt XS_PDTA optional in same corrected for flexure addition XSH_MOD_CFG_OPT_2D_ lt arm gt XS_PMC2 mandatory physical model configuration file XSH_MOD_CFG_OPT_AFC_ lt arm gt XS_PMCA_ optional in same corrected for flexure addition SPECTRAL_FORMAT_TAB_ lt arm gt XS_GSFT Static spectral format table BP_MAP_RP_ lt arm gt XS_GBPM optional but Static bad pixel m
18. der not to introduce additional bias from unstable atmospheric conditions e good algorithmic quality In order to make use of the xshoo 2 x improvements we started the compilation with data from 2013 01 and later The master response curve for the NIR arm averaged from all compiled re sponse curves and its 10 envelope are displayed in Figure 2 We then took this master response curve and compared it to individual response curves from the earlier epochs Again the good in the above sense response curves were Selected interactively 16 and averaged That average together with the 10 envelope is plotted for the UVB arm in Figure 3 Then the existing master response curve was multiplied by a fudge factor to account for differences in the normalization procedure for the master flats at that time it is displayed as the blue line It is obvious that adopting that master response curve originally defined for the 2013 time range also for the earlier epochs is possible within the 10 range It has small systematic variations possibly due to instrumental drifts change of coating properties or flat field lamp energy distribution and possibly also to different pipeline versions but it is well defined in terms of spline robustness and also step size in particular in the VIS and NIR telluric windows We have tested the validity of our approach with XSHOOTER observations of the flux standard star Feige 110 These data have been obser
19. e gt As given by the observatory staff to assess the match with user defined constraints for SM data was aborted and an additional quality issue occurred see section Data Quality For observing mode STARE we have processed every single input raw file separately While processing of STARE data could also be done with all files from a single template stacked together it is not always reasonable to do so since there might be cases when the template sequence was de signed to follow a time variability pattern Since it is impossible to automatically recognize this case this case it was decided to process the STARE data always one by one A manual co addition or stacking of these IDPs by the user is straightforward Release Notes Depending on the observing mode the data reduction uses the standard XSHOOTER pipeline reci pes xsh s ired slit stare xsh Scired slit nod orxsh scired slit oriset Find a description of these science recipes in the Pipeline User Manuals Find the pipeline version used for processing in the header of the product file under HIERARCH ESO PRO REC1 PIPE ID or in the QC database key pipe_id The version for the initial dataset the historical batch until 2013 12 31 was xshoo 2 3 12dewv1 despite its name this was a stable version All recipe parameters including the extraction methods were set by the pipeline defaults The only exceptions are purely technical Find more details in the User Ma
20. e corresponding to the seeing disk exposure time integrated and the value corresponding to the slit width 12 Table 4 QC flags is 0 otherwise 1 HIERARCH ESO 1 seeing seeing lt slit width flux losses if seeing disk QC SEEING mean truncated of TEL FWHM START and TEL FWHM END 2 airmass below threshold value higher probability of flux losses QC AIRM MEAN 1 8 for higher airmass if the ADC mean of did not work properly TEL AMBI START and TEL AMBI END calibration otherwise always no 4 mean gt 0 1 If very small or negative gt QC COUNTS MEAN counts extraction went wrong or just mean of IDP indicates low signal column FLUX_REDUCED 5 mean gt 0 If negative gt extraction went QC SNR MEAN column SNR 6 saturated number of saturated quality issue if too many QC NPIXSAT pixels pixels below 1000 saturated pixels UVB VIS or 20 000 NIR 7 aborted TPL EXPNO if not data reduction is likely TPL EXPNO template TPL NEXP suffering e g from upcoming TPL NEXP clouds during observation this is mean reduced counts not flux The final QC flag is composed of all binary values For instance 0000000 is a pristine product 1011100 is a product that might have some slit losses and also has negative flux indicating a localization or extraction issue The seeing check 1 is a rather coarse quality check and meant to be indicative only the two header keys TEL FWHM START and END refer to th
21. e first raw file The same is true for the airmass check 2 the underlying values are also from the first raw file Taken together they indicate mainly how reliable the flux scale is With 1 and or 2 being 1 it becomes likely that the flux scale for spectra from the different arms may show jumps Only if check 3 returns 0 the flux calibration is likely to be free from slit losses but still not necessarily be of photometric quality since the transparency is not strictly controlled for XSHOOTER observations Flags 4 and 5 assess the extraction quality They are not entirely independent but not redundant either Their underlying values are averaged across the whole spectral range Flag 6 indicates possible saturation issues while 7 is a technical indicator of template abortion which might alert the user to unusual observing conditions that might explain abnormal product properties Template abortion almost always indicates degradation of observational conditions or instrumental problems and results often in data sequences which are incomplete at least for good automatic processing or violate quality assumptions of the pipeline In case there were no quality issues discovered other than flag 7 being set we have delivered the product as IDP If flag 7 was found to be set and another quality issue out of 4 5 or 6 was found we have decided to not deliver the product since these products were always found to be useless Science product
22. eter values meaning several times during the night with or without comments In those cases the user should always carefully check that the listed comment applies to the IDP with the closest pre vious timestamp QC related information QCFLAG e g 0010010 QC flag composed of 7 bits see Table 4 QC_COMMENT Free text Automatically added comment if a quality issue is discov ered by the processing system there are no human provided comments All XSHOOTER spectra come separated by arm like the raw data Typically there are three spectra from the UVB VIS and NIR arm but there are exceptions e in STARE mode the number of input raw files and therefore the number of output prod uct files 2 might be different from arm to arm e in all modes it might have happened that a template was aborted or the pipeline processing of one product failed in which case less than three products are available The three products have essentially no spectral gap The UVB product has the nominal signal trun cated beyond 556nm to suppress a spurious pseudo absorption at 570nm arising from artifacts in the flat field the suppression is controlled by the parameter cut uvb spectrum TRUBE Rejected or failed processing Under the following conditions the pipeline processing always fails and no data products exist e NODDING data sets with only one input file e OFFSET data sets with no OBJECT frames e OFFSET data sets with no SKY frames
23. ferences Science data with the PROGID starting with 60 or 060 have been de selected considering them as test data Data taken at daytime with obviously wrong SCIENCE tag have been ignored Other wise the header tag SCIENCE has been blindly accepted from the raw data originally defined by the PI thus including sometimes standard stars intended by the PI for use as flux calibrator but labeled wrongly This is often evident from the OBJECT header key Also there are rare cases when test observations were executed under the SCIENCE label Some very short exposures with no sig nal fall into that category but have not been suppressed All data have been processed up to the flux calibration step The flux calibration was done with master response curves There is no raw science data selection based on quality Likewise we have not considered OB grades the observations might have any grade between A and D if taken in SM or X in VM The availability or non availability of a particular file in this release does not infer any claim about the data quality For the observing modes NODDING and OFFSET we have combined all raw files taken in one tem plate and from one arm in a single product file Note that we have combined only within a tem plate multiple observations of the same OB or observations across OBs have not been co added In the automatic quality checks of the products we have flagged those rare cases when a templat
24. h profiles come from the STARE mode Left target is properly centred no issue with localization and extraction Right Source is off center center being at 0 pipeline creates a negative spectrum by confusing sky and target The difference in FWHM is caused by seeing For NODDING and OFFSET data the extraction method assumes a point source If true the cross dispersion profile is well defined as a single positive signal and two correlated negative signals Figure 13 a b Quite some spectra actually contain complex sources multiple targets complex background extended sources Figure 13c This is the main source for abnormal spectra nega tive flux negative SNR The pipeline does not recognize these situations and the user should always carefully check the 2D spectra in case of doubts On the other hand the pipeline can safely extract spectra without continuum from an emission line object with a cross dispersion profile in Figure 13d Such profile is not unusual Figure 13 Cross dispersion profile in NODDING mode From left to right a the target is properly centred and well behaved The two negative signals come from the nodding pattern b The target is properly centred and well behaved The negative peaks appear double reflecting a jittered NODDING pattern c Bad profile shape useless extraction Possible reasons are complex background invalid NODDING pattern extended multiple sources d No continuum signal but extraction l
25. ified master calibrations It is important to note that the reduction process itself is automatic while the quality assessment and certification of the master calibrations was and still is human supervised The overall data content will grow with time as new data are being acquired and processed approximately with monthly cadence and with a delay of 1 or 2 months These products are called Internal Data Products IDPs since they are produced in house as op posed to externally provided data products The IDPs cover many different science cases defined by their parent programmes in the case of XSHOOTER many hundreds The data format follows the ESO 1D spectroscopic standard for phase 3 data products Each spec trum is a multi column binary table There is one product file for each set of input raw files The set of input raw files is template based for the NODDING and OFFSET data and file based for STARE data The XSHOOTER IDPs always come per arm UVB VIS or NIR This data release offers science grade data products with the instrumental signature and sky back ground removed flux calibrated with error estimates and quality flags and with a list of known shortcomings They are considered to be ready for scientific analysis They are expected to be use ful for any kind of medium to high resolution spectroscopic research including abundance and line profile studies and radial velocity studies For studies of energy distributions the re
26. in red line and its 10 envelope The scaled master response curve from 2013 is overplotted in blue Versions of master response curves We have identified several periods of validity for the mas ter response curves separated by breakpoints One breakpoint occurred on 2013 01 15 and is related to the onset of using the pipeline versions xsh 2 x from that date on for the calibrations in particular for the creation of the master flats Those versions use a different normalization proce dure than the versions xsh 1 x used before Because both the science data and the standard stars used for computing the response curves are flat fielded the master response curve and the flat fields used need to be aligned in terms of pipeline algorithms The second breakpoint applies to the NIR arm only and reflects the exchange of the NIR flat lamp along with its significantly changed chromatic energy distribution Table 3 Versions of master response curves by 2009 10 begin of UVB VIS NIR pipeline versions xsh 1 x old operations XS_GRSF_090930A_ lt arm gt fits normalization algorithm for master 2013 01 15 flats 10 by 2013 01 15 now for XS_GRSF_130125A_ lt arm gt fits upgrade to xsh 2 x implementing a UVB and VIS new normalization for the master 2013 01 15 2013 11 flats and a much better definition 05 NIR across telluric windows 2013 11 06 now XS_GRSF_131106A_NIR fits exchange of NIR flat lamp modified
27. m a total of 5214 data points between 2009 10 01 and 2013 12 31 We distinguish between data for the high gain mode the standard mode the low gain mode in both cases all binnings and slits and high gain mode 1x1 bin 0 9x11 slit the most frequently used setting The data plotted in this figure are accessible under the URL given below XSHOOTER IDPs mean SNR vs mean product counts 3 selected runs 400 NOD VIS arm 1 hg 2x2 0 7x11 2 hg 1x1 0 4x11 u 3 hg 1x1 0 9x11 mean SNR 20000 40000 60000 80000 100000 mean reduced Figure 11 Same as Figure 10 for three selected runs again NODDING products from the VIS arm only Runs 2 and 3 have delivered data for young stellar objects panel C while run 1 belongs to panel D stellar evolution The data plotted in this figure are accessible under the URL given below Process quality control The quality of the data reduction is monitored with quality control QC parameters which are stored in a database The database is publicly accessible through a browser and a plotter interface t QC parameters are used to monitor the reduction quality The most obvious check is the SNR versus signal control plot the signal being expressed as the mean across the entire reduced spectrum before flux calibration and the SNR being the mean over the entire spectral range An example for NODDING VIS arm is displayed in Figure 10 The blue crosses represent the low gain
28. mode which is often used to obtain very high SNR of bright targets by combining many input spectra For that mode the record SNR found so far is in excess of 1000 quite remarkable for a 16 product that has been automatically pipeline processed without any fine tuning The red filled circles represent the workhorse high gain mode which is able to deliver SNR of up to 300 or more The spread in the achieved SNR for a given value of mean_reduced is likely to come from the spectral slope of the products This is illustrated by the next figure which displays these parameters for three selected observing runs Their SNR slopes are well confined due to their targets having roughly the same spectral slopes Known features and issues This is a non exhaustive list which might evolve with time Please check also Sect 6 of the Pipeline User Manual and Sect 7 6 of the Reflex Tutorial22 Issues Localization and extraction Sometimes STARE data might have the object off center with the effect that sky and target get confused by the pipeline which uses pre defined localization and ex traction windows see Figure 12 The spectrum then has negative values23 The method used for localization of STARE targets is MANUAL which is the most robust one but sometimes the wrong choice Check the User Manual for more information Figure 12 Cross dispersion profile all examples are cut out from the second overview plot collapsed in X direction Bot
29. nual sections 10 15 10 17 for version 12 1 Pipeline Description Information about the XSHOOTER pipeline including downloads and manual can be found under the URL http www eso org sci software pipelines The QC pages contain valuable information about the XSHOOTER data their reduction and the pipeline recipes Data Reduction and Calibration The main reduction steps of the XSHOOTER Echelle science data are the following STARE observations Each STARE frame is reduced individually multiple STARE observations from the same template are not stacked Reduction steps are e The input spectrum gets bias corrected UVB and VIS arm or dark corrected NIR arm e Cosmic rays are detected on the input spectrum using Laplacian edge detection e The order table is used to locate the inter order regions where the background is fit with a polynomial which is then subtracted from the science data e The master flat frame is divided into the science frame e The sky background is subtracted sky subtract TRUE sky method MEDIAN e The science frame is rectified order by order e The spectrum is localized localize method MANUAL A simple sum extraction is done on the 2D rectified orders No optimal extraction was performed although the pipeline offers it since it has artefacts e The extracted 1D orders are merged into a single spectrum e The spectrum is finally flux calibrated using a master response curve The localization wa
30. of type ANCILLARY lt subtype gt Once you have renamed the files they carry the names as also listed in the ASSON1 ASSON2Z etc header keys of the first extension of the IDP See further down for a description of the file types and their content Then the easiest way to identify all files belonging together is the timestamp in their name e g 2009 10 05T07 45 26 840 which is common to all file names belonging together 3 http www eso org qc PHOENIX XSHOOTER processing html 4 http archive eso org wdb wdb adp phase3 spectral form Data Selection Data selection is entirely rule based It is organized along the following criteria e instrument XSHOOTER or SHOOT as used until March 2010 e observing technique DPR TECH ECHELLE SLIT lt obsmode gt with obsmode being one of STARE NODDING or OFFSET not ECHELLE IFU e category DPR CATG SCIENCE e type DPR TYPE OBJECT No selection is made on the basis of the observing mode visitor or service nor on settings XSHOOTER ECHELLE settings are defined by the combination of arm UVB VIS or NIR binning 1x1 1x2 or 2x2 and slit width We have processed only those science data for which certified master calibrations exist in the arc hive Depending on their type these master calibrations exist at daily frequency or less They were all processed by the Quality Control Group but earlier in time close to the date of acquisition in order to provide quality feedback to the
31. ooks ok and spectrum likely to be valid from an emission line object 22 http www eso org sci software pipelines go to KSHOOTER Reflex Tutorial 23 In those cases it might happen that the extracted spectrum is quite useful after correcting for the wrong sign 17 OFFSET data depend even more critically on the observer s definition of the Observing Block Figure 14 The pipeline blindly trusts the labels OBJECT and SKY in the raw data The user should always check carefully if the OFFSET products have a cross dispersion profile like in Figure 14 a b well behaved or e g in c ill defined Remember that OFFSET data without OBJECT or without SKY are suppressed Figure 14 Cross dispersion profile in OFFSET mode a proper behaviour object and sky well defined b target off centre the extraction still works fine c odd behaviour object and or sky not well defined It is not possible to tell from the header if the source is pointlike or not Furthermore it is impossible to check whether the OBJECT and SKY labels are correctly set by the PI In case of doubt the user should always check the 2D ancillary files and the VIS QC plot displaying the cross dispersion profile Saturation and gaps Saturated pixels in the raw data get ignored by the pipeline If there are too many so that no unsaturated pixels can be found by the pipeline for a given wavelength bin the flux in this wavelength range is set to zero In the QUAL
32. quality flag per wavelength bin propagated throughout the reduction 1 2 3 4 5 6 7 File Size The unbinned XSHOOTER products have about 1 5 MB size The ancillary 2D files have the same spectral coverage and come as 15 29 MB files depending on arm Files are always uncompressed Acknowledgment text According to the ESO data access policy all users of ESO data are required to acknowledge the source of the data with an appropriate citation in their publications Find the appropriate text here 4 All users are kindly reminded to notify Mrs Grothkopf esodata at eso org upon acceptance or 24 http archive eso org cms eso data access policy html 21 publication of a paper based on ESO data including bibliographic references title authors journal volume year page numbers and the programme ID s of the data used in the paper 22
33. rors and quality flags These products are combined into the final binary table FITS file which is the delivered main data product with the wavelength scale as first column and all other products as further columns see File Structure below There is also an ancillary FITS file e 2D not extracted but rectified spectrum not flux calibrated with the 2D information of the signal available for quality checks There are further ancillary files for each product e a text file with OB related information about the product file including OB classification and comments and QC information Table 2 e one or two in the case of VIS arm overview plots displaying the spectrum a smoothed version the unfluxed version and the SNR for the VIS arm the second graphical file con tains an overview plot of all three arms PEROSA 2 Content of CA README text file parameter values i ssid meaning OB related information SM_VM SM or VM Data taken in Service Mode or Visitor Mode VM data are less constrained in terms of OB properties have no user constraints defined and therefore no grades formally graded X meaning unknown ambient conditions checked against user constraints OB_COMMENTS Free text Any optional comments added by the night astronomer mje together with the approximate UT hh mm truncated after 200 characters Note that OBs might have been executed match at the price of far less controlled quality param
34. roughly monthly intervals and with a delay of 1 or 2 months when all master calibrations for the corresponding time interval are available How to organize The downloaded data come with their technical archive names S aS eo Filename Progress ADP ZUT4 U9 U511U_Z1_VUDYUMS isnt OE a Rh sh ADP 2014 05 08T10_27_00 610 txt Completed 100 370 0bytes of 370 0bytes 221 0bytes s ADP 2014 05 08T10_27_00 677 fits Completed 100 10 1MB of 10 1MB 5 1MB s ADP 2014 05 08T10_27_00 830 png Completed 100 142 0KB of 142 0KB 5 3KB s ADP 2014 05 08T10_27_00 630 fits Completed 100 27 9MB of 27 9MB 15 2MBis ADP 2014 05 08T10_27_00 723 png Completed 100 136 6KB of 136 6KB 70 5KB s ADP 2014 05 08T10_27_00 743 pna Completed 100 63 8KB of 63 8KB 37 8KBIs ADP 2014 05 08T10_27_00 930 txt Completed 100 370 0bytes of 370 0bytes 211 0bytesis ADP 2014 05 08T10_27_00 993 fits Completed 100 1 3MB of 1 3MB 646 3KB s ADP 2014 05 08T10_27_00 510 txt Completed Te ADP 2014 05 08T10_27_00 550 fits Completed 100 14 8MB of 14 8MB 5 3MB s Th ADP 2014 05 08T10_27_00 737 fits Completed 100 1 3MB of 1 3MB 1 4MBis ADP 2014 05 08T10_27_00 803 png C ompleted 100 156 3KB of 156 3KB 80 1KB s ADP 2014 05 08T10_27_00 457 fits Completed 100 21 6MB of 21 6MB 13 8MBis ADP 2014 05 08T10_27_00 583 txt Completed 100 370 0bytes of 370 0bytes 235 0bytes s ADP 2014 05 08T10_27_01 007 png Completed 100 149 3KB of 149 3KB 73 0KB s ADP 201
35. s There has been some internal quality control on the pipeline processing of the science data monitoring e the quality of the associations of calibration data checking that the master calibrations used are not more than a few days away from the science data e score flags for various properties of the products among them the number of saturated pixels see above Table 4 e on demand QC reports and quick look overviews 13 This information has largely been used to improve and fine tune the reduction process An individual one by one inspection of the products has not been done Quality flags The XSHOOTER pipeline supports pixel based quality flags not to be confused with the QC flag that applies to the whole file propagated through the entire calibration and reduction procedure They come as co added bit codes and are available per wavelength bin in the spectrum table as column QUAL see Figure 7 below The possible values for the bit codes are defined in the User Manual sect 11 3 Some important values are listed here Table 5 List of important quality flags 2 12 4096 Pixel saturation UVB or VIS 2 19 524288 2 20 1048576 Extrapolated flux in NIR 2421 2097152 Raw pixel value zero or negative saturation in NIR 2 22 4194304 Interpolated flux during standard extraction The logarithm of the quality flags is plotted in blue as lowest panel in the preview plots labeled log Q 20 Figure 7 Qualit
36. s done with the method MANUAL localize slit position 0 0 localize slit hheight 2 0 It is the most robust method but not always the correct one if the object is off centre The position of the peak signal can easily be checked on the preview plot 2 for VIS arm data 6 Under the XSHOOTER link in http www eso org sci software pipelines There you will also find the link to the Reflex tutorial the current version is under XSHOOTER Reflex Tutorial 7 Namely generate SDP format TRUE to generate the output format of the IDPs and dummy association keys 4 for the technical ASSOC ASSON keys 8 http www eso org qc XSHOOTER pipeline pipe gen html NODDING observations They get combined within the same template per arm the pipeline assumes constant transparency and equal exposure times for all input frames Reduction steps are e All input spectra are bias corrected UVB and VIS arm N input frames are combined alternating object top object bottom where N must be even e Cosmic rays are detected on each input spectrum using Laplacian edge detection e The recipe combines the science frames taken at the same position if e g sequence was AA BB data are combined as A B doing median stacking Sstack method median e Pairs are subtracted e g A B e The master flat frame is divided into each subtracted science frame pair e ach science frame pair is rectified order by order e Since correct sky by median TRUE the recipe calc
37. searcher Should keep in mind that the XSHOOTER instrument is a medium to high resolution spectrograph designed for stability and throughput It is usually not operated to achieve high precision spectro photometry unless the wide slit is used Slit losses have to be expected and they may differ from arm to arm There has been no attempt to correct for telluric absorption lines this might affect certain science applications Disclaimer Data have been pipeline processed with the best available calibration data However 1 http www eso org sci facilities paranal instrtuments xshooter html 2 http www eso org sci observing phase3 p3sdpstd pdf please note that the adopted reduction strategy may not be optimal for the original scientific pur pose of the observations nor for the scientific goal of the archive user You may also want to consult the on line version of this documentation which is a living docu ment and has further useful links and more details Release Content The XSHOOTER_ECHELLE IDP release is a stream release The overall data content is not fixed but grows with time as new data are being acquired and processed The data are tagged as XSHOO TER_ECHELLE in the ESO archive user interface The first data were published under XSHOOTER_ECHELLE in May 2014 with XSHOOTER data from the begin of operations October 2009 until the end of 2013 The archive is being filled chrono logically New data are being added at
38. ulates and subtracts the median pixel value for each wavelength column in the rectified frame from the column pixel values e For each subtracted science frame pair the recipe forms A B shifted B A and then combines them using combinenod method MEAN e The standard extraction method is used to extract the spectrum extract method NOD e The extracted 1D orders are merged into a single spectrum e The spectrum is finally flux calibrated using a master response curve Note that in case of odd number of input files the pipeline rejects the last one and processes all others Nodding with one input frame always fails OFFSET observations SKY and OBJECT frames get combined within the same template per arm the pipeline assumes constant transparency and sky brightness and equal exposure times for all input frames Reduction steps are e Cosmic rays are detected on each input frame using Laplacian edge detection e SKY frames are subtracted from OBJECT frames in pairs e The master flat frame is divided into each sky subtracted science frame e Each sky subtracted science frame is rectified order by order e The rectified science frames are stacked combinenod method MEAN e The standard extraction method is used to extract the spectrum e The extracted 1D orders are merged into a single spectrum e The spectrum is finally flux calibrated using a master response curve Master Calibrations used for data reduction Table 1 Set of m
39. ved in NODDING mode with the widest slit size 5 and have been processed with the corresponding science recipe including the final step of the flux calibration with the master response curves for the three arms A comparison of the flux calibrated XSHOOTER spectrum which should in this case have photometric quality to the tabulated energy distribution is reassuring Figure 4 We did also a comparison to a flux calibrated UVES spectrum of the same star Figure 5 Again the agreement is very good 14 Defined by the 10 envelope of the best response curves see Figure 2 15 Bumps and wiggles in the response curve are inacceptable if they do not correspond to instrument features but are due to imperfect records of the standard star energy distribution or to algorithmic flaws 16 Amounting to several hundred individual curves per arm master_response_NIR period3 also marked 10 est N zZ oO a Nn i m LAMBDA Figure 2 Selection of good response curves and construction of master response curve For the NIR arm we show the selected individual response curves taken after 2013 01 in purple and the master response curve thin blue line derived from averaging them with its 10 envelope in bold RESPONSE ee es es So 450 500 LAMBDA Figure 3 Propagation of master response curve back into 2009 2012 For the UVB arm we show the average of all selected response curves from the pre 2013 period plotted as th
40. y flags in the 2D product of the NIR arm The colour map has been chosen to display bits 2 20 extrapolated flux and 2 21 raw pixel zero or negative white patches Previews The preview plots have been originally developed as quick look plots for process quality control It was felt that they might also be useful to the archive user They are delivered as ancillary files along with the IDPs There are two plots 1 the main preview plot Figure 8 one per arm 2 the overview plot Figure 9 only for the VIS arm 20 With bit 2 22 being suppressed in the plots but not in the data in order to avoid the plots being swamped 14 XSHOOTER QC Report SCIENCE_VIS Date 2010 03 11 mode SLIT_NOD_VIS slit 0 4x11 bin 1x1 RuniD 084 C 0952 A product r SHOOT 2010 03 12T09 09 22 296_tpl_0000 fits OB 387834 target PDS 399 fluxed spectrum reduced S N qual wavelength in nm amp spectral windows ull info rebin 0 5nm A Q j w o o g bani a E O S ao l 3 x lt m amp 0 Pe hae log OQ Urtititi ti titi tits tists ists is ts tsisisisis ists ts ts 540 560 580 600 620 640 660 680 700 720 740 760 780 800 820 840 860 880 900 920 940 960 960 10001020 EXPTIME 15 0 s Astart 533 7 nm NPIX_SAT DATANCOM 8 Aend 1020 0 nm E template 20 20 complete tot_EXPT 120 0 s Abin 0 02 nm QC_S N mean_r 29105 score_bil 1010000 AIRMASS 1 163 SL
Download Pdf Manuals
Related Search