VLBA Pipeline: Outline of Data Reduction Heuristics 1 Introduction
Contents
1. 1 2 Software The VLBA pipeline is Written in python and e Uses Obit and AIPS tasks to do the data processing and Uses AIPS data structures for intermediate data and Writes FITS images and AIPS FITAB format calibrated datasets The pipeline scripts are publically available for checkout from a Subver sion SVN repository https svn cv nrao edu svn VLBApipeline AIPS http www aips nrao edu index shtml and Obit http www cv nrao edu bcotton Obit html are installed on all NRAO Linux computers and available for installation via download to non NRAO computers 1 3 Prototype Comparison The Mojave project was selected as an initial set of observations This com prises more than 150 datasets each roughly 24 hours in duration observing sources to track morphological changes over time The observations are snapshots mostly at 2cm Ku band with some 3cm X band observations This extended project has the advantage that the data have already been calibrated and imaged by experts and that the resultant images are publicly available for direct comparison with the images produced by the pipeline The FITS images of the Mojave project are available at http www nrao edu 2cmsurvey For consistency between epochs the Mojave project necessarily has lim itations on the data that they fully reduce The VLBA pipeline has no such limitations and about 3500 individual images were produced from data taken between August 20032. for opacity corrections Apply PC table Plot results Manual phase cal even to tweak up PCals Determine and apply manual phase cals Manual phase cal solution interval min Manual phase cal smoothing time hr Plot the phase and delays from manual phase cal Edit using editList Table to apply edit list to doEOPCor True Opacity Tsys correction doOpacCor True OpacSmoo 0 25 Apply phase cal corrections doPCcor True doPCPlot True doManPCal True manPCsollnt None manPCSmoo None doManPCalPlot True Special editing list doEditList False editFG 2 editList EditList Do median flagging doMedn True Median editing mednSigma 10 0 Median sigma clipping level mednTime Wind 1 0 Median window width in min for median flagging mednAvgTime 10 0 60 Median Averaging time in min mednAvegFreq 0 Median 1 gt avg chAvg chans 2 gt avg all chan 3 gt avg chan and IFs mednChAvg 1 Median number of channels to average Flag Suspect data doF lags True UVELG editing elLim 10 0 Min allowed source elevation deg flag0 2 0 if gt 1 flag data near zero fringe rate Editing doClearTab True Clear cal edit tables doGain True Clear SN and CL tables gt 1 doF lag True Clear FG tables gt 1 doBP True Clear BP tables doCopyFG True Copy FG 1 to FG 2quack doQuack False Quack data quackBegDrop 0 1 Time to drop from start of each scan in min quack
3. and December 2011 These images typically had dynamic ranges peak rms of 25 35dB Roughly 3200 images were available for direct comparison The comparison was excellent On average the in tegrated fluxes for the pipeline were just over 5 lower than the images in the Mojave catalog as predicted 2 The Process The pipeline processing uses the following processes Many of the default processing parameters are frequency dependent and may be overridden and the various steps may be turned on or off The following gives an overview of the processing Details are docu mented in the main pipeline processing script VLBAContPipe py and in the routines called in VLBACals py see python online documentation The processing is driven by a parameter script which is initially automatically generated but may be modified for detailed control of the processing pa rameters are described in the Appendix These are described in more detail in the VLBA Pipeline User Manual 1 Data retrieved from the archive Pre DifX data may be either multiple FITS IDI format files or a single AIPS UVFITS data file Data from the DifX correlator are in a single FITS IDI format file Data converted to AIPS format Multiple FITS IDI format files can be concatenated Flag Data at low elevations and at low fringe rates are flagged using AIPS UVFLG Initial data filtering The data are edited with a running median window Obit MednFlag to flag de
4. ee xm Phase calibration of all sources The source models are used to determine the phase corrections for all sources and these are applied to the cumulative calibration table Obit tasks Calib CLCal are used Pe piis Sor 7 Imaging and production of results a Imaging Each source for which previous calibration was successful is then imaged This final imaging may use phase and possibly amplitude self calibration and the imaging uses multiple resolutions 2 to help recover extended emission Obit task Imager is used for the imaging b Saving images Final and calibration images are written to FITS files c Saving visibility data The averaged and calibrated uv data and the tables from the initial data are written to AIPS FITAB format FITS files d Reports Statistics of the images are determined and an HTML page con structed to simplify viewing the results An XML file manifest is generated for re ingestion into the archive e Cleanup All AIPS data files are deleted 3 The Products e Calibrated u v dataset with calibration and flagging tables in AIPS FITAB format Tables from initial data and averaged visibilities per input dataset e FITS Images one per source observed plus calibration images e Diagnostic plots several per image e Reports and logs created during the process e Meta data for a VOTable to describe the products The file set comprising all files and the meta data are stored in a
5. in min FOV Field of view radius in deg solP Int phase self cal solution interval min solAInt amp phase self cal solution interval min findSolInt Solution interval min for Calib findTimeInt Maximum timerange large gt scan CalAvgTime Time for averaging calibrated uv data min Parameter lt 1GHz P 20cm L 13cm S 6cm C 3cm X manPCsolInt 0 25 0 5 0 5 0 5 0 5 manPCSmoo 10 0 10 0 10 0 10 0 10 0 delaySmoo 0 5 0 5 0 5 0 5 0 5 bpsolint 1 10 60 15 60 10 60 10 60 10 60 FOV 0 4 3600 0 4 3600 0 2 3600 0 2 3600 0 1 3600 solP Int 0 10 0 25 0 25 0 25 0 25 solA Int 3 0 3 0 3 0 3 0 3 0 findSolInt 0 1 0 25 0 25 0 5 0 5 findTimelInt 10 0 10 0 10 0 10 0 10 0 CalAvgTime 10 60 10 60 10 60 10 60 10 60 Parameter 2cm Ku 1cm K 9mm Ka 7mm Q 3mm W manPCsolInt 0 5 0 2 0 2 0 1 0 1 manPCSmoo 10 0 10 0 10 0 10 0 10 0 delaySmoo 0 5 0 5 0 5 0 5 0 5 bpsolint 1 10 60 10 60 10 60 5 60 5 60 FOV 0 05 3600 0 05 3600 0 04 3600 0 04 3600 0 02 3600 solP Int 0 25 0 25 0 25 0 1 0 1 solA Int 3 0 3 0 3 0 3 0 3 0 findSolnt 0 5 0 3 0 2 0 1 0 1 findTimelInt 10 0 10 0 10 0 10 0 10 0 CalAvgTime 10 60 5 60 5 60 5 60 4 60 12
6. EndDrop 0 0 Time to drop from end of each scan in min quackReason Quack Reason string Bandpass Calibration doBPCal True Determine Bandpass calibration bpBChan1 1 Low freq channel initial cal bpEChan1 0 Highest freq channel initial cal 0 gt all bpDoCenter1 None Fraction ofchannels in 1st overrides bpBChan1 bpEChan1 bpBChan2 1 Low freq channel for BP cal bpEChan2 0 Highest freq channel for BP cal 0 gt all bpChWid2 1 Number of channels in running mean BP soln bpdoAuto False Use autocorrelations rather than cross bpsolMode A amp P Band pass type A amp P P PIA bpsolint1 None BPass phase correction solution in min bpsolint2 10 0 BPass bandpass solution in min specIndex 0 0 Spectral index of BP Cal doSpecPlot True Plot the amp and phase across the spectrum Amp phase calibration parameters Reference antenna List of Reference antenna for fringe fitting Image calibrators List of target sources List of failed target source process Image targets Output calibrator image class Output target temporary image class Output target final image class Weighting robust parameter Max number of clean iterations Minimum CLEAN flux density Minimum Allowed SNR Delay solution for phase self cal Average poln in self cal Average IF in self cal Max number of phase self cal loops Min flux density peak for phase self cal Max number of Amp phase self cal loops Min f
7. VLBA Pipeline Outline of Data Reduction Heuristics Gareth Hunt Bill Cotton amp Jared Crossley September 28 2012 1 Introduction The VLBA Pipeline was designed to take uncalibrated VLBA visibility data directly from the NRAO archive and to create a file set for reingestion into the archive or for direct use by end users This file set contains reference images with associated diagnostic plots reports scripts and log files plus calibrated visibility data with associated tables The scripts can be used to set non default values to processing parameters and used to repeat part or all of the processing if the default processing is inadequate 1 1 Scope The scope of the present version of the pipeline is e VLBA data only It may work with the inclusion of other telescopes if all of the VLBA calibration tables are available e 1 15 GHz The pipeline has been used on continuum data sets with frequencies as high as 43 GHz with robust results e Calibrated fluxes Calibration uses standard external calibration and does not include coherence losses e Continuum imaging Spectral line data sets can have the continuum calibration done but no spectral cubes are made Corrections based on the pulsed cal system may need to be turned off if this system was not used e Imaging including self calibration Multiresolution imaging with self calibration is done e No polarization No polarization calibration imaging is currently implemented
8. e Final Image Clean selfcal Stokes Tl Stokes to image doKntrPlots True Contour plots Final outDisk 0 FITS disk number for output 0 cwd doSaveUV True Save uv data doSavelmg True Save images doSaveTab True Save Tables doCleanup True Destroy AIPS files copyDestDir j Destination directory for copying output files empty string gt do not copy Diagnostics doSNPlot True Plot SN tables etc doDiagPlots True Plot single source diagnostics prtLv 2 Amount of task print diagnostics doMetadata True Save source and project metadata doHTML True Output HTML report B Band dependent Parameters This section lists the default band dependent parameters used in the VLBA Pipeline scripts They are only explained briefly but experienced users should have no difficulty recognizing their use and functionality It is clearly possible to re run or re start the pipeline using different values than the defaults Note that the VLBA has two receiver bands below 1GHz 90cm and 50cm The band dependent parameters are the same for both bands Note also that 9mm Ka is included for completeness in the software but there is no receiver 11 Parameter Description manPCsolInt Manual phase cal solution interval min manPCSmoo Manual phase cal smoothing time hr delaySmoo Delay smoothing time hr bpsolint1 BPass phase correction solution
9. luded Uses Obit task BPass d Calibrator phase calibration Phase corrections on a short time scale are determined for the calibrator sources using the source models for each This phase correction is then applied to the data needed in the next step Obit tasks Calib CLCal are used Calibrator amplitude calibration Longer time amplitude solutions are determined for the calibrator sources In able to prevent poor weather or other conditions at a small number of antennas from skewing the amplitude scale a subset of the antennas with the most stable set of fitted gains are used to stabilize the flux density scale The average gain for these antennas is divided into all gain solutions The strong enough calibrator sources have the solutions determined for them applied in the calibration table Other sources use a smoothed version of the amplitude calibration solutions io ie f Calibrate and average data Calibration is applied and the data are averaged in frequency and possibly time Subsequent steps use the averaged data Uses Obit task Splat Self calibration of all sources An initial self calibration to get models of all sources is performed Phase self cal is always used and also amplitude self cal if the peak in the image is above a given threshold Imaging uses Obit task SCMap h Data clipping Data with amplitudes significantly in excess of the sum of the CLEAN components for each source are flagged
10. lux density peak for amp phase self cal Number of additional imaging multiresolution tapers List of tapers in pixels Make ref pixel tangent to celest sphere for each facet F Allow negative components in self cal model Search for good calibration reference antenna Solution interval min for Calib Maximum timerange large gt scan Name or list of continuum cals No cal model No target model yet Search for OK cals if contCals not given Minimum fraction of acceptable solutions Minimum test SNR refAnt 0 refAnts 0 Imaging calibrators contCals and targets doImgCal True targets failTarg dolmgTarget True outCclass TCalSC outT class ITImgsC outIclass TClean Robust 0 0 Niter 500 minFlux 0 0 minSNR 4 0 solMode DELA avgPol True avelF False maxPSCLoop 6 minFluxPSC 0 05 maxASCLoop 1 minFluxASC 0 2 nTaper 1 Tapers 20 0 0 0 do3D False noNeg False Find good calibration data doFindCal True findSolnt None findTimelInt None contCals None contCalModel None targetModel None If need to search for calibrators doFindOK True minOKFract 0 5 minOKSNR 20 0 failTarg list of failed sources Delay calibration Determine apply delays from contCals Delay smoothing time hr Determine smooth apply amplitudes from contCals Stablize gains with best antennas Fraction of antenna to use in stabilization List of antennas to exclude from stabilization Li
11. single directory For approved pipeline use this directory is stored on the lustre file system in NRAO Socorro From there it is ingested directly into the NRAO archive Sources that did not image acceptably are added to the failTargets list This is referenced in the HTML Report A General Parameters This section lists the default global parameters used in the VLBA Pipeline scripts They are only explained briefly but experienced users should have no difficulty recognizing their use and functionality It is clearly possible to re run or re start the pipeline using different values than the defaults Several parameters are actually placeholders for derived intermediate products failTarg contCalModel targetModel although in principle con tCalModel could be user supplied These are initialized as specified here at the beginning of the pipeline process but may be overridden in the parameter script Quantization correction doQuantCor True Do quantization correction QuantSmo 0 5 Smoothing time hr for quantization corrections QuantFlag 0 0 If gt 0 flag solutions lt QuantFlag use 0 9 for 1 bit 0 8 for 2 bit Parallactic angle correction doPACor True Make parallactic angle correction Total Electron Content TEC correction doTECor True Make TEC correction Earth Orientation Parameters EOP correction Make EOP correction Make Opacity Tsys gain correction Smoothing time hr
12. st of antennas to always include in stabilization calibrate and average cont calibrator data AIPS class of calibrated averaged uv data Time for averaging calibrated uv data min First IF to copy Highest IF to copy First Channel to copy Highest Channel to copy Average all channels Average all channels Phase calibration of all targets in averaged calibrated data Phase calibrate contCals data with self cal Phase target data with self cal determination instrumental polarization from instPolCal Defaults to contCals EVPA calibration doDelayCal True delaySmoo None Amplitude calibration doAmpCal True doStable True stableFract 0 667 stableBadAnts stableGoodAnts Apply calibration and average doCalAvg True avgClass UVAve CalAveTime None CABIF 1 CAEIF 0 CABChan 1 CAEChan 0 chAvg 10000000 avgFreq 1 doPhaseCal True doPhaseCal2 True Instrumental polarization cal doInstPol False instPolCal None Right Left phase doRLCal False RLCal None Set RL phases from RLCal also needs RLCal RL Calibrator source name if given a list of triplets name R L phase deg 1GHz RM rad m 10 Clip excessive visibilities doClipFlag True Clip flag visibilities above sum of CCs clipFactor 1 25 Factor above sum of CCs to clip clipTime 0 25 Time in min for which the data is to be averaged before clipping Final Image Clean doImgFullTarget Tru
13. the fringe fitting Pulse calibration The pulse cal signals are used to align the phases and delays of the various parts of the electronics Since these are based on phase measurements from discrete tones the delays are ambigu ous This ambiguity is resolved using fringe fit results for the best calibrator scan Obit tasks PCCor CLCal are used for this Manual phase calibration There are generally residuals delay and phase errors after correc tion by the pulse calibration these are corrected using delays and phases determined for the best calibrator scan and applied to all data Obit tasks Calib CLCal are used for this 6 Calibration from visibility data a Initial calibrator self calibration eH All sources deemed to be calibrators are self calibrated to provide initial images for further calibration Phase calibration is applied and amplitude as well if the peak in the image exceeds a frequency dependent minimum value Imaging uses Obit task SCMap Delay calibration Group delay fits are made using a fringe fit on the calibrator sources using the source models derived in the previous step Obit tasks Calib CLCal are used for the fringe fitting correction c Bandpass correction A bandpass correction for the amplitudes and phases in each channel is determined from the best calibrator scan and the model derived for that calibrator from the cross correlation data No spectral index correction is inc
14. viant data such as when an antenna is late on source Standard external calibration a 1 2 bit sampling correction Uses AIPS task ACCOR Parallactic angle correction Phases are corrected for the effects of parallactic angle Uses AIPS task CLCOR c Ionospheric correction TEC Relevant ionospheric models are downloaded from the Web and applied using AIPS TECOR to correct for the Total Electron Content TEC given by the model d Earth Orientation Parameters EOP The most recent IERS earth orientation parameters UT1 UTC b NH NS ww NS Nae position of pole are downloaded from the Web and used by AIPS CLCOR to correct the VLBA correlator model with the final values Tsys atmosphere gain correction The amplitudes are converted to Jy using measured system tem peratures standard gain curves and atmospheric opacity correc tions estimated from the system temperatures Uses AIPS task APCAL These gains are smoothed before application to the data Calibrator selection Calibrator sources are then determined by doing a fringe fit on all sources to determine which ones reliably give solid detections The reference antenna is picked on the basis of strong source detections The best calibration scan is then selected on the basis of the fringe fit signal to noise estimates This scan is the one involving the largest number of antennas and with the highest average SNR Obit task Calib is used for
Download Pdf Manuals
Related Search