Python bindings for CPL recipes
Contents
1. HDUlist or str ora list of them or dict Data input frames tag str Overwrite the tag attribute optional threaded boo1 overwrite the threaded attribute optional loglevel int set the log level for python logging optional logname str set the log name for the python logging Logger optional de fault is cpl recipename output_dir str Set or overwrite the out put_dir attribute optional param dict overwrite the CPL parameters of the recipe specified as keys with their dictionary values optional calib dict Overwrite the calibration frame lists for the tags specified as keys with their dictionary values optional env dict overwrite environment variables for the recipe call optional Returns The object with the return frames as pyfits HDUList objects Return type cpl Result Raise exceptions ValueError If the invocation parameters are incorrect 3 6 Runtime environment 13 Python bindings for CPL recipes Release 0 4 Raise exceptions 10Error If the temporary directory could not be built the recipe could not start or the files could not be read written Raise cpl CplError If the recipe returns an error Raise cpl RecipeCrash If the CPL recipe crashes with a SIGSEV or a SIGBUS Note If the recipe is executed in the background threaded True and an exception occurs this exception is raised whenever result fields are accessed See Also2. Parallel execution 14 Chapier 3 The Recipe interface CHAPTER FOUR PARALLEL EXECUTION The library allows a simple parallelization of recipe processing The parallelization is done using independent processes and thus does not depend on parallelization features in the CPL or the recipe implementation To specify that a recipe should be executed in the background the threaded attribute needs to be set to True This may be done either in the recipe constructor as a recipe attribute or as a parameter of the execution call Each of the following three recipes will start a background process for the BIAS calculation Create a threaded recipe rl cpl Recipe muse_bias threaded True resulti rI biasl fits bias2 fits bidas3 fits Prepare a recipe for background execution r2 cpl Recipe muse_bias r2 threaded True result2 2 7 biasi fits bias2 tits bias3 tlits Execute a recipe in background r3 cpl Recipe muse_bias result3 r3 biasl fits bias2 fits bias3 fits threaded True If the threaded attribute is set to True the execution call of the recipe immediately returns while the recipe is executed in the background The current thread is stopped only if any of the results of the recipe is accessed and the recipe is still not finished The result frame of a background recipe is a subclass of threading Thread This interface may be used
3. Their minimum and maximum value will be set to None In order to assing a FITS file to a tag the file name or the pyfits HDUList is assigned to the calibration attribute gt gt gt muse_scibasic calib MASTER_BIAS MASTER_BIAS_0 fits Using pyfits HDUList is useful when it needs to be patched before fed into the recipe gt gt gt master_bias pyfits open master_bias_0 fits gt gt gt master_bias 0 header HIERARCH ESO DET CHIP1 OUT1 GAIN 2 5 gt gt gt muse_scibasic calib MASTER_BIAS master_bias_0 fits Note that pyfits HDUList objects are stored in temporary files before the recipe is called which may produce some overhead Also the CPL then assigns the random temporary file names to the FITS keywords HIERARCH ESO PRO RECm RAWn NAME which should be corrected afterwards if needed To assign more than one frame put them into a list gt gt gt muse_scibasic calib BADPIX_TABLE badpixl fits badpix2 fits All calibration frames can be set in one step by assigning a dict to the parameters In this case frame that are not in the map are set are removed from the list and unknown frame tags are silently ignored The key of the map is the tag name the values are either a string or a list of strings containing the file name s or the pyfits HDUList objects gt gt gt muse_scibasic calib MASTER_BIAS master_bias_0 fits BADPIX_TABLE badpix_1
4. e Pyfits 1 2 Binary packages e Python CPL comes with the Ubuntu distribution since 12 04 The latest version for the current release can be found in a special Ubuntu repository 12 10 e Debian package in Wheezy http packages debian org wheezy python cpl 1 3 Source code Current release python cpl 0 4 tar gz Python Package Index Git repository To access do a git clone git github com olebole python cpl git This gives you the current version in the subdirectory python cpl To update to the current version of an existing repository doa git pull inthe python cpl1 directory For more detailed information check the manual page of git 1 and the github page of the project 1 4 Compilation For compilation a C compiler is needed additionally to the software mentioned above The installation follows the standard procedure used in python On default the installation path usr local If using a non standard installation path add the directory PREFIX lib python2 7 site packages 1ib64 python2 7 site packages on 64 bit systems to your environment variable PYTHONPATH where where PREFIX is the installation path for the package In the source directory of cpl python run python setup py install prefix PREFIX There are other options available as well use the he1p option to list them Python bindings for CPL recipes Release 0 4 1 5 Test suite The test suite can be downloaded as tar file python c
5. fits badpix_2 fits J In a recipe call the calibration frame lists may be overwritten by specifying them ina dict 12 Chapter 3 The Recipe interface Python bindings for CPL recipes Release 0 4 gt gt gt res See Also muse_scibasic calib MASTER_BIAS master_bias_1 fits cpl FrameConfig 3 6 Runtime environment For debugging purposes the runtime environment of the recipe may be changed The change may be either done by specifying the Recipe env attribute of as a parameter on the recipe invocation The change will have no influence on the environment of the framework itself Note Some variables are only read on startup like MALLOC_CHECK_ changing or deleting them will have no effect Recipe env None Environment changes for the recipe This is adict with the name of the environment variable as the key and the content as the value It is possible to overwrite a specific environment variable Specifying None as value will remove the variable gt gt gt muse_flat env MUSE_RESAMPLE_LAMBDA_LOG 1 gt gt gt muse_flat env MUSE_TIMA_FILENAME tima fits In a recipe call the runtime environment may be overwritten as well gt gt gt res muse_flat env MUSE_PLOT_TRACE true 3 7 Recipe invocation Recipe __call__ data ndata Call the recipes execution with a certain input frame Parameters raw pyfits
6. in a list of output frames res muse_scibasic rawl fits raw2 fits raw3 fits J for i h in res PIXTABLE_OBJ h writeto pixtable i fits 1 1 If we call the recipe with a list containing a single frame we get a list with a single frame back res muse_scibasic rawl fits res PIXTABLE_OBJ 0 writeto pixtablel fits The bias recipe always returns one MASTER BIAS regardless of number of input frames So we always get a single frame back muse_bias cpl Recipe muse_bias res muse_bias biasl fits bias2 fits bias3 fits res MASTER_BIAS writeto master_bias fits Note This works well only for MUSE recipes Other recipes dont provide the necessary information about the recipe 21 Python bindings for CPL recipes Release 0 4 7 2 Run statistics 7 3 7 4 In A ddition to the result frames the cp 1 Result object provides the attribute cp1 Result stat which contains several statistics of the recipe execution cpl Result return_code The return code of the recipe Since an exception is thrown if the return code indicates an error this attribute is always set to 0 cpl Result stat user_time cpl CPU time in user mode in seconds Result stat sys_time CPU time in system mode in seconds cpl Result stat memory_is_empty Flag whether the recipe terminated with freeing all available Memory If th
7. init the search path for CPL recipes from the esorex startup gt gt gt cpl esorex init Otherwise you will need to explicitely set the recipe search path gt gt gt cpl Recipe path store 01 MUSE recipes List available recipes gt gt gt cpl Recipe list muse_quick_image 0 2 0 0 3 0 m se_scipost 0 2 07 0 3 0 Python bindings for CPL recipes Release 0 4 muse_scibasic 0 2 0 0 3 0 muse_flat 0 2 0 0 3 0 muse_subtract_sky 0 2 0 0 3 0 muse bias 0 2 0 0 3 07 muse_ronbias 0 2 0 0 3 0 1 muse_fluxcal 0 2 0 0 3 0 muse focus 0 200 0 3 0 5 muse_lingain 0 2 0 0 3 0 muse_dark 0 2 0 0 3 0 muse_combine_pixtables 0 2 0 0 3 0 muse_astrometry 0 2 0 0 3 0 muse_wavecal 0 2 0 0 3 0 muse_exp_combine 0 2 0 0 3 0 muse_dar_correct 0 2 0 0 3 0 muse standard 0 20 0 3 0 17 muse_create_sky 0 2 0 0 3 0 muse_apply_astrometry 0 2 0 0 3 0 muse_rebin 0 2 0 0 3 0 Create a recipe specified by name gt gt gt muse_scibasic cpl Recipe muse_scibasic By default it loads the recipe with the highest version number You may also explicitely specify the vers
8. is accessed 16 Chapier 4 Parallel execution CHAPTER FIVE THE CPL PARAMETER CLASS class cpl Parameter name Runtime configuration parameter of a recipe Parameters are designed to handle monitor control data and they provide a standard way to pass information to the recipe The CPL implementation supports three classes of parameters a plain value a value within a given range or a value as part of an enumeration When a parameter is created it is created for a particular value type In the latter two cases validation is performed whenever the value is set Attributes value The value of the parameter or None if set to default default The default value of the parameter readonly name The parameter name readonly Parameter names are unique They define the identity of a given parameter context The parameter context readonly The context usually consists of the instrument name and the recipe name separated by a dot The context is used to associate parameters together range The numeric range of a parameter or None if the parameter range is unlimited readonly sequence A list of possible values for the parameter if the parameter are limited to an enumeration of possible values readonly The following example prints the attributes of one parameter gt gt gt print name muse_scibasic param cr name name er gt gt gt print fullname muse_scibasic param cr fullname fullname mus
9. method 22 L level cpl esorex CplLogger attribute 28 line cpl CplError attribute 23 line cpl RecipeCrash attribute 23 list cpl Recipe static method 9 load_rc in module cpl esorex 27 load_sof in module cpl esorex 27 localvars cpl RecipeCrash attribute 24 log cpl cpl Result attribute 22 log cpl CplError attribute 23 log in module cpl esorex 28 LogList class in cpl logger 26 M MALLOC_CHECK_ 13 max cpl FrameConfig attribute 19 md5sum cpl dfs ProcessingInfo attribute 31 md5sums cpl dfs ProcessingInfo attribute 32 memory_dump cpl Recipe attribute 11 memory_is_empty cpl cpl Result stat attribute 22 min cpl FrameConfig attribute 19 msg cpl CplError attribute 23 msg in module cpl esorex 28 N name cpl dfs ProcessingInfo attribute 31 name cpl Parameter attribute 17 next cpl CplError attribute 23 O output cpl Recipe attribute 11 output_dir cpl Recipe attribute 10 37 Python bindings for CPL recipes Release 0 4 P param cpl dfs ProcessingInfo attribute 31 param cpl Recipe attribute 11 Parameter class in cpl 17 params cpl RecipeCrash attribute 24 path cpl Recipe attribute 9 pipeline cpl dfs ProcessingInfo attribute 31 ProcessingInfo class in cpl dfs 31 PYTHONPATH 3 R range cpl Parameter attribute 17 raw cpl dfs ProcessingInfo attribute 31 Recipe class in cpl 9 RecipeCrash 23 return_code
10. number of frames or None if not specified frames List of frames file names or pyfits HDUList objects that are assigned to this frame type See Also Recipe calib 19 Python bindings for CPL recipes Release 0 4 20 Chapter 6 The cpl FrameConfig class CHAPTER SEVEN EXECUTION RESULTS 7 1 Result frames class cpl Result Calling cpl Recipe __call___ returns an object that contains all result production frames in attributes All results for one tag are summarized in one attribute of the same name So the muse_bias recipe returns a frame with the tag MASTER_BIAS in the according attribute res muse_bias res MASTER_BIAS writeto master_bias The attribute content is either apyfits HDUList ora list of HDU lists depending on the recipe and the call If the recipe produces one out put frame of a tag per input file the attribute contains a list if the recipe was called with a list and if the recipe was called with a single input frame the result attribute will also contain a single input frame If the recipe combines all input frames to one output frame a single pyfits HDUList es returned independent of the input parameters The following examples will illustrate this muse_scibasic cpl Recipe muse_scibasic Only single input frame so we get one output frame res muse_scibasic raw fits res PIXTABLE_OBJ writeto pixtable fits List of input frames results
11. present or None the operation will block until the recipe termi nates 22 Chapter 7 Execution results Python bindings for CPL recipes Release 0 4 A thread can be cpl Result join ed many times Like in the foreground execution the output frames may be retrieved as attributes of the cpl Result frame If any of the attributes is accessed the calling thread will block until the recipe is terminated If the recipe execution raised an exception this exception will be raised whenever an attribute is accessed 7 5 CPL Exceptions exception cpl CplError retval res logger None Error message from the recipe If the CPL recipe invocation returns an error it is converted into acpl CplError exception and no frames are returned Also the error is notified in the log file The exception is raised on recipe invocation or when accessing the result frames if the recipe was started in background cp1 Recipe threaded set to True Attributes code The CPL error code returned from the recipe msg The supplied error message filename The source file name where the error occurred line The line number where the error occurred log Log lines of the recipe that lead to this exception See Also cpl logger LogList next Next error or None exception cpl RecipeCrash br_file Recipe crash exception If the CPL recipe crashes with a SIGSEV or a SIGBUS the C stack trace is tried to conserved in this exc
12. sum ProcessingInfo __init__ source datapaths None Parameters e source str or pyfits HDUList or pyfits PrimaryHDU or pyfits Header Object pointing to the result file header e datapaths dict Dictionary with frame tags as keys and directory paths as values to provide a full path for the raw and calibration frames Optional 32 Chapter 10 cp1 dfs DFS header parsing CHAPTER ELEVEN RESTRICTIONS FOR CPL RECIPES Not every information can be retrieved from recipes with the standard CPL functions Only MUSE recipes provide additional interfaces that allow the definition of input calibration and output frames All other interfaces will have the following restrictions 1 The Recipe calib attribute is not filled with templates for calibration frames After recipe creation this attribute is empty Also no check on the required calibration frames may be done before calling the recipe Anything that is set here will be forwarded to the recipe In the cpl esorex support directly assigning the recipe calibration files from the SOF file with recipe calib cpl esorex read_sof file will also put the raw input file into Recipe calib unless Recipe tags and or Recipe tag are set manually The standard recipe interface does not provide a way to distinguish between raw input and calibration files The Recipe tags attribute is set to None The Recipe tag attribute is not initially set If this attribute is no
13. Python bindings for CPL recipes Release 0 4 Ole Streicher November 26 2012 10 11 CONTENTS Installation 1 1 Breqwisites sonene g BAS e A Re nn ER He we 1 2 Binary packages a et ee PRS a st he SURES AA AS 1 3 SOURCECOdE u 4 4 5 ae kun ee anhand ee ae ee de Boe LA Compilation 42462 24 426 ar wa ar Be EG ERE EHS SEES 15 TWEst Suite ek ea Seen eRe EA Ee e amp SE Rhee A Ae Tutorial 2 1 Simple example cornisas 46 bea Oe ee eee ee A 2 2 Quick pide oa a 1 2 508 SS es RS Oe das Se eA Se h eS e oe The Recipe interface Sell Static members 4 2 see Saeed Sr Eee eS A ee a a 3 2 CONSCI e 2 G4 e ek a es bebe ae 3 3 Common attributes and methods nn 34 Recipe parameters eea ne t cea s Ma ne Be en ee al ee 39 Recipe frames u ae AG Ben See BS BD a Bae Boa GS BBA oe aa 3 6 Runtime environment 6 544 044 5 6b eee he 3 1 Recipeinvocation ss 4 4 eh a ee GR A me Bed Be od hbo e Boa ew A ee da Parallel execution The cpl Parameter class The cpl FrameConfig class Execution results dell JResultirames 2414 Ass Be a eh aan 1 2 Runst tistios euro 40 Buch Ba en eA ae ee OE ee S 7 3 Execution lo sea 4 nr A RR Aw EOE Bae Ce a Ge eS TA Thread control sosa Au aan Ge neh Ge ea AG ea ae a T OCPL BXcepvoms 22 5 a e E a ed A Log messages Sal Pythonstyle lop eine zu o a E AA E Ah 2 Los message lists pi 4 4 Sach fb dee ee en ee eee bene ea be er cpl esorex EsoRex
14. SOREX ESOREX LEGACY SUPPORT EsoRex is a standard execution environment for CPL recipes provided by ESO 9 1 Support for configuration and SOF files cpl cpl cpl esorex init source None Set up the logging and the recipe search path from the esorex rc file Parameters source str or file Configuration file object or string with file content If not set the esorex config file esorex esorex rc is used esorex load_rc source None Read an EsoRex configuration file Parameters source str or file Configuration file object or string with file content If not set the EsoRex config file esorex esorex rc is used These files contain configuration parameters for EsoRex or recipes The content of the file is returned as a map with the full parameter name as key and its setting as string value The result of this function may directly set as cpl Recipe param attribute import cpl myrecipe cpl Recipe muse_bias myrecipe param cpl esorex load_rc muse_bias rc esorex load_sof source Read an EsoRex SOF file Parameters source str or file SOF Set Of Files file object or string with SOF file content These files contain the raw and calibration files for a recipe The content of the file is returned as a map with the tag as key and the list of file names as value The result of this function may directly set as cpl Recipe calib attribute import cpl myrecipe cpl Rec
15. arameters from the FITS header of a CPL processed file This is done through the FITS headers that were written by the DFS function called within the processing recipe name Recipe name version Recipe version string pipeline Pipeline name cpl_version CPL version string tag Tag name calib Calibration frames from a FITS file processed with CPL The result of this function may directly set as cpl Recipe calib attribute import cpl myrecipe cpl Recipe muse_bias myrecipe calib cpl dfs ProcessingInfo MASTER_BIAS_0 fits calib Note This will not work properly for files that had pyfits HDUList inputs since they have assigned a temporary file name only raw Raw input frames Note This will not work properly for files that had pyfits HDUList inputs since they have assigned a temporary file name only param Processing parameters The result of this function may directly set as cpl Recipe param at tribute import cpl myrecipe cpl Recipe muse_bias myrecipe param cpl dfs ProcessingInfo MASTER_BIAS_0 fits param 31 Python bindings for CPL recipes Release 0 4 md5sum MDS sum of the data portions of the output file header keyword DATAMDS md5sums MDS sums of the input and calibration files dict with the file name as key and the corresponding MDS sum as value Note Due to a design decision in CPL the raw input files are not accompanied with the MDS
16. cpl cpl Result attribute 22 S sequence cpl Parameter attribute 17 set_maxthreads cpl Recipe static method 9 signal cpl RecipeCrash attribute 24 sys_time cpl cpl Result stat attribute 22 T tag cpl dfs ProcessingInfo attribute 31 tag cpl FrameConfig attribute 19 tag cpl Recipe attribute 10 tags cpl Recipe attribute 10 temp_dir cpl Recipe attribute 10 threaded cpl Recipe attribute 10 threadid cpl esorex CplLogger attribute 28 29 time cpl esorex CplLogger attribute 28 29 U user_time cpl cpl Result stat attribute 22 V value cpl Parameter attribute 17 version cpl dfs ProcessingInfo attribute 31 version cpl Recipe attribute 10 W warning cpl logger LogList attribute 26 38 Index
17. e muse_scibasic cr gt gt gt print context muse_scibasic param cr context context muse muse_scibasic gt gt gt print sequence muse_scibasic param cr sequence sequence dcr none gt gt gt print range muse_scibasic param cr range range None gt gt gt print default muse_scibasic param cr default default dcr gt gt gt print value muse_scibasic param cr value value None See Also Recipe param 17 Python bindings for CPL recipes Release 0 4 18 Chapter 5 The cp1 Parameter class CHAPTER SIX THE CPL FRAMECONFIG CLASS class cpl FrameConfig tag min_frames 0 max_frames 0 frames None Frame configuration Each FrameConf ig object stores information about one the data type a recipe can process They are used for defining the calibration files However since this information is not generally provided by CPL recipes it contains only dummy information except for the MUSE recipes The objects stores a frame tag a unique identifier for a certain kind of frame the minimum and maximum number of frames needed Attributes tag Category tag name The tag name is used to distinguish between different types of files An examples of tag names is MASTER BIAS which specifies the master bias calibration file s min Minimal number of frames or None if not specified A frame is required if the min is set to a value greater than 0 max Maximal
18. eption The stack trace is obtained with the GNU debugger gdb If the debugger is not available or if the debugger cannot be attached to the crashed recipe the Exception remains empty When converted to a string the Exception will return a stack trace similar to the Python stack trace The exception is raised on recipe invocation or when accessing the result frames if the recipe was started in background cpl Recipe threaded set to True Attributes elements List of stack elements with the most recent element the one that caused the crash at the end Each stack element isa collections namedtuple with the following attributes filename Source file name including full path if available line Line number if available 7 5 CPL Exceptions 23 Python bindings for CPL recipes Release 0 4 func Function name if available params Dictionary parameters the function was called with The key here is the parameter name the value is a string describing the value set localvars Dictionary of local variables of the function if available The key here is the parameter name the value is a string describing the value set signal Signal that caused the crash 24 Chapter 7 Execution results CHAPTER EIGHT LOG MESSAGES We provide CPL log messages in two different ways via Python logging and as a list of messages in the cpl Result object For convienience simple terminal messages and predefined log file o
19. er msg False This variable is a Cp 1 Logger instance that provides a convienience file handler similar to the file logging functionality of the CPL It basically does the same as import logging log logging getLogger log setLevel logging INFO ch logging FileHandler filename ch setLevel logging INFO ch setFormatter logging Formatter 3 asctime s levelname 7s log addHandler ch The following attributes control the format of the log file messages CplLogger filename Log file name CplLogger dir Directory name that is prepended to the log file name CplLogger level Log level for output to the terminal Any of DEBUG INFO WARN ERROR OFF CplLogger format Output format See Also 28 Chapter 9 cpl esorex EsoRex legacy support Python bindings for CPL recipes Release 0 4 logging LogRecord attributes Key mappings in the logging output CplLogger time If True attach a time tag to output messages CplLogger component If True attach the component name to output messages CplLogger threadid If True attach a thread tag to output messages 9 2 Convienence logging control 29 Python bindings for CPL recipes Release 0 4 30 Chapter 9 cpl esorex EsoRex legacy support CHAPTER TEN CPL DFS DFS HEADER PARSING class cpl dfs ProcessingInfo source datapaths None This class contains support for reading input files and p
20. fter call ing it Default is False This may be also set as an attribute or specified as a parameter when calling the recipe 3 3 Common attributes and methods These attributes and methods are available for all recipes Recipe _ name Recipe name Recipe _ file None Shared library file name Recipe _ author__ Author name Recipe _ email__ Author email Recipe _ copyright__ Copyright string of the recipe Recipe description Pair synopsis description of two strings Recipe version Pair versionnumber versionstring of an integer and a string The integer will be increased on development progress Recipe cpl version Version of the CPL library that is linked to the recipe as a string Recipe cpl_description Version numbers of CPL and its libraries that were linked to the recipe as a string Recipe output_dir Output directory if specified or None The recipe will write the output files into this directory and return their file names If the directory does not exist it will be created before the recipe is executed Output files within the output directory will be silently overwritten If no output directory is set the recipe call will result in pyfits HDUList result objects The output directory may be also set as parameter in the recipe call Recipe temp_dir Base directory for temporary directories where the recipe is executed The working dir is created as a subdir with a random file name If set to None t
21. he system temp dir is used Defaults to Recipe threaded Specify whether the recipe should be executed synchroniously or as an extra process in the background See Also Parallel execution Recipe tag Default tag when the recipe is called This is set automatically only if the recipe provided the information about input tags Otherwise this tag has to be set manually 10 Chapier 3 The Recipe interface Python bindings for CPL recipes Release 0 4 Recipe tags Possible tags for the raw input frames or None if this information is not provided by the recipe Recipe output Return a dictionary of output frame tags Keys are the tag names values are the corresponding list of output tags If the recipe does not provide this information an exception is raised Recipe memory_ dump If set to 1 amemory dump is issued to stdout if the memory was not totally freed after the execution If set to 2 the dump is always issued Standard is 0 nothing dumped 3 4 Recipe parameters Recipe parameters may be set either via the Recipe param attribute or as named keywords on the run execution A value set in the recipe call will overwrite any value that was set previously in the Recipe param attribute for that specific call Recipe param This attribute contains all recipe parameters It is iteratable and then returns all individual parameters gt gt gt for p in muse_scibasic param print p name p value p default nif
22. in contain the name or the fullname with context 3 4 Recipe parameters 11 Python bindings for CPL recipes Release 0 4 gt gt gt muse_scibasic param nifu 1 xbox 11 resample True See Also cpl Parameter 3 5 Recipe frames There are three groups of frames calibration calib frames input raw frames and result product frames Calibration frames may be set either via the Recipe calib attribute or as named keywords on the run execu tion A value set in the recipe call will overwrite any value that was set previously in the Recipe calib attribute for that specific call Input frames are always set in the recipe call If their tag name was not given the tag name from Recipe tag is used if the recipe provides it Recipe calib This attribute contains the calibration frames for the recipe It is iterable and then returns all calibration frames gt gt gt for f in muse_scibasic calib print f tag f min f max f frames TRACE_TABLE 1 1 None WAVECAL TABLE 1 1 None ASTER_BIAS 1 1 master _bias_0 fits ASTER_DARK None 1 None GEOMETRY_TABLE 1 1 None BADPIX_TABLE None None badpix_l1 fits badpix_2 fits IASTER_FLAT None 1 None Note Only MUSE recipes are able to provide the full list of calibration frames and the minimal maximal number of calibration frames For other recipes only frames that were set by the users are returned here
23. ion number gt gt gt muse_scibasic cpl Recipe muse_scibasic version 0 2 07 List all parameters gt gt gt print muse_scibasic param Parameter nifu default 99 Parameter xbox default 15 Parameter ybox Parameter passes default 2 Parameter thres Parameter resample default False Parameter dlambda default none default 40 default 4 5 default 1 25 Parameter cr Set a parameter gt gt gt muse_scibasic param nifu 1 Print the value of a parameter None if the parameter is set to default gt gt gt print muse_scibasic param nifu value N List all calibration frames gt gt gt print muse_scibasic calib FrameDef TRACE_TABLE value None FrameDef MASTER_BIAS value None FrameDef WAVECAL TABLE value None FrameDef MASTER_DARK value None FrameDef BADPIX_TABLE value None r FrameDef GEOMETRY_TABLE value None FrameDef MASTER_FLAT value None Set calibration frames with files gt gt gt muse_scibasic calib MASTER_BIAS MASTER_BIAS 01 fits gt gt gt muse_scibasic calib MASTER_FLAT MASTER_FLAT 01 fits gt gt gt muse_scibasic calib TRACE_TABLE TRACE_TABLE 01 fits gt gt gt muse_scibasic calib GEOMETRY_TABLE geometry_table fits You may also set ca
24. ipe muse_bias myrecipe calib cpl esorex read_sof file muse_bias sof Note The raw data frame is silently ignored wenn setting cpl Recipe calib for MUSE recipes Other recipes ignore the raw data frame only if it was set manually as cpl Recipe tag or in cpl Recipe tags since there is no way to automatically distinguish between them 27 Python bindings for CPL recipes Release 0 4 9 2 Convienence logging control cpl esorex msg cpl esorex CplLogger msg True This variable is a Cpl Logger instance that provides a convienience stream handler similar to the terminal logging functionality of the CPL It basically does the same as import logging log logging getLogger log setLevel logging INFO ch logging StreamHandler ch setLevel logging INFO ch setFormatter logging Formatter levelname 7s message s log addHandler ch The following attributes control the format of the terminal messages CplLogger level Log level for output to the terminal Any of DEBUG INFO WARN ERROR OFF CplLogger format Output format See Also logging LogRecord attributes Key mappings in the logging output CplLogger time If True attach a time tag to output messages CplLogger component If True attach the component name to output messages CplLogger threadid If True attach a thread tag to output messages cpl esorex 1log cpl esorex CplLogg
25. is information is not available this flag ist setto None Execution log cpl Result log List of log messages for the recipe See Also cpl logger LogList cpl Result error If one or more error was set during the recipe run the first error is stored in this attribute The following errors are chained and can be accessed with the cpl CplError next attribute Note An error here does not indicate a failed recipe execution since a failed execution would result in a non zero return code and an exception would be thrown See Also epl CplErr r Thread control If the recipe was called in the background see Parallel execution the result object is returned imme diately and is dervived from threading Thread Its interface can be used to control the thread execution cpl Result isAlive Returns whether the recipe is still running cpl Result join timeout None Wait until the recipe terminates This blocks the calling thread until the recipe terminates either normally or through an unhandled exception or until the optional timeout occurs When the timeout argument is present and not None it should be a floating point number specifying a timeout for the operation in seconds or fractions thereof As join always returns None you must call isAlive after join to decide whether a timeout happened if the recipe is still running the join call timed out When the timeout argument is not
26. legacy support 9 1 Support for configuration and SOF files ee 9 2 Convienence logging COMO s sosse a Daran ae a Se a do cpl dfs DFS header parsing Restrictions for CPL recipes 11 1 Technical Background a 3 reo ea a eo A ne 15 17 19 21 21 22 22 22 23 25 25 26 27 27 28 31 33 12 Feedback Index 35 37 Python bindings for CPL recipes Release 0 4 This is a non official python module to access CPL recipes It is not meant as part of the CPL or the MUSE pipeline software but may be useful for testing and analysis See Also http www eso org sci software cpl The Common Pipeline Library CPL consists of a set of C libraries which have been developed to standardise the way VLT instrument pipelines are built to shorten their development cycle and to ease their maintenance The Common Pipeline Library was not designed as a general purpose image processing library but rather to address two primary requirements The first of these was to provide an interface to the VLT pipeline runtime environment The second was to provide a software kit of medium level tools which allows astronomical data reduction tasks to be built rapidly ESO A PDF version of the user manual can be downloaded from here CONTENTS 1 Python bindings for CPL recipes Release 0 4 2 CONTENTS CHAPTER ONE INSTALLATION 1 1 Prequisites e Python 2 6 or 2 7
27. libration frames with pyfits HDUList objects This is especially useful if you want to change the file on the fly gt gt gt import pyfits gt gt gt wavecal pyfits open WAVECAL_TABLE 01_flat fits Chapter 2 Tutorial Python bindings for CPL recipes Release 0 4 gt gt gt wavecal 1 data field wlcc00 1 01 gt gt gt muse_scibasic calib WAVECAL TABLE wavecal To set more than one file for a tag put the file names and or py fits HDUList objects into a list gt gt gt muse_scibasic calib MASTER_BIAS MASTER_BIAS 202i fits i 1 for i in range 24 To run the recipe call it with the input file names as arguments The product frames are returned in the return value of the call If you don t specify an input frame tag the default first one of the recipe is used gt gt gt res muse_scibasic Scene_fusion_1 fits Run the recipe with a nondefault tag use raw data tag as argument name gt gt gt res muse_scibasic raw SKY sky_newmoon_no_noise_1 fits Parameters and calibration frames may be changed for a specific call by specifying them as arguments gt gt gt res muse_scibasic Scene_fusion_1l fits param nifu 2 calib MASTER_FLAT None WAVECAL_TABLE WAVECAL TABLE noflat fits The results of a calibration run are pyfits HDUList objects To save them use output tags as attribu
28. lled 3 1 Static members Recipe path Search path for the recipes It may be set to either a string or to a list of strings All shared libraries in the search path and their subdirectories are searched for CPL recipes On default the path is set to the current directory The search path is automatically set to the esorex path when cpl esorex init is called static Recipe list Return a list of recipes Searches for all recipes in in the directory specified by the class attribute Recipe path or its subdirecto ries static Recipe set_maxthreads n Set the maximal number of threads to be executed in parallel Note This affects only threads that are started afterwards with the threaded True flag See Also Parallel execution 3 2 Constructor Recipe __init__ name filename None version None threaded False Try to load a recipe with the specified name in the directory specified by the class attribute Recipe path or its subdirectories Parameters e name str Name of the recipe Required Use cpl Recipe list to get a list of available recipes Python bindings for CPL recipes Release 0 4 e filename str Name of the shared library Optional If not set Recipe path is searched for the library file e version int or str Version number Optional If not set the newest version is loaded e threaded 0001 Run the recipe in the background returning immediately a
29. olution of the log time stamp is one second The fields Logging LogRecord args logging LogRecord exc_info and logging LogRecord lineno are not set Also due to limi tations in the CPL logging module level filtering is done only after the creation of the log entries This may cause performance problems if extensive debug logging is done and filtered out by logging Logger setLevel In this case the cpl Recipe __call___ parameter loglevel may be used See Also 25 Python bindings for CPL recipes Release 0 4 cpl esorex msg and cpl esorex log EsoRex like convienience logging 8 2 Log message lists The cpl Result object as well as a cpl CplError have an attribute cpl Result log resp cpl CplError log that contains the list of all log messages class cpl logger LogList List of log messages Accessing this list directly will return the logging LogRecord instances Example res muse_bias bias_frames for logrecord in res log print 3s ts entry funcname entry msg To get them formatted as string use the error warning info or debug attributes res muse_bias bias_frames for line in res log info print line error Error messages as list of str warning Warnings and error messages as list of str info Info warning and error messages as list of str debug Debug info warning and error messages as list of str 26 Chapter 8 Log messages CHAPTER NINE CPL E
30. pl test 0 4 tar gz There are a number of tests defined in test TestRecipe py To run them you need first to compile the recipe in test iiinstrumentp cd test iiinstrumentp bootstrap configure make CA python TestRecipe py Compiling the test recipe needs an installed CPL development environment The tests may print a memory cor ruption detection by glibc This is normal since the tests also check this behaviour in the recipe 4 Chapter 1 Installation CHAPTER TWO TUTORIAL 2 1 Simple example The following code takes BIAS input file names from the command line and writes the MASTER BIAS to the file name provided with the o option from optparse import OptionParser import sys import cpl parser OptionParser usage sprog files parser add_option o output help Output file default master_bias fits parser add_option b badpix table help Bad pixel table opt filenames parser parse_args if not filenames parser print_help sys exit cpl esorex init muse_bias cpl Recipe muse_bias muse_bias param nifu 1 muse_bias calib BADPIX_TABLE opt badpix_table res muse_bias filenames res MASTER_BIAS writeto opt output 2 2 Quick guide Input lines are indicated with gt gt gt the python prompt The package can be imported with gt gt gt import cpl If you migrate from Esorex you may just
31. ribute 10 __file__ cpl Recipe attribute 10 __init__ cpl Recipe method 9 __init__Q cpl dfs ProcessingInfo method 32 __name__ cpl Recipe attribute 10 C calib cpl dfs ProcessingInfo attribute 31 calib cpl Recipe attribute 12 code cpl CplError attribute 23 component cpl esorex CplLogger attribute 28 29 context cpl Parameter attribute 17 cpl module 1 7 16 17 19 26 32 cpl dfs module 31 cpl esorex module 27 cpl Result class in cpl 21 cpl_description cpl Recipe attribute 10 cpl_version cpl dfs ProcessingInfo attribute 31 cpl_version cpl Recipe attribute 10 CplError 23 D debug cpl logger LogList attribute 26 default cpl Parameter attribute 17 description cpl Recipe attribute 10 dir cpl esorex CplLogger attribute 28 E elements cpl RecipeCrash attribute 23 env cpl Recipe attribute 13 environment variable MALLOC_CHECK_ 13 PYTHONPATH 3 error cpl cpl Result attribute 22 error cpl logger LogList attribute 26 F filename cpl CplError attribute 23 filename cpl esorex CplLogger attribute 28 filename cpl RecipeCrash attribute 23 format cpl esorex CplLogger attribute 28 INDEX FrameConfig class in cpl 19 frames cpl FrameConfig attribute 19 func cpl RecipeCrash attribute 23 info cpl logger LogList attribute 26 init in module cpl esorex 27 isAlive cpl cpl Result method 22 J joinO cpl cpl Result
32. t set manually the tag is required when executing the attribute Accessing the attribute Recipe output raises an exception 11 1 Technical Background CPL recipes register all their parameter definitions with the CPL function cpl_parameterlist_append All registered parameters may be retrieved from the recipe structure as a structure which contains all defined parameters For frames such a mechanism does not exist although components of the infrastructure are implemented The CPL modules cp1_recipeconfig allows the definition of input raw and output frames for a recipe How ever this module is only half way done has no connection to the recipe definition and is not mandantory for CPL recipes The MUSE pipeline recipes with the exception of those contributed by ESO implement a central frameconfig registry which allows to access this meta information from the Python interface 33 Python bindings for CPL recipes Release 0 4 34 Chapter 11 Restrictions for CPL recipes CHAPTER TWELVE FEEDBACK Bug reports should be made on the developer web page Send python specific questions to python cpl liska ath cx Questions regading CPL should be mailed to cpl help eso org 35 Python bindings for CPL recipes Release 0 4 36 Chapter 12 Feedback Symbols __author__ cpl Recipe attribute 10 __call__ cpl Recipe method 13 __copyright__ cpl Recipe attribute 10 __email__ cpl Recipe att
33. tes gt gt gt res PIXTABLE_OBJECT writeto Scene_fusion_pixtable fits They can also be used directly as input of other recipes gt gt gt muse_sky cpl Recipe muse_sky gt gt gt res_sky muse_sky res PIXTABLE_OBJECT If not saved the output is usually lost During recipe run a temporary directory is created where the pyfits HDUList input objects and the output files are put into This directory is cleaned up afterwards To control message verbosity on terminal use debug info warn error or off gt gt gt cpl msg level debug 2 2 Quick guide 7 Python bindings for CPL recipes Release 0 4 8 Chapter 2 Tutorial CHAPTER THREE THE RECIPE INTERFACE class cp1 Recipe name filename None version None threaded False Pluggable Data Reduction Module PDRM from a ESO pipeline Recipes are loaded from shared libraries that are provided with the pipeline library of the instrument The module does not need to be linked to the same library version as the one used for the compilation of python cpl Currently recipes compiled with CPL versions from 4 0 are supported The list of supported versions is stored as cpl cpl_versions The libraries are searched in the directories specified by the class attribute Recipe path or its subdirec tories The search path is automatically set to the esorex path when cpl esorex init is ca
34. to control the thread execution The simples way to use parallel processing is to create a list where the members are created by the execution of the recipe The following example shows the parallel execution of the muse_focus recipe muse_focus cpl Recipe muse_focus threaded True muse_focus calib MASTER_BIAS master_bias fits Create a list of input files files MUSE_CUNGC202i fits i for i in range 20 30 Create a list of recipe results Note that for each entry a background process is started results muse_focus f for f in files HR Save the results The current thread is stopped until the according recipe is finished for i res in enumerate results res FOCUS_TABLE writeto FOCUS_TABLE_202i fits 1 1 When using parallel processing note that the number of parallel processes is not limited by default so this feature may produce a high load when called with a large number of processes Parallelization in the recipe itself or in the CPL may also result in additional load 15 Python bindings for CPL recipes Release 0 4 To limit the maximal number of parallel processes the function cpl Recipe set_maxthreads can be called with the maximal number of parallel processes Note that this function controls only the threads that are started afterwards If the recipe execution fails the according exception will be raised whenever one of the results
35. u None 99 cr None dcr xbox None 15 ybox None 40 passes None 2 thres None 4 5 sample None False dlambda None 1 2 On interactive sessions all parameter settings can be easily printed by printing the param attribute of the recipe gt gt gt print muse_scibasic param Parameter nifu default 99 Parameter cr default dcr Parameter xbox default 15 Parameter ybox default 40 Parameter passes default 2 Parameter thres default 4 5 Parameter sample default False Parameter dlambda default 1 2 Y To set the value of a recipe parameter the value can be assigned to the according attribute gt gt gt muse_scibasic param nifu 1 The new value is checked against parameter type and possible value limitations provided by the recipe Hyphens in parameter names are converted to underscores In a recipe call the same parameter can be specified as dict gt gt gt res muse_scibasic param nifu 1 To reset a value to its default it is either deleted or set to None The following two lines gt gt gt muse_scibasic param nifu None gt gt gt del muse_scibasic param nifu will both reset the parameter to its default value All parameters can be set in one step by assigning a dict to the parameters In this case all values that are not in the map are reset to default and unknown parameter names are ignored The keys of the map may conta
36. utput in a style similar to the original CPL messages 8 1 Python style logging The preferred and most flexible way to do logging is the use of the logging module of Python A basic setup similar to the style used in esorex is import logging log logging getLogger log setLevel logging INFO ch logging FileHandler cpl_recipe log ch setLevel logging INFO fr logging Formatter reated s levelname s name s 3 message s y M oe c Hs oe ch setFormatter fr log addHandler ch The default basic log name for CPL log messages in the recipes is cpl recipename The log name can be changed with the logname parameter of col Recipe ___call___ to follow own naming rules or to separate the output of recipes that are executed in parallel res muse_focus f logname cpl muse_focus 02i i 1 threading True for i f in enumerate inputfiles To the basic log name the function name is appended to allow selective logging of a certain function The following sample line logging getLogger cpl muse_sky muse_sky_create_skymask setLevel logging DEBUG will log the debug messages from muse_sky_create_skymask additionally to the other messages Note Since the log messages are cached in CPL they may occur with some delay in the python log module Also log messages from different recipes running in parallel may be mixed in their chronologi cal order The res
Download Pdf Manuals
Related Search