Photometric Assisted Spectral Extraction :: User Guide :: v0.2 :: phAse regroups programs in C++ and Python to 1)- reduce and calibrate FORS1 LSS/MOS images into master frames. 2)- use Deep CFHT references to define the extraction profils of the detected sources. 3)- extract those sources from the science master frame. ================ Presets: ======== Python: All the targets are listed in a Django-MySQL database (DB), as the search rolls. This DB is used to give a name to each pointing (N~3 frames). The DB used for SNLS is 'snlspord', and it must be installed. Tables 'supernovae' from the 'snls' base and 'VLT_ACQ' and 'VLT_MASTER' from the 'spectro' base are used. (see DBtools.py) For graphical outputs, the python module 'MatPlotLib' is used, which must be installed, but works with python2.4, and NOT python2.2! c-libs: To manipulate FITS images, the 'cfitsio' library must be installed. The $CFITSIOLIBDIR and $CFITSIOINCDIR variables must be defined. To invert matrices, the 'cernlib' library must be installed (possibly not trivial). Only cholesky_invert and cholesky_solve routines are used. Also, the $PHASEPATH variable must be set to the phAse install directory. It is used in the Makefiles, and to find the standard star spectra and the extinction law. The deep reference images from CFHT and the lightcurves of the spectroed objects are assumed to be present. Directories of those external inputs, and those of VLT raw and product files are mainly defined in /src/ReducProd/conventions.py, but also in other files, at different levels (see WhereIsWhat section). ================= Contents: ========= Default directories under $PHASEPATH are /src, /bin, /calibs and /logs. The C++ binaries built from the '/src/*/Makefile's are sent to /bin. /calibs contains default calibration functions. /logs will contains the processing logs. Python routines performs calls to those C++ binaries, and rule the organization of products. ** C++ directories are under /src : /reduce -> combining routines to built master frames. /VLTprofil-> routine to integrate a master frame in a broad filter. /extract -> extraction routines, for standard stars and science. /calib -> routine to built a master response from individual ones. /get_ref_lines->routine to caracterize standard star abs. lines. /dumpfits -> routines to extract fits extensions to ASCII tables. ** Python directories are under /src : /ReduceProd -> create cards, assoc. calibs and do masters. /PhotoPriors-> compute profils, match them and extract. /HTML -> create the graphs and log web pages. Binaries and executables python scripts display a usage message for a call without arguments. 'addDATES.csh' does not, but 'more' of it is equivalent. ** We tried to comment the codes so that it might be understood by others, but some parts have changed a lot, and migth change again, and may not be very easy to read through. ================== PROCEDURE: ========== Assuming we want to produce the 2004 observations : ESO frames from a given night are stored under $SPECdir/YYMMDD/. *** Record the FORS1 frames in $SPECdir : *** /src/ReducProd >./addDATES.csh 04 => Add FORS1*.fits files under /$SPECdir/04*/rawdata/ to 'VLT_ACQ' table of 'spectro' base. ( $SPECdir is defined in addDATES.csh !) *** Group them per type, setup and time lag : *** The MMJD date limits of this set is 360 to 720. (MMJD=1 at 030101). /src/ReducProd >./writeCards LSS 360 720 ../../logs/ /src/ReducProd >./writeCards MOS 360 720 ../../logs/ => creates the cards under $SPECdir/CARDS/[BIAS,FLAT,WAVE,STD,SKY] of the groups of frames to be later combined together. The time interval for grouping are defined in 'conventions.py'. The lists of cards created are saved in /logs/. An alternative way is to call successively BiasCards.py ... StdCards.py. *** To produce the Bias master frames, then Flats, then others : *** /src/ReducProd>./DOmasterBIAS.py LSS T /src/ReducProd>./DOmasterBIAS.py MOS T => combine all the $SPECdir/CARDS/BIAS/LSS*T*.sof corresponding master bias under $SPECdir/MASTER/BIAS/LSS*T*_combN.fits. T stands for trimester, and is the default bias grouping time. The set of bias card of biases are of the form LSS_BIAS_T5.sof for the 5th trimester. /src/ReducProd>./DOmasterFLAT.py LSS 1.01_300V /src/ReducProd>./MatchWaveCalibs.py LSS 1.01_300V WAVE /src/ReducProd>./DOmasterWAVE.py LSS 300V /src/ReducProd>./MatchStdCalibs.py LSS 300V STD /src/ReducProd>./DOmasterSTD.py LSS 300V /src/ReducProd>./DOextractSTD.py MASTER/STD 300V 360 720 => Create all Grism 300V master calibrations and associate them to the further calibration sets of frames *.sof. If the Standard star is taken in MOS, MOS equivalent calls must be done. Exemple froms of card names are : $SPECdir/CARDS/FLAT/LSS_FLAT_1.01_300V_378.sof $SPECdir/CARDS/FLAT/LSS_WAVE_1.01_300V_377.sof $SPECdir/CARDS/FLAT/LSS_STD_1.01_300V_381.sof Images have similar names, appending '_combN.fits', N being the number of frames in the set (1 for wave and std), and are under $SPECdir/MASTER/[BIAS, FLAT, WAVE, STD]. The extracted Standard star spectra are stored under e.g. : $VLTdir/REDUCE_V2/STD/LSS_STD_1.01_300V_381/LSS_STD_1.01_300V_381_2Dresidu.fits together with some control tables. *** To obtain the response function model : *** We propose to combine the responses derived from single standard star acquisitions together : /src/ReducProd>./RespCards.py $SPECdir/REDUCE_V2/STD 360 720 ../../logs/RespCards.list /src/ReducProd>./DOmasterRESP.py 300V 360 720 => Create the response cards and produces the combined response functions from all 300V extracted standard spectra. /src/ReducProd>./MatchSkyCalibs.py LSS 04 MASTER/SKY /src/ReducProd>./DOmasterSKY.py LSS 04 360 720 => Create the combined images corresponding to the *04*.sof's found under $SPECdir/MASTER/SKY/*04*_combN.fits. Exemple of science card name, in the case of SNLS survey, is $SPECdir/CARDS/SKY/04D4it_629.sof and the combined spectrogram is $SPECdir/CARDS/SKY/04D4it_629_comb5.fits *+*+*+*+* This ends the Reduction Processing *+*+*+*+* Now, to perform the science extraction : from /src/PhotoPriors /src/PhotoPriors>./PrepareSN $SPECdir/MASTER/SKY 04 => create the Deep CFHT slit cuts, the VLT profiles, matches them, and performs multi-PSF extraction of all $SPECdir/MASTER/SKY/*04*_combN.fits images. Priors and Products are stored, e.g., in $SPECdir/REDUCE_V2/SKY/04D4it_629/04D4it_629_2Dresidu.fits /"_PHOTtrc.dat, /"_REFextr.dat, /"_PSFspec.dat,... *** To create web pages summarizing the observations *** /src/HTLM>./MakeSNpages.py REDUCE_V2/SKY 04 => creates a $SPECdir/WEB/*04* dir, copies and written informations files for all $SPECdir/REDUCE_V2/SKY/*04*_2Dresidu.fits file previously created by /src/PhotoPriors/PrepareSN.py, itself calling /bin/SkyExtract. /src/HTML>./summarySN $SPECdir/WEB => creates the summary htlm page of obs, grouped par field and year, with links to individual pages. ====================== WHERE IS WHAT: ============== Listing the hardcoded non internal entries, we account for : - /src/reduce/make_Nstd.cc & make_Nsky.cc extinctio law : CTIO : '/calibs/atmoext_ctio.dat' - /src/VLTprofil.cc Photo. filters : '/calibs/Filters/cfht_[GRIZ]_filter.dat' - /src/ReducProd/conventions.py std.stars spectra : STDSTARlist='/calibs/FORS_Std/FORS_STD.list' ESO obs logs for DIMM seeing : LOGSdir='~/SNLS/ObsLogs/' Root VLT data dir: SPECdir='/sps/snls/snls-spectro/VLT/' - /src/PhotoPriors/SN_values.py lightcurves : LCdir='/sps/snls/LIGHTCURVES-v1.2-rev10' also defined in - /src/PhotoPriors/traceConventions.py with others... - /src/PhotoPriors/stampsConventions.py deep images path : REFdir='/sps/snls/REFERENCES/' and revision, seasons ... Listing all the cuts would probably fill a full page. The lower level C++ binaries usually allow manual setting through command options. The major Thresholds can thus be tunned, changing the command string in the higher level Python script. Other choices (polynomial orders, pixel margins, minor thresholds) can only be changed through the code, and re-compilation. *** Default Rejection Levels are : *** -'cosmic' rejection on bias , flat, wave, std: 8 sigma on sky : 5 sigma on wave :... - Source detection : flux : 8sigma psf : Fgauss>0.66*Fmax color: RMS(r,i,z)>0.05, arm/core. - SN proximity : dHost < 0.2 * FWHMobs. *** Internal Calibration Files *** Under '/calibs/' : - Arc line weigth : from 'arc_lines.dat' : - Instrumental responses 300V & 300I : default, from MIDAS. : 'fors1_300V_resp.dat' & 'fors1_300I_resp.dat' - sky lines and telluric bands: 'telluric_bands.dat' et 'sky_lines.dat' - Photometric filters : under /calibs/Filters' : cfht. ======================= USELESS FILES : =============== phAse is still under development, thus some scripts are not used, but may be one day ( i.e. src/ReducProd/MatchCalibs.py or src/calib/Syst_Nresp.cc). ========================== TROUBLE SHOOTING : ================== - MOS Flat Fields In MOS mode, for the flat fields combining, a relative flux normalisation is applied, based on the central slitlet projected flux ratios between frames. Appling this normalisation to other slitlets is not relevant. The ratios should be computed/applied in wavelength space, or computed for each slitlet. Also, out of the illuminated region, the lamp spectra is made by the average diffuse light, and the spectral normalisation varies a lot depending on the local level of diffuse light. Implementing a solution is not easy, maybe using a diffuse spectra for each slitlet. Those regions are discarded later, so it is of an esthetical concern.