;+ ; NAME: ; lpipe (version 2022.07) ; ; PURPOSE: ; Fully-automated reduction of LRIS imaging and spectroscopy. ; ; CALLING SEQUENCE: ; lpipe, [settings] ; ; INPUTS AND KEYWORDS (most common; see manual for more): ; directories - List or search string of subdirectories. (Usually blank.) ; General: ; data - Location of raw data (current directory if unspecified) ; /help - Print some basic help information ; reducer - Specify name of user to add to the file headers ; Step control: ; mode - Mode to process (imaging or spectroscopy) ; start - Start with this step, skipping previous ones ; stop - End with this step, skipping subsequent ones ; step - Do only this step ; /redo - Repeat actions, overwriting any existing files ; File control: ; camera - Camera to process (red or blue) ; chip - Chip to process (left or right) ; files - Specific file numbers to process ; filters - Specific filters to process ; gratings - Specific gratings to process ; targets - Specific targets to process ; date - Specific night or month to process (e.g. '150423') ; Options: ; preparation/formatting ; /prepareall - Bias-subtract multislit and dark files (still ignores specpol) ; /skipfocus - Skip frames associated with focusing ; flat-fielding ; flatrequire - Specify the number of images needed to make a flat-field ; /norebin - Don't rebin if no flats exist in same binning as data ; /noreflat - Do not perform super sky flat correction for images ; altflatdir - Give an alternative flat-field directory (archival flats, etc) ; /forceflat - Produce all flatfields even if not needed for science data ; /forcereflat - Tell the pipeline to use reflat step even if not enough files/fields ; /ignoredichroic- Allow dichroic differences when flat-fielding images ; /rightchipdead - Normalize flats using left CCD ; cosmic ray cleaning ; /nocrclean - Do not zap cosmic rays at all ; crzeal - Adjust cosmic ray cleaning (default=1.0, >1.0:more aggressive) ; sky subtraction ; skyalgorithm - Use a different sky-subtraction algorithm ('kelson') ; summing ; /nosum - Do not coadd any repeat spectra (e.g., for time series) ; tracing ; /norbalign - Don't attempt to align red and blue traces of the same target ; spectrum extraction ; /noreaddsky - Rely on 2D sky subtraction for sky-line removal. ; wavelength calibration ; altarcdir - Give an alternative arc-lamp wavelength solution directory ; /forcewavcal - Solve wavelength solution even for marginal lamp combinations ; /novalidatewavcal - Do not validate wavelength solution. ; flux-calibration ; altresponsedir - Give an alternative flux-calibration directory ; /notelluric - Skip telluric correction ; red-blue connection ; /requirepair - Do not write out final spectrum without both red+blue ; astrometry ; /distort - Apply distortion solution (not well-tested, may not work) ; photometry ; /nophotometry- Skip zeropoint calculation and attenuation adjustment ; /photometric - Use zeropoint solution to calibrate non-SDSS/PS1 fields ; visualization/output ; /nodisplay - Don't pop up display windows (for remove/server reductions) ; /nofailuresum- Hide the summary of unprocessed files printed at the end ; /debug - Print out more info; turn off some exception handling ; /timer - Print out processing times ; /shortnames - Exclude the six-digit date string from output files ; Other ; /continuous - Keep running, assimilating new data as it appears ; /configure - Write configuration file to disk for even more control ; ; ; ADDITIONAL OPTIONS: ; If any of the following files are found in the directory where lpipe ; is run, it will change the default behavior. (These have not been tested ; recently.) ; ; lpipe.par - Contains various defaults (mainly directory paths) ; catalog.txt - Source positional catalog. Use to auto-correct the object ; name in the header based on position (independent of TARGNAME.) ; brokenlamps.txt - List of arc lamps that are not functioning. ; userobjectpos.txt - List of files and extraction positions (y-coordinates). ; Enables the user to specify which object(s) to extract. ; Now handled visually by lrisapertures, or see below to set manually ; seeing.txt - A file with only two numbers, the minimum and maximum seeing ; experienced in arcseconds. (obsolete?) ; extendedsources.txt - List of imaging fields (TARGNAMEs) containing large ; galaxies or other sources for which median-filtered sky subtraction is ; not appropriate. Scalar background subtraction is used instead. ; These will also be excluded from constructing supersky flats. ; badflattargets.txt - List of additional targets not to use in making a ; supersky flat (e.g., fields with lots of scattered light.) ; badfiles.txt - List of files not to process at all. ; ; ; ; OUTPUTS: ; The following files are written to disk (from most to least processed): ; ; Imaging - ; * lris[object]_[filter].fits - Reduced, stacked image ; stars[object].cat - Photometric catalog for a target ; coadd[object]_[filter].[l|r].fits - Reduced, stacked image, one chip only ; eazfp[b|r]yymmdd_nnnn[l|r].fits - Individual aligned image ; (one exposure) ; eazfp[b|r]yymmdd_nnnn[l|r].tca[t|l|z]-Calibrated list of stars in image ; using another exposure of this field ; eazfp[b|r]yymmdd_nnnn[l|r].cal - Calibrated list of stars in image ; using SDSS imaging of this field ; eazfp[b|r]yymmdd_nnnn[l|r].caz - Calibrated list of stars in image ; using a zeropoint/airmass solution ; eazfp[b|r]yymmdd_nnnn[l|r].cat - Uncalibrated list of stars in image ; azfp[b|r]yymmdd_nnnn[l|r].fits - Image solved directly against an ; astrometric catalog catalog ; ; Spectroscopy - ; * lrisyymmdd[object].spec - Flux-calibrated spectrum of an object, ; combining blue/red and all exposures. ; ASCII table (read with readspec.pro) ; cels[b|r]yymmdd_nnnn[l|r].[obj].spec - Flux/wave-calibrated spectrum ; els[b|r]yymmdd_nnnn[l|r].[obj].spec - Wave-calibrated 1D spectrum ; (not flux-calibrated) ; xls[b|r]yymmdd_nnnn[l|r].[obj].spec - Extracted, uncalibrated 1D spectrum ; ls[b|r]yymmdd_nnnn[l|r].[obj].fits - Summed 2D spectrum ; szfp[b|r]yymmdd_nnnn[l|r].fits - Sky-background subtracted 2D spectrum ; ; ; Both - ; zfp[b|r]yymmdd_nnnn[l|r].spec - Cosmic ray-cleaned image ; fp[b|r]yymmdd_nnnn[l|r].spec - Flat-fielded image, split between the ; disconnected left/right chips ; fp[b|r]yymmdd_nnnn.spec - Flat-fielded image ; p[b|r]yymmdd_nnnn.spec - Overscan-subtracted, bad-pixel ; marked, coverted to simple format, ; updated header (otherwise raw) image ; ; Only the starred (*) files are written to the current directory. Other ; files are written to the imredux/ or spredux*/ directories. Processed ; calibrations are written to the flats/, arcs/, or response/ directories. ; If using KOA file name conventions, yymmdd_nnnn becomes yymmdd.nnnnn. ; ; ; ; COMMENTS: ; This code is meant to be fully automated, producing science-quality ; output with a single command line with zero intervention. Reduces ; both LRIS-B and LRIS-R data (including LRIS-R1 with some limitations) in ; either imaging or spectroscopy mode. ; ; The production of images is quite high-level and includes photometric ; calibration of the field (although the accuracy of this has not been ; robustly tested). Astrometric distortion corrections are currently ; in development and may not always work correctly: please report any ; difficulties with the astrometry. ; ; Full spectroscopic extraction and reduction (including flux calibration) ; is also available. ; ; The program works in a series of steps following standard CCD reduction ; techniques, automatically recognizing calibration files, matching ; calibrations together and science frames with appropriate calibrations, ; etc. Users looking for more oversight over the reductions can run each ; step individually with the "step" command. ; ; If the reduction is interrupted, the program will resume without ; incident (although any failed steps may be repeated); each task is ; run independently of all others. ; ; The program tries to correct for observer mistakes (such as inconsistent ; windowing or poor-quality calibrations) or instrument problems where ; possible. Some situations require user attention to obtain a reliable ; reduction - see the troubleshooting page if the pipeline does not return ; the results you expect. ; ; Filenames for the input raw images are expected to be in one of the ; standard formats output by the telescope or KOA archive. ; Compressed (.gz) files are also acceptable. They can be either in the ; current IDL working directory or in a different directory as specified ; by the 'data' keyword at the IDL prompt at runtime. ; ; The code runs in a series of steps in the following order: ; ; 1. "Prepare" the data by converting the multi-header extension FITS files ; to a standard single frame, correcting/flagging known pixel defects, ; cropping unused area of the chip, bias-subtracting using the overscan, ; and adding extra information to the header. The X and Y axes ; for spectroscopic frames are transposed for easier viewing of traces on ; standard monitors and to match 1D spectrum plots (wavelength=x-axis). ; Output: p[b|r]*.fits (in imredux/ and spredux2d/ subdirectories) ; ; 2. Create flat-fields. The program searches through all science exposures ; and determines what flat fields are needed, then attempts to make the ; best possible flat for each case. In the case of imaging flats, ; stars are identified and masked out. For spectroscopic flats, a response ; correction is applied (these are pixel flats only). Regions of a ; spectroscopic flat with low signal (common for LRIS-B in the UV) are ; not flat-fielded in the wavelength direction (slit correction only). ; Output: flats/l[b|r]*flat*.fits ; ; 3. Flat-correct data. Divide each image by the flatfield. A more ; refined cropping is also done at this stage, depending on the placement ; of the image area during the run (which is variable.) ; Output: fp[b|r]*.fits ; ; 4. "Re"-flatten imaging data by constructing a super-sky flat using a ; median of the science images in a given configuration. This is ; customized for each exposure (the exposure itself is excluded from the ; flat-field to avoid self-division, which can create peculiar image noise ; statistics). For LRIS-R1 (pre-2009) I- or RG850-band data this process ; will be subtractive rather than divisive (fringe correction), but this ; is not implemented yet. (This can be slow and isn't helpful unless a ; great deal of imaging is acquired during the night, and is off by ; default.) ; Output: ifp[b|r]*.fits (imaging only) ; ; 5. Split the data into separate right and left chips. (Because of the ; physical gap and rotation between the LRIS chips, they must be treated ; independently after this point.) A standard astrometric distortion is ; added at this point for imaging exposures. ; Output: [i]fp[b|r]*[l|r].fits ; ; 6. Removes cosmic rays, using the independent routines pzap.pro and ; pzapspec.pro. See those programs for more information. ; This can be a time-consuming process. ; Output: zfp[b|r]*[l|r].fits (the "i" is removed from imaging.) ; ; At this point different procedures are applied for imaging and spectroscopy. ; For imaging: ; ; 7. Solve astrometry of the field against the best available online catalog ; (SDSS or USNO-B1.0), using the independent autoastrometry.py code. ; (Also requires sextractor to be installed.) ; All images are solved against a common star catlog, then each image is ; solved in a relative sense (optionally including distortion correction) ; against the image nearest to it, starting from the central image. ; Output: azfp[b|r]*[l|r].fits (catalog alignment) ; eazfp[b|r]*[l|r].fits (relative alignment) ; ; 8. Produce photometric solution of the field. The GSFC IDLastro aper.pro ; program is used to measure photometry of point sources in each field ; for comparison of different exposures (on the same target) to establish ; the relative zeropoints. ; Output: eazfp[b|r]*[l|r].[t|][cat,cal,caz] and other text files ; ; 9. Stack exposures. A weighted, masked median is performed for each field. ; Requires swarp to be installed and properly linked. ; Output: coadd[object].[filter].[l|r|]fits ; lris[b|r][object]_[filter].fits ; ; For spectroscopy: ; ; 7. Subtract night-sky emission lines by fitting low-order polynomials in ; vertical blocks passing across the data. Can introduce artifacts in ; some situations (e.g. poor flatfielding or 0.7" slit). Slow. ; Output: szfp[b|r]*[l|r].fits ; ; 8. Sum exposures. This uses only the header information: in the event of ; guiding loss this will not be desired! ; Output: ls[b|r]*.fits (in spredux1d/ subdirectory) ; ; 9. Determines the approximate trace position of all objects, starting with ; standard-star observations. ; Output: ls[b|r]*.trace, ls[b|r]*.profile, objectpos.dat ; ; 10. Extract trace to produce a 1D spectrum of the object of interest. ; Output: xls[b|r]*.spec ; ; 11. Wavelength-calibrate arc frames. ; Output: fp[b|r]*[l_r].sol ; ; 12. Applies and adjusts wavelength calibration. ; Output: els[b|r]*.spec ; ; 13. Flux-calibrate the 1D spectrum using observations of standard stars. ; Output: cels[b|r]*.spec ; ; 14. Combine all spectra of a single object in a single grating. ; (This is an optional step and is now skipped by default.) ; ; 15. Connect red and blue sides of an object to produce final spectrum. ; Output: lrisyymmdd_[object].spec ; ; ; ; EXAMPLES: ; 1. In a directory containing a full night worth of LRIS data, enter: ; ; IDL> lpipe ; ; This will automatically execute all of the steps above. ; Depending on the quantity and binning of data from your run and the ; speed of your computer, the reduction of data from an entire night ; takes 20 minutes up to about three hours. ; This defaults to reducing only the right chip, ignoring the left. ; ; ; 2. Fully reduce only spectroscopy, only the red camera: ; ; IDL> lpipe, mode='s', camera='red' ; ; ; 3. Run only the "prepare" step (bias subtraction/formatting), on raw data ; stored in a separate directory: ; ; IDL> lpipe, step='prepare', data='/scr3/user/lris/20120501/' ; ; ; 4. Run all the 2D spectroscopy reduction steps, but don't do any later ; (1D) reductions. ; ; IDL> lpipe, mode='s', stop='skysubtract' ; ; ; 5. Reduce all the imaging, include both left and chips (full field): ; ; IDL> lpipe, mode='i', chips='rl' ; ; ; 6. Convert KOA-style filenames, ; move raw data to another directory, then run lpipe, and plot all ; the spectra for quick viewing: ; ; IDL> lrisrename, 'L?.*.fits' ; IDL> $mv [b|r]*_????.fits raw/ ; IDL> lpipe, data='raw/' ; IDL> lrisplotall ; ; 7. Reprocess the extraction and all subsequent steps of a target: ; ; IDL> lpipe, mode='s', start='extract', target='J1910+1234', /redo ; ; ; 8. Display some information at the command line: ; ; IDL> lpipe, /help ; ;- ; ; ; INSTALLATION: ; ; Create the subdirectory 'lpipe' somewhere on your hard drive (probably ; in your IDL directory), and unpack the contents of this installation file ; there (e.g., tar -xvf lpipe.tar.gz). You will need to tell IDL about ; the existence of this new directory by editing the IDL_PATH system variable: ; add the string ":+/path/to/lpipe:+/path/to/lpipe/dependencies/" to whatever ; paths are stored there currently, replacing "/path/to/" with the actual path. ; (The variable will need to be edited in your .bashrc, .cshrc, or .idlenv file ; to be available for future use.) The GSFC IDLastro routines must also be ; installed (and visible within IDL_PATH); see http://idlastro.gsfc.nasa.gov/ ; for programs and instructions. ; ; In order to process images, you will also need to have autoastrometry, ; swarp and sextractor installed (this requirement will eventually be removed ; via a simplification of the astrometric solver method). If the latter two ; cannot simply be called via "swarp" and "sex", you may need to edit the ; file lpipe.par AND also edit the dependencies/autoastrometry.py file ; to indicate the actual commands to call these routines in the global ; variables at the top of the code. The standard UNIX routine wget is also ; used to download star catalogs. These are not necessary for spectrosopic ; reductions. ; ; For a Caltech astronomy cluster matchine, it is recommended to simply point ; your IDL path variable to the latest release version in Daniel Perley's ; home directory (see citsetup.txt for more info.) This will ensure you ; always have the most up-to-date version. ; ; More detailed installation instructions are given in installation.txt. ; ; ; ; ; TROUBLESHOOTING: ; ; While this pipeline is designed to deal with all possible observing ; circumstances (including many common mistakes), much more testing and ; development will be required before this ideal is fully reached. ; Despite best efforts, the program may crash if it encounters an ; unanticipated situation or has problems accomplishing its goals and is ; unable to proceed. If you encounter problems, try e-mailing Daniel Perley ; (d.a.perley@ljmu.ac.uk) for assistance, after checking the below. ; ; If the pipeline does not crash, but does not process any files: ; ; * Is the pipeline unable to find a necessary function or subroutine? ; Double-check that you have set up IDL_PATH correctly and that the GSFC ; IDLAstro libraries are installed (see installation). ; ; * Are the data files in a different directory? ; If the raw data files are not in the current working directory, you ; need to point the pipeline to the directory containing raw data with ; the "data" keyword in lpipe.par. ; ; * Are you using old LRIS data? ; The pipeline assumes files follow the standard convention as written ; at the telescope (since ~2010) or the KOA filename convention. If ; your files are named using a different convention (e.g. older data) ; then you can rename them by executing the following command (assuming ; only raw data is in the current directory): ; IDL> lrisrename, '*.fits' ; If important files have the same sequence number this will cause them ; to be skipped because they don't have unique ID's. Instead use: ; IDL> lrisresequence, '*.fits' ; ; * Did you already run the pipeline once on these data, but want to re-run? ; If you want to re-do a step, it will not overwrite existing ; files by default (the pipeline is designed to be able to automatically ; resume from wherever it left off if interrupted or if new data is ; added.) To override this behavior, set the /redo keyword, which will ; overwrite files from any previously-attempted steps. Be sure to set ; the "start" or "step" keywords to the step you want to start at unless ; you want to restart the pipeline from the beginning for all files. ; ; * Is the pipeline not progressing beyond a certain stage of operation? ; Likely, this is because of the lack of an appropriate calibration file ; (e.g. a flat-field, arc, or flux-standard taken in the same setting.) ; You may be able to trick the pipeline into processing a file using ; a calibration taken in some other setting by editing header keywords, ; though obviously this should be used with caution. ; ; ; ; If the pipeline halts or crashes during operation: ; ; * Did it report a 'file not found' or 'variable is undefined' error? ; If so, a subroutine, configuration file, or catalog it needed was not ; found at the expected location. Generally this is an installation ; problem, but you can inspect the paths in lpipe.par to make sure ; that the directories and files specified there actually exist. ; ; * Did an external call (autoastrometry, swarp, sex) fail? ; Check that you have specified the paths correctly and actually have the ; required software installed for astrometry/coadding (see "installation", ; above.) Alternatively if you don't care about reducing images, specify ; mode='s' at runtime to process only the spectroscopy, which uses only ; IDL routines and requires no external packages (except IDLastro). ; ; * Did it encounter some other error while processing a file? ; Check which file the program was working on when it crashed. If the ; file is not essential, try deleting it and re-running the pipeline ; starting with the current step (or delete ALL of the file's precursors ; with the same file number and rerun the pipeline.) Please report the ; error (and perhaps include a copy of the deleted file) so the problem ; can be identified and fixed in future releases. ; ; * Did it fail to generate an appropriate flatfield? ; If science frames are not being flat-fielded, setting the /forceflat ; option and rerunning starting from makeflat may help. Otherwise, ; you may be able to obtain calibrations from a different night and ; copy them to this directory. If desparate, you may be able to trick ; the pipeline into processing a file using a calibration taken in ; some other (closely related) setting by editing header ; keywords, though obviously this should be used with caution. ; ; * Did wavelength calibration fail? ; It is common for some arc wavelength solution attempts to fail, usually ; because the observations were taken before the Cd and Zn lamps warmed ; up fully, although the arc might also be saturated because of too much ; binning (red 2x2 arcs are problematic) or too long an exposure time. ; This is fine as long as other, good arcs exist and solve ; correctly. If all arc calibrations fail for a given setting, the ; pipeline cannot perform most of the 1D spectroscopic reductions. If ; there are no good calibrations in the required setting, you can try to ; find an archival arc somewheresomewhere in the same setting. ; If good arcs exist and aren't being solved, possibly the central ; wavelength is off by enough to prevent automatic line matching from ; succeeding. Editing the CWAVE parameter in the file headers may fix ; things. ; ; * Is it failing to perform flux calibration? ; It is also possible that no useable flux calibration standard is available. ; Check that the data directory contains a flux standard in the ; correct configuration. If one exists, it is possible that its processing ; was stopped at an earlier stage (e.g. lack of appropriately binned flat, ; wavelength calibration failure) - rerunning the pipeline might highlight ; this if so, and the solutions above may apply here as well. If the response ; solution of the standard was successfully produced (see response/ directory) ; it is possible that its configuration was not the same as your science data: ; double-check this. If so, you may need to obtain a standard from elsewhere. ; ; ; If processing completes, but the results are problematic: ; ; * Did it extract the wrong object, or no object at all? ; The pipeline does its best to determine which source was yours based on ; the location and brightness of its trace, relative to that of standards. ; If it guesses wrong or if you need more control over the placement of ; the aperture or the background window, the lrisapertures tool provides ; a visual way to inspect and modify your object apertures. Run: ; IDL> lrisapertures ; This offers an intuitive visual interface to change, add, or subtract ; extraction apertures. (Aperture settings are stored in the file ; userobjectpos.txt, which can also be edited by hand.) Be sure to edit ; the corresponding red file to match the blue, and vice versa. ; Then, re-run the pipeline starting with start='extract' (and /redo). ; You can also inspect (although not edit) the apertures/traces within ; DS9 (open the FITS 2D spectrum and then load [filename].tracepos.reg) ; or view see every file at once by opening "lristraces.html" in a web ; browser or loading profiles[b|r].ps. ; You can update these plots as follows: ; IDL> lrisplotprofiles, camera='b' ; (or camera='r') ; ; * Does one (or a few) particular output spectra look bad? ; Aperture placement issues: ; If just a few spectra show problems (in particular, square-wave-like ; "oscillations" or other jumps) this probably originates from the fact ; that the background aperture was placed over another object. ; The pipeline may also have selected the wrong object for extraction or ; extracted different objects on the red and blue sides. In either case, ; the solution is to simply type "lrisapertures" and reposition the ; apertures and/or background locations (see above item). ; In a future release the background placement will automatically avoid ; contaminating objects to reduce the incidence of this problem. ; Tracing issues: ; For very faint objects the tracing may have failed. You can ; check the trace placement using the tracing .html files, by loading ; the appropriate extraction region file on top of the 2D spectrum in ; DS9, by re-running the profile plotter directly (lrisplotprofiles), ; using the 2D view within lrisapertures, or using lrisvalidate. ; There is no simple solution to tracing issues if encountered. ; ; * Do many spectra, or all spectra in a particular configuration, look bad? ; If all your spectra show similar issues the issue is more likely to be ; connected to the calibrations. ; Wavelength calibration: ; Issues with wavelength calibration will propagate to the flux ; calibration. Very poor red-blue matching or telluric corrections are ; tell-tale signs of wavelength calibration problem. You can also look ; or signs of issues in the wavelength calibration in the output logs, ; or in the wavelength solution plots (spredux1d/*wavsol.ps). ; It is common for some individual arc frames to have poor wavelength ; solutions (because of non-warmed lamps) but the pipeline avoids using ; these and they don't cause any problems. However, if the wavelength ; calibration for all arcs shows issues this is a major problem. ; There is no easy solution for this, but you can delete any bad arcs ; (and acquire new ones or use archival arcs), or override the ; wavelength solution by writing your own polynomial terms into the ; *arc.sol files. ; On rare occasions, the instrument itself can be the source or problems: ; an individal arc lamp can malfunction and not actually turn on when the ; header keywords say that it is on, or the grating angle can not be what is expected. ; Workarounds exist (edit brokenarcs.txt or the WAVELEN keywords in the ; file headers.) ; Flux calibration: ; The flux calibration itself is usually robust except when the ; wavelength calibration fails in a key region, e.g. around the dichroic. ; If the wavelength calibration is good but flux calibration is not, you ; can inspect individual standard star's flux solutions and delete any ; bad ones. If your target was observed at a very different airmass from ; your standards or in very different seeing conditions, this is ; difficult to calibrate well due to differential slit loss and ; extinction. You may just have to live with a sub-optimal flux ; calibration unless you can acquire a new standard-star observation. ; If your flux calibration star was poor (faint, too many lines, no bright ; and no sky lines for flexure correction) this will also lead to ; smaller issues. ; ; * Is the output spectrum noisier than you expected, or does it contain ; narrow artifacts? ; - Possibly the aperture is too large or the sky aperture contains ; another object that's being subtracted from yours (use lrisapertures). ; If the slit drifted during observations (loss of guiding) this will ; also cause difficulties. ; - Poor sky subtraction may be a culprit in some gratings. You can ; try turning off 2D sky subtraction within lrisapertures. ; - Residual cosmic rays not removed by the pipeline may remain in your ; final object spectrum. You can try filtering these more aggressively ; by adjusting the nsigma parameter in pzapspec, but watch out to be ; sure narrow features aren't zapped. ; - Especially if the 0.7" slit was used, flexure in the spatial ; direction can cause significant noise due to residual flat-field ; variations associated with tiny slit defects. A tool exists to remove ; these (reflatslit) but it is not incorporated in the pipeline or ; documented yet. ; ; * Is the cosmic ray removal malfunctioning? ; In some circumstances bright nebular emission lines can be zapped, ; especially in conditions of good seeing and binned data. Real cosmic ; rays can also be missed. If you are having problems with an image or ; set of images, you can adjust the cosmic ray cleaning aggressiveness ; with the crzeal parameter (default 1.0; increase to make it more ; aggressive or decrease to make it less aggressive). Some cosmic rays ; (those that streak purely vertically or horizontally) are not well ; cleaned by the algorithm and difficult to remove in any setting. ; ; * Do the reduced images look bad? ; The most common source of this is bad flat-fielding. Especially if ; twilight flats were not taken, the routine may have attempted to ; construct a supersky flat, which is difficult to do with Keck due ; to bright stellar halos and (especially) galaxies. There is a way ; to specify fields not to use in supersky flats (badflattargets.txt). ; Fields with bright galaxies are also susceptible to sky over- ; subtraction during stacking (within swarp). Sky subtraction can be ; turned off for these fields (currently manually by modifying the swarp ; file, but a semi-automated method will be coming soon.) Other times, ; a bad image can result from unavoidable instrument issues such as ; scattered light from a bright star outside the field. ; ; * Were certain files skipped and not reduced? ; If the pipeline encountered a non-fatal problem processing an ; individual image or spectrum (such as a failed wavelength calibration, ; inability to flatfield or calibrate, or no trace to extract) then it ; will simply not process that file any further. For most such cases a ; summary of the affected files and the problems will be printed out at ; the end of processing. If a file is not being ; processed and you do not see it in the final summary, you can simply ; rerun the pipeline (without deleting any files and without setting the ; redo flag) and it will try to repeat any failed steps of this nature, ; so you can see the issues during runtime without them getting lost in ; other notifications from successful steps. ; ; ; The risk of encountering pipeline issues can be significantly reduced ; by following standard observing procedures, below. ; ; ; ; OBSERVING TIPS: ; ; Spectroscopy - ; ; Slit: The 0.7" slit has many defects, and because this slit moves around ; due to instrument flexure this inevitably results in extremely poor ; flat-fielding and sky subtraction. Use the 1.0" or 1.5" slit instead. ; (However, a new routine is now available to correct 0.7" slit data ; using the sky lines. For now it needs to be run manually.) ; ; Binning: Do not bin the red side by more than 2x1. Further binning ; (e.g. 2x2) causes severe saturation issues on bright arc lines needed ; for automated wavelength calibration, and greatly increases the ; likelihood of false-positive cosmic-ray removal. ; ; Arcs: Please turn the mercury lamp on during red-side arc exposures (in ; addition to the standard neon and argon lamps). If using a setup with ; the D560 dichroic where continuous wavelength coverage is desired, tune ; the central wavelength during setup to be sure that the bright Hg ; line appears on the red side as well as the blue - this is needed for an ; accurate wavelength solution across the D560 dichroic. Always make sure ; the lamps are adequately warmed, and try to obtain calibrations not at ; the horizon location. If you change the configuration during the night, ; it is also a good idea to take at least one new arc before changing it ; again: it is possible that a component may not be where you think it is. ; ; Flats: Halogen flats are poor (uneven illumination, variable spectrum, ; significant scattered red light present on the blue side) and should not ; be relied on. Spectroscopic dome flats are superior: turn on the ; spectral dome lamp and get flats as you would for imaging. Note that ; the UV cannot be pixel-calibrated with either the halogen OR with dome ; flats - the lamp is not hot enough. Twilight flats and deuterium-lamp ; flats should provide sufficient UV signal. Preliminary support for ; these has now been added to the pipeline, but they leave some line ; residuals so please use with caution. ; ; Standards: The list of recognized spectrophotometric flux-calibration ; standards can be viewed in the 'refdata' subdirectory as the file ; specstandards.dat, or online at: ; http://astro.caltech.edu/~dperley/programs/lpipe/specstandards.txt ; Of these standards, the hot white dwarfs essentially always make much ; better choices than features with strong absopriton lines. If no ; clean standard is available the flux calibration will be worse. ; Any standards not on the list above will not be recognized as standards ; (if you have a favorite standard not on this list, please suggest it.) ; Ideally the same standard should be observed twice at significantly ; different airmasses in each setup. ; Additionally, it is recommended to take arcs alongside observations of ; bright standards where a short integration time (<20 sec) is employed ; or for standards observed in bright twilight. This will ensure the ; standard has a good wavelength solution if refinement based on ; night-sky OH emission is impossible, which in turn will help ensure ; accurate flux-calibration. ; ; ; Imaging - ; ; Windowing: You can window the red side to Image Area on the XLRIS CONFIG ; menu to save readout and processing time with essentially no compromise. ; One can window it even more to save more readout time at the expense ; of FOV (use the ETC menu). Windowing is fully integrated into the ; pipeline, and images of different windowing settings can be processed ; without difficulty even if the windowing of the flats differs from that ; of the target. ; ; Flats: In general, twilight flats produce the most reliable results, so ; try to get them if possible. Observing in the U-band REQUIRES twilight ; flats since there are not enough dark-sky counts and the dome lamp is ; not hot enough to emit in the UV. (Twilight flats do seem to create ; artifacts with the glue defects in some filters, however.) Also, note ; that the mirror does not return precisely to its original state after ; changes in and out of spectroscopy mode, changing the vignetting and ; dust pattern. Try to avoid changing the configuration between flats ; and science frames, or take sufficient science frames that super-sky ; correction flats ('reflat' step) can be constructed reliably. Of ; course, this is not always possible. ; ; Standards: If you are imaging any uncalibrated fields and conditions are ; photometric, try to get at least two measurements of ; Landolt photometric standard fields (preferably two observations of ; the same standard field) per filter at significantly different ; airmasses. This can be done during twilight. While the program now ; uses imaging of science fields in SDSS to help calculate the zerpoint ; solution, these can be susceptible to crowding when calculating the ; aperture correction and Landolt fields are useful to verify them. ; ; Science fields: Always specify that you want the source placed at the ; "LRIS-B imaging position" to avoid the chip gap. A test exposure is ; usually not necessary if the operator is trustworthy, but a short ; exposure (< ~30s) will help significantly with calibration later if a ; moderately deep (e.g. SDSS) calibration of the field is not available, ; especially in the redder filters (z- / RG850- band in particular). ; When dithering, try to make sure that every image in the ; dither sequence is at a different vertical position (as seen on the ; live DS9 display) by e.g. dithering 1" vertically to accompany each ; otherwise horizontal dither. LRIS-R3 has some bad columns (horizontal ; lines in the viewer orientation) which one should avoid keeping over ; the same horizontal location (helpful, since these columns actually ; fluctuate in intensity and do not always flat-field out.) ; ; Organization - ; ; Use save_state and restore_state (NOT the GUI) to store your LRIS ; configurations. This includes the CCD parameters and will ensure that ; you do not forget to change the binning and windowing which are NOT ; included in the XLRIS save/load options. ; ; ; ; RELATED PROGRAMS: ; ; To check what calibrations (etc.) you have or need, you can type: ; IDL> lriscat ; This will print out a catalog table listing all relevant LRIS settings ; for each file in the current directory. You can specify particular ; files (including extracted spectra or intermediate-reduction files) ; by specifying the file wildcard string, e.g., 'spredux1d/ls*.fits'. ; A python version also exists that can be run outside IDL: download it ; at http://www.astro.caltech.edu/~dperley/programs/lriscat.py ; ; To re-create output/diagnostic plots produced by the pipeline, run e.g.: ; IDL> lrisplotall ; final plots of reduced spectra ; IDL> lrisplottraces ; 2D spectra showing extraction apertures ; IDL> lrisplotprofiles ; profile plots ; ; To change trace extraction or background settings for specific objects: ; IDL> lrisapertures ; ; To inspect and validate final output spectra: ; IDL> lrisvalidate ; ; Some steps can be run manually on single images, e.g.: ; IDL> lrisfluxcal, 'science.spec', 'fluxcalstar.spec' ; flux-calibrate ; IDL> lrisconnect, 'red.spec', 'blue.spec' ; R-B connect ; ; Detailed pipeline reference information will soon be moved online. ; ; -----------------------------------------------------------------------------