;+ ; NAME: ; lpipe (version 2024.02) ; ; PURPOSE: ; Fully-automated reduction of LRIS imaging and spectroscopy. ; ; CALLING SEQUENCE: ; lpipe, [settings] ; ; INPUTS AND KEYWORDS (most common; see manual for more): ; directories - List or search string of subdirectories. (Usually blank.) ; General: ; data - Location of raw data (current directory if unspecified) ; /help - Print some basic help information ; reducer - Specify name of user to add to the file headers ; Step control: ; mode - Mode to process (imaging or spectroscopy) ; start - Start with this step, skipping previous ones ; stop - End with this step, skipping subsequent ones ; step - Do only this step ; /redo - Repeat actions, overwriting any existing files ; File control: ; camera - Camera to process (red or blue) ; chip - Chip to process (left or right) ; files - Specific file numbers to process ; filters - Specific filters to process ; gratings - Specific gratings to process ; targets - Specific targets to process ; date - Specific night or month to process (e.g. '150423') ; Options: ; preparation/formatting ; /prepareall - Bias-subtract multislit and dark files (still ignores specpol) ; /skipfocus - Skip frames associated with focusing ; flat-fielding ; flatrequire - Specify the number of images needed to make a flat-field ; /norebin - Don't rebin if no flats exist in same binning as data ; /noreflat - Do not perform super sky flat correction for images ; altflatdir - Give an alternative flat-field directory (archival flats, etc) ; /forceflat - Produce all flatfields even if not needed for science data ; /forcereflat - Tell the pipeline to use reflat step even if not enough files/fields ; /ignoredichroic- Allow dichroic differences when flat-fielding images ; /rightchipdead - Normalize flats using left CCD ; cosmic ray cleaning ; /nocrclean - Do not zap cosmic rays at all ; crzeal - Adjust cosmic ray cleaning (default=1.0, >1.0:more aggressive) ; sky subtraction ; skyalgorithm - Use a different sky-subtraction algorithm ('kelson') ; summing ; /nosum - Do not coadd any repeat spectra (e.g., for time series) ; tracing ; /norbalign - Don't attempt to align red and blue traces of the same target ; spectrum extraction ; /noreaddsky - Rely on 2D sky subtraction for sky-line removal. ; wavelength calibration ; altarcdir - Give an alternative arc-lamp wavelength solution directory ; /forcewavcal - Solve wavelength solution even for marginal lamp combinations ; /novalidatewavcal - Do not validate wavelength solution. ; flux-calibration ; altresponsedir - Give an alternative flux-calibration directory ; /notelluric - Skip telluric correction ; red-blue connection ; /requirepair - Do not write out final spectrum without both red+blue ; astrometry ; /distort - Apply distortion solution (not well-tested, may not work) ; photometry ; /nophotometry- Skip zeropoint calculation and attenuation adjustment ; /photometric - Use zeropoint solution to calibrate non-SDSS/PS1 fields ; visualization/output ; /nodisplay - Don't pop up display windows (for remove/server reductions) ; /nofailuresum- Hide the summary of unprocessed files printed at the end ; /debug - Print out more info; turn off some exception handling ; /timer - Print out processing times ; /shortnames - Exclude the six-digit date string from output files ; Other ; /continuous - Keep running, assimilating new data as it appears ; /configure - Write configuration file to disk for even more control ; ; ADDITIONAL OPTIONS: ; Further options can be set by creating certain text files in the working ; directory, for example to indicate bad files not to process. See manual.txt ; for options, although note that this is an old feature that has not been ; tested on recent data. ; ; OUTPUTS: ; Numerous final files are written to disk throughout operation. Intermediate ; files are written to a series of subdirectories created during operation; ; final data products are written to the present working directory. ; The final imaging products are coadded exposures of each field in each filter; ; the final spectroscopy products are coadded 1D spectra of the brightest source ; near the trace position for each object. ; ; COMMENTS: ; See manual.txt in the docs/ subdirectory or the online documentation for ; detailed information on the pipeline, including a description of the workflow, ; best-practice guidance, additional examples, and troubleshooting tips. ; For inline help, type "lpipe, /help" within IDL. ; ; Note that a variety of supporting codes are available - including a detailed ; GUI for setting the extraction aperture(s) and a viewer for examining 1D ; spectra simultaneously with the response function and check plots. ; ; EXAMPLES: ; Process an entire night of LRIS data: ; IDL> lpipe ; ; Reprocess a specific exposure by file number: ; IDL> lpipe, file=105, /redo ; ; Carry out only the prepare step (overscan subtraction) ; IDL> lpipe, step='prepare' ; ; See manual.txt for additional examples. ; ; NOTES: ; The in-code documentation header has been substantially truncated, but the ; full version is still available in manual.txt. ; Website: https://sites.astro.caltech.edu/~dperley/programs/lpipe.html ; Paper: https://arxiv.org/abs/1903.07629 ; ; HISTORY: ; Authored by Daniel Perley (d.a.perley@ljmu.ac.uk). ; ;- @autolris ; Step-specific functions have been moved to autolris.pro pro lpipe, directories, datadirectory=datadirectory, modestr=modestr, camerastr=camerastr, chipstr=chipstr, files=files, filters=filters, gratings=gratings, targets=targets, dates=dates, start=start, stop=stop, skip=skip, only=only, step=step, nocrclean=nocrclean, noreflat=noreflat, notelluric=notelluric, redo=redo, nofringe=nofringe, continuous=continuous, photometric=photometric, help=help, nodisplay=nodisplay, distort=distort, debugmsg=debugmsg, nophotometry=nophotometry, configure=configure, reducername=reducername, timer=timer, forceflat=forceflat, forcewavcal=forcewavcal, novalidatewavcal=novalidatewavcal, nosum2d=nosum2d, crzeal=crzeal, skyalgorithm=skyalgorithm, norbalign=norbalign, shortnames=shortnames, prepareall=prepareall, altflatdir=altflatdir, altarcdir=altarcdir, altresponsedir=altresponsedir, norebin=norebin, nofailuresummary=nofailuresummary, noreaddsky=noreaddsky, skipfocus=skipfocus, requirepair=requirepair, reducestandards=reducestandards, flatrequire=flatrequire, rightchipdead=rightchipdead, wildchar=wildchar, setredwav=setredwav, forcereflat=forcereflat ; Load default parameters and interpret user arguments. common lrisauto, lpar common lrisfail, prepfail, flatfail, catastrofail, relastrofail, fullastrofail, extractfail, wavsolfail, wavsolwarn, fluxcalfail common lristop, runct !quiet = 1 ; Set up / check for the config file if keyword_set(configure) then autolrissetup ; this is normally not run now ; Load settings from the config file autolrisdefaults idlversion = !version.release lpar.lpversion = '2024.02' if n_elements(reducername) gt 0 then lpar.reducer = reducername idlver = fix(strsplit(idlversion,'.',/extract)) if n_elements(idlver) lt 2 then begin print, ' Cannot identify IDL version' endif else begin if idlver[0] lt 8 or (idlver[0] eq 8 and idlver[1] lt 1) then begin print, ' WARNING: Old IDL version ('+idlversion+')' print, ' LPipe is not tested for IDL <8.1.' print, ' Pipeline will attempt to proceed, but you may need to update your installation.' endif endelse if n_elements(runct) eq 0 then runct = 0 if runct eq 0 then begin print, 'LPIPE version '+lpar.lpversion+' - new this version: ' ; 2024.02: print, ' Accounts for repositioned flat screen for images, too.' ; 2023.06: print, ' Accounts for repositioned flat screen for spec. flats.' print, ' Partial backwards compatibility for old IDL versions.' ; 2022.07: ; print, ' Optional image reflattening procedure works again.' ; 2022.06: ; print, ' Correction of most faulty red header values' ; print, ' Note: Red headers may require further repair using' ; print, ' editheader for accurate pipeline processing.' ; print, ' See README_lrisred4.txt for further info' ; print, ' Arc solver retry loop' ; print, ' Extensive new documentation of dependency codes' ; print ,' /notelluric option re-enabled to skip telluric corr.' ; 2022.05: ; print, ' Some improvements to cosmic ray cleaning (use caution' ; print, ' - some strong nebular lines may be zapped)' ; 2022.04: ; print, ' Red CCD reduction functionality mostly restored.' ; print, ' Not all gratings have been tested.' ; 2021.09: ; print, ' Misc. debugs in addition to the following:' ; print, ' Limited support for new LRIS red chip.' ; print, ' Not all features are re-enabled.' ; print, ' Red-blue trace matching not performed.' ; print, ' Reductions may not be compatible with future versions' ; print, ' BETA version, may cause unexpected behavior!' ; print, ' May need to edit header keywords (WAVELEN, TRAPDOOR) ; 2021.06: ; features same as .09 ; print, ' Automatically supports R400 grating ONLY - no others' ; print, ' Need to specify red CCD overscan with setredoverscan' ; 2020.09: ; print, ' Support for spectroscopy using left (secondary) chip.' ; print, ' Specify /rightchipdead to use left chip to normalize.' ; print, ' Small modifications to arc line matching.' ; 2020.08: ; print, ' Support for use of outer amplifier on LRIS-red due ' ; print, ' to CCD malfunction experienced during shutdowns.' ; ; print, ' better halogen flats,' ; print, ' more conservative cosmic-ray rejection on standards' ; ; better tracing+extraction,' ; print, ' better support for obscure gratings, documentation,' ; print, ' trace alignment improvements, single-camera output,' ; print, ' background aperture avoidance, photometry bug fixes' ; Numerous large changes:' ; print, ' header management, images are transposed, redefined' ; print, ' physical coordinates, flat field construction,' ; print, ' processes gzipped raw images, better extraction,' ; print, ' full support for KOA images, mass-reduction tools,' ; print, ' validation tool, extensive bug fixes and improvements ; 'R+B alignment fixes' ; PS1 lookup, larger sky window,' ; print, ' 12-char targnames' ; print, 'LPIPE version '+lpar.lpversion+' - new this version: can skip 2D summation; consistent' ; print, ' y-axis WCS for spectra referenced to amplifier edge; automatic' ; print, ' red/blue aperture alignment to avoid chimeric spectra' ; 'automatic keyword recovery;' ; print, ' improved flat-fielding, blue windowing support,' ; print, ' separate wavapply step, user control options, & more.' ; print, ' specify individual files to reduce at command line;' ; print, ' Use caution if re-running data from old versions.' if keyword_set(help) eq 0 then begin print, 'Type lpipe, /help for more info' wait, 0.25 endif endif runct = runct + 1 ; 'combine' has been deprecated ; 'makefringe', 'rmfringe' - come after flatten allsteps = ['prepare', 'makeflat', 'flatten', 'reflatten', 'split', 'crclean', 'skysubtract', $ 'sum', 'trace', 'extract', 'wavcal', 'wavapply', 'response', 'fluxcal', 'combine', 'connect', $ 'astrometry', 'photometry', 'stack'] steps = allsteps stephelp = ['Converts multi-fits to simple fits, bias subtracts, adds header info',$ 'Creates flat fields',$ 'Divides by flat fields',$ 'Create customized super-skyflats for each exposure (set noreflat=0)',$ 'Separates right and left chips',$ 'Removes cosmic rays',$ 'Subtracts sky lines from 2D spectra', $ 'Sums 2D spectra of the same target', $ 'Traces spectra', $ 'Extracts spectra', $ 'Wavelength-calibrates arc spectra', $ 'Applies wavelength calibration and flexure correction', $ 'Solves response function of flux standards', $ 'Flux-calibrates spectra including telluric correction', $ 'Combines 1D spectra', $ 'Connects blue/red spectra', $ 'Solves for astrometric offset/rotation of images', $ 'Solves for zeropoint and transmission variations of images', $ 'Co-adds images'] stepstr = '' for i = 0, n_elements(steps)-1 do stepstr += ' '+steps[i] if keyword_set(help) then begin print, ' LPIPE - LRIS automatic processing pipeline (v '+clip(lpar.lpversion)+')' print, ' Command-line options:' print, " mode = 'i' for imaging only, 's' for spectroscopy only, default is both" print, " camera = 'b' for blue only, 'r' for red only, default is both" print, " chip = 'r' for right, 'l' for left, 'both' for both, default is right" print, " step = step to execute individually (see information below)" print, " start = step to start with" print, " stop = step to end with" print, " data = location of raw data subdirectory" print, " file = specific files to process (ex: '10,11' or '30-40')" print, " target = specific target to process" print, " /redo - overwrite subsequent files (otherwise pipeline resumes where it left off)" print, " /nosum - Do not coadd spectra of same target (e.g., for spectral time series)" print, " crzeal = Adjust aggressiveness of cosmic ray cleaning. (1.0 = default)" print, " skyalg = Sky-line subtraction algorithm. Can be 'median' (Default) or 'kelson'. print, " (See the documentation for even more options.)" ; print, " To control tracing of individual 2D spectra, run lrisapertures" ; print, " To remove problematic files from processing, create the file 'badfiles.txt'" print, " Additional tools:" ; print, " lriscorrectheaders - run to repair missing values from headers before processing" print, " lriscat - run to print out a catalog of your data files" print, " lrisapertures - visually select/modify source/background extraction regions" print, " lrisvalidate - inspect and approve your final output spectra" print, ' Steps are:' for i = 0, n_elements(steps)-1 do print, ' '+clip(steps[i],12)+' '+stephelp[i] print, ' For additional help and more options, go to http://www.astro.caltech.edu/~dperley/programs/lpipe.html' return endif if n_elements(step) eq 1 then begin ; print list of steps for convenience if clip(step[0]) eq '1' then begin str = ' ' for i = 0, n_elements(steps)-1 do str += ' '+clip(steps[i]) print, str return endif endif close, /all if n_elements(step) gt 0 and n_elements(stop) gt 0 then if step ne stop then begin print, 'Both step and stop are set: conflicting requests.' return endif if n_elements(step) gt 0 and n_elements(start) gt 0 then if step ne start then begin print, 'Both step and start are set: conflicting requests.' return endif if n_elements(datadirectory) gt 0 then lpar.datadir = datadirectory if strlen(lpar.datadir) gt 0 and strmid(lpar.datadir,strlen(lpar.datadir)-1,1) ne '/' then $ lpar.datadir = lpar.datadir + '/' ; this seq naming convention is not rational, since it conflicts with that used in the headers. ; It's just a list of files/filters/targets/gratings to process, stored as an array. ;if n_elements(files) eq 0 then lpar.fileseq = -1 if n_elements(files) eq 1 then lpar.fileseq = clip(files) if n_elements(files) gt 1 then begin for i = 0, n_elements(files)-1 do lpar.fileseq += clip(files[i]) + ',' lpar.fileseq = strmid(lpar.fileseq,0,strlen(lpar.fileseq)-1) endif if lpar.fileseq ne '' then dum = strseq(lpar.fileseq) ; make sure it doesn't crash etc. ; else if n_elements(files) eq 1 then lpar.fileseq = strseq(files) ; only allows one filter, target, or grating at a time for now. Could parse a comma list in lrisselect. if n_elements(filters) gt 0 then lpar.filterseq = filters else lpar.filterseq = '' if n_elements(targets) gt 0 then lpar.targetseq = targets else lpar.targetseq = '' if n_elements(gratings) gt 0 then lpar.gratingseq = gratings else lpar.gratingseq = '' if n_elements(noreflat) eq 0 then noreflat = 1 ; default is off redo = keyword_set(redo) nocrclean = keyword_set(nocrclean) noreflat = keyword_set(noreflat) notelluric = keyword_set(notelluric) nodistort = 1 ; keyword_set(distort) eq 0 ; not yet an option nofringe = keyword_set(nofringe) nophotometry = keyword_set(nophotometry) lpar.debug = keyword_set(debugmsg) forceflat = keyword_set(forceflat) alwaysreaddsky = keyword_set(noreaddsky) eq 0 datenames = keyword_set(shortnames) eq 0 if keyword_set(forcereflat) then noreflat = 0 ; Use with caution: archival flats may be better. prepareall = keyword_set(prepareall) cd, current=pwd dirtree = strsplit(pwd,'/',/extract,count=nd) lastdir = dirtree[nd-1] if lastdir ne '' then lastdir += '/' ; whether or not a slash is on the end is very ; confusing, need to rethink this. if lastdir eq 'imredux/' or lastdir eq 'spredux/' or lastdir eq 'spredux1d/' or lastdir eq 'spredux2d/' or lastdir eq 'dark/' or lastdir eq 'flats/' or lastdir eq 'response/' or lastdir eq 'arcs/' then begin print, 'Currently in a reduction subdirectory.' print, "Type cd, '..' and rerun." ; could reinterpret this as run with the mode set to whatever this directory is... return endif if n_elements(altflatdir) gt 0 then begin check = file_test(altflatdir) for a = 0, n_elements(altflatdir)-1 do begin if altflatdir[a] eq '' then continue ; silent if check[a] eq 0 then print, ' WARNING: '+altflatdir[a]+' does not exist' endfor ; bad directories are harmless, but remove them anyway (and add the /) wok = where(check, ct) if ct gt 0 then altflatd = altflatdir[wok] for a = 0, n_elements(altflatd)-1 do begin ; if strmid(altflatd[a],strlen(altflatd[a])-1) ne '/' then $ ; add trailing slash if needed altflatd[a] = file_search(altflatd[a], /fully_qualify, /mark_dir);strmid(altflatd[a],0,strlen(altflatd[a])-1) endfor endif if n_elements(altarcdir) gt 0 then begin check = file_test(altarcdir) for a = 0, n_elements(altarcdir)-1 do begin if altarcdir[a] eq '' then continue ; silent if check[a] eq 0 then print, ' WARNING: '+altarcdir[a]+' does not exist' endfor wok = where(check, ct) if ct gt 0 then altarcd = altarcdir[wok] for a = 0, n_elements(altarcd)-1 do begin altarcd[a] = file_search(altarcd[a], /fully_qualify, /mark_dir) endfor endif ; --- Process mode options --- ; By default do both imaging and spectroscopy. if n_elements(modestr) eq 0 then begin modestr = 'is' if n_elements(step) gt 0 then begin ; for certain mode-specific steps also set modestr if step eq 'skysubtract' or step eq 'extract' or step eq 'fluxcal' $ or step eq 'combine' or step eq 'connect' $ then modestr = 's' if step eq 'astrometry' or step eq 'photometry' or step eq 'stack' $ then modestr = 'i' endif endif modestr = strlowcase(modestr) if modestr eq 'is' or modestr eq 'i,s' or modestr eq 'im,sp' then modes = ['im', 'sp'] if modestr eq 'si' or modestr eq 's,i' or modestr eq 'sp,im' then modes = ['sp', 'im'] if n_elements(modes) eq 0 then begin if strmid(modestr,0,1) eq 'i' then modes = ['im'] if strmid(modestr,0,1) eq 's' then modes = ['sp'] endif if n_elements(modes) eq 0 then begin print, 'Cannot recognize mode request: ', modestr return endif lpar.displayoff = keyword_set(nodisplay) ; --- Process camera options --- if n_elements(camerastr) eq 0 then camerastr = 'br' camerastr = strlowcase(camerastr) if camerastr eq 'br' or camerastr eq 'b,r' or camerastr eq 'both' then cameras = ['blue', 'red'] if camerastr eq 'rb' or camerastr eq 'r,b' then cameras = ['red', 'blue'] if n_elements(cameras) eq 0 then begin if strmid(camerastr,0,1) eq 'b' then cameras = ['blue'] if strmid(camerastr,0,1) eq 'r' then cameras = ['red'] endif if n_elements(cameras) eq 0 then begin print, 'Cannot recognize camera request: ', camerastr return endif ; --- Process chip options if n_elements(chipstr) eq 0 then chipstr = 'r' chipstr = strlowcase(chipstr) if chipstr eq 'lr' or chipstr eq 'l,r' then chips = ['l', 'r'] if chipstr eq 'rl' or chipstr eq 'r,l' or chipstr eq 'both' or chipstr eq 'b' then chips = ['r', 'l'] if n_elements(chips) eq 0 then begin if strmid(chipstr,0,1) eq 'l' then chips = ['l'] if strmid(chipstr,0,1) eq 'r' then chips = ['r'] endif if n_elements(chips) eq 0 then begin print, 'Cannot recognize chip request: ', chipstr return endif ; --- Process step options if n_elements(start) gt 0 then begin w = (where(steps eq start, ct)) [0] if ct eq 0 then begin print, "Invalid starting step '", start, "' print, "Must be one of: ", steps return endif steps = steps[w:*] endif if n_elements(stop) gt 0 then begin w = (where(steps eq stop, ct)) [0] if ct eq 0 then begin print, "Invalid stopping step '", stop, "' print, "Must be one of: ", steps if n_elements(start) gt 0 then print, 'Note that start is also set.' return endif steps = steps[0:w] endif if n_elements(step) gt 0 then only = step if n_elements(only) gt 0 then begin w = (where(steps eq only, ct)) [0] if ct eq 0 then begin print, "Invalid step '", only, "'" print, "Must be one of: ", steps if n_elements(start) gt 0 then print, 'Note that start is also set.' if n_elements(stop) gt 0 then print, 'Note that stop is also set.' return endif steps = steps[w] endif if n_elements(skip) gt 0 then begin w = (where(allsteps eq skip, ct)) [0] if ct eq 0 then begin print, "Invalid skip step '", skip, "' print, "Must be one of: ", steps return endif w = where(steps ne skip, ct) if ct eq 0 then begin print, 'Note: skipped step '+skip+' is not in sequence.' endif else begin steps = steps[w] endelse endif ; For imaging, check if autoastrometry, sextractor, swarp are installed and functioning if total(modes eq 'im') ge 1 then begin if total(steps eq 'astrometry') gt 0 or total(steps eq 'photometry') gt 0 or total(steps eq 'stack') gt 0 then begin if file_test('temp.txt') then file_delete, 'temp.txt' cmd = getsexpath()+' -d '+' > temp.txt' spawn, cmd c = countlines('temp.txt') if c eq 0 then begin print, getsexpath() print, cmd print, 'Error: Sextractor is not installed or not configured.' print, ' Cannot run image alignment steps.' print, " Configure or set mode='s' or stop='crclean'" return endif endif if total(steps eq 'stack') gt 0 then begin if file_test('temp.txt') then file_delete, 'temp.txt' if lpar.swarpcommand ne '' then begin cmd = lpar.swarpcommand+' -d > temp.txt' spawn, cmd endif c = countlines('temp.txt') if c eq 0 then begin print, cmd print, 'Error: Swarp is not installed or not configured.' print, ' Cannot run image coadds.' print, " Configure or set mode='s' or stop='photometry'" return endif endif if total(steps eq 'astrometry') or total(steps eq 'stack') gt 0 then begin if file_test('temp.txt') then file_delete, 'temp.txt' cmd = lpar.autoastrocommand+' > temp.txt' spawn, cmd c = countlines('temp.txt') if c eq 0 then begin print, 'Error: Autoastrometry is not installed or not configured.' print, ' Cannot run image alignment steps.' print, " Configure or set mode='s' or stop='crclean'" return endif if c eq 22 then begin print, 'Error: Python is not configured.' print, ' Cannot run image alignment steps.' print, " Configure or set mode='s' or stop='crclean'" endif endif endif photometric = keyword_set(photometric) ; overwrites variable, which is OK if photometric and ((lpar.fileseq[0] ge 0 and n_elements(lpar.fileseq) lt 4) or $ (lpar.targetseq[0] ne '' and n_elements(lpar.targetseq) lt 2)) then begin logprint, 'WARNING: Not enough files/filters to expect a reliable photometric solution.' logprint, ' photometry step should be run with no file filters.' ; photometry step really shouldn't be run at all in these circumstances. ; note that this makes no attempt to see if the targets are valid / have observations. endif ; Investigate the data and rename files to modern format if necessary. ; (no longer automatic - user should do this themselves using lrisrename now.) ;noldfiles = countfiles(lpar.datadir+'lred*.fits') + countfiles(lpar.datadir+'lblue*.fits') ; ;+ countfiles(lpar.datadir+'rfoc*.fits') + countfiles(lpar.datadir+'bfoc*.fits') ;if noldfiles gt 0 then begin ; print, 'Image file names are in old format. Updating...' ; autolrisrename ;endif dirs = '' if n_elements(directories) ge 1 then begin for d = 0, n_elements(directories)-1 do begin adddir = file_search(directories[d], /test_dir, /mark_dir, count=check) if check eq 0 then print, directories[d]+' is not a valid directory' else $ dirs = [dirs, adddir] endfor if n_elements(dirs) eq 1 then return if n_elements(dirs) gt 1 then dirs = dirs[1:*] endif if n_elements(directories) eq 1 then dirs = file_search(directories, /test_dir, /mark_dir) cdir = file_search('./',/fully_qualify) for d = 0, n_elements(dirs)-1 do $ if dirs[d] ne '' and strmid(dirs[d],strlen(dirs[d]-1)) ne '/' then dirs[d] += '/' for d = 0, n_elements(dirs)-1 do begin ver = getlrisversion(datadir=dirs[d]+lpar.datadir, /quiet, filenametype=filenametype) if ver le 0 then ver=getlrisversion(datadir=dirs[d]+lpar.spworkingdir2d, /quiet, filenametype=filenametype) if ver le 0 then ver=getlrisversion(datadir=dirs[d]+lpar.spworkingdir1d, /quiet, filenametype=filenametype) if ver le 0 then ver=getlrisversion(datadir=dirs[d]+lpar.imworkingdir, /quiet, filenametype=filenametype) if ver gt 0 then break endfor if ver eq 0 then begin print, ' Found no files to process.' if total(steps eq 'prepare') gt 0 then begin if file_test('raw') then print, " Specify data='raw/'" else $ print, " If raw files are in a different directory, try specifying: data='/path/to/files/'" endif return endif lpar.lrisversion = ver if n_elements(wildchar) gt 0 then begin lpar.wildchar = wildchar endif if filenametype eq 'koa' then begin lpar.koa = 1 lpar.wildchar = '??????.?????' ; five-digit file ID with a dot endif ;if n_elements(dates) eq 1 then lpar.wildchar = clip(dates,6)+'_????' else lpar.wildchar = '??????_????' if n_elements(dates) eq 1 then begin ; date filtering is implemented crudely for now by modifying wildchar lpar.wildchar = repstr('??????',dates) endif if keyword_set(rightchipdead) then begin if n_elements(chips) gt 1 or chips[0] ne 'l' then print, " Right chip marked as dead: Changing chip option to 'l'" chips = ['l'] endif skiprbalign = keyword_set(norbalign) if lpar.lrisversion eq 1 then skiprbalign = 1 ; rbalign doesn't work on LRIS-R1 yet ; or lpar.lrisversion eq 4 ;and n_elements(norbalign) eq 0 if redo then lpar.overwrite = 1 else lpar.overwrite = 0 ; if keyword_set(nocrclean) eq 0 then flag = 1 this doesn't do anything if n_elements(modes) eq 1 then begin ; remove redundant steps if mode is set to only do imaging or spectroscopy spsteps = where(steps ne 'astrometry' and steps ne 'photometry' and steps ne 'stack',ct) if modes[0] eq 'sp' and ct gt 0 then steps = steps[spsteps] imsteps = where(steps ne 'trace' and steps ne 'wavcal' and steps ne 'fluxcal' and steps ne 'combine' and steps ne 'connect',ct) if modes[0] eq 'im' and ct gt 0 then steps = steps[imsteps] endif ndir = n_elements(dirs) nsteps = n_elements(steps) nmodes = n_elements(modes) ncameras = n_elements(cameras) nchips = n_elements(chips) npipeiter = 1L if keyword_set(continuous) eq 0 then maxpipeiter = 1L else maxpipeiter = 100000L logprint, logfile='lpipelog.txt' logprint, ' --- LPIPE version '+lpar.lpversion+' started UT '+tstamp(/ut)+ ' ---', /logonly if n_elements(start) gt 0 then logprint, ' start='+clip(start), /logonly if n_elements(stop) gt 0 then logprint, ' stop='+clip(stop), /logonly if n_elements(step) gt 0 then logprint, ' step='+clip(step), /logonly if n_elements(modestr) gt 0 then if strlen(modestr) lt 2 then logprint, ' mode='+clip(modestr), /logonly if n_elements(camerastr) gt 0 then if strlen(camerastr) lt 2 then logprint, ' cam='+clip(camerastr), /logonly if n_elements(files) gt 0 then logprint, ' file='+clip(files), /logonly if n_elements(filters) gt 0 then logprint, ' filter='+clip(filters), /logonly if n_elements(gratings) gt 0 then logprint, ' grating='+clip(gratings), /logonly if n_elements(dates) gt 0 then logprint, ' date='+clip(dates), /logonly logprint, ' Run with IDL version ', idlversion, /logonly ; could print arguments and such... times = [systime(/seconds)] tstep = [''] while npipeiter le maxpipeiter do begin for istep = 0, nsteps-1 do begin instep = steps[istep] logprint, instep for d = 0, n_elements(dirs)-1 do begin if dirs[d] ne '' then begin cd, dirs[d] logprint, dirs[d] endif for icam = 0, ncameras-1 do begin camera = cameras[icam] ca = strmid(cameras[icam],0,2) if instep eq 'prepare' then begin ; the mode setting is passed on to the routine itself. autolrisprepare, modestr=modestr, camstr=ca, prepareall=prepareall, skipfocus=skipfocus, setredwav=setredwav endif for imode = 0, nmodes-1 do begin mo = strmid(modes[imode],0,2) if mo eq 'im' then begin if instep eq 'makeflat' then autolrismakeimflat, cam=camera, chip='', forceflat=forceflat if instep eq 'flatten' then autolrisimflatten, cam=camera, chip='', ignoredichroic=ignoredichroic, altflatdir=altflatd if lpar.lrisversion eq 1 and nofringe eq 0 and cameras[icam] eq 'red' then begin if instep eq 'makefringe' then autolrismakefringe, chip=ch, cam=ca if instep eq 'rmfringe' then autolrisrmfringe, chip=ch, cam=ca endif if instep eq 'reflatten' and noreflat eq 0 then autolrisreflatten, chip='', cam=ca, filter=['RG850','I','R','V','G'], forcereflat=forcereflat endif if mo eq 'sp' then begin if instep eq 'makeflat' then autolrismakespecflat, cam=camera, chip='', forceflat=forceflat, flatrequire=flatrequire, rightchipdead=rightchipdead if instep eq 'flatten' then autolrisspecflatten, cam=camera, chip='', altflatdir=altflatd, norebin=norebin endif if instep eq 'split' and (lpar.lrisversion eq 2 or lpar.lrisversion eq 3 or cameras[icam] eq 'blue') then begin ; and n_elements(chips) eq 2 if n_elements(chips) eq 1 then ch = chips[0] else ch = '' autolrissplit, cam=camera, mode=mo, ch=ch, nodistort=nodistort endif for ichip = 0, nchips-1 do begin if (lpar.lrisversion eq 1 or lpar.lrisversion eq 4) and camera eq 'red' then ch = '' $ else ch = strmid(chips[ichip],0,1) if nocrclean eq 0 and instep eq 'crclean' then begin if mo eq 'im' then autolriscrcleanim, cam=camera, chip=ch, setzeal=crzeal if mo eq 'sp' then autolriscrcleanspec, cam=camera, chip=ch, czeal=crzeal endif if mo eq 'im' then begin if instep eq 'astrometry' then autolrisastrometry, chip=ch, cam=camera, nocrclean=nocrclean, nodistort=nodistort if camera eq 'blue' then bl = 1 else bl = 0 if camera eq 'red' then re = 1 else re = 0 ;if n_elements(chips) eq 1 then begin if instep eq 'photometry' and nophotometry eq 0 then autolrisphotometry, chip=ch, camera=camera, photometric=photometric ;endif else begin ; eventually need to re-enable this? ; if instep eq 'photometry' and ichip eq 1 then autolrisphotometry, camera=camera ;endelse if instep eq 'stack' then autolrisstack, chip=ch, cam=camera, live=live, final=(ichip eq nchips-1), nophotometry=nophotometry, datenames=datenames endif if mo eq 'sp' then begin if instep eq 'skysubtract' then autolrisskysubtract, chip=ch, cam=camera, nocrclean=nocrclean, skyalg=skyalgorithm if instep eq 'wavcal' then autolriswavcal, chip=ch, cam=camera, novalidatewavcal=novalidatewavcal, forcewavcal=forcewavcal if instep eq 'sum' then autolrissum, chip=ch, cam=camera, nosum=nosum2d if instep eq 'trace' then autolristrace, chip=ch, cam=camera, skiprbalign=skiprbalign if instep eq 'extract' then autolrisextract, chip=ch, cam=camera, alwaysreaddsky=alwaysreaddsky ; should merge with first part of wavcal if instep eq 'wavapply' then autolriswavapply, chip=ch, cam=camera, altarcsoldir=altarcd ; should merge with wavcal endif endfor if mo eq 'sp' then begin ; not really necessary if instep eq 'response' then autolrisresponse, chip='r', cam=camera if instep eq 'fluxcal' then autolrisfluxcal, chip='r', cam=camera, notelluric=notelluric, reducestandards=reducestandards, altresponsedir=altresponsedir ; could move this in the chip loop if n_elements(step) gt 0 then $ if instep eq 'combine' and step eq 'combine' then autolriscombine, camera=camera, live=live ; only if specifically requested endif endfor ; mode endfor ; camera ; Camera has ceased to be relevant for the following for imode = 0, nmodes-1 do begin mo = strmid(modes[imode],0,2) if mo eq 'im' and instep eq 'photometry' then autolrisphotcat ; OK that this happens after stacking. if mo eq 'sp' and n_elements(cameras) ge 2 and instep eq 'connect' then autolrisconnect, live=live, datenames=datenames, requirepair=requirepair ; a redblue imaging alignment procedure would also go here endfor if dirs[d] ne '' then cd, cdir endfor ; dir times = [times, systime(/seconds)] tstep = [tstep, instep] endfor ; step logprint logprint, 'Processing complete.' if keyword_set(nofailuresummary) eq 0 then lrisfailuresummary, dirs, altflatdir=altflatd lrisfailuresummary, /reset if keyword_set(continuous) then begin logprint, 'Pipeline iteration complete.' logprint, 'Continuous mode is active.' logprint, 'Pausing for 30 seconds, then will loop from the beginning. Press Ctrl+C to stop.' for s = 30, 0, -1 do begin wait, 1.0 endfor endif npipeiter += 1 endwhile ;while !window ge 0 do wdelete, !window ; cleanup- should ultimately put these things in imredux if file_test('temp*.*') gt 0 then file_delete, file_search('temp*.*') if file_test('det.*') gt 0 then file_delete, file_search('det.*') if file_test('cat.*') gt 0 then file_delete, file_search('cat.*') if file_test('match*') gt 0 then file_delete, file_search('match*') if file_test('sex*') gt 0 then file_delete, file_search('sex*') ; this expunges user preferences, of course... if file_test('swarp.xml') gt 0 then file_delete, 'swarp.xml' if file_test('default.swarp') gt 0 then file_delete, 'default.swarp' if keyword_set(timer) and nsteps ge 1 then begin logprint, ' Processing time summary: ' for s = 1, nsteps do begin logprint, ' ', clip(tstep[s],15), ' ', fpr(times[s]-times[s-1],6.2)+'s' endfor endif logprint, ' --- LPIPE version '+lpar.lpversion+' completed UT '+tstamp(/utc)+' ---', /logonly logprint, /close clearerr = check_math() ; hide arithmetic warnings !quiet = 0 !p.multi = 0 !p.position = 0 end ; add some tools ; pro setobjectpos, name, y, width, bakmin, bakmax, tracefile ; pro setbadfile, cam, number