Directory Tree Confirmation (as of Sep'08)
This document describes running checks and confirmations on the pipeline output
directory trees. In particular, we want the 'use' links
under /home/galex/fltops/mains/ and /home/galex/fltops/visits/ to
point at the most recent pipeline reduction for
a given coadd or visit.
Summary of Checks and Corrections:
- Check that forest directory ( /home/galex/fltops/... ) 'use' link points at most recent reduction based on date in 'Out.txt' file.
- Delete any use links that point at a '-try' directory that is not the most recent.
- ... more to be documented ...
For checking visits:
/home/galex/tab/bin/pipecheck visit.list -confirm [ -no72 ] [ -CheckDirPred ] [ -MakeChanges ] >! visit_list.out
For checking coadds:
/home/galex/tab/bin/pipecheck tile.list -confirm owc=d [ -no72 ] [ -MakeChanges ] >! tile_list.out
Where 'visit.list' is a list of current '-try' directories of visits, and 'tile.list'
is a list of tile names for coadds. Normally, the program only checks directories
which have not been modified within the last 72 hours-- avoid this limit by using the
'-no72' option. Run the program first without the '-MakeChanges' option. When you are
ready to actually change the directories and the use links, run again with this option.
For visit checking, this program looks at all the '-try' directories for a given
tile name and visit number on all the known data disks (under 01-vsn and 00-visits).
The program checks the date written at the end of the 'Out.txt' log. The directory
with the most recent date written in the log file is assumed to be directory to which
all the 'use' links should point. For coadd checking, all possible '-img' and '-try'
coadd directories on all the known data disks (under 01-vsn and 01-main) are tested.
Lists of the tiles or visit directories with errors and warnings are written out to files
with the '-error' and '-warning' extensions. You can use these listings to rerun the
program with '-MakeChanges' on only those cases which had problems. This avoids having
to rerun the entire input list when you decide to execute the directory modifications.
Reading the log file:
Using the command line syntax above, you will get output messages to the '.out' files.
Error messages and warnings are printed when use links do not point at the most recent
reduction. The messages for each tile or visit begin with the line " #:Confirming ... ".
A listing of all directories and links associated with that tile or visit are shown
first. The a list of the actual data paths (see "NOTE: Real path ...") with a date shown
for each one. This date is from the 'Out.txt' file. The path with the most recent
date is considered the current (correct) data for each tile or visit. The paths and
links are then shown in groups and then warning and error messages are shown. Finally,
a set of commands is given for fixing any possible problems. If '-MakeChanges' is
not used, then you will see the words 'Not executing:' for each command, otherwise
the commands are actually executed and directories are changed. Normally, the only
changes made will be to 'use' links. Sometimes data directories are moved into
'/erase/' subdirectories for later deletion, but no real data is actually deleted by
Things to do:
- check for try directories of the same visit divided between different disks [ done 9/2008 -- moved when needed ]
- check for coadds dated later but not as up-to-date as earlier coadds [ done 9/2008 -- best use link override ]
- identify duplicate directories (same Out.txt date) and decide which to erase [ done 10/2008 -- older directory sent to erase/ ]
- clean out (delete) unneeded files from old try directories (sep'08) [ in progress 9/2008 -- ]