Directory Tree Confirmation (as of Sep'08)

General comments:
This document describes running checks and confirmations on the pipeline output directory trees. In particular, we want the 'use' links under /home/galex/fltops/mains/ and /home/galex/fltops/visits/ to point at the most recent pipeline reduction for a given coadd or visit.
Summary of Checks and Corrections:
  1. Check that forest directory ( /home/galex/fltops/... ) 'use' link points at most recent reduction based on date in 'Out.txt' file.
  2. Delete any use links that point at a '-try' directory that is not the most recent.
  3. ... more to be documented ...

Program usage:

For checking visits:

    /home/galex/tab/bin/pipecheck visit.list -confirm  [ -no72 ] [ -CheckDirPred ] [ -MakeChanges ] >! visit_list.out

For checking coadds:

    /home/galex/tab/bin/pipecheck tile.list -confirm  owc=d  [ -no72 ] [ -MakeChanges ] >! tile_list.out

Where 'visit.list' is a list of current '-try' directories of visits, and 'tile.list' is a list of tile names for coadds. Normally, the program only checks directories which have not been modified within the last 72 hours-- avoid this limit by using the '-no72' option. Run the program first without the '-MakeChanges' option. When you are ready to actually change the directories and the use links, run again with this option.


Method:
For visit checking, this program looks at all the '-try' directories for a given tile name and visit number on all the known data disks (under 01-vsn and 00-visits). The program checks the date written at the end of the 'Out.txt' log. The directory with the most recent date written in the log file is assumed to be directory to which all the 'use' links should point. For coadd checking, all possible '-img' and '-try' coadd directories on all the known data disks (under 01-vsn and 01-main) are tested.
Output files:
Lists of the tiles or visit directories with errors and warnings are written out to files with the '-error' and '-warning' extensions. You can use these listings to rerun the program with '-MakeChanges' on only those cases which had problems. This avoids having to rerun the entire input list when you decide to execute the directory modifications.
Reading the log file:
Using the command line syntax above, you will get output messages to the '.out' files. Error messages and warnings are printed when use links do not point at the most recent reduction. The messages for each tile or visit begin with the line " #:Confirming ... ". A listing of all directories and links associated with that tile or visit are shown first. The a list of the actual data paths (see "NOTE: Real path ...") with a date shown for each one. This date is from the 'Out.txt' file. The path with the most recent date is considered the current (correct) data for each tile or visit. The paths and links are then shown in groups and then warning and error messages are shown. Finally, a set of commands is given for fixing any possible problems. If '-MakeChanges' is not used, then you will see the words 'Not executing:' for each command, otherwise the commands are actually executed and directories are changed. Normally, the only changes made will be to 'use' links. Sometimes data directories are moved into '/erase/' subdirectories for later deletion, but no real data is actually deleted by this program.


Things to do: