Marco Morais                                                    06/13/08

This file contains a procedure for loading records into the TOAST database.
This procedure might be encapsulated into a single script that has
the capability to both update and bootstrap the database from scratch.

--------------------------------------------------------------------------------
                        Database Update Procedure
--------------------------------------------------------------------------------

1. Copy the following files from the GALEX cluster to
   the same directory on sherpa. The overcountrate
   shutdowns file is the only one that is in use
   by applications on sherpa (gmap), so try and do
   that update as quickly as possible to avoid
   errors from other processes trying to access that
   file.

/home/galex/fltops/mps/build/use/SURVEYS/MPS-overcountrate-shutdowns-list.fits
/home/galex/fltops/logs/mission_status/mission_status.fits
/home/galex/fltops/logs/mission_status/coadds-status-dat.fits
/home/galex/fltops/mps/ref/use/db/mission-pdb.fits
/home/galex/fltops/mps/ref/use/db/mission-tdb.fits

2. Run the following script without any arguments.
   You will have to be in the same directory as the
   script for it to work since it references another
   sql file.  Currently only the 'krl' and 'mmorais'
   users will have the proper database permissions
   to do this on sherpa.

/home/galex/mmorais/checkouts/soda/mps/gips/gi5/tools/db/toast_insert_fits.pl

3. Run the following script.

psql -d toast -f \
/home/galex/mmorais/checkouts/soda/mps/gips/gi5/tools/db/toast_create_obs.sql

4. At this point you will have a 'mission_obs' table
   in the 'toast' database.  This table is the essence
   of what is copied into the 'obs_sum' table and is
   publically available for query by users such as the
   web-based TOAST.  If we wanted to do any validation
   of the observations prior to copying the records from
   'mission_obs' to 'obs_sum', then we would do it at
   this step.  In the absence of anything better,
   simply query for the count of the number of records
   in the 'mission_obs' table.

psql -d toast -c 'select count(*) from mission_obs;'

   The number of records returned by the query should
   be approximately 35k.  If it is not, then you should
   immediately abandon this procedure and go back and
   figure out what went wrong.  If it is, then you
   should continue to the next step.

5. Copy the records from the 'mission_obs' to the
   'obs_sum' table.  The script should run in a matter
   of seconds, but for a brief instant of time while
   this script is running any queries made by users of
   the database will report 0 records.

psql -d toast -f \
/home/galex/mmorais/checkouts/soda/mps/gips/gi5/tools/db/toast_create_obs_post.sql

6. Create the observation name and observation status
   report files that are available from the following
   urls on sherpa.

   http://sherpa.caltech.edu/gips/ref/galex_obs_names.txt
   http://sherpa.caltech.edu/gips/ref/galex_obs_status.csv
   http://sherpa.caltech.edu/gips/ref/galex_obs_status.fits

   Run the following scripts.  You will have to be in the
   same directory as the script in order for the 'make'
   step that copies the new files to the web directory
   to work properly.

/home/galex/mmorais/checkouts/soda/mps/gips/gi5/tools/db/toast_make_obs_names_file.pl
/home/galex/mmorais/checkouts/soda/mps/gips/gi5/tools/db/toast_make_obs_status_file.pl
sudo make install

   At this point, the files available from sherpa should
   match the records that are queried using web-based
   TOAST.

7. Verify that the update is complete by comparing the 'Database
   Reference Date' value reported in the web-based TOAST with
   the same date stored in the first HDU of the mission status
   FITS file.

   Use the following command to obtain the reference date from 
   the mission status FITS file:

    /home/galex/ops/bin/printfits -i \
      /home/galex/fltops/logs/mission_status/mission_status.fits | \
      grep REFDAT

   The date is given in the form YYMMDDThhmmss. This date should
   be identical to the date value reported in the page returned
   when you run a query such as the one below (paste URL into the
   browser and verify).

http://sherpa.caltech.edu/gips/tools/run_toast.pl?&ra=120.0&dec=30.0&coord_sys_type=eqj2000&scitarg_diam=0&search_method_type=point_search&point_fovr=0.5

   Ed Sabol of GSFC said that he would set up a cron job
   to periodically check for new files on sherpa, but that
   until he does that we should contact him when the
   contents of the files changes.

--------------------------------------------------------------------------------
                        Database Creation Procedure
--------------------------------------------------------------------------------

!!! WARNING !!!

  This step should only be done in situations where you want
  to create a database from scratch.  Creating the database
  from scratch is only necessary in extreme situations 
  (eg disk failure when there is no possibility to recover
  the state of the database).

  Run the following script.  You will have to be in the
  same directory as the script in order for it to work
  properly since it references a sql file.  You will also 
  have to change the name of the 'db_owner' variable in the 
  script to match your username.  Currently only the 'krl' 
  and 'mmorais' users will have the proper database 
  permissions to create new databases on sherpa.

/home/galex/mmorais/checkouts/soda/mps/gips/gi5/tools/db/toast_init_db.sh

  At this point you can begin populating the database with 
  records using the 'Database Update Procedure' described
  above.