Batch Flat Fielding Words

The Batch Queue

Separate routines which flatten spectroscopic or (non-SCAN) direct data in batch mode have been written. The microvaxes are set up so that the batch queue is two jobs wide. However, you should set up and run only one batch flattening job at a time. Do not start another batch flattening job until the first one has finished. To see what jobs are running in the batch queue, type:

$ SHOW QUEUE/ALL SYS$BATCH

Specification of the List of Frames to be Flattened

The list of frames operated upon by the batch flattening words can be specified in three ways. The first option is to operate on all frames from the correct camera. (i.e., don't want to flatten a red side of double spectrograph frame with a blue side flat). For direct frames, all frames from the same camera with the correct filter code are used. This applies to all frames with the names that indicate that they were obtained by reading a CCD through OBSERVING, i.e., they have prefixes of S, K, M, C, or F followed by a number (the frame number), followed by a filter code.

The next choice is to specify a range of frames, all of which are assumed to be of the same type (i.e., all from SNAP or EXPOSE -- i.e., starting with S, or all from C (CONSEC), etc.). Again, for direct frame flattening the filter code must be correct.

The last choice is to specify a list of N frames.

In all cases, a check is made that the camera code is correct, that the filter code is correct (for direct frames only), and that the size of the frame (rows$\times$columns) is the same size as the flat field used.

Before batch flat fielding begins, the number of frames which will be flattened is typed on the terminal; to see the actual frames used, look in the log file after the batch job is finished.

Flattening for Direct Frames (DFLAT)

To use DFLAT (the batch flat fielding routine for direct images) you must have a suitable flat field frame. This can be created by using IAVERAGE to average multiple exposures of the dome light source or CONSEC to take a series of exposures and average them. You may also need to use CLEAN (a FIGARO word) to remove any cosmic ray hits that fell on the flat field frames. This flat field image will be used only for frames taken with the same camera port (i.e., the same dewar) and with the same filter.

DFLAT begins by asking which filter is to be flattened, (answer should be a one letter filter code) and what is the name of the flat field image. You must then specify the set of frames to be flattened in one of the ways described above. You may also specify a dark exposure, which will be scaled and subtracted away. The name of the frame used as the dark exposure CANNOT be DARK (i.e. do not use DARK.dst). One way to generate a suitable dark exposure is to take a long exposure with the shutter closed (i.e. have the shutter control on local to force it to be closed), then median filter (with the FIGARO command MEDFILT) the resulting frame to eliminate cosmic ray hits. The dark frame subtraction should only be used for CCDs with luminescing amplifiers or other serious problems, as it increases the noise slightly.

You must then define a region on the flat field frame which will be used for the normalization of the flat field image. The region is defined in terms of its size in pixels and the location of its center. This selected region is imaged and you must indicate that is in fact a suitable region.

The actual algorithm used for flattening is written below. There are two constants. C1 is the mean (from ISTAT) of the (flat field frame -- erase line) over the selected region. C2 is the ratio of the exposure time of the image to be flattened to the exposure time of the dark frame. C1 is typed out on the screen.

$$\hbox{Final flat fielded image}(x,y) = {{C1*(\hbox{Image}(x,... ...rk}(x,y)-\hbox{erase}))}\over{(\hbox{Flat}(x,y)-\hbox{erase})}}$$ (1)

These calculations are carried out as floating point operations, and the result is stored as an integer frame with name DA123F.dst. (i.e., D is added to the front of the image name.) Note that there can be overflows which corrupt the data as a result of the decision to use only integer 2d frames. For example, if you have 20,000 dn in a pixel, and you scale it by 1.5, you have exceeded 32767, and the number will be stored as -32767.

Once the set of frames to be flattened is specified, DFLAT checks available disk space to see if there is enough room for the number of frames to be flattened. If not, a message is typed, and you can change the choice of frames to be flattened. DFLAT then starts a batch job running in the background to carry out the flat fielding. To see whether the job is finished, type SHOW QUEUE/ALL. If there is a job in there called *FLAT* then flat fielding is not yet finished.

If you examine the algorithm used in DFLAT, you will see that if your flat field frame has a large variation in response across the CCD, overflows (i.e., final pixel values exceeding 32767) from valid non-saturated pixels of original data are possible. One way to avoid this is to choose the region used to normalize the frame as that where the sensitivity is in fact the lowest, rather than the highest. This makes all values of the C1/(flat(x,y)-erase) always one or less, so that no pixel which is not originally 32767 becomes larger than 32767 in the flattened frame. However, if the variations in sensitivity across the chip are very large, this may result in less information (i.e., loss of dynamic range) in the final flattened image.

Because of this problem, DFLAT asks if you want floating point or integer frames as output. If you choose floating point, the frames are twice as large and hence need twice as much disk space.

Flattening for Spctroscopic Frames (SFLAT)

Spectroscopic frames are not rotated; they are in the same orientation as when they were created by reading out the CCD. Thus, the dispersion direction is along the columns (y), while the slit lies along the rows (x).

SFLAT assumes that a single flat field frame exists in the disk for a particular grating and setting. You must create this flat by using IAVERAGE, CONSEC, or whatever you so desire on the set of dome flat field exposures. SFLAT begins by asking a series of questions to specify the set of frames to be flattened. These must have the same camera code as the flat field frame. (There is no dark subtraction for spectroscopic frames)

The normalization is defined by specifying an interval in x (i.e., x begin and x end) (these should be two integers) to be used in the flat field frame. The region to be used is displayed on the Peritek, but a region no larger than 512x512 will be displayed even if the chosen region is larger (in either dimension). This region is extracted to form a 1d spectrum, which is then smoothed with a 30 point boxcar. This flat field frame is then divided by this spectrum (all with erase properly subtracted ) to define the normalized flat field frame. This frame has pixels whose values are almost 1.0 throughout the entire frame. The algorithm is then

$$\hbox{Final flat fielded frame}(x,y) = {{(\hbox{Image}(x,y)-\hbox{erase})}\over{(\hbox{normalized flat field frame}(x,y))}}$$ (2)

After setting everything up, SFLAT checks the available disk space, and if it is sufficient, the flat fielding will proceed in batch mode. The names of the flattened frames have P appended as the first letter, so they are PA123F.dst.

Because of the way the spectroscopic flat fields are normalized, and because the range of values in an original spectroscopic data frame is generally smaller than in a direct frame, overflows are not a serious problem here. No option for real output files is provided in SFLAT.