9.- Data Reduction Machine

9.1 - General Information

As an add-on to the WIRC System, a reduction machine is configured for the data pipeline.
The idea is that this machine does not affect and it is completelly transparent to the acquisition machine.
There is a task running on this machine for getting all the new images acquired through the net, automatically, using sftp. This means that the reduction machine pulls the data out instead of having the acquisition machine pushing it.
In this way, if the link is not working, or the reduction machine is down, the acquisition machine is not affected at all. We may say that the acquisition machine does not even know of the existance of the reduction machine.
The system works basically in this way:

There is a database on the ArcVIEW system, a module called DBS. This database is a general purpose module which holds variables of general interest for all the rest of the system. This DBS module has, besides the internal port for the other modules, an external TCP/IP port for servinbg external requests.
One of the things the database can keep track of, is the images that has been written to disk. If set for doing so, the FITS writter module can send to the Database any new image that was written to disk. In this way, the DBS mantains a list with all the images that have been written. There is a set of commands that the DBS module accept for handling this list of images.

In the Data Reduction Machine, then, can run any software for communicating with the DBS module on the acquisition machine, so it can know which images have been written and where they are; this is the only information this machine needs in order to pull the desired data out.
Any software that can use a socket for communicating can then do this.

As examples, two "applications" has been created, intended onlly for starting and demo purposes; more complex applications can be built in the future by the interested parts:

a) Iraf scripts: a set of Iraf scripts for polling the data has been created. Documentation on this Iraf scripts can be found on the Scripting section, and on the scripts files itselfs ($HOME/irafpkgs/arcview/imageList/*)

b) an ArcVIEW-based application was created too. An specific arcview module was created for remotelly getting the data. This application is a module that can be loaded on any ArcVIEW application. This module basically knows how to talk with the DBS module through the socket. The idea then, the reduction machine can starts a local ArcVIEW with just this module (IMPUL module) loaded. This module will talk with the remote DBS module and can talk also with a local DBS module, if running. So in a more complex situation, it can bring the images from the acquisition machine, and in turn send the names of the images gotten to the local DBS, which in turn will keep track of all the images now on the local disk. A third machine can now run another IMPUL module and get the images from the local machine, forming a daisy-chain. Of course normal application will not requiere such complexity, but it can be naturally done.
A little GUI was created for handling this module on the reduction machine. Documentation on that GUI can be found here

Another thing that now can be done os to mix the rpovided Iraf scripts with the provided ArcVIEW module provided. Since the Iraf scripts can talk to a DBS, if the local ArcVIEW is running a DBS module (besides the IMPUL one), the script can talk to the local DBS and process the images as they are arriving, so we will have  a task just getting images (ArcVIEW) and another postprocessing it in parallel (Iraf scripts).

As was said before, all this are just demos that can be improved, modified or completelly replaced as needed.
For a more general information, the list of available DBS commands for handling the image list through the net are described here


9.2- Starting the Data Reduction Software for pulling the data

It is important to note that both machines, the acquisition and the reduction machine has exactly the same software.
This means that both machines can be run as either reduction or acquisition machine, meaning that the purpose of the reduction machine is dual: in normal times, it will be used as data reduction machine, but in case the image acquisition machine dies, for any reason, it can be swapped and act as data acquisition machine.
There is nothing to do in order to configure the machine as reduction or acquisition; it will detect inmediatelly if the acquisition PCI card is installed or not, and will autoconfigure according to that.
So for starting the software for getting the images:

a) If you will use that iraf scripts, then just start a cl windows and type "getimages" on it ("epar getimages" for configuration)
b) If you are going to use the GUI, then just start ArcVIEW in the same way it is done on the  acquisition software, and it will start ArcVIEW configured as  an "image puller" rather than as an "image acquisition" system


9.3 - Technical Information

The WIRC data reduction machine is a dual Xeon Processor running at 3.06 GHz. It has 3 GB of RAM and about 130 GB of free HD.
It is a dual head monitor, configured as a unique monitor (Twin View).
It has a SONY DRU500A DVD writter drive, which speeds are:
DVD-R, DVD+R: 4x
DVD-RW, DVD+RW: 2x

Operating system: Linux RedHat 8.0
Software packages installed for recution purposes includes Iraf 2.12 and IDL 5.6