|
LFC - Troubleshooting for Observers |
This page represents an attempt to document some of the known problems that have been encountered in the first year of observing with LFC. Most of these can be fixed without loss of observing time if addressed correctly. If you encounter an error not shown on this page, please contact us so that we can fix the problem and alert future observers.
Troubleshooting Errors (click on the error description for diagnosis):
Upon running the "filter move" command, the filter wheel appears to get to the correct position, but an error message is returned that indicates "filter wheel didn't make it into fine lock".
This will normally happen a few times in the course of an observing night, and is not a major error. Filter wheel moves are always executed in two steps: first, a coarse move is done to put the right filter into rough position. Then, using a fine position sensor on the wheel, a servo loop is run that iterates small adjustments to the wheel's location until a highly accurate position is achieved. Sometimes, the servo loop needs more iterations to converge than are used by default, in which case you'll see this error message.
The Fix: repeat the "filter move" command - e.g., if you received the error on a "filter move 1", just retype "filter move 1" once or twice more until proper convergence is achieved. If the move does not converge after several attempts, you may need to run the "filter home" command.
Upon running the "filter move" command, the filter wheel never appears to get the correct filter into place, and reports an error messgage that the "filter wheel seems to be lost near position xxx"
When the filter wheel executes its initial move from one position to the next, it commands the motors to move in open loop for a specified distance. if, for some reason (e.g. slippage in the mechanical mechasnism) the wheel doesn't end up close to the requested position, the electronics cannot detect where it is out of position or by how much. The solution in this event is to re-home the filter wheel using the "filter home" command.
Upon running the "filter move" command, a failure message is reported that includes several repetitions of "timed out"
Although a "filter move" seemed to trigger this error, the real problem is related to a communications breakdown between mcdcom and the camera electronics. This communications link is established via fiberopitc lines that run up to the PF cage, and connect to the camera electronics at the "utility" board in the Leach electronics. It is possible to get the utility board to hang in a state where it stops listening for commands, and this is the cause of the "timed out" message - mcdcom is timing out in its attempts to communicate.
The Fix: If the utility board is indeed hung, it will require a power cycle of LFC to bring it back to life. Before doing so, however, you should run the following diagnosic commands to verify that the utility board is in fact the problem.
First, run the command "utility status". This will query the utility board to report information on various temperatures and voltages. If things are running correctly, the output should look like this:
+40 V at +40.1
CCD temp at -89.9
Elapsed time 0.000
+15 V at +15.3
CCD set temp at -89.9
Requested time 6.000
-15 V at -15.0
Electronics temp at +37.4
+5 V at +4.87
Heater percentage 24
Heater rate $??1000
Note that the voltages need not be exact, as long as they are close to accurate. Also, the CCD temperatures in the LFC readout are not meaningful. If, rather than seeing this message you see another repetition of the "timed out" message, it is likely that there is a utility board problem.
As a second check, run the command "synch". This is a low level command that is equivalent to "pinging" the camera electronics. If this times out then there is definitely a comminucations problem, and a reboot of the camera electronics is required. This should be done together with the observing assistant, carefully following the instructions in the LFC detailed technical reference. This operation requires someone to go up into the PF cage and takes 10-15 minutes. Should this problem occur during your run, please fill out a pink sheet and contact an LFC pundit (mrm@astro, ras@astro).
The shutter appears to have successfully opened to start an exposure, but when the exposure time is up an error message is reported and the shutter fails to close.
This problem is caused by the faulty activation of a limit switch in the shutter mechanism, which causes one of the shutter blades to stop before it is fully opened. The software then prevents the other blade from closing, in order to avert possible damage to the shutter. Unfortunately, this exposure will be lost.
Should this occur during your run, we ask that you do the following:
This problem appears to have been solved as of Jan 2001.
Having just completed an exposure, the CCDs appear to read out successfully, but an error is reported when mcdcom tries to write fits files to disk.
This problem is likely to occur if your run extends over multiple nights - it means that you have filled up one of the disks on oasis. Fortunately, your exposure is not lost beacuse the data are stored in a memory buffer.
The Fix: At the mcdcom prompt, simply change directories to an area on a partition which is not full. Then issue a "wf" command and your exposure will be written to the new disk location (you will be prompted to do this). Subsequent exposures will then be stored in the new directory.
The guider has been working fine, but suddnely seems to lose communication with the TCS or becomes generally unresponsive. No error messages are reported, but guiding seems to stop, and the family of "telescope ___" commands in mcdcom stop working.
If you see this problem occurring, you should run a "df" to see if the disk on which the lfc home directory resides - oasis:/scr6/ - is full. When the disk fills, lfcguide will misbehave becasue certain read/write operations to the disk fail.
The Fix: If you have been writing your data to /scr6 or someplace in the ~lfc home directory then you should move it to a different location, as /scr6 contains system files and is not intended for data storage. If the disk fills up even in the absence of data files (which one shouldn't expect to happen under reasonable circomstances) you will need to free up some space by other means. One possible solution is to quit the guider and then compress the file lfcguide.out, and then restart. Once some disk space is freed up, the guider should start functioning normally again, although you may need to quit and restart.
I'm having trouble moving / resizing the boxes in the guider GUI that control the location of the guide star and the sky background.
Make sure that when clicking on the box to perform these actions, you are clicking *on the box edges*, that is, on the line drawn at the edge of the box itself. If you try to move the box by dragging a point located inside the box it will not work.
A mosaic readout seems to have failed with error code -4; the error message reads "ccd_endrdc big: read: Directory not empty."
A problem has been reported on occasion where a mosaic readout fails and an exposure is lost. The error output from mcdcom reads as follows:
Cleared mosaic 1 time
Exposure started, ^C to pause
Exposure complete, time 2.00 sec
Opened /dev/astro0, fileno 3
Beginning mosaic read of 2080 x 4128 x 6 (51517440 pixels)...
nread: Directory not empty
ccd_endrdc big: read: Directory not empty
CCD device 0 readout failed, err=-4
Failed, error code = -4, elapsed = 31.861000
Opened /dev/astro0, fileno 3
Map: 0 1 2 3 4 5 -1 -1 tot=6
ccd.019>
In most cases, the appearance of this message indicates that the LFC host computer oasis did not have enough memory available to read out the image. This can happen if one runs mora on the oasis computer and displays over the network to rhea, because the image buffers take up large amounts of memory. Check to make sure you are running mora on rhea, or that there are not multiple copies of mcdcom running. Unfortunately, you cannot recover the exposure that was lost, but these measures should prevent further loss of data. If the memory usage on oasis is low and you are still seeing this error on readout there is some sort of problem with the software, and you should report it to an LFC pundit.
How do I change the filter table so that mcdcom has the right filters listed in the right slots?
Simply edit the file "/usr/ccd/config/filters.def" on oasis. The first character should be the filter slot 0-3 in the wheel, and a string afterwards describes the filter in that slot.
When moving to filter zero only, the software reports that the wheel is lost.
This problem has been reported fairly often, and will require a software fix. The filter wheel is most likely not in fact lost, and typically one can revive it by issuing a single
"filter bump -1000"
command. Do not issue multiple "filter bump"s or you might actually succeed in getting the wheel lost. After bumping the filter, then retry a "filter move 0". If this doesn't work, try homing the filter wheel, and if this in turn doesn't work contact an expert.
Page created by R. Simcoe (ras@astro.caltech.edu) - 1 / 27 / 01