NSFCAM2 Troubleshooting

NSFCAM2 Troubleshooting Guide

NSFCAM2 has a small set of repeating problems. This document is to provide a reference for dealing with those problems.

It is important that symptoms are clearly identified, so that the correct solution can be found.

Problem: motors are not responding

nsfcam2 uses a motor server daemon program to control the motors via a socket connection to the digiport which has serial connections to the motors used to control the nsfcam2 mechanisms. Sometimes the connection between this server daemon program and the digi gets damaged, and needs to be reset. We do this by resetting the server daemon program via the nsfcam2 GUI.

Step 1: Using the nsfcam2 GUI, click on the Setup tab, then deselect the "Enable Mechanism Control" checkbox (which is 3rd in a series of checkboxes used to enable instrument functionality, after "Enable T-Controller" and "Enable T-Monitor"). This will disconnect the instrument control software from the motor server daemon program.

Step 2: Using the nsfcam2 gui, click on the System tab, then click the "Restart motor server" button. This will turn kill the motor server daemon program, then restart it.

Step 3: Using the nsfcam2 gui, click on the Setup tab, then select the "Enable Mechanism Control" checkbox that we deselected in Step 1. At this point it will take up to 30 seconds for connections to the digiport and the motors to be made, status evaulated, &c., and then the motor server daemon program should report back that the mechanisms are ready, but that the state is unknown. The mechanisms can then be homed.

If at this point the motors are not working, something else is wrong. Check that the digiport is on (in an nsf2pc ssh session, ping "digi" to see if it's up and on the network. If it isn't, check power. If it is, cycle power on the digi, wait 30 seconds for the digi to boot, then repeat steps 1-3.

If the above steps don't work, check that the cables to the smartmotors are still in place and functional. Then repeat steps 1-3.

Problem: The GUI has frozen or otherwise quit working.

Step 1: Kill the GUI. First try the Quit button. If that doesn't work, quit the associated xterm session, either by going to the xterm session and using Ctrl-C or, if that doesn't work, clicking the "X" shaped button in the upper right hand corner of the xterm session.

Step 2: Start the GUI using the NSF2 START button located on the task bar (the horizontal menu along the bottom of the screen). If the xterm and GUI don't show up, or the xterm shows up then dissappears, go to starting the system from zero. If the GUI does show up, click the connect button. If nothing happens, indicating the GUI isn't connecting, quit the GUI and retry. If this doesn't work, go to starting the system from zero. If this does work, make sure an instance of DV is running, and take a test image. If that works, set up and run as normal.

Problem: The GUI is running, images can be taken, but we're not getting any temperatures and/or motors are unresponsive.

In the case where temperatures or motors are unresponsive, we have two possible problems: either the temperature or motor daemon has failed, or the digi needs to be reset.

Step 1: In the Setup tab, disable the system you're having a problem with (unclick the "Enable T-Controller" and "Enable T-Monitor" checkbuttons if it's a problem with temperatures, unclick the "Enable Mechanism Control" checkbutton if it's a problem with the motors).

Step 2: In the System tab, click either the "Restart temp server" or "Restart motor server" button, depending on the system you're having problems with. Wait 20 seconds after this.

Step 3: In the Setup tab, re-enable the system you disabled in step 1. If this is the temperatures system, enable T-Monitor first, then enabel T-Controller. If this is the motor control system, wait 30 seconds after enabling, to see that the motors show the "unknown" condition.

Step 4: From here you should see temperatures and/or be able to home motors. If not, go to starting the system from zero.

Starting The System From Zero

Step 1: Reboot the NSF2PC

Step 2: Reset the digi (see Resetting The Digi)

Step 3: Reset the HP Power supply for the electronics (this is in coolracks, must be done manually).

Step 4: Once the PC is booted, click the NSF2 START button, bring up the GUI, click the connect button, configure, test, and continue.

If none of this works, call Charles or Bobby.

Resetting The Digi
Sometimes when things get reset, the digiport will think that something is still connected to it, and in that case we need to reset it. This can be done via a simple telnet session into the digiport, and issuing a small set of commands.

NOTE: the digi is on an internal subnet accessible by the nsf2pc only!!! You cannot access it from any computer other than nsf2pc!!!

Step 1: Use the "NSF2 SSH" button (next to the "NSF2 START" button) to start and SSH session from NSF2PC. From here, all commands are executed in this session.

Step 2: run "telnet digi" to telnet to the digiport

Step 3: login as "root"

Step 4: enter password, it is "dbps"

Step 5: run the command "kill tty=1-8"

General List of Known Issues

    * Rebooting of nsf2pc: any reboots of nsf2pc require cycling the power on the electronics.
    * Cycling power on the electronics: In the NSFCAM2 portion of coolracks there is an HP power supply (looks similar to the SPEX HP power supply). There are two buttons, one in the bottom left hand corner, and one to the right side, which is red. Use the red button to turn the power off, wait five seconds, turn the power back on.
    * Motor problems: The most frequent problem we encounter is with motor control. This will occur when the user goes to change filters, and never completes.
      Typically this involves the sockets to the digi port getting stale. The most likely solution to fix this is by restarting the NSFCAM2 motor control daemon. To do this, log in as root (to nsf2pc, using the command "su -" (the "-" is necessary to set up the correct path), and run "service nsf2smd stop", then "service nsf2smd start" This should stop then start the daemon that controls the motors.
    * Readout problems: We have had the problem where the electronics quit reading out. This results in the GUI showing a pop-up window saying there was a read failure. To fix, quit the GUI, cycle power on the electronics, start the GUI, and you should be able to proceed normally.