Software startup check
Contents
1. AM E Te 5 SN e TEE 5 4 2 Remote operation SUQUUS EEN 5 4o PEP SIUS EE E EE deeniocutes Asante 5 AA Inside of fedome ccc ccc ccc secs eeceeceeceeceeceeaeceeeeeeceeseesgeseeseseesaeeeeeseeseeseeseesensaes 6 4 5 Communication GEIRG Zelescopne 6 4 5 1 Complete command chan T 4 52 Commands ON UINA EE T e le ue e ne e E CN 9 AG Ee e E 9 9 Fi nalstan p SUS EE 10 Doc Rer PANIC SW MN 02 PANIC software Issue 1 1 startup check Date 17 07 2015 Page 4 10 1 INTRODUCTION AND SCOPE 1 1 General The communication of the PANIC instrument control software ICS GEIRS RD 1 with the CAHA 2 2m telescope runs via multiple interfaces and is unfortunately prone to outages A startup guide shall allow the quick check and troubleshooting of the control system It is recommended to do the startup check about 1h before starting any observation to have enough time to fix complicated problems 1 2 Contents This document contains a guide how to check the functionality of the 2 2m telescope control from the PANIC instrument control software before the observations and which actions are necessary to establish a correct communication with the telescope For the moment it is restricted to the 2 2m telescope The document aims at the CAHA staff and possibly experienced observers It is TBC if parts or the whole document can be transferred to a user manual for general observers It covers only simple diagnosis tools and issues and not all problems2. Figure 5 the call of the telescope commands by GEIRS is not working Ask the IT group for help to check this interface It may help to look at the responses of the telescope interface to GEIRS by logging in as ssh obs22 panic22 and check the logfile with cd GEIRS log and fgrep answer debug obs22 log fgrep tele o If no output is shown or it also contains error messages ask the IT group for help to check the t_command scripts 4 6 GEIRS readout Description In some cases GEIRS cannot allocate the memory required for the readout data This is normally prevented by checking the available memory and freeing it when needed However in rare cases this can fail right after the startup of the software although the free memory seems large enough Diagnosis Read an image with GEIRS by clicking Read in the GUI Check if the read fails and if there is an error output in the instrument shell containing the message unable to allocate Memory for Buffer Doc Rer PANIC SW MN 02 PANIC software Issue 1 1 startup check Date 17 07 2015 Page 10 10 Fix Check the free memory by going to a terminal and calling free The output lists several lines similar as follows total used free shared buffers cached Mem 66059796 49001800 17057996 Q 208032 47378640 buffers cache 1415128 64644668 Swap 2103292 80600 2022692 Note the value in the first line of numbers below free here 17057996 Divide it by 1000 and round up to a nice
3. log Move again the filter wheels between Ks and Z and check if messages for focus events appear as shown in Figure 5 WWII duagalll AMAAL in tte om propru ofl Sun Jun 28 19 15 37 2015 from xt22 caha es au d ot of fun Co anic22 gt ssh posets login Sun Jun 28 19 15 46 2015 from panic22 caha es ui a Ui of fun Ibs226 Ne ei J cadeve1 L0G T22 eigent yhome cadeve 1 LDG TZ2 gt tail f t_command log_panic22 log 2 12 30 END command t_dfocus_22 Code 31 12 30 _ ___ ___________ __ 13 39 13 39 INI command t_dfocus 0 054000 9 13 45 t_dfocus Finished with focus at 9 13 45 t_dfocus Requested focus position 19 13 45 t_dfocus Limits 21 978 22 008 19 13 45 t_dfocus OK 19 13 45 END command t_dfocus_22 Code 9 19 13 45 ___ ______ _____________ OO fake be de 96 15 19 96715 19 96715 19 96715 1 96715 1 96715 96715 96715 96715 b i Shell a Shell No 2 Figure 5 Logfile of the t_command scripts on posets for focus commands Fix e If log entries are written and indicate that the called commands are correct the communication with ultra3 is not working Reboot ultra3 or ask the IT group for help s If no log output appears or errors about the command itself appear call a telescope command directly on posets At the prompt enter t_command t_dfocus 0 04 d o If the output lists no errors and is similar to the one in
4. number here e g 18000 Go to the directory of the GEIRS installation GEIRS trunk rxxxM yy where rxxxM yy is the highest available combination of numbers Go to the subdirectory bin and clear the memory by calling geirs_dropcaches m mmmm where mmmm is the memory number determined before here 18000 When the command has finished try again to read an image 5 FINAL STARTUP STEPS If the telescope properly executes the commands from GEIRS and the dome is clear the system is ready for the observations The final steps to bring the telescope to operational status are e switching on the tracking of the telescope s opening the dome when it is allowed by the weather conditions e opening the main mirror cover e launching the OT Then observations can be performed as usual
5. that may occur 2 CONTROL ARCHITECTURE The control architecture of the telescope during the observations consists of the following layers and interfaces to the upper level e Observation tool OT RD 2 High level control software for observer running on panic22 e GEIRS Readout and instrument control software running on panic22 interface with OT via command server e posets Computer to process the telescope control commands from GEIRS interface t_command Tcl scripts e ultra3 Computer to handle the telescope and dome control commands interface EPICS database e tecs22 Computer to communicate with the telescope interface TECS e PEP Control computer of the telescope hardware 3 START REQUIRED PROGRAMS To operate PANIC at the T2 2m from the remote control room at least the following programs have to be started and running e DOMEC Dome and telescope remote control GUI s GEIRS PANIC instrument control software All other programs and systems are running by default The OT is optional and not required for the functional check described here Doc Rer PANIC SW MN 02 PANIC software Issue 1 1 startup check Date 17 07 2015 Page 5 10 The DOMEC is usually started look for it on the desktops of the xt22 terminal instrument control terminal If it is nowhere to be found start it by opening a terminal window log in as ssh X obs22 ultra3 and start it with start _domec amp Be aware that it may hang up at some poi
6. Description The PEP may hang often sometimes daily and then needs a restart Verify that the PEP is running Diagnosis Have a look at the TECS GUI of the telescope control computer Figure 1 Doc Rer PANIC SW MN 02 PANIC software Issue 1 1 startup check Date 17 07 2015 Page 6 10 13 25 03 8 19 09 00 1 Ji 2457202 2979 13 25 03 7 37 13 48 MI 00 00 00 0 Figure 1 GUI of the TECS at the telescope control terminal Check that the values of ST and UT are updated about every second and the UT is correct If it is frozen the PEP needs to be restarted Fix First check in the DOMEC that the hydraulics and drives are off red color If not switch off drives and hydraulics If it does not work restart the DOMEC see section 3 and try again ONLY THEN use a browser and go to the webpage respep caha es may already be opened somewhere To switch the PEP power off and back on Click on the plug symbol in the line of PEP wait for a few seconds and click the symbol again Wait about a minute and check that the TECS GUI elements are updated correctly If that does not fix it close the TECS GUI and restart it by right clicking on the empty desktop area and select TECS 2 2m from the context menu see Figure 4 Start the current version Check if now the time values are up to date 4 4 Inside of the dome Description Check if the inside of the dome is clear of persons and items if not already done when setting t
7. Prepared by PANIC software startup check Doc Ref PANIC SW MN 02 Issue 1 1 Date 17 07 2015 Page 1 10 Software startup check Bernhard Dorner Max Planck Institut fur Astronomie Revised by Code PANIC SW MN 02 Issue Ver 1 1 Date 17 07 2015 No of pages 10 Doc Rer PANIC SW MN 02 PANIC software Issue 1 1 startup check Date 17 07 2015 Page 2 10 Document Change Log Version Date Chapters Comments affected Issue 1 0 08 07 2015 4 Complete revision of checks TT eee Issue 1 1 17 07 2015 4 6 Added fix for GEIRS memory TT ee E List of acronyms and abbreviations GEIRS Generic Infrared Software GUI Graphical User Interface Instrument Control Software Observation Tool PANIC PAnoramic Near Infrared camera for Calar Alto List of supporting documents The following documents provide additional information about topics addressed in this document They are referenced as RDx in the text RD Nr Doc Title and Issue PANIC SW DCS 01 GEIRS Installation and User s Manual PANIC SW MN 01 PANIC Observation tool user manual RD 1 RD 2 Doc Rer PANIC SW MN 02 PANIC software Issue 1 1 startup check Date 17 07 2015 Page 3 10 Contents 1 introduction and SCODE aise des citterectectsaveaacscencenecenaveclertdadsuebedgeucicbesieiadanadamdeluaninevaeeneedens nade 4 Pes ee 1 E 4 Ms ee 4 CZE eine E eer UE 4 3 Slalt EGUITER Proglas EE 4
8. erver Move again the filter wheels between Ks and zZ and check if messages for focus commands appear as shown in Figure 3 Doc Ret PANIC SW MN 02 PANIC software Issue 1 1 startup check Date 17 07 2015 Page 8 10 Figure 3 Teleserver output on ultra3 for focus commands Fix e If the teleserver output appears but the telescope still does not execute the command reset the TECS interface On the telescope control computer right click on the empty desktop area and select RESET tecs dacs tvg from the context menu Figure 4 Then try again moving the filters a few times between zZ and Ks If the errors persist try rebooting ultra3 or ask the IT group for help There could be more difficult problems in the command chain starting at ultra3 e If the teleserver output does not appear continue to check the communication with posets as written below SI E SCH DM Et E EN i brun i eit ail el JOEREN SPARE TV Guider 2 2m _ Utilities i Properties workstation Info Help Exit Figure 4 Context menu of the desktop of the telescope control terminal computer Doc Rer PANIC SW MN 02 PANIC software Issue 1 1 startup check Date 17 07 2015 Page 9 10 4 5 3 Commands on posets Diagnosis Open a terminal window and log in as ssh obs22 panic22 then as ssh posets Have a look into the t_command logfile by going to cd cadevel LOG T22 and calling tail f t command log panic22
9. he remote control switch Diagnosis Open the webpage of the webcams inside the 2 2m dome In the DOMEC switch on the lights Have a look at the webcams check that no person is inside the dome and the area around the telescope is clear of any items Switch the lights off again Note When the startup is done early it is recommended to do this check again before switching on the telescope tracking Fix Go up to the 2 2m building gently but firmly ask interfering persons to leave the dome or call the dome phone and or remove obstructing items 4 5 Communication GEIRS telescope The communication path from GEIRS to the telescope passes multiple interfaces so debugging requires having a look at various places The detailed way how to progress is explained in the following Doc Ret PANIC SW MN 02 PANIC software Issue 1 1 startup check Date 17 07 2015 Page 7 10 4 5 1 Complete command chain Description Check if GEIRS can control the telescope Diagnosis Have a look at the DOMEC and check if Hydraulics and Drives are on green If not switch on Hydraulics and Drives If it does not work restart the DOMEC as noted in section 3 and try again If it still does not work the drives may have to be recentered In the GEIRS control GUI select the wheel macro Z While the macro runs observe the GEIRS instrument shell for errors and error popup windows Should the macro complete without errors select the filter Ks Agai
10. n observe the instrument shell output but also the Focus value in the TECS GUI Figure 1 The focus should move by several tens of microns If everything works the system is OK and you have to execute the remaining preparation steps in section 5 However if any error similar as in Figure 2 appears the focus correction sent by GEIRS is not executed Continue to analyze the error as below ng with process es Focus irh movement keier error 31 from Tetz ae d analyse_waitd4dpid exit status 74 if geirs error E te ned ma TetS JERROR start prorcess and check returned with error exit statuss H E tecs 4 error returned from TeCS V eel Macro time 45 sec jwheel E_tecs 74 error returned from TeCS JERR OR wheel E_tecs 74 error returned from TeCS jalpha 48415 374000 delta 134026 964000 hrangle 0 000012 air d ele terminated ok a ERROR ATTENTION expected focus correction returned error abs focus p d 21 993mm current 21 939mm SC ATTENTION expected focus correction returned error abs focus pos expecterc ag current 21 939mm CS vcirstzsmgmEEEEIrabrrEEIEEREeTTECEE EES ther om Figure 2 Example of an error when the telescope does not execute the focus correction after a filter change 4 5 2 Commands on ultra3 Diagnosis To narrow down the error check the communication from GEIRS up to ultra3 To do this open a terminal window log in as ssh obs22 ultra3 and re start the telescope server with start _teles
11. nt so if any button is not working or not updated quit and relaunch it as noted before GEIRS is launched from the panic computers If it is not yet running open a terminal window on the instrument control terminal log in as ssh X obs22 panic22 and start GEIRS with start panic new Set the RO Electronics and Motors to IP addresses and the Telescope to EPICS Start the GUI by clicking on Al In the Controls GUI open an instrument shell with Modules New InstrShell Wait until the initialization is complete and the filter wheel buttons are not greyed out For more information refer to RD 1 4 SYSTEM CHECKS 4 1 Overview The following minimal checks are required 1 Remote operation status Is the telescope in remote operation 2 PEP status Is the PEP running 3 Inside of the dome Is the dome clear 4 Communication GEIRS telescope Can GEIRS control the telescope You can find details for every step and fixes for common problems in the following sections 4 2 Remote operation status Description Verify correct status of the telescope operation switch Diagnosis Have a look at the DOMEC GUI and check if the light for Remote is green If it is red the switch at the telescope control console is still in position Local Fix Go up to the 2 2m building check that the inside of the dome is clear and no person is there and change the switch position to Remote 4 3 PEP status
Download Pdf Manuals
Related Search