PARK User Manual - reflectometry.org
Contents
1. from commonsvrUti mport service serviceFil Ht EE HE HE HE TE EH FE HE EH HE HE HE HEH Ht EE EE HE HE TE EH FE HE HH HE HE HE HEH def echo s0 sn 0 0 print s0 FE HE EE HE HE HEH EH HH HE HE HE HE HEH FE EE EE HE HE HE EH FE HE HH HE HE HE HEH FILENAME os path join os getcwd echo xml CHO_SERVICE Echo service FE EE TE FE HE HE EE EH HH HH HE HE HE HEH Het aE EH EH TE FE EH HH HE HE HE HE HEH if name main__ serviceFile echo FILENAME ECHO_SERVICE service echo ECHO_SERVICE AE aE AE aE AE AE aE E aE aE aE AE aE HE aE PE AE AE aE aE HE E AE a aE aE aE AE AE aE AE AE HEHE aE aE HE HE aE HE Ha aE aH aE EE H A utility function service is defined in commonsvrUtil py which will handle the communication with the PARK working server It reads the input from the standard input and calls the given service The corresponding serviceFile will read the input from a file and is normally used for the debugging The first parameter for these two utility functions is a service function and the last one is the description string about the service The service function has the following format Function_name s0 sn 0 Where s0 an xml string that specified in the lt inpu2. The procedure to install PARK python source code is the same as in Unix under cygwin unzip park x y z_src zip setup py install It will install PARK in site packages park The executable file is also available for Windows platform parkServer exe User Manual for GUI Client see Simple Instructions for more details especially for Windows Pre requisite python gt 2 40 wxPython gt 2 8 matplot svn version Enter park Run the client application Spython parkClient py A GUI program will popup D For Windows executable files the client application is parkClient exe parkWin is a program that runs on a single node in Windows platform It first starts the PARK server and then starts the PARK client The reflectometry and boxmin modules are needed to run parkClient py Extract these two models from svn co svn svn danse us boxmin svn co svn svn danse us reflectometry Then go to the reflectometry run configure make This will create a python directory under home directory Copy the python boxmin and python reflectometry to park Sometimes the backend may need to be changed Edit park client_aui model refl mixbackend py to select the available backend in matplotlib which could be found at lib python2 5 site packages matplotlib backends All the source code of PARK can be accessed via svn co svn svn danse us park Directory Structure park Home directory for PARK park servers
3. Directory for PARK servers park services Directory for PARK services park parkUI Directory for client side script based UI application park client_aui Directory for client side wx matplotlib based GUI application version 0 3 park parkAui Directory for client side wx matplotlib based GUI application version 0 4 and above park examples Some examples Example to develop applications on PARK 1 Develop the client side applications to submit the request and represent the results 2 Develop the server side applications to do the real work 1 Client Side Application The client side application needs to build the job request submit the request then parse and represent the results The echoClient py in parkUI is such a simple client side application which will use the echo service to testing the network connection and environment setting usr bin env python aE aE aE EEE HH import time import traceback import socket FE FEAE aE ae AE aE aE AE aE aE AE aE aE AE aE ae AE HE a AE aE aE aa AE HE aaa aE aE aaa aaa aE E HE E AE IE E AE AE AE E AE E AE AE AE E FE AE AE AE E E FE AE AE AE AE E FE AE AE AE AE E FE AE AE AE AE AE FE AE AE AE AE AE FE FE AE AE AE AE AE E H from replyEcho import ParkReplyEcho from networkThread import NetworkThread HHPHEEEEREEEEREE EERE EERE EERE EERE EERE EERE EEE HEE HEH HEH EH FH default PARK servers DEFAULT_SERVERS socket gethostname localhost compufans ncnr
4. EE HE HEE EE EE HE EE HEE EE E E E HEE E E E E E E HEE E E H if name main ue main AE aE aE aE AE AE aE aE aE aE AE AE aE AE aE aE aE AE E aE aE HE AE a aE HE aE AE HE aE aE aH aaa HaHa Ea aE aa This client side application is running in the following order 1 Create a new thread to handle the network such as submit the request to the server and receive the reply from the server This network thread has been defined in NetworkThread class It can be initialized in the main thread by NetworkThread server reply where Server atupleof server port where server is the string name of the server and port the number of port that the server will accept the request Reply an object to handle the reply from the server It inherits from reply ParkRep1ly class The user needs to subclass this class and at least implement the function _OnReply self msg where msg is the string received from the server The network thread will automatically call the corresponding function of reply ParkReply class to handle the network message In this example implementation we only print out the reply message as defined in class replyEcho ParkReplyEcho usr bin env python FE aE AE aE FE aE AE aE FE aE AE aE FE FE AE aE FE FE AE aE FE HE FE FE AE FE HE aE FE FE EE FE AE FE aE FE AE FE FE EEE aE EE aaa E EHE ww The class to handle the reply from the park server ww FEAE AE aE E AE aE aE AE aE AE AE AE HE FE aE EE HE HE FE H
5. 0 For all cases Most services may need numpy scipy Unix Linux Mac OS 1 Download park x y z tar gz or park x y z tar bz2 or park x y z sre zip from http chemnuc 20 umd edu DANSE download index html 2 Unzip the file tar xvzf park x y z tar 2Z 3 Make the installation cd park x y z make install or setup py install install purelib home_directory_of_park The command make install is equivalent to setup py install install purelib It will install park in directory park 4 Delete the build files optional cd rm r park x y Z rm park x y Z tar gZ 5 Setup environments to run park optional We assume that park is installed in park cd park config Edit hosts file so only the available nodes are included as the working nodes There are several other examples that can be use where the only the local host is used as the working nodes whose number of cpus may be 1 2 4 8 16 files hosts local hosts local2 hosts local4 hosts local8 hosts locall6 6 Start the server cd park python parkServer py or parkServer py When there is no error the service server is ready to accept the service request The use can connect to the server and submit the service request Make sure that the services are executable files For example in Unix system use chmod u x py to make the services executable 7 Testing optional Using the echo client to testing the connection cd park parkUI python echoClien
6. E HE E HE HE A HE HA HE HE AE HHA HE a HH aE from reply import ParkReply FE aE AE aE aE aE aE aE FE aE AE aE AE aE HE aE FE FE HE aE FE AE HE aE AE FE HE aE FE FE HE FE FE AE HE FE AE FE aE aE AE FE FE AE FE FE AE FE EEE aE EE Ht TE aE a aE a aE a aE EE a FE HE EE FE FE HE TE HE EE EE FE aE HE aE aE EEE EE FE FE EE EE EE HEE HE E E E E E class ParkReplyEcho ParkReply ww This is the class to handle the network message It is used by the network thread Not used directly by the user ww FEAE AE aE E AE aE aE EAE AE AE AE HE FE AE EE HE HE FE HE HE HE HE aE HEH aE Ha AE AE HE AH HEE aE def __ init__ self super ParkReplyEcho self __init__ FE aE AE aE FE aE AE aE FE aE AE aE FE HE AE aE FE FE AE aE FE AE HE aE FE FE AE aE FE FE EE FE AE FE aE AE FE FE AE FE FE a EE EE EEE def _OnReply self msg ww Handler for the normal reply from the PARK server The subclass must implement this function ww print Reply from parkServer msg FE aE AE aE ae aE AE aE ae aE AE aE AE HE FE aE FE FE AE aE FE AE HE aE FE FE HE aE FE FE Ea FE FE FE aE HE aE HEHEHE Ea Ea EEE E EH 2 Start the network thread prepare and submit the job request The network thread will be starting to handle the network by call its start method 2 1 Then the main thread needs to prepare the job request 2 2 For the network thread use asynchronous model to communicate with the server the job request will be st
7. PARK User Manual PARK is a pluggable framework for distributed computing It is implemented in pure python provides a simple and easily managed distributed environment for computational intensive computing The architecture of current PARK implementation is shown as in Fig 1 ServiceServer i Cluster Management _ Service Server Master Node WorkingServer work Nodes a N 7 7 N 7 S 7 Fig 1 The architecture of current PARK implementation PARK is implemented in the standard client server style and map reduce model further information see MapReduce Simplified Data Processing on Large Clusters at http labs google com papers mapreduce html The main part of PARK is the service management server which can provide the job management and cluster management for the distributed computing The service management server starts the working server on each working node in the cluster manage the request job queue and the load balance Server installation The PARK server includes the service management server and the service working server The service management server will run in the master node which can be connected to the outside client It will start and stop the service working server manage the job and cluster The service working server runs on each working node within the cluster provides the services to the service management server Pre requisite python gt 2 4
8. educe function will be called on the server once with the output from cnt map functions as the input For the map functions are not running simultaneously in distributed environment the reduce function will be called on the server each time when there is the reply from the working node 2 Server Side Application Map function This kind of application will run in the working nodes The PARK working nodes run this map function as a separated thread or process All the map function outputs to the standard out will be collected by the PARK working server and send it back to the PARK service server The PARK service server stores this information in the archive system passes it to the reduce function and sends the results to the client The map functions services are under park services directories Echo service park services svrcommon echo py is a simple service which only echoes the string passed to it The function is usr bin env python EAE aE aE aE AE aE aE E aE aE HE EA HE aE EAT EE EAE AE aE E aE aE AE AE aE aE aE aE aE aE aE AE HE aE aE EEE AE AE AE aE aT aE aE aE AE aE aE AE aE aE aE a AE HE aE aE aE aE E send back the user s xml request
9. nist gov default port number used in park DEFAULT_PORT 5400 DEFAULT WAITING TIME WAITING_TIME 150 i ee AE AE AE FE AE AE AE AE AE FE FE TE AEA FE aE aE aE AE AE aE aE AE aE AE AE AE AE AE aE AE AE AE aE AE AE AE AE EE aE AE AE E AE AE AE AE AE AE AE AE AE AE AE AE AE AE AE FE AE AE AE AE AE AE AE AE AE AE AE AE AE FE EAEE EH def main server DEFAULT_SERVERS 0 DEFAULT_PORT reply ParkReplyEcho networkThread None 1 networkThread NetworkThread server reply 2 1 networkThread start print n Start network Thread 232 request mmu lt session version 0 2 1 name wwchen type 7 user wwchen email wwchen nist gov priority 0 gt lt group name groupl gt lt dataSet gt lt dataSet gt lt reduce classname Chisq gt lt task emd echo py gt lt bufsize value 3000 gt lt task gt lt joblist name Jjoblistl1l ent 5 gt lt input gt This is a echo example lt input gt lt joblist gt lt group gt lt session gt www 2 3 reply GetOutQueue put request print Apn Main Thread is sleeping 2 4 time sleep WAITING_TIME except print NetWorkThread exception traceback format_exc if networkThread is not None print n Stop network Thread FSA 2 5 networkThread stop print n a Main Thread is closing HEE FE HE HE EEE EEE EEE HEE HEE HE
10. ored in the queue provide by reply handler 2 3 The user doesn t need to submit the request to the server directly Then the main thread sleeps for a while so the network thread can receive the reply from the server and call the reply handler to parse the reply properly 2 4 In the real application the main thread may in an infinite loop to prepare the job request and the reply is handled in a separated thread Once all the work has been done the main thread call the st op method of the network thread to stop the communication with the server and exit the program 3 XML format for the job request 2 2 The job request sent to PARK server must be a valid XML string with the proper format Among which the most import ones are highlighted in 2 2 They are lt reduce classname Chisq gt classname for the reduce function lt task cmd echo py gt lt task gt file name for the map function lt joblist ent 5 gt cnt is the total number of map function that should run in the working nodes lt input gt This is a echo example lt input gt This is the string sent to the map function lt joblist gt There can have multiple lt joblist gt lt joblist gt in one job request The reduce function must be accessible for the PARK server and the map function must be accessible for the working nodes Theoretically the map function will be run total ent times with the given input on the working node then the r
11. t gt lt input gt tag in the job request The service function needs to parse this xml string to get the input parameters sn A integer representing the serial number of current job within the cnt number of requested jobs Within the service function any outputs to the standard output will be collected and sent back to the client In echo service it will send back the input string as shown in 0 Recommendations for distributing service 1 Put the service under correct directory currently put it on any subdirectories under park services 2 Make sure that the service works locally for example using serviceFile to make sure the service works correctly then changing to service for the service application 3 Make sure that the service is executable in Unix Linux environment
12. t py or echoClient py This will echo the contents of the request which will be sent to the service applications as a String in the real application The application will parse the input string do the real work and then send its results to the client 8 Shut down the service server by Ctrl C or kill command Please use kill without 9 command which will also stop the working server program Otherwise the working server will continue to work even the service server is killed Error shooting 1 Make sure that python and its environments are set correctly 2 For cluster with multiple working nodes make sure that RSH defined in park servers environ py is set to the remote shell command The default is rsh and it may be changed to other remote shell command such as ssh Make sure that this remote shell command can start the remote command without the password 3 Errno 2 No such file or directory park config hosts no configure file hosts 4 ERROR 111 Connection refused or socket error 111 Connection refused a The working server doesn t start b netstat make sure that the port is not used 5 ERROR 98 Address already in use a The address is used b The server cannot be started immediately after it is shutdown 6 No such file or directory a Make sure the file is in the proper directory b Using servers switchFD py to change the r n string used in windows to r in Unix Windows OS
Download Pdf Manuals
Related Search