Latest AEDCS User Manual
Contents
1. 4 Click the Enable button to enable this variable set Note If a variable set is not enabled the values will not be used during data mapping 5 Enter a value in the Variable Set Name field This should be a unique name that helps identify this set of variables Tip Variable sets can be used to manage variables that change between projects or experiments Planning can be done for upcoming experiments by creating a new variable set and keeping it disabled until the experiment starts 6 Click the Add Variable button to add a new variable entry 7 Add a Key and Value for the variable 19 of 25 8 Repeat steps 6 for each variable you want to set Add New Variable Set o Enable Vanable Set Name UMNEXF 1 Add Variable Cancel Save 9 Click the Save button to save the variable set and close the window and display the result of the operation 20 of 25 Operation Result x Result SUCCESS Ok 10 Verify that the variable set appears on the list and that the variable set has a Current 11 State of Enabled Note Refreshing the browser page may be required to see the change Configure Variables UMNEXPI Enabled Edit Delete Add Variable Set Configuring the AEDCS Data Mapping To configure the data mapping for a client perform the following steps Note The client must be connected prior to configuring the data mapping for it 1 Open Google2. 02 214 zip Navigate to the temporary directory where the ZIP file is located For example cd tmp Copy the WAR file to your webapps Tomcat directory For example cp tmp aedcs aedcs war opt tomcat webapps 12 of 25 12 13 14 15 16 Switch to the user account you want to run Tomcat as For example su lt user account gt Where lt user account gt the Linux user account you want to run the AEDCS process as Set the JAVA HOME environment variable to the path where the Oracle JDK is installed If this variable is not set Tomcat may not start For example export JAVA_HOME opt jdk Set the TOMCAT HOME environment variable to the symbolic link we created for Tomcat For example export TOMCAT HOME opt tomcat Start Tomcat as follows cd opt tomcat bin startup sh Open a web browser and connect to the following URL http lt central server ip gt lt port gt aedcs Where lt central server ip gt the IP address of the host running the AEDCS Central Server lt port gt the port Tomcat is running on Note You may need to open a port in the Linux firewall if you are having trouble connecting on step 9 Windows Central Server Installation System Requirements Oracle Java Development Kit JDK 7u45 Tomcat 7 0 47 Windows 7 32 bit or 64 bit 50MB free space for AEDCS server software Enough free space to keep a cache of all experiment data Note This guide d
3. B Experiment 1 i E Experiment 11 a E Experiment 12 I E Experiment 13 j E Experiment 14 I E Experiment 15 i E Experiment 16 I E Experiment 17 I E Experiment 18 j E Experiment 19 I E Experiment 20 f E Experiment 21 E Experiment 22 I E Experiment 4 I E Experiment 5 I E Experiment 6 L a Public Selected Models Action Refresh Create Subdirectory Rename Delete Upload Download Verify Files 3 Click the Upload button to start uploading data from this directory on the Central Server to nees org 25 of 25
4. Name imageprocessor start stop Verify Cancel Send 2 Click the Start button 3 Click the Send button to send a message to the client Starting a client will instruct the client software to send data to the Central Server Once all the data from the client is sent the client goes into a synchronized state Stopping Data Transfer a Client to Central Server To stop a client from sending data follow these steps 1 Click the Command button on the Configure Clients page The following dialog will appear Send Command to Client x Name imageprocessor start stop Verify Cancel Send 24 of 25 2 Click the Stop button and press Send to send a command to stop transferring data from the given client to the Central Server Data Upload Prior to data upload it is expected steps in the Data Upload Preparation have been completed Also it is expected that data has been transferred to the Central Server from clients Here are the steps to upload data 1 Navigate to the Configure Upload page Note It is expected that your NEES username password and project ID have already been entered The project you intend to upload data for should be shown in the Central Server Files section 2 Expand the directory tree under Central Server Files and highlight the directory you want to upload as shown below Central Server Files a P8 L a Analysis ad Documentation
5. Settings View Log Configure Variables Central Server Files SE P g Configure Upload About Help 2 Click the Edit Settings button to bring up the Pen Settings dialog Edit Upload Settings o NEES Username your nees username NEES Password Enable Automatic Upload Cancel Save 3 Enter your nees org username and password in the Edit Upload Settings window 4 Enter the project ID of the data you want to upload 5 Decide if you want to enable automatic upload of data as it is gathered from data acquisition systems If so check the Enable Automatic Upload checkbox 6 Click the Save button to close the window and save your data Note the Central Server Files area will refresh and show a workspace for the give project ID It will be empty by default until data is uploaded to the Central Server or downloaded from 23 of 25 nees org 7 Create the directory structure required for your data using the Create Subdirectory and Rename buttons in the dialog Now you are ready to start transferring data from the data acquisition systems to the Central Server Transfer Data from a Client to Central Server To start transferring data from a client to the Central Server follow these steps 1 To start data transfer from a client to the Central Server click the Command button on the Configure Clients page The following dialog will appear Send Command to Client x
6. outside the scope of this document There are many resources on the internet that describe how to properly install these components Linux Central Server Installation System Requirements e Linux Kernel 3 0 e 50MB free space for AEDCS server software e Enough free space to keep a cache of all experiment data 11 of 25 Installation Steps 1 10 11 Open a terminal Change to the root user su Download and install the latest version of the Oracle Java Development Kit JDK in the opt directory of Linux For example cd opt tar xzf home user Downloads jdk 7u51 linux x64 tar gz Create a symbolic link to the latest directory of the JDK as follows In s opt jdk1 7 0 51 opt jdk Download and install the latest Apache Tomcat in the opt directory of Linux For example cd opt tar xzf home user Downloads apache tomcat 7 0 50 tar gz Change the ownership of the extracted directory to the user you are running as For example chown R lt user account gt opt apache tomcat 7 0 50 Where lt user account gt the Linux user account you want to run the AEDCS process as Create a symbolic link to the latest Tomcat as follows In s opt apache tomcat 7 0 50 opt tomcat Download the latest stable release of AEDCS from the NEES website to a temporary location e g tmp Extract the contents of the latest stable release of AEDCS For example cd tmp unzip aedcs 2013 12
7. Automated Experiment Data Centralization System User Manual em Version 1 0 NEES and the University of Minnesota 1 of 25 Table of Contents Document Versioning Introduction Audience Deployment Options Single Machine Configuration Client Server Configuration Client Installation Windows Client Installation System Requirements Linux Client Installation System Requirements Central Server Installation Prerequisites Linux Central Server Installation Installation Steps Windows Central Server Installation Installation Steps System Operation Overview Configuring an AEDCS Client Configure the AEDCS Central Server Configuring Pen starting the AEDCS Server starting an AEDCS Client Configuring the AEDCS Variables Configuring the AEDCS Data Mapping Throttling TCP IP Traffic Performing File Verification Data Upload Preparation Transfer Data from a Client to Central Server stopping Data Transfer a Client to Central Server Data Upload 2 of 25 3 of 25 Document Versioning 11 27 2013 Michael Boldischar Initial Version 9 11 2014 Michael Boldischar Updated for software 1 changes A of 25 Introduction Each NEES site has one or more data acquisition systems Many of these systems are designed to capture sensor data and store to a local hard drive The purpose of the Automated Experiment Data Centralization System AEDCS is to automatically aggregate data from NEES data acquisition systems This tool
8. Chrome and navigate to the Central Server website See Starting the AEDCS Server section for more information on this URL Click on the Configure Clients button to open the page listing all the connected clients It should look similar to this 21 of 25 Configure Clients en Ten Home Configure Clients imelapse Edit Command Log Throttle Configure Variables Help 3 Choose the client from the list that you wish to configure For that client click the Edit button to bring up the data mapping pop up It should look similar to this Tip Using variable sets effectively in data mappings can save a lot of work when switching projects and experiments Develop a common process for saving data e g common directory structure Then use variables in data mapping source and destination paths Throttling TCP IP Traffic Performing File Verification To stop a client from sending data click the Command button on the Configure Clients page The following dialog will appear Send Command to Client x Name imageprocessor start Stop Verify Cancel send Click the Verify button and press Send to send a command to perform file verification Data Upload Preparation To start uploading data to nees org follow these steps Itis expected that experiment data has 22 of 25 1 Navigate to the Configure Upload screen shown below Configure Upload Configure Clients en Edit
9. a good check to make sure the data was uploaded properly and the client data matches the server data 9 Upload to NEEShub This feature is currently satisfied by running the Pen tool separately Pen can be used to upload data to NEEShub from the Central Server Automated Experiment Data Centralization System 1 Map 4 Read T Write Date Acquisition Data from Deltas to Data to PEN Filesystem Filesystem MEEShub 9 Upload te 2 Run on rei B Verify us as to Acquisition Cent Data Sysiem uui Inlegrity Server 3 Watch 2 6 Reoenve Filesystem for Chen Data Deltas Figure 4 AEDCS Functional Analysis Configuring an AEDCS Client 16 of 25 Configure the AEDCS Central Server The steps in this section are optional These steps are for advanced users only 1 Open the Tomcat server xml using a text editor This file is found in the Tomcat conf directory 2 Locate the section of the server xml file with the Connector XML element It should look similar to the following wi C Abin apache tomcat 7 0 47 conf server xml Notepad bos File Edit Search View Encoding Language Settings Macro Run Plugins Window A cH RB sls el 4 DD a C h y RAS 1 AAA m bea a server mi lt A Connector represents an endpoint by which requests are received and responses are returned Documentation at Java HITE Connector docs config http html blocking amp non blocking Java AJP Co
10. bles button 8 of 25 You must be logged on as an Administrator to make most of these changes Performance Visual effects processor scheduling memory usage and virtual memory Settings User Profiles Desktop settings related to your logon Startup and Recovery System startup system failure and debugging information 6 Inthe Environment Variables window click on the New button under the System variables section to add a variable for JAVA_HOME System Properties ee Variable name JAVA HOME Variable value C Program Files Java jdk1 7 0_ 45 System variables Variable Value ComSpec C Windows system32 ond exe FP_NO HOST C NO WIROOTDIR 32 C Program Files IVI Foundationi vI NMIDAQmsxSwitchDir C Program Files National Instruments 7 Set the Variable name to JAVA HOME 9 of 25 8 Set the Variable value field to the path to the Oracle JDK directory For example C Program Files Java jdk1 7 0 45 9 Hit OK on the New System Variable window to save the new variable 10 Hit OK on the Environment Variables window to save the settings 11 Hit OK on the System Properties window to save the settings 12 Navigate to the directory where the AEDCS ZIP file was extracted 13 Open the file aedcs properties with a text editor e g Notepad Each property in this file is on a separate line H
11. cle Java Development Kit JDK 7u45 Steps to complete this installation are outside the scope of this document The following steps are required to properly configure an AEDCS Client 7 of 25 1 Log into the AEDCS client you want to configure 2 Left click on the Windows Start Menu to bring it up 3 Inthe Start Menu right click on Computer and choose Properties from the drop down menu 4 Inthe Properties menu choose Advanced System Settings J Jr Control Panel System and Security System 4 Search Control Panel Control Panel Home View basic information about your computer Ey Device Manager Windows edition amp Remote settings Windows 7 Enterprise amp System protection Copyright 2009 Microsoft Corporation All rights reserved By Advanced system settings Service Pack1 System Rating em Windows Experience Index Processor Intel R Core TM i7 2760QM CPU 2 40GHz 2 38 GHz Installed memory RAM 2 00 GB System type 32 bit Operating System Pen and Touch No Pen or Touch Input is available for this Display Computer name domain and workgroup settings See also Computer name menomonie Change settings Action Center i Full computer name menomonie Windows Update Computer description Performance Information and Took Workgroup WORKGROUP 5 In the Advanced tab of the System Properties window click the Environment Varia
12. der File folder File folder Executable Jar File PROPERTIES File WAR File PROPERTIES File Windows Batch File SH File 105 KB 1KB 8 361 KB 1 KB 1 KB 1 KB 4 Navigate to the Tomcat installation directory It should look similar to the following c S CON di t Computer Local Disk O bin apache tomcat 7 047 FAET Organize Include in library Share with New folder MM she Favorites Name Date modified Type Size BE Desktop uli bin 11 27 2013 10 08 File folder a Downloads I cont 11 27 2013 10 08 File folder Recent Places a lib 11 27 2013 10 08 File folder di logs 10 18 2013 210 PM File folder Libraries I temp 11 27 201310 08 File folder Documents di webapps 11 27 2013 10 08 File folder a Music work 10 18 2013 210 PM File folder Pictures __ LICENSE 11 27 2013 10 08 File 57 KB E Videos _ NOTICE 11 27 2013 10 08 File 2 KB _ RELEASE NOTES 11 27 2013 10 08 File 9 KE ME Computer H RUNNING tet 11 27 2013 10 08 Text Document 17 KB J 11 items 5 Navigate to the webapps subdirectory 6 Right click in the webapps directory and choose Paste from the drop down menu After this operation the aedcs war file should be copied to the current directory ond OU Local Disk C bin apache tomcat 7 0 47 webapps Organize Include in library Share with New folder u sir Favorites Name Date modi
13. ee Figure 2 for an example of this configuration USB Drive Internal Hard Drive Desktop Computer AEDCS AEDCS Secure Client Server Digital Card Project Warehouse Other Removable Media Figure 2 AEDCS running on a single host Client Server Configuration The client server configuration is the standard deployment configuration for a NEES site Data may be collected by multiple systems and stored on separate host hard drives All the data from these separate systems is aggregated by the AEDCS Central Server From the Central Server the data is uploaded to NEEShub See Figure 3 for an example of this configuration 6 of 25 on Krypton T 3 on om AEDCS Central Project 30 System 7 Server Warehouse Still Image at Acquisition 39 System u Other Data gt ie Om Acquisition 5 9 System a Figure 3 AEDCS running with multiple clients and a single server Client Installation Windows Client Installation System Requirements e Windows 7 32 bit or 64 bit e 50MB of free space for AEDCS client software e TCP IP network connection to Central Server Prerequisites e Installation of Oracle Java Development Kit JDK 7u45 Steps to complete this installation are outside the scope of this document Linux Client Installation System Requirements e 50MB of free space for AEDCS client software e Linux Kernel 3 0 e TCP IP network connection to Central Server Prerequisites e Installation of Ora
14. ere is an example Wi CAbin aedcs aedcs properties Notepad mE File Edit Search View Encoding Language Settings Macro Run Plugins Window x ssa sees bHlacd as 2s BSl21 8e unne gt aedcs properties Ll 3 2 2 2 2 2 SSS SSS SSS SS SSS SSS SS SS SSS SSS SSS SSF SSF SSF SS SES SESS SST 2 2 2 222 2 22 22 E EE EEEE This is the configuration file for AFDCS clients Es 2 2 2 2 2 2 2 2 22 2 202 2 22 20202 2022 2022 222 2022 22 222 2 2 2 2 2 RRE EE ER EEEE E 2 2 2 22 2 2 2 2 2 EE E HE This is the unique name for the data acquisition system client daq name timelapse This is the IP address of the Tomcat server If this is a single host EN configuration use localhost or 127 0 0 1 for this value server ip 192 168 0 117 This is the port of Tomcat The default port is 8080 server port 30350 This is the sleep time for common worker threads in the client thread sleeptime 1000 This needs to be set to the maximum POST size for Tomcat Performance can be increased by increasing the max post size max byte size 2097152 length 764 lines 21 In 10 Col 11 3el 13 14 Change the value for the property daq name to a unique name for this data acquisition system For example a system that gathers data from a timelapse camera might be named timelapse This name will appear on the Central Server website 15 Change the value for the property server ip to the IP address of the Central Server For exam
15. fied BE Desktop _ di docs 11 27 2013 10 08 File folder ig Downloads J examples 11 27 2013 10 08 File folder E Recent Places L host manager 11 27 2013 10 08 File folder di manager 11 27 2013 10 08 File folder a Libraries ROOT 11 27 2013 10 08 File folder Documents _ aedes war 11 27 2013 11 15 WAR File 8 361 KB al Music amp items System Operation Overview AEDCS is designed to perform the following functions during an experiment See Figure 4 1 Map acquisition data to PEN AEDCS has a data mapping configuration for each client This describes where the data comes from on the client and where it goes on the server 15 of 25 2 Run on acquisition system AEDCS can be run on any number of data acquisition systems 3 Watch filesystem for changes AEDCS watches specified files on data acquisition systems for changes 4 Read data from filesystem Once file changes are detected the data is read from the client filesystem 5 Send data deltas to Central Server The data changes deltas are streamed from the client to the Central Server over TCP IP 6 Receive data Deltas The Central Server receives the deltas via Tomcat HTTP requests 7 Write data deltas to filesystem The Central Server writes the changes to the filesystem 8 Verify data integrity Once data is done being collected the integrity can be verified by sending the client a VERIFY command This is
16. nnector docs config ajp html APR HITP AJP Connector docs apr html Define a non S5L HITP 1 1 Connector on port 8080 eh lt Connector port 8080 protocol HTTP 1 1 eonnectionTimeout 20000 redirectPort 8443 gt lt A Connector using the shared thread pool gt Ann lt Connector executor tomcatThreadPool port 8080 protocol HITP 1 1 connectionTimeout 270000 redirectPort 8443 gt gt lt Define a 55L HTTP 1 1 Connector on port 8443 This connector uses the JSSE configuration when using APR the IM length 6577 lines 143 In Col 1 Sel 127 Dos Windows 3 Set your tomcat connector to the following lt Connector port 8080 protocol HTTP 1 1 connectionUploadTimeout 36000000 disableUploadTimeout false connectionTimeout 60000 redirectPort 8443 maxPostSize 0 gt Configuring Pen The current version of AEDCS leverages the NEES Pen tool to upload data to NEEShub Pen must be run on the same host as the AEDCS Central Server Note AEDCS integration with the NEES Pen tool is in progress This will allow automatic upload of data to NEEShub as data is gathered from clients without needing the Pen upload tool 17 of 25 Starting the AEDCS Server To start the AEDCS Central Server follow these steps 1 Start Tomcat on the AEDCS Central Server by executing standard Tomcat startup scripts located in the Tomcat bin directory For Windows this is startup bat For Linux
17. oes not include steps for installing Tomcat or Oracle There are many resources on the internet which describe how to configure these tools 13 of 25 Installation Steps 1 Download the latest stable release of AEDCS from the NEES website to the directory on your computer where you want to install it e g C bin 2 Extract the contents of the AEDCS ZIP file into this directory The resulting directory should look similar to the following G m t Computer Local Disk bin aedcs Share with New folder 5 Organize Include in library UY Favorites ME Desktop m Downloads r os l Recent Places Libraries Documents al Music E Pictures E Videos jE Computer ci Network bh 14 items 3 Right click on the file called aedcs war and choose Copy from the drop down menu Name Date modified JE aedcs wht doc di javadoc i lib di report di script D test dt zip aedes jar _ aedcs properties _ aedes war _ log4 properties Ea start client bat _ start client sh 14 of 25 11 27 2913 1115 11 27 2613 11 15 11 27 2013 11 15 1172729131115 11 27 2013 11 15 11 27 2013 11 03 11 27 2013 11 15 11 27 2013 11 03 11 27 2913 1115 11 27 2013 11 15 11 27 2913 11 15 11 27 2913 11 15 11 27 2013 11 15 27 2013 11 15 File folder File folder File folder File folder File folder File fol
18. ple 192 168 0 117 without the quotes 16 Optional Update the max byte size value Increasing this value will improve transfer speeds The default value for this is 2097152 bytes Here is an example where the 10 of 25 size has been increased to 200MB per request max byte size 209715200 Warning This value must also be changed in your Tomcat configuration before making this modification Otherwise errors will occur since Tomcat and the clients will not agree on the maximum POST size for data This value must match the maxPostSize configured in the Configure the AEDCS Central Server section 17 Optional Change the server port value This value must correspond with the port Tomcat is running on The default port for Tomcat is 8080 Warning Changing this value requires a change in the Tomcat configuration otherwise the client will not be able to connect to the Central Server 18 Save the aedcs properties file in your text editor 19 Close your text editor Each AEDCS client needs a unique name e g krypton or dag Central Server Installation Prerequisites Here are the prerequisites for installing the AEDCS Central Server software e Oracle Java Development Kit JDK 7u45 e Tomcat 7 0 47 Examples in this document assume Tomcat is installed in the following path C bin apache tomcat 7 0 47 NEES Pen Google Chrome Instructions for installing these software components is
19. the startup script is called startup sh Open Google Chrome on a networked computer type the following in the address bar and press the Enter key http lt central server ip gt lt port gt aedcs Where lt central server ip gt the IP address of the host running the AEDCS Central Server lt port gt the port Tomcat is running on This will open the Central Server configuration page If you Starting an AEDCS Client The following steps can be used to start an AEDCS client 1 Navigate to the directory where the AEDCS ZIP was extracted Run the start client script corresponding to your current operating system For Linux run start client sh For Windows run start client bat This will start the client in a shell Configuring the AEDCS Variables Variables are useful in AEDCS for managing paths with elements that change for each experiment but retain the same structure Configured variables are global for one Central server Variable expansion is performed on each data mapping where the variables are found 1 Open Google Chrome and navigate to the Central Server website See Starting the AEDCS Server section for more information on this URL 18 of 25 2 Click on the Configure Variables button to open the page 3 Click the Add Variable Set button to bring up the variable set window Add New Variable Set o Enable Variable Set Name Add Variable Cancel Save
20. uses a hub and spoke approach to gather data from various sources See Figure 1 As the data acquisition systems collect data and update the filesystem the AEDCS detects changes and sends file deltas changes to a central server Also the AEDCS uses acquisition system specific rules to map the data to the appropriate location in the Project Warehouse directory structure NEES Site NEEShub y nA Experiment amp 28 Project ri storage ms a Warehouse in if Figure 1 AEDCS Architecture Audience This manual is intended for the following users e NEESIT Staff e NEES Researchers NEES IT Staff is defined as a person who works at a NEES site where NEES experiments are executed This person is responsible for IT operations and data upload at the NEES site A NEES researcher is an individual responsible for uploading experiment data to NEESHub AEDCS software developers are individuals who are interested in compiling and creating custom configurations for AEDCS 5 of 25 Deployment Options Single Machine Configuration In this configuration a single desktop computer is responsible for gathering data and uploading This configuration can be used for a small research project where data is gathered and stored on removable media Also this configuration may be used if data is gathered offline and uploaded after an experiment is completed In this configuration both the AEDCS client and AEDCS server are installed on the same host S
Download Pdf Manuals
Related Search