pdf version here
Contents
1. Megatest User Manual Matthew Welland Document Number TSETAGEM 01 1 Revision Number v1 36 Jan 29 2012 Matt Welland Megatest User Manual Megatest User Manual 2011 Matthew Welland All rights reserved Megatest is free software released under the General Public License v2 0 Please see the file COPY ING in the source distribution for details Email mattOkiatoa com Web www kiatoa com fossils megatest This document is believed to be acurate at the time of writing but as with any opensource project the source code itself is the reference It is the responsibility of the end user to validate that the code will perform as they expect The author assumes no responsibility for any inaccuracies that this document may contain In no event will Matthew Welland be liable for direct indirect spe cial exemplary incidental or consequential damages resulting from any defect or omission in this document even if advised of the possibility of such damages This document is a snapshot in time and Megatest software has likely changed since publication This document and Megatest may be improved at any time without notice or obligation TSETAGEM 011 Jan 29 2012 ii Revision v1 36 Free Software License GPL v2 0 Megatest Megatest User Manual User Manual Megatest document Revision History Notable revisions of the software are occasionally documented here Version Author Descr2. A flow specifies the tests to run the order and dependencies and is managed by a running megatest process 12 Flow Specification and Running Not released yet 12 1 Write your flow file flows lt flowname gt config Flow lt flowname gt flowconfig turn on item level dependencies itemdeps on flowsteps 4 lt testname gt lt predecessor gt Run the test copydata copydata Run the test setup after copydata completes with PASS WARN or WAIVE setup copydata once the test setup completes successfully run siml sim2 and sim3 siml setup sim2 setup sim3 setup 12 2 Run the flow megatest runflow lt flowname gt FIELD1 vall FIELD2 val2 runname wk32 4 13 Monitor based running 13 1 Monitor logic Note The monitor is usable but incomplete as of Megatest v1 31 Click on the Monitor button on the dashboard to start the monitor and give it a try TSETAGEM 011 Jan 29 2012 Page 9 of 16 Revision v1 36 Free Software License GPL v2 0 Megatest User Manual Megatest User Manual Reference NN Ha done 14 Reference 14 1 Configuration file Syntax Note whitespace is preserved including at the end of line Ensure your entries only have whitespace at the end of line when needed to avoid problems 14 1 1 Sections section name This creates a section named section name 14 1 2 Variables VARX has this value TSETAGEM 011 Jan 29 2012 Page 10 of
3. 1 testconfig file 2 0 6 ke AR sar e RNA oe es 14 145 2 Command lines syaa aniier ele Ala a aa 15 A Data 16 B References 16 TSETAGEM 011 Jan 29 2012 v Megatest User Manual Megatest User Manual Introduction 1 Introduction 1 1 Megatest design philosophy Megatest is intended to provide the minimum needed resources to make writing a suite of tests and implementing continuous build for software design engineering or process control via owlfs for example without being specialized for any specific problem space Megatest in of itself does not know what constitutes a PASS or FAIL of a test In most cases megatest is best used in conjunction with logpro or a similar tool to parse analyze and decide on the test outcome 1 2 Megatest architecture All data to specify the tests and configure the system is stored in plain text files All system state is stored in an sqlite3 database Tests are launched using the launching system available for the distributed compute platform in use A template script is provided which can launch jobs on local and remote Linux hosts Currently megatest uses the network filesystem to call home to your master sqlite3 database 2 Installation 2 1 Dependencies Chicken scheme and a number of eggs are required for building megatest See the file utils instal lall sh for an automated way to install the dependencies on Linux 2 2 Build and Install Run make test to create the megatest ex
4. 16 Revision v1 36 Free Software License GPL v2 0 Megatest User Manual Megatest User Manual Environment variables The variable VARX will have the value has this value 14 1 3 Includes include filename The file named filename will be included as if part of the calling file NOTE This means no section can be named include with the whitespace 14 1 4 Setting a variable by running a command VARNAME system ls tmp The variable VARNAME will get a value created by the Unix command ls tmp All lines of output from the command will be joined with a space 14 1 5 Notes e Some variables are infered as lists Each token on the line separated by whitespace will be member of the list e Comments lines starting with and blank lines are ignored 14 2 Environment variables Variable Purpose MT_CMDINFO Conveys test variables to the megatest test runner MT_TEST_RUN_DIR Directory assigned by megatest for the test to run MT_TEST_NAME Name of the test corrosponds to the directory name under tests MT_ITEM_INFO Iterated tests will set this to a sequence of key values KEY val MT_RUN_AREA HOME Directory where megatest was launched from and where the tests code can be found MT_RUNNAME Name of this run as set by the runname parameter MT_MEGATEST Path Filename to megatest executable Found either from called path or but using
5. not the underscore test running machine Typical usage is to control the host or run queue for launching tests These values will not be seen by the test when it runs disks string of letters a valid path writable by the yes The disk usage balancing numbers and test launching process and algorithm is to choose the underscore by the test process disk with the least space for each test run TSETAGEM 011 Jan 29 2012 Page 12 of 16 Revision v1 36 Free Software License GPL v2 0 Megatest User Manual Megatest User Manual runconfigs config file 14 4 runconfigs config file section variable value required omment default string of letters any no variables set in this section will be numbers and available for all runs defining the same underscore variable in another section will override the value from the default section field 1 value field2value string of letters any no the values in this section will be set for numbers and any run where field1 is field 1 value field2 underscore is field2 value and fieldN is fieldN value Example a test suite that checks that a piece of software works correctly for different customer configurations and locations each of which is done as a separate release regression run The fields CUSTOMER and LOCATION were chosen The following runconfigs config file would set some variables specific to runs for megacorp in India and femtocorp in the Co
6. status Y You can now run megatest and the created test directory will contain the new files run_simulation html and run_simulation log If you are using the dashboard you can click on the run and then push the View log button to view the log file in firefox TSETAGEM 011 Jan 29 2012 Page 3 of 16 Revision v1 36 Free Software License GPL v2 0 Megatest User Manual Megatest User Manual Simple Test with Multiple Steps 5 4 Simple Test with Multiple Steps To run multiple steps simply add them to the main csh file Here we add a step to test cpu2 The second step that tests cpu2 will only run after the step that tested cpul completes bin tcsh x run the cpul simulation The step name is run_simulation The commandline being run for this step is runsim cpul The logpro file to validate the output from the run is runsim logpro SMT_MEGATEST runstep run_simulation_cpul logpro runsim logpro runsim_cpul amp amp SMT_MEGATEST runstep run_simulation_cpu2 logpro runsim logpro runsim cpu2 SMT_MEGATEST test status state COMPLETED status 6 Simple Test with Multiple Steps Some in Parallel 6 1 The Makefile A good way to run steps in parallel within a single test especially when there are following steps is to use the Unix Make utility Writing Makefiles is beyond the scope of this d
7. the exectuable keyword in the setup section lt field1 gt The field values as set on the megatest runall command line e g field1 abc TSETAGEM 011 Jan 29 2012 Page 11 of 16 Revision v1 36 Free Software License GPL v2 0 Megatest Megatest User Manual User Manual megatest config 14 3 megatest config section variable value required comment setup max_concurrent_jobs if variable is not defined no no limit on jobs executable full path to megatest binary no Use only if necessary megatest will extract the location from where it used to launch and add append that to the PATH for test runs runsdir full path to where the link no Because your runs may be tree to all runs will be spread out over several disk created partitions a central link tree is created to make finding all the runs easy fields string of letters string of letters numbers at least numbers and and underscore one underscore jobtools launcher command line used to no launch jobs the job command megatest execute will be appended to this workhosts list of hostnames to run n a jobs on NOT SUPPORTED RIGHT NOW jobgroups string of letters number no Control number of jobs numbers and allowed to concurrently run underscore in categories See jobgroup in testconfig env string of letters any string no These are set on the test override numbers and launching machine
8. 000 0 5 7 2 Rewrite your main csh for iteration oss ooo ee ee 5 7 3 Tests with Inter test dependencies o e sosse 5 7 4 Rolling up Miscellaneous Data o o e e 5 EY Role UP Runs o A A A da 6 TSETAGEM 011 Jan 29 2012 iv Revision v1 36 Free Software License GPL v2 0 Megatest User Manual Revision v1 36 Free Software License GPL v2 0 Megatest User Manual Contents 8 Dashboard 6 9 Generating an OpenDocument Spreadsheet from the Database 8 10 Introspection 9 10 1 Getting previous test paths o o e ess ss 9 11 Flows 9 12 Flow Specification and Running Not released yet 9 12 1 Write your flow fe srp sanra oae ps Ya wba ae a A ba ea Pa bares 9 12 2 R n ThE TOW scan 0 r a gl Stn ot ch oh wate Ge SR ade aetna oes the oes Eh heh Mad ba 9 13 Monitor based running 9 13 1 Monitor logic tos otero ae ph oes oe ech Se SM VER ae ee ts ee Me et ge dc 9 14 Reference 10 14 1 Configuration file Syntax e 10 14D SECHONS a e e ae re je At fer eR Es AR dai 10 14 127 Variables 05 A E Ee A at 10 14 13 Includes lot a ts 11 14 1 4 Setting a variable by running a command 11 ALI NOLES 6 tl AT e 11 14 2 Environment variables o e 11 14 3 megatest config eeeeeerer ee 12 14 4 runconfigs config file so ooo 20 2 20 2 rr rr rs es ss 13 145 WANE tests ast oa ran A A EE fate AS hig A ae Rc 14 14 5
9. New entries are keyed on the category and variable If a new record is inserted with a category and variable that have already been used the new record will replace the old record Where value expected and tol are specified the behavior is as follows e If value expected and tol are numbers then status is calculated as PASS if expected tol lt value lt expected tol e If value and expected are numbers and tol is gt lt gt or lt then value is compared with expected using the operator given by tol e If status is specified its value overrides the above calculations 7 5 Rolling up Runs To roll up a number of tests in a sequence of runs to a single run use the rollup command megatest rollup sysname ubuntu fsname nfs datapath none runname rollup_ww38 All keys must be specified and the runname is the name of the run that will be created All paths are kept original inside the database When remove runs is used to delete runs the data is not deleted 1f there are rollups that refer to the data 8 Dashboard gt dashboard TSETAGEM 011 Jan 29 2012 Page 6 of 16 Revision v1 36 Free Software License GPL v2 0 Megatest Megatest User Manual User Manual Dashboard Ma Megatest dashboard sysname ubuntu ubuntu ubuntu ubuntu fsname nfs nfs nfs nfs datapath none n
10. duration 9s Logfile wasting_time html Actions j O o vewog sem e close Set fields Comment STATE MI ec STARTED RUNNING REMOTEHOSTST _kuLeD lee da O IO Ju check O WAED Test Steps Test Data Stepname Start End Status Time Category Variable Value Expected Tol Status Units Commel wasting time 23 05 20 23 05 28 0 8 0s faz bar 10 0 8mA 0 this foo abl 3 2 1 3 8 1 pass 0 foo alb 1 2 1 2 pass Amps This foo bal 1 2 1 2 lt fail Check foo bar 1 2 1 9 gt fail 0 foo bla 1 2 1 9 pass 0 foo bra 1 2 pass silly stuff 0 foo rab 1000000000 010000000000 01000000000 0fail 0 9 Generating an OpenDocument Spreadsheet from the Database And OpenDocument multi paned spreadsheet can be generated from the megatest db file by running extract ods megatest extract ods results ods runname You can optionally specify the keys for your database to limit further the runs to extract into the spreadsheet The first sheet contains all the run data and subsequent sheets contain data rolled up for the individual tests TSETAGEM 011 Jan 29 2012 Page 8 of 16 Revision v1 36 Free Software License GPL v2 0 Megatest User Manual Megatest User Manual Introspection 10 Introspection 10 1 Getting previous test paths o o o megatest test paths target runname testpatt itempatt status PASS 11 Flows
11. ecutable You may wish to copy the executable to a centrally accessible location 3 Setup 3 1 Create megatest config Create the file megatest config using the megatest config template from the tests directory At a minimum you need the following Fields are the keys under which your test runs are organized fields fieldl TEXT field2 TEXT TSETAGEM 011 Jan 29 2012 Page 1 of 16 Revision v1 36 Free Software License GPL v2 0 Megatest User Manual Megatest User Manual Create runconfigs config jobtools The launcher launches jobs to the local or remote hosts the job is managed on the target host by megatest comment out launcher to run local only An example launcher nbfake can be found in the utils directory launcher nbfake The disks section specifies where the tests will be run As you run out of space in a partition you can add additional disks entries Format is name path to area disks disk1 tmp 3 2 Create runconfigs config This file is used to set environment variables that are run specific You can simply create an empty file to start runconfigs config 3 3 Create the tests directory and your first test mt megatest config runconfigs config tests mytest main sh testconfig 3 4 Create the testconfig file for your test setup runscript main sh 3 5 Create your test running script main sh bi
12. iption Date v1 25 matt converted to new document template Jan 29 2012 TSETAGEM 011 Jan 29 2012 iii Revision v1 36 Free Software License GPL v2 0 Megatest User Manual Megatest User Manual Contents Contents 1 Introduction 1 1 1 Megatest design philosophy o oo e 1 1 2 Megatest architecture soo rr rr rr rr rr rr rr es ss 1 2 Installation 1 2011 Dependencies as o E AAA AA A A ea ee od 1 2 2 Build and Install o ee 1 3 Setup 1 3 1 Create megatest config 0 0 0000 rss ss eee 1 3 2 Cr ate rinconfigs config s ar seck A A A eee a a ee 2 3 3 Create the tests directory and your first test o o 2 3 4 Create the testconfig file for your test soo oss ss rr ss rss sees ss 2 3 5 Create your test running script mainsh o o 2 3 6 Run megatest and watch your TUN progress som sees ss ess ss e 2 4 Choose Flow or Unstructured Run 3 5 How to Write Tests 3 5 1 A Simple Test with one Step esse 0 000000 000000 3 5 2 Create your testconfig file so ooo 2 2 0 0 000200 000000 3 5 3 Create the main esh script 2 o e e 3 5 4 Simple Test with Multiple Steps o o e 4 6 Simple Test with Multiple Steps Some in Parallel 4 6 1 The Makefile cid a is E a A ee 4 6 2 Themain cshfile o 4 7 Simple Test with Iteration 4 7 1 Update your testconfig file for iteration
13. n bash SMT_MEGATEST runstep mystepl sleep 20 echo Done m mystepl is done SMT_MEGATEST test status state COMPLETED status PASS m This is a comment 3 6 Run megatest and watch your run progress TSETAGEM 011 Jan 29 2012 Page 2 of 16 Revision v1 36 Free Software License GPL v2 0 Megatest User Manual Megatest User Manual Choose Flow or Unstructured Run megatest fieldl abc field2 def runname 2011week08 4a runall o watch megatest list runs OR use the dashboard dashboard amp 4 Choose Flow or Unstructured Run A flow is a structured and specifically sequenced set of tests See the Flows chapter to understand the difference 5 How to Write Tests 5 1 A Simple Test with one Step mkdir simpletest cd simpletest 5 2 Create your testconfig file testconfig setup runscript main csh 5 3 Create the main csh script Note Using csh is NOT recommended Use bash perl ruby zsh or anything other than csh We use csh here because it is popular in the EDA industry for which Megatest was originally created bin tcsh x run the cpul simulation The step name is run simulation The commandline being run for this step is runsim cpul The logpro file to validate the output from the run is runsim logpro SMT_MEGATE SMT_MEGATE Y runstep run simulation logpro runsim logpro runsim cpul test status state COMPLETED
14. ocument but here is a minimal example that will run runsim cpul and runsim cpu2 in parallel For more information on make try info make at the Linux command prompt Example Makefile to run two steps in parallel RTLDIR path to rtl CPUS cpul cpu2 run simulation CPUS html RTLDIR S CPUS S MT_MEGATEST runstep run simulation CPUS logpro runsim logpro runsim _ 6 2 The main csh file bin tcsh x run the cpul and cpu2 simulations in parallel The j parameter tells make how many jobs it may run in parallel make j 2 SMT_MEGATEST test status state COMPLETED status 7 Simple Test with Iteration Since no jobs run after the cpul and cpu2 simulations in this test it is possible to use iterated mode TSETAGEM 011 Jan 29 2012 Page 4 of 16 Revision v1 36 Free Software License GPL v2 0 Megatest User Manual Megatest User Manual Update your testconfig file for iteration 7 1 Update your testconfig file for iteration setup runscript main csh items CPU cpul cpu2 7 2 Rewrite your main csh for iteration bin tcsh x run the cpu simulation but now use the environment variable SCPU to select what cpu to run the simulation against SMT_MEGATEST runstep run simulation logpro runsim logpro runsim_ CPU As of version 1 07 Megatest automatically converts a status of 0 to PASS any other number to FAIL and directl
15. ok Islands and New Zealand runconfigs config default ENCRYTION true megacorp india TESTPATH nfs testing megacorp_runs femtocorp cook_islands ENCRYTION false TESTPATH afs kiatoa testing cook_islands femtocorp new_zealand TESTPATH afs kiatoa testing new_zealand megacorp new_zealand TESTPATH nfs testing megacorp_runs Running megatest like this megatest CUSTOMER megacorp LOCATION new_zealand run name week12_2011_runl runall Would set ENCRYPTION true TESTPATH nfs testing megacorp_runs TSETAGEM 011 Jan 29 2012 Page 13 of 16 Revision v1 36 Free Software License GPL v2 0 Megatest Megatest User Manual User Manual Writing tests 14 5 Writing tests 14 5 1 testconfig file section variable value required comments setup runscript name of script to yes The script must be executable and either execute for this provide the full path or put a copy at the test top of your test directory requirements waiton list of valid test no This test will not run until the named tests names are state completed and status PASS jobgroup items any list of values no The test will be repeated once for each valid item with the variable name set to the value If there is more than one variable then the test will be run against all unique combinations of the values eztests any stepname no Use in addition to or instead of run
16. one none none runname 2011WW38 2 2011WW37 3 2011WW37 1 2011WW36 7 f lineitem_pass lineitem_fail _ singletest singletest2 Lo exit_1 sqlitespeed NOT_STARTE L salitespeed au NOT_STARTE m sqlitespeed at NOT_STARTE salitespeed as NOT_STARTE _ sqlitespeed ar NOT_STARTE sqlitespeed aq NOT_STARTE salitespeed ap NOT_STARTE _ sqlitespeed ao NOT_STARTE sqlitespeed an NOT_STARTE m sqlitespeed al NOT_STARTE salitespeed a Nor starre Pass pass ass _ i salitespeed a nor sar ASS PASS PASS Quit lt Left Up Down v Right gt i gt Pushing one of the buttons on the main dashboard will bring up the test specific dashboard Values are updated in semi real time as the test runs TSETAGEM 011 Revision v1 36 Jan 29 2012 Free Software License GPL v2 0 Page 7 of 16 Megatest User Manual Megatest User MaGealerating an OpenDocument Spreadsheet from the Database Megatest Run Info Test Info Test Meta Data sysname ubuntu Testname runfirst Author matt fsname nfs Item path b 2 Owner bob datapath none Current state COMPLETED Reviewed 1 1 1965 runname 2011WW38 2 Current status PASS Tags first single Test comment Description This test must Test id 112 be run before the other tests Remote host and Test Run Info Hostname zeus Uname a Linux 2 6 38 11 generic pae 48 Ubuntu SMP Fri Jul 29 20 51 21 UTC 2011 i686 i386 GNU Linux Disk free 70906928 CPULoad 0 0 Run
17. s itempatt and testpatt for filters testpatt any value is wildcard itempatt any value is wildcard showkeys Print the keys being used for this database force Test will not re run if in the PASS WARNING The force sw CHECK or KILLED using force will force the run to be launched xterm Launch an xterm instead of run the test The xterm will have the environment that the test would see remove runs Remove a run test or subtest from the database and the disk Cannot be undone Requires testpatt itempatt runname and all keys be specified Test helpers runstep Used inside a test to run a step record the start and end of the step and optionally analyze the output using logpro logpro If using logpro to acess the PASS FAIL status of the step you specify the logpro file with this parameter TSETAGEM 011 Jan 29 2012 Page 15 of 16 Megatest User Manual Megatest User Manual Data A Data B References TSETAGEM 011 Jan 29 2012 Page 16 of 16 Revision v1 36 Free Software License GPL v2 0
18. script valid command for easy implementation of steps If lt stepname gt logpro exists it will be applied to the lt stepname gt log and resulting exit code will be used to determine PASS FAIL WARN TSETAGEM 011 Jan 29 2012 Page 14 of 16 Revision v1 36 Free Software License GPL v2 0 Megatest Megatest User Manual User Manual Writing tests 14 5 2 Command line Revision v1 36 Free Software License GPL v2 0 switch or param parameter purpose comments h brief help runall run all tests runtests test1 test2 run one or more tests step stepname record a step requires state and status test status record the test status requires state and status setlog logfilename set the logfile name for a test path is assumed to be relati set toplog logfilename set the logfile name for the top test in an each sub test can have its o iterated test run m comment sets acomment for the step test or run runname a zA Z0 9_ directory in which this run will be stored in the test run area state any value Set the step or test state this is stored in For tests Megatest recogni the state field in the steps or tests table respectively status any value Set the step or test status this is stored in For tests Megatest recognis the status field in the steps or tests table respectively list runs any value is wildcard Respect
19. y uses the value of a string passed in SMT_MEGATEST test status state COMPLETED status 7 3 Tests with Inter test dependencies Sometimes a test depends on the output from a previous test or it may not make sense to run a test is another test does not complete with status PASS In either of these scenarios you can use the waiton keyword in your testconfig file to indicate that this test must wait on one or more tests to complete before being launched In this example there is no point in running the system test if the cpu and mem tests either do not complete or complete but with status FAIL testconfig for the system test setup runscript main csh waiton cpu mem 7 4 Rolling up Miscellaneous Data Use the load test data switch to roll up arbitrary data from a test into the test_data table Fields are category variable value expected tol units comment status SMT_MEGATEST load test data lt lt EOF foo bar 1 2 1 9 gt foo rab 1 0e9 10e9 1e9 foo bla 1 2 1 9 lt foo bal 1 2 1 2 lt Check for overload foo alb 1 2 1 2 lt Amps This is the high power circuit test toos abLl Lil O TSETAGEM 011 Jan 29 2012 Page 5 of 16 Revision v1 36 Free Software License GPL v2 0 Megatest User Manual Megatest User Manual Rolling up Runs foo bra 1 2 pass silly stuff faz bar 10 8mA this is a comment EOF
Download Pdf Manuals
Related Search