Home

Fast Multi Layer Perceptron with Genetic Algorithm User Manual

1. 20362 P T383 2 09885 4 604384 F045 2 61725 9 L 0 0 a 78 0 501168 Fig 22 The files error left and weights right output of the xorTrain experiment The file weights has one column and 9 rows These values are the weights of the connections between the network layers e Connections between input and hidden nodes having 2 input nodes and 2 hidden nodes we have 4 connections plus 2 bias values for each of the two hidden nodes first 6 values in the file e Connections between hidden and output nodes having 2 hidden nodes and 1 output node we have 2 connections plus the bias value for the output node last 3 values in the file 39 DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved 4 1 2 Regression FMLPGA Test use case DAta Mining amp Exploration Program The file weights can be copied into the input file area File Manager of the workspace in order to be re used in future experiments for example in this case the test use case This is because it represents the stored brain of the network trained to calculate the XOR function v File Manager Vorkspace MLPGAExp Ei Dow amp Edit File Type Last Access D gt FMLPGA _Train_weights tt ascii 2013 09 03 1 FMLPGA Train_weights txt Ej gt xor_run csv MA SERIA IT ARE Ej gt xorcsy csy 2012 09 03 v My Experiments Vorkspace MLPGAExp Exp
2. for Pattern Recognition Oxford Neural Computation Data Mining Introductory and Advanced Topics Prentice Hall Mining the SDSS archive I Photometric Redshifts in the Nearby Universe Astrophysical Journal Vol 663 pp 752 764 The Fourth Paradigm Microsoft research Redmond Washington USA Artificial Intelligence A modern Approach Second ed Prentice Hall Pattern Classification A Wiley Interscience Publication New York Wiley Neural Networks A comprehensive Foundation Second Edition Prentice Hall A practical application of simulated annealing to clustering Pattern Recognition 25 4 401 412 Probabilistic connectionist approaches for the design of good communication codes Proc of the ISCNN Japan Approximations by superpositions of sigmoidal functions Mathematics of Control Signals and Systems 2 303 3 14 no 4 pp 303 314 Genetic Algorithm Modeling with GPU Parallel Computing Technology Neural Nets and Surroundings Proceedings of 22nd Italian Workshop on Neural Nets WIRN 2012 Smart Innovation Systems and Technologies Vol 19 Springer http adsabs harvard edu abs 2012arXiv1211 5481C Astrophysical data mining with GPU A case study genetic classification of globular clusters Nuclear Instruments and Methods in Physics Research A Vol 720 p 92 94 Elsevier http adsabs harvard edu abs 2013arXiv1304 0597C Program Author Ronald Fisher Bishop C M Bishop C M Svensen M
3. 0 497965 Worst 0 507175 10 Best 9 20 39 40 59 60 79 80 Best 0 Best 9 Best 0 Best 6 Best D Best 9 Best 0 99 Best 9 100 119 120 130 140 159 160 170 180 299 200 210 220 28 230 240 250 260 279 Best Best Best Best Best Best Best Best Best Best Best Best Best Best Best Best Best Best mound to to be bon th tb bt ob ob ob tb om bn oooo 0 n cd ao 0a ODO oO On oO oO oO 8 47769 467536 458977 448234 439231 439165 439133 439099 439099 439097 439097 439097 439097 439097 439934 439034 438979 438979 438979 438979 438979 4385979 2438774 438774 438774 438774 438774 Worst Worst Worst Worst Worst Worst Worst Worst Worst Worst Worst Worst Worst Worst Worst Worst Worst Worst Worst Worst Worst Worst Worst Worst Worst Worst Worst 0 483175 muh wtb bt bt bt bt be ot bm tb ob md oooo0 on go oO oO nD OO va oO oO oO 8 0 9 0 0 G 9 0 0 476203 469144 448798 439409 440529 452294 439231 4391 439097 2499993 439097 439097 439097 439097 439034 438979 439014 438979 438979 438979 5 438774 438774 444847 439367 438774 DAta Mining amp Exploration Bain Program out Ty Tal etto weights is feat o
4. In the case of classification functionality the following output files are obtained in all use cases TRAIN TEST RUN NOTES default prefix default prefix default prefix FFMLPGA_ Train FFMLPGA_ Test FFMLPGA_Ru n info experiment status log trainerrorsixt_ error trend table _trainerrors jpeg po erortrendplot _trainoutput csv _testoutput csv _weights txt trained network weights to be moved in the File Manager tab area through GUI button AddInWS to be loaded during a test run experiment evaluation of training performance io evaluation of test performance Tab 2 output file list in case of classification type experiments DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved 20 F WARE Li DAta Mining amp Exploration nes Program 3 5 TRAIN Use case In the use case named Train the software provides the possibility to train the FMLPGA The user will be able to use new or existing already trained MLP weight configurations adjust parameters set training parameters set training dataset manipulate the training dataset and execute the training experiments There are several parameters to be set to achieve training dealing with network topology and learning algorithm In the experiment configuration there is also the Help button redirecting to a web page dedicated to support the user with deep information about all parameters a
5. the sum of input and output nodes MUST be equal to the total number of the columns in this file e GPU or CPU It is a file generated by the model during training phase It contains the resulting network topology as stored at the end of a training session Usually this file should not be edited or modified by users just to preserve its content as generated by the model itself Accepted entries are 0 serial type execution on CPU 1 parallel type execution on GPU If left empty its default is 1 GPU e input nodes this parameter is a field required It is the number of neurons at the first input layer of the network It must exactly correspond to the number of input columns in the dataset input file Training File field except the target columns 24 DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved WARE DAta Mining amp Exploration ne a Program e hidden layers It is the number of hidden layers of the MLP network It can assume only two values 1 or 2 As suggestion this should be selected according the complexity of the problem usually 1 layer would be sufficient If left empty its default is 1 e 1st hidden layer nodes this parameter is a field required It is the number of neurons of the first hidden layer of the network As suggestion this should be selected in a range between a minimum of 2N 1 where N is the number of
6. Fig 6 Example of a SLP to calculate the logic AND operation iii S Fig 7 A MLP able to calculate the logic XOR operation ccccccccccccsssveeecccccceeee ne eesseecceeeeeaaaessseseeeeesaaaaaeeeeees S Fig 8 A typical Genetic Algorithm optimization research MECHANISM cccccccccccccccceeeeseseeseeeseeeeeeeeeeeeeeennaaas 10 Fig 9 An example of genetic cross over operator application iii 11 Fig 10 An example of genetic mutation operator application iii 12 Fig 11 A MLP network trained by a Genetic Algorithm iii 13 Fig 12 Steps of the algorithm related to genetic algorithm evolution iii 14 Fig 13 The sigmoid function and its first derivative iii 15 Fig 14 The computing time comparison between FMLPGA training execution on CPU and GPU platforms The experiment was based on 1000 input patterns each one composed by 10 parameters 17 Fig 15 The computing time speedup for FMLPGA on CPU and GPU platforms The function axis is referred to the different chromosome selection type used during training evolution of genetic population 18 Fig 16 The content of the xor csv file used as input for training test USC Cases 19 Fig 17 The content of the xor_run csv file used as input for Run use case iii 19 Fig 18 The starting point with a Workspace FMLPGAExp created and two data files uploade
7. amp Williams C K I Dunham M D Abrusco R et al Hey T et al Russell S Norvig P Duda R O Hart P E Stork D G Haykin S Donald E Brown D E Huntley C L Babu G P Murty M N Cybenko G Cavuoti et al Cavuoti et al Tab 4 Reference Documents DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved DAta Mining amp Exploration Date 1936 1995 1998 2002 2007 2009 2003 2001 1999 1991 1993 1989 2012 2013 43 WARE al IA ID AI A2 A3 A4 AS A6 A7 A8 A9 A10 AII AI2 A13 A14 A15 A16 A17 A18 A19 Ved A let Ve Title Code SuiteDesign VONEURAL PDD NA 0001 Re12 0 project_plan_ VONEURAL PLA NA 0001 Rel12 0 statement_of_work_VONEURAL SOW NA 0001 Rel1 0 MLP_user_manual VONEURAL MAN NA 0001 Rel1 0 pipeline_test_VONEURAL PRO NA 0001 Rel 1 0 scientific_example_VONEURAL PRO NA 0002 Rel 1 1 frontend_VONEURAL SDD NA 0004 Rel1 4 FW_VONEURAL SDD NA 0005 Rel2 0 REDB_VONEURAL SDD NA 0006 Rel1 5 driver VONEURAL SDD NA 0007 Rel0 6 dm model VONEURAL SDD NA 0008 Rel2 0 ConfusionMatrixLib_VONEURAL SPE NA 0001 Rel1 0 softmax_entropy_ VONEURAL SPE NA 0004 Rel1 0 VONeuralMLP2 0_VONEURAL SPE NA 0007 Rel1 0 dm_model VONEURAL SRS NA 0005 Rel0 4 FANN_MLP_VONEURAL TRE NA 0011 Rel1 0 DMPlugins DAME TRE NA 0016 Rel0 3 BetaRelease_ReferenceGuide DAME MAN NA 0009 Re
8. experiment In particular we will leave empty the not required fields labels without asterisk The meaning of the parameters for this use case are described in section 3 1 1 of this document As alternative you can click on the Help button to obtain detailed parameter description and their default values directly from the webapp We give xor csv as training dataset specifying e Number of input nodes 2 because 2 are the input columns in the file e Number of hidden nodes first level 2 as minimal number of hidden nodes no particularly complex network brain is required to solve the XOR problem Anyway we suggest to try with different numbers of such nodes by gradually incrementing them to see what happens in terms of training error and convergence speed e Number of output nodes 1 because the third column in the input file is the target correct output for input patterns 37 DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved DAME Application DAta Mining amp Exploration Program App Manuals v Model Manuals v Cloud Services v Science Cases v RESOURCE MANAGER Fie Editor EJ Workspace v File Manager Workspace n New Workspace mipgaExp C Dow Edit File F Rename Workspace j Upicad CR Experiment 3 Delete AS xorcsy PA myfirstWS 3 i gt 4 we Em aa H PN xor_run csv PA secondWs id Note mipgaExp i Experiment
9. good properties o Centered at zero Anti symmetric f net f net Faster learning Overall range and slope are not important O O O O finet 5 f net net Fig 13 The sigmoid function and its first derivative Scaling input and target values Standardize o Large scale difference error depends mostly on large scale feature o Shifted to Zero mean unit variance Need to be done once before training Need full data set Target value o Output is saturated In the training the output never reach saturated value e Full training never terminated o Range 1 1 1s suggested Number of hidden nodes Number of hidden units governs the expressive power of net and the complexity of decision boundar y Well separated gt fewer hidden nodes From complicated density highly interspersed gt many hidden nodes Heuristics rule of thumb o Use a minimum of 2N 1 neurons of the first hidden layer N is the number of input nodes o More training data yields better result DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved 2 3 4 DAta Mining amp Exploration Program o Number of weights lt number of training data o Number of weights number of training data 10 o Adjust number of weights in response to the training data Start with a large number of hidden nodes then decay prune weights Number of h
10. population of chromosomes for example by using normal or uniform statistical distributions The method proceeds by performing cyclic variation and combination of the initial population looking for the best population best problem solution At each evolution stage the output chromosomes are obtained by applying several genetic operators to the input population and by evaluating through a specific fitness function the goodness of the new generated population The fitness function has the basic role to give a method to discard worst chromosomes from the population achieving the evolution to the next generation of the best candidates only exactly like Nature works with its species following the Darwin s law Typical genetic operators are cross over and mutation Genetic Algorithm HARVEST SEED POPULATION Fig 8 A typical Genetic Algorithm optimization research mechanism In the design of a GA to solve an optimization problem three steps are considered as strategic to obtain better results in the better time the chromosome representation codified in some way the choice of the fitness function and the method to choose chromosome reproduction The latter in particular deals with the issue that no random choices can be applied to select chromosomes to be combined by the genetic operator employed in the algorithm Otherwise the convergence could result too slow The choice must be driven in some way Usual rules are the so called
11. the layer will use the same activation function type The options are 0 sigmoid 1 threshold 2 linear 3 hyperbolic tangent By default the hyperbolic tangent function is used 3 6 2 Classification with FMLPGA Test Parameter Specifications In the case of Classification_FMLPGA with Test use case the help page is at the address http dame dsf unina it FMLPGA_help html class_test e input dataset this parameter is a field required Dataset file as input It is a file containing input and target columns It must have the same number of input and target columns as for the training input file For example it could be the same dataset file used as the training input file e weights file this parameter is a field required It is a file generated by the model during training phase It contains the resulting network topology as stored at the end of a training session Usually this file should not be edited or modified by users just to preserve its content as generated by the model itself 29 DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved WARE DAta Mining amp Exploration ni a Program e GPU or CPU It is a file generated by the model during training phase It contains the resulting network topology as stored at the end of a training session Usually this file should not be edited or modified by users just to preserve its content
12. the population an integer between 10 and 60 Remember that each chromosome is a solution of the problem At each iteration generation this parameter indicates how many chromosomes should be considered in the population of the GA If left empty the default value is 20 e elitism rate The parameter user defined related to this elitism mechanism defines the number of copies of the winner chromosome to be transmitted unchanged in the population of the next generation If left empty its default is 2 e tournament participants This is the number of chromosomes in the population to be engaged in the so called Ranking Selection This is in practice used only in case of ranking selection function choice Among this number of participants the first two chromosomes with higher fitness value are chosen to generate childs 3 5 2 Classification with FMLPGA Train Parameter Specifications In the case of Classification_FMLPGA with Train use case the help page 1s at the address http dame dsf unina i FMLPGA_help html class_train e input dataset this parameter is a field required This is the dataset file to be used as input for the learning phase of the model It typically must include both input and target columns where each row is an entire pattern or sample of data The format hence its extension must be one of the types allowed by the application ASCH FITS CSV VOTABLE More specifically take in mind the following simple rule
13. the various use cases are and sometimes must be different in terms of internal file content representation Remind that in all DAME models it is possible to use one of the following data types e ASCII extension dat or txt simple text file containing rows patterns and columns features separated by spaces normally without header e CSV extension csv Comma Separated Values files where columns are separated by commas e FITS extension fits tabular fits files e VOTABLE extension votable formatted files containing special fields separated by keywords coming from XML language with more special keywords defined by VO data standards For training and test cases a correct dataset file must contain both input and target features columns with input type as the first group and target type as the final group SOLE di 0 0 0 D l 1 LLP Ji Fig 16 The content of the xor csv file used as input for training test use cases As shown in Fig 14 the xor csv file for training test uses cases has 4 patterns rows of 2 input features first two columns and one target feature third column The target feature is not an input information but the desired output to be used in the comparison calculation of the error with the model output during a training test experiment ba ar xor_run csv rl Fig 17 The content of the xor_run csv file used as input for Run use case In Fig 15 the xor_run csv f
14. DAta Mining amp Exploration ft YO Dipartimento di Scienze Fisiche isnruronazionareaiastrorisica S Q Vago gt 5 we i sCALTECH Umi V i a vapoli X eder i m CRI 9 evsitd di F oT TIA i OSSERVATORIO ASTRONOMICO di CAPODIMONTE SI i I ED ii It S 7 DI me eo ma Fast Multi Layer Perceptron with Genetic Algorithm User Manual DAME MAN NA 0012 Issue 1 3 Author M Brescia A Solla Doc FMLPGA UserManual DAME MAN NA 0012 Rel1 3 DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved DAta Mining amp Exploration WARE Mac Program INDEX I delie ui 4 2 FMLPGA Model Theoretical Overview ciccsscsnacecsvssuwcccndnscmessesdcuassitnaceassis cautions E ERE EARE AEE 5 Zed Malti Layer Torce pt Olies E E R 5 22 TGA E E See ee eia 10 Zi IVE PE AC TEC AN RU CS ea E E E E 14 2 3 1 Selection of neuron activation function 14 Za SCalimownput and target VA DES ccssssesssasasssncereczeacesaceassanaghanaananssaseanescocesansansagagiansenaacaancseed ce 15 Foes Nombro hidden NOS iii ina n i 15 2 5 4 N mbDerof hidden layers prices ineeie Rana OAE aAA ai enirar 16 3 Useofthe web application Model M inisee ne n aeien nan E Ea aaea eaaa reaa ie 17 3 1 The fastest parallel GPU based version FMLPGA 17 al WC 18 CO PRAIA 19 aa 20 dii RN 21 3 5 1 Regressio
15. Depending on the code used to represent the chromosome genes typical is the binary code the original value of a gene is replaced by the other character There are two different mutation criteria unbiased and biased Both criteria can be applied to an entire chromosome as well as to a single gene 101011010101 101011110101 Fig 10 An example of genetic mutation operator application In the single gene unbiased mutation standard mutation type one position inside the chromosome string is randomly selected Then the gene related to the selected chromosome string position is replaced In the unbiased chromosome mutation the entire selected chromosome is completely replaced by a new chromosome randomly generated In the biased chromosome mutation the selected chromosome is totally replaced by a new chromosome obtained by the sum of original chromosome and a new one randomly generated in this sense biased Generally neither the cross over and mutation operators are always applied to all population members but the first usually with a certain pre defined probability typical is 70 of population members while the second with a lower probability and only to members with associated poor fitness This procedure serial application of selected genetic operators 1s applied until the new population has exactly the same member number of the previous one And the entire cycle of population generation is iterated until the chro
16. Finished Er OK My Experiments Workspace mipgaExp Wi CopiadiSCHEDAD doc MR maxfotojpg Experiment P xorTrain Status ended Documents v Type esv esv Last Access 2010 12 02 2010 12 02 Last Access 2010 12 03 Fig 20 The xorTrain experiment status after submission MLPGAExp Experiment xorTran po Downlcad Addinve Status Last Access ended 2043 09 63 File Type Description FHLPGA _Train_trainerrors txt azc training error log file FRILPGA Train_trainoutoutcsy csv training output file FMLPGA Train _trainerrors jpeo ieg Image Tile FHLPGA Train_traincontusionmat other FRMLPGA Train loo ASC File log M Delete x training output pseudo confusion ri Fig 21 The xorTrain experiment output files Logout Me Info v XM Dele gt Delete O Mostra tutti i download 2 The content of output files obtained at the end of the experiment available when the status is ended is shown in the following note that in the training phase the file train_out is not much relevant The file error reports the training error after a set of iterations indicated in the first column the error is the MSE of the difference between network output and the target values DAMEWARE FMLPGA Model User Manual 38 This document contains proprietary information of DAME project Board All Rights Reserved fe ees out ii E eror 1 E walaras E tetou 2 0 Best
17. It contains the resulting network topology as stored at the end of a training session Usually this file should not be edited or modified by users just to preserve its content as generated by the model itself Accepted entries are 0 serial type execution on CPU 1 parallel type execution on GPU If left empty its default is 1 GPU e input nodes this parameter is a field required It is the number of neurons at the first input layer of the network It must exactly correspond to the number of input columns in the dataset input file Training File field except the target columns e hidden layers It is the number of hidden layers of the MLP network It can assume only two values 1 or 2 As suggestion this should be selected according the complexity of the problem usually 1 layer would be sufficient If left empty its default is 1 e 1st hidden layer nodes this parameter is a field required It is the number of neurons of the first hidden layer of the network As suggestion this should be selected in a range between a minimum of 2N 1 where N is the number of input nodes e Ist activation function It is the choice of which activation function should be associated to neurons of the Ist hidden layer After this choice all neurons of the layer will use the same activation function type The options are 0 sigmoid 1 threshold 2 linear 3 hyperbolic tangent By default the hyperbolic tangent function is used 32 DAMEWA
18. RE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved WARE DAta Mining amp Exploration ni a Program e 2nd hidden layer nodes It is the number of neurons of the second hidden layer of the network As suggestion this should be selected smaller than the previous layer By default the second hidden layer is empty not used e 2nd activation function It is the choice of which activation function should be associated to neurons of the 2nd hidden layer After this choice all neurons of the layer will use the same activation function type The options are 0 sigmoid 1 threshold 2 linear 3 hyperbolic tangent By default if the 2nd layer is activated the hyperbolic tangent function is used e output activation function It 1s the choice of which activation function should be associated to neurons of the output layer After this choice all neurons of the layer will use the same activation function type The options are 0 sigmoid 1 threshold 2 linear 3 hyperbolic tangent By default the hyperbolic tangent function is used 3 7 2 Classification with FMLPGA Run Parameter Specifications In the case of Classification_FMLPGA with Run use case the help page is at the address http dame dsf unina i FMLPGA_help html class_run e input dataset this parameter is a field required Dataset file as input It is a file containing input and tar
19. Rights Reserved DAta Mining amp Exploration Program After execution the experiment xorTest will show the output files available v File Manager Workspace FMLPGAExp Eh Dow gt Edit File Type Last Access SC Dele ne FMLPGA_Train_weights tt ascii 2013 09 03 x Gi amp xcr_runcsy csv 2013 09 03 x xorcsv csv 2043 09 83 x l v My Experiments Workspace FMLPGAExp Experiment Status Last Access MM Delete b xorTrain ended 2013 09 03 x 4 xorTest ended 2013 08 03 x gt Download Addinws File Type Description a 3 FMLPGA_Test_testconfusionmatri other testing output pseudo confusion ir Lt FMLPGA_Tast log ASCI File log ts n FMLPGA_Test_testoutput csv csv testinto output file Il a d FMLPGA Test params xmi xmi Experiment Configuration File Sai Fig 25 The xorTest experiment output files 41 DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved WARE al IA DAta Mining amp Exploration Program Ved A debt Ve 5 Appendix References and Acronyms Abbreviations amp Acronyms A amp A Meaning A amp A Meaning Al Artificial Intelligence HW Hardware ANN Artificial Neural Network KDD Knowledge Discovery in Databases ARFF Attribute Relation File Format IEEE Institute of Electrical and Electronic Engineers ASCII American Standard Code for INAF Istituto Nazionale di Astrofisica Informati
20. Xx A ri on xor_run c 1 0 x mow B a x B v My Experiments Workspace imipgaExp Experiment Status Last Access XM Delete Fig 18 The starting point with a Workspace FMLPGAExp created and two data files uploaded 36 DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved DAta Mining amp Exploration Program 4 1 1 Regression FMLPGA Train use case Let suppose we create an experiment named xorTrain and we want to configure it After creation the new configuration tab is open Here we select Regression FMLPGA as couple functionality model of the current experiment and we select also Train as use case IAME Application User bresciamax gmail com Mode Manuals Other Services v Science Cases RESOURCE MANAGER Experiment Setup DM Workspace FMLPGAExp Select a Running Train a ode Experiment xorTrain Select a Regression FMLPGA w Functionality De Field is Required input dataset x Xx GPU or CPU input nodes 2 hidden layers tat activation function 2nd hidden layer nodes 2nd activation function output activation function selection function error treshoid epochs error log frequency crossover rate mutation rate population size elitism rate tournament partecipants Submit Fig 19 The xorTrain experiment configuration tab Now we have to configure parameters for the
21. as generated by the model itself Accepted entries are 0 serial type execution on CPU 1 parallel type execution on GPU If left empty its default is 1 GPU e input nodes this parameter 1s a field required It is the number of neurons at the first input layer of the network It must exactly correspond to the number of input columns in the dataset input file Training File field except the target columns e hidden layers It is the number of hidden layers of the MLP network It can assume only two values 1 or 2 As suggestion this should be selected according the complexity of the problem usually 1 layer would be sufficient If left empty its default is 1 e 1st hidden layer nodes this parameter is a field required It is the number of neurons of the first hidden layer of the network As suggestion this should be selected in a range between a minimum of 2N 1 where N is the number of input nodes e Ist activation function It is the choice of which activation function should be associated to neurons of the Ist hidden layer After this choice all neurons of the layer will use the same activation function type The options are 0 sigmoid 1 threshold 2 linear 3 hyperbolic tangent By default the hyperbolic tangent function is used e 2nd hidden layer nodes It is the number of neurons of the second hidden layer of the network As suggestion this should be selected smaller than the previous layer By default
22. d 36 Fig 19 The xorTrain experiment configuration tab iii 37 Fig 20 The xorTrain experiment status after SUDMISSION iii 36 Fig 21 The xorTrain experiment output files iii 38 Fig 22 The files error left and weights right output of the xorTrain experiment 39 Fig 23 The file weights copied in the WS input file area for next purposes i 40 Fig 24 The xorTest experiment configuration tab note weights file inserted 40 Fig 25 The xorTest experiment output files iii 41 3 DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved F WARE Li DAta Mining amp Exploration ns Program 1 Introduction he present document is the user guide of the data mining model FMLPGA Fast Multi Layer Perceptron trained by Genetic Algorithms as implemented and integrated into the DAMEWARE This manual is one of the specific guides one for each data mining model available in the webapp having the main scope to help user to understand theoretical aspects of the model to make decisions about its practical use in problem solving cases and to use it to perform experiments through the webapp by also being able to select the right functionality associated to the model based upon the specific problem and related data to be explore
23. d to select the use cases to configure internal parameters to launch experiments and to evaluate results The documentation package consists also of a general reference manual on the webapp useful also to understand what we intend for association between functionality and data mining model and a GUI user guide providing detailed description on how to use all GUI features and options So far we strongly suggest to read these two manuals and to take a little bit of practical experience with the webapp interface before to explore specific model features by reading this and the other model guides All the cited documentation package is available from the address http dame dsf unina it dameware html where there is also the direct gateway to the webapp As general suggestion the only effort required to the end user is to have a bit of faith in Artificial Intelligence and a little amount of patience to learn basic principles of its models and strategies By merging for fun two famous commercial taglines we say Think different Just do it casually this is an example of data text mining DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved L WARE F DAta Mining amp Exploration pe PE Program 2 FMLPGA Model Theoretical Overview This paragraph is intended to furnish a theoretical overview of the FMLPGA model associated to single or m
24. d has been reached Offspring Decoded strings New generation Population chromosomes Genetic Fitness Operators Evaluation Selection Mating Pool Reproduction Manipulation Fig 12 Steps of the algorithm related to genetic algorithm evolution 2 3 MLP Practical Rules The practice and expertise in the machine learning models such as MLP are important factors coming from a long training and experience within their use in scientific experiments The speed and effectiveness of the results strongly depend on these factors Unfortunately there are no magic ways to a priori indicate the best configuration of internal parameters involving network topology and learning algorithm But in some cases a set of practical rules to define best choices can be taken into account 2 3 1 Selection of neuron activation function e If there are good reasons to select a particular activation function then do it o linear o threshold o Hyperbolic tangent 14 DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved 2 3 2 2 3 3 DAta Mining amp Exploration vas Mac Program o sigmoid General good properties of activation function o Non linear o Saturate some max and min value o Continuity and smooth o Monotonicity convenient but nonessential o Linearity for a small value of net Sigmoid function has all the
25. d to the number of input columns in the dataset input file Training File field except the target columns e hidden layers It is the number of hidden layers of the MLP network It can assume only two values 1 or 2 As suggestion this should be selected according the complexity of the problem usually 1 layer would be sufficient 21 DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved WARE DAta Mining amp Exploration eel Program If left empty its default is 1 e 1st hidden layer nodes this parameter is a field required It is the number of neurons of the first hidden layer of the network As suggestion this should be selected in a range between a minimum of 2N 1 where N is the number of input nodes e Ist activation function It is the choice of which activation function should be associated to neurons of the Ist hidden layer After this choice all neurons of the layer will use the same activation function type The options are 0 sigmoid 1 threshold 2 linear 3 hyperbolic tangent By default the hyperbolic tangent function is used e 2nd hidden layer nodes It is the number of neurons of the second hidden layer of the network As suggestion this should be selected smaller than the previous layer By default the second hidden layer is empty not used e 2nd activation function It is the choice of which activation function shou
26. e by applying any reproduction with a chromosome randomly chosen between those with poor fitness can introduce gain in the next generations Usually but not always in the tournament selection two winners are chosen at a time i e from two tournaments and recombined to generate two sons put in the next generation As said before two are typical genetic operators employed in the chromosome recombination The Cross over or unbiased chromosome cross over criterion on two m length chromosomes A and B brakes original parents at a fixed position jJ j is the number of genes or sub string length arbitrarily fixed a priori So the m j genes of chromosome A are queued to the first j genes of chromosome B while the m j genes of chromosome B are queued to the first j genes of chromosome A Fig 9 An example of genetic cross over operator application Example A 10010001 B 00110111 m 8 and j 5 gt after cross over gt A 10010111 B 00110001 After the cross over application the two sons A and B are obtained from their parents by reciprocally reverting last m j 3 genes 11 DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved DAta Mining amp Exploration WARE Mac Program After cross over usually it is applied another genetic operator mutation It consists into the roughly change in only one gene inside the chromosome
27. e logical XOR function between two binary variables As known the XOR problem is not a linearly separable problem so we require to obtain a neural network able to learn to identify the right output value of the XOR function having a BoK made by possible combinations of two input variable and related correct output This is a very trivial problem and in principle it should not be needed any machine learning method But as remarked the scope is not to obtain a scientific benefit but to make practice with the web application Let say it is an example comparable with the classical print lt Hello World gt on standard output implementation problem for beginners in C language As first case we will use the FMLPGA model associated to the regression functionality The starting point is to create a new workspace named FMLPGAExp and to populate it by uploading two files e xor csv CSV dataset file for training and test use cases e xor_run csv CSV dataset file for run use case Their content description is already described in section 3 of this document DAME Application User brescia oacn inaf it LogOut 2 App Manuals v Model Manuals v Cloud Services v Science Cases v Documents v Info v RESOURCE MANAGER Upload in mipgaExp Workspace Workspace v File Manager Workspac er New Workspace mipgaExp j Dow Edt File yp Last Ac X Ocke 7 Rename Workspace i Upload Gj Experiment 3 Delete p gt XOF CSV 05 30 x mb x
28. eir segments As all genetic operators the crossover is not always applied in the genetic recombination but with an associated probability this parameter Instead the breaking point inside the chromosome where to apply crossover is selected randomly If left empty its default is 0 9 1 e 90 26 DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved F WARE Li DAta Mining amp Exploration nes Program e mutation rate probability percentage a real value between 0 and 1 of the mutation operator application during the population generation process The mutation operator makes a single change in a gene of a chromosome replacing it with a new random value selected in a fixed range see parameter chromosome perturbation value If left empty its default is 0 4 1 e 40 e population size This is the number of chromosomes which compose the population an integer between 10 and 60 Remember that each chromosome is a solution of the problem At each iteration generation this parameter indicates how many chromosomes should be considered in the population of the GA If left empty the default value is 20 e elitism rate The parameter user defined related to this elitism mechanism defines the number of copies of the winner chromosome to be transmitted unchanged in the population of the next generation If left empty its default is 2 e tournament pa
29. eriment Status Last Access a xorfrain ended 2013 09 03 gt Bownilcad Addinyys File Type Description i FNLPGA_Train_trainerrors jpec jpeo image file i FMLPGA_Train_trainconfusionmat other i FNLPGA_Train ioo ASCII File joo i FMLPGA_ Train weights txt asci M Deea x Deiete training output pseudo confusion r 7 trained network weionht fils Fig 23 The file weights copied in the WS input file area for next purposes So far we proceed to create a new experiment named xorTest to verify the training of the network For simplicity we will re use the same input dataset file xor csv but in general the user could use another dataset uploaded from scratch or extracted from the original training dataset through file editing options RESOURCE MANAGER Experiment Setup DJ Workspace FMLPGAExp Experiment xorTest Selecta Functionality Selec t ing Select a Running Tesi gt Mode Regression_FMLPGA v Field is Required input dataset weights file GPU ar CPU input nodes hidden layers tst hidden layer nodes ist activation function 2nd hidden layer nodes 2nd activation function output activation function t JXOCCSY v FMLPGA_ Train_weights w hm 1 2 Submit Fig 24 The xorTest experiment configuration tab note weights file inserted DAMEWARE FMLPGA Model User 40 Manual This document contains proprietary information of DAME project Board All
30. erwise it takes the deactivated value typically 1 gt out P x w x w 0 input output Fig 6 Example of a SLP to calculate the logic AND operation Neurons with this kind of activation function are also called artificial neurons or linear threshold units as described by Warren McCulloch and Walter Pitts in the 1940s A Perceptron can be created using any values for the activated and deactivated states as long as the threshold value lies between the two Most perceptrons have outputs of 1 or 1 with a threshold of 0 and there is some evidence that such networks can be trained more quickly than networks created from nodes with different activation and deactivation values SLPs are only capable of learning linearly separable patterns In 1969 in a famous monograph entitled Perceptrons Marvin Minsky and Seymour Papert showed that it was impossible for a single layer Perceptron network to learn an XOR function Although a single threshold unit is quite limited in its computational power it has been shown that networks of parallel threshold units can approximate any continuous function from a compact interval of the real numbers into the interval 1 1 So far it was introduced the model Multi Layer Perceptron Olit input output hidden Fig 7 A MLP able to calculate the logic XOR operation DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Righ
31. generalization capability Despite of these considerations it should be always taken into account that neural networks application field should be usually referred to problems where it is needed high flexibility quantitative result more than high precision qualitative results Second learning type unsupervised is basically referred to neural models able to classify cluster patterns onto several categories based on their common features by submitting training inputs without related DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved WARE jj DAta Mining amp Exploration A dii Program desired outputs This is not the learning case approached with the MLP architecture so it is not important to add more information in this document 2 2 The Genetic Algorithms The Genetic Algorithms GAs are computational methods inspired by the natural evolution law discovered by Darwin They are particularly powerful to solve problems where the solution space is not well defined The algorithm hence consists mainly in the cyclic exploration of the parameter space carefully going towards the best solution In a GA each element of a population is the so called chromosome composed by a set of genes features that represents its DNA Each DNA is in practice one possible solution to the problem The starting point of the method consists in the random generation of a
32. get columns It must have the same number of input and target columns as for the training input file For example it could be the same dataset file used as the training input file e weights file this parameter is a field required It is a file generated by the model during training phase It contains the resulting network topology as stored at the end of a training session Usually this file should not be edited or modified by users just to preserve its content as generated by the model itself 33 DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved WARE DAta Mining amp Exploration ni a Program e GPU or CPU It is a file generated by the model during training phase It contains the resulting network topology as stored at the end of a training session Usually this file should not be edited or modified by users just to preserve its content as generated by the model itself Accepted entries are 0 serial type execution on CPU 1 parallel type execution on GPU If left empty its default is 1 GPU e input nodes this parameter 1s a field required It is the number of neurons at the first input layer of the network It must exactly correspond to the number of input columns in the dataset input file Training File field except the target columns e hidden layers It is the number of hidden layers of the MLP network It can assume only t
33. gorithm If left empty its default is 1000 e error log frequency Frequency in steps of training error storage in a log file to evaluate the trend in the learning error during generation cycles If left empty its default is 10 e cross over rate probability percentage a real value between O and 1 of the cross over operator application during the population generation process Crossover happens when two chromosomes break themselves at the same point inside the string coding the gene vector and exchange their segments As all genetic operators the crossover is not always applied in the genetic recombination but with an associated probability this parameter Instead the breaking point inside the chromosome where to apply crossover is selected randomly If left empty its default is 0 9 1 e 90 e mutation rate probability percentage a real value between O and 1 of the mutation operator application during the population generation process The mutation operator makes a single change in a gene of a chromosome replacing it with a new random value selected in a fixed range see parameter chromosome perturbation value If left empty its default is 0 4 1 e 40 23 DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved F WARE Li DAta Mining amp Exploration nes Program e population size This is the number of chromosomes which compose
34. he choice of which activation function should be associated to neurons of the output layer After this choice all neurons of the layer will use the same activation function type The options are 0 sigmoid 1 threshold 2 linear 3 hyperbolic tangent By default the hyperbolic tangent function is used e selection function It is the choice of the selection function used to evolve chromosomes of genetic population at each training iteration The options are 0 fitting 1 ranking 2 standard roulette v1 3 optimized roulette v2 By default the standard roulette function is used e error threshold This is the threshold of the learning loop This is one of the two stopping criteria of the algorithm Use this parameter in combination with the number of iterations If left empty its default is 0 01 e epochs Number of training epochs done in batch learning mode This is the second stop condition of the algorithm If left empty its default is 1000 e error log frequency Frequency in steps of training error storage in a log file to evaluate the trend in the learning error during generation cycles If left empty its default is 10 e cross Over rate probability percentage a real value between O and 1 of the cross over operator application during the population generation process Crossover happens when two chromosomes break themselves at the same point inside the string coding the gene vector and exchange th
35. idden layers One or two hidden layers are OK so long as differentiable activation function o But one layer is generally sufficient More layers gt more chance of local minima Single hidden layer vs double multiple hidden layer o single is good for any approximation of continuous function o double may be good some times Problem specific reason of more layers o Each layer learns different aspects DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved DAta Mining amp Exploration vas Mac Program 3 Use of the web application model The Multi Layer Perceptron MLP is one of the most common supervised neural architectures used in many application fields It 1s especially related to classification and regression problems and in DAME it is designed to be associated with such two functionality domains The description of these two functionalities is reported in the Reference Manual A18 available from webapp menu or from the beta intro web page In the following are described practical information to configure the network architecture and the learning algorithm in order to launch and execute science cases and experiments The GA learning rule is one of the options as available in DAME webapp to be associated with the MLP network 3 1 The fastest parallel GPU based version FMLPGA The last version released on DAMEWARE exploits the new HW inf
36. ile is shown valid only for Run use case experiments It is the same of xor csv except for the target column that is not present This file can be also generated by the user starting from the xor csv As detailed in the GUI user Guide A19 the user may in fact use the Dataset Editor options of the webapp to manipulate and build datasets starting from uploaded data files IMPORTANT NOTE in case of classification experiment the target columns of input data file must be at least two 2 class problem or more than two 19 DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved F WAGE i g DAta Mining amp Exploration e Program 3 4 Output In terms of output different files are obtained depending on the specific use case of the experiment In the case of regression functionality the following output files are obtained in all use cases TRAIN TEST RUN NOTES default prefix default prefix default prefix FFMLPGA_ Train FFMLPGA_ Test FFMLPGA_Ru info log log dog log experiment status log error trend table _trainerrors jpeg error trend plot _testoutput csv _weights txt trained network weights to be moved in the File Manager tab area through GUI button AddInWS to be loaded during a test run experiment I evaluation of training performance o evaluation of test performance Tab 1 output file list in case of regression type experiments
37. input nodes e 1st activation function It is the choice of which activation function should be associated to neurons of the Ist hidden layer After this choice all neurons of the layer will use the same activation function type The options are 0 sigmoid 1 threshold 2 linear 3 hyperbolic tangent By default the hyperbolic tangent function is used e 2nd hidden layer nodes It is the number of neurons of the second hidden layer of the network As suggestion this should be selected smaller than the previous layer By default the second hidden layer is empty not used e 2nd activation function It is the choice of which activation function should be associated to neurons of the 2nd hidden layer After this choice all neurons of the layer will use the same activation function type The options are 0 sigmoid 1 threshold 2 linear 3 hyperbolic tangent By default if the 2nd layer is activated the hyperbolic tangent function is used e output nodes this parameter is a field required It is the number of neurons in the output layer of the network It must correspond to the number of target columns as contained in the dataset input file filed Training File It is mandatory to set 2 neurons at least 25 DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved DAta Mining amp Exploration Program e output activation function It 1s t
38. l1 0 BetaRelease_ GUL UserManual DAME MAN NA 0010 Rell 0 Program Author DAME Working Group Brescia Brescia DAME Working Group D Abrusco D Abrusco Cavuoti Manna Fiore Nocella d Angelo Cavuoti Di Guido Cavuoti Skordovski Skordovski Cavuoti Skordovski Laurino Di Guido Brescia Brescia Brescia Tab 5 Applicable Documents DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved DAta Mining amp Exploration Date 15 10 2008 19 02 2008 30 05 2007 12 10 2007 17 07 2007 06 10 2007 18 03 2009 14 04 2010 29 03 2010 03 06 2009 22 03 2010 07 07 2007 02 10 2007 20 02 2008 05 01 2009 30 11 2008 14 04 2010 28 10 2010 03 12 2010 44 DAta Mining amp Exploration Program 000 DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved 45 DAta Mining amp Exploration Program DAME Program we make science discovery happen 46 DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved
39. ld be associated to neurons of the 2nd hidden layer After this choice all neurons of the layer will use the same activation function type The options are 0 sigmoid 1 threshold 2 linear 3 hyperbolic tangent By default if the 2nd layer is activated the hyperbolic tangent function is used e output activation function It 1s the choice of which activation function should be associated to neurons of the output layer After this choice all neurons of the layer will use the same activation function type The options are 0 sigmoid 1 threshold 2 linear 3 hyperbolic tangent By default the hyperbolic tangent function is used 22 DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved DAta Mining amp Exploration Program e selection function It is the choice of the selection function used to evolve chromosomes of genetic population at each training iteration The options are 0 fitting 1 ranking 2 standard roulette v1 3 optimized roulette v2 By default the standard roulette function is used e error threshold This is the threshold of the learning loop This is one of the two stopping criteria of the algorithm Use this parameter in combination with the number of iterations If left empty its default is 0 01 e epochs Number of training epochs done in batch learning mode This is the second stop condition of the al
40. m networks with sigmoidal non linearity and two layer of weights can approximate any decision boundary to arbitrary accuracy Thus such networks also provide universal non linear discriminate functions More generally the capability of such networks to approximate general smooth functions allows them to model posterior probabilities of class membership Since two layers of weights suffice to implement any arbitrary function one would need special problem conditions or requirements to recommend the use of more than two layers Furthermore it is found empirically that networks with multiple hidden layers are more prone to getting caught in undesirable local minima Astronomical data do not seem to require such level of complexity and therefore it is enough to use just a double weights layer i e a single hidden layer What is different in such a neural network architecture is typically the learning algorithm used to train the network It exists a dichotomy between supervised and unsupervised learning methods In the first case the network must be firstly trained training phase in which the input patterns are submitted to the network as couples input desired known output The feed forward algorithm is then achieved and at the end of the input submission the network output is compared with the corresponding desired output in order to quantify the learning quote It is possible to perform the comparison in a batch way after an entire input patter
41. mosome with desired good fitness is founded Now having introduced the computing model architecture and the learning algorithm we are ready to combine them into the FMLPGA model 12 DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved DAta Mining amp Exploration Program di ARAIA VTN ARA E le le LP oe DA ra P e ed N lt soluzioni buone Learning rule Aa gt based on GAs Fig 11 A MLP network trained by a Genetic Algorithm In the DM model the GA technique is used to change weights related to hidden layers during the learning phase corresponding to the backward step of BP algorithm In this case the chromosomes are represented by the weight vectors associated with neuron layers where the single neuron weight is a gene of the string By evolving populations of network weights through the application of described genetic operators the model is able to converge generally faster than weight gradient descendent of BP to the best solution In other words at each step of genetic evolution a population of neural networks identified by its weight vectors is generated The fitness function in this case is simply the calculated error of MLP output by applying standard feed forward propagation of the input patterns through the network layers The winner chromosome will be therefore the MLP weight configura
42. mp Exploration ni a Program For example it could be the same dataset file used as the training input file e weights file this parameter is a field required It is a file generated by the model during training phase It contains the resulting network topology as stored at the end of a training session Usually this file should not be edited or modified by users just to preserve its content as generated by the model itself e GPU or CPU It is a file generated by the model during training phase It contains the resulting network topology as stored at the end of a training session Usually this file should not be edited or modified by users just to preserve its content as generated by the model itself Accepted entries are 0 serial type execution on CPU 1 parallel type execution on GPU If left empty its default is 1 GPU e input nodes this parameter is a field required It is the number of neurons at the first input layer of the network It must exactly correspond to the number of input columns in the dataset input file Training File field except the target columns e hidden layers It is the number of hidden layers of the MLP network It can assume only two values 1 or 2 As suggestion this should be selected according the complexity of the problem usually 1 layer would be sufficient If left empty its default is 1 e 1st hidden layer nodes this parameter is a field required It is the number of neurons of the fi
43. n set submission or incremental the comparison is done after each input pattern submission and also the metric used for the distance measure between desired and obtained outputs can be chosen accordingly problem specific requirements usually the euclidean distance is used After each comparison and until a desired error distance is unreached typically the error tolerance is a pre calculated value or a constant imposed by the user the weights of hidden layers must be changed accordingly to a particular law or learning technique After the training phase is finished or arbitrarily stopped the network should be able not only to recognize correct output for each input already used as training set but also to achieve a certain degree of generalization 1 e to give correct output for those inputs never used before to train it The degree of generalization varies as obvious depending on how good has been the learning phase This important feature is realized because the network doesn t associates a single input to the output but it discovers the relationship present behind their association After training such a neural network can be seen as a black box able to perform a particular function input output correlation whose analytical shape is a priori not known In order to gain the best training it must be as much homogeneous as possible and able to describe a great variety of samples Bigger the training set higher will be the network
44. n with FMLPGA Train Parameter SpecificationS 21 3 5 2 Classification with FMLPGA Train Parameter SpecificationS 24 i TE ii 27 3 6 1 Regression with FMLPGA Test Parameter SpecificationS 2T 3 6 2 Classification with FMLPGA Test Parameter Specifications 29 o Rintra 31 3 7 1 Regression with FMLPGA Run Parameter SpecificationS 31 3 7 2 Classification with FMLPGA Run Parameter SpecificationS 33 I E ABI RR banc ndesessaa Quanta nacneacenses nacosiesaacageeteadanameaasenees T EA EEEE T 36 d l Regression AOR om ONG Rina in 36 4 1 1 Regression FMLPGA Train use case 37 4 1 2 Regression FMLPGA Test use case 40 5 Appendix References and AcronymS 42 DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved WARE DAta Mining amp Exploration ni a Program TABLE INDEX Tab 1 output file list in case of regression type experiments iii 20 Tab 2 output file list in case of classification type experiments iii 20 ARRONE ERRE 42 TaD 4 Reference DOCUMENIS snmnnnnnia aaa 43 TDD DGC POC A iii 44 FIGURE INDEX Fig 1 the MLP artificial and biologic brains iii 5 zar 6 VAIO NCN CROAZIA 6 FOA Lina 6 AIA RR AA as 7
45. nd their default values We remark that all parameters labeled by an asterisk are considered required In all other cases the fields can be left empty default values are used 3 5 1 Regression with FMLPGA Train Parameter Specifications In the case of Regression FMLPGA with Train use case the help page is at the address http dame dsf unina it FMLPGA_ help html regr_train e input dataset this parameter is a field required This is the dataset file to be used as input for the learning phase of the model It typically must include both input and target columns where each row is an entire pattern or sample of data The format hence its extension must be one of the types allowed by the application ASCH FITS CSV VOTABLE More specifically take in mind the following simple rule the sum of input and output nodes MUST be equal to the total number of the columns in this file e GPU or CPU It is a file generated by the model during training phase It contains the resulting network topology as stored at the end of a training session Usually this file should not be edited or modified by users just to preserve its content as generated by the model itself Accepted entries are 0 serial type execution on CPU 1 parallel type execution on GPU If left empty its default 1s 1 GPU e input nodes this parameter is a field required It is the number of neurons at the first input layer of the network It must exactly correspon
46. normal experiments on new datasets In the experiment configuration there is also the Help button redirecting to a web page dedicated to support the user with deep information about all parameters and their default values We remark that all parameters labeled by an asterisk are considered required In all other cases the fields can be left empty default values are used 3 7 1 Regression with FMLPGA Run Parameter Specifications In the case of Regression_FMLPGA with Run use case the help page is at the address http dame dsf unina i FMLPGA_help html regr_run e input dataset this parameter is a field required Dataset file as input It is a file containing input and target columns 31 DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved WARE DAta Mining amp Exploration ni a Program It must have the same number of input and target columns as for the training input file For example it could be the same dataset file used as the training input file e weights file this parameter is a field required It is a file generated by the model during training phase It contains the resulting network topology as stored at the end of a training session Usually this file should not be edited or modified by users just to preserve its content as generated by the model itself e GPU or CPU It is a file generated by the model during training phase
47. on Program Step function y lifx gt 0 0ifx lt 0 Fig 2 the step function The step function is the most similar to biological neuron reaction to external stimuli In this case in fact it uses a constant activation threshold to propagate the signal It is useful only in problem solving where it is needed a crisp classification between two well identified classes Ramp function y xifxis internal to 1 1 Fig 3 the ramp function It is also known as identity function in the 1 1 range Useful only when the network output is unbounded Sigmoid function y 1 1 e x Fig 4 the sigmoid function DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved DAta Mining amp Exploration Program This function is the most frequent in the MLP model It is characterized by its smooth step between 0 and 1 with a variable threshold But this restricted domain 0 1 is also its limitation It in fact can be used only in problems where expected outputs are numbers in this range Hyperbolic tangent function y tanh x Fig 5 the tanh function The hyperbolic tangent is very similar to the sigmoid except for the output range 1 1 and that it across the origin of axes It therefore extends the admissible range of network output For the output units activation functions suited to the distribution of the ta
48. on Interchange Bok Base of Knowledge JPEG Joint Photographic Experts Group BP Back Propagation LAR Layered Application Architecture BLL Business Logic Layer MDS Massive Data Sets CE Cross Entropy MLP Multi Layer Perceptron CSV Comma Separated Values MSE Mean Square Error DAL Data Access Layer NN Neural Network DAME DAta Mining amp Exploration OAC Osservatorio Astronomico di Capodimonte DAPL Data Access amp Process Layer PC Personal Computer DL Data Layer PI Principal Investigator DM Data Mining REDB Registry amp Database DMM Data Mining Model RIA Rich Internet Application DMS Data Mining Suite SDSS Sloan Digital Sky Survey FITS Flexible Image Transport System SL Service Layer FL Frontend Layer SW Software FW FrameW ork UI User Interface GPU Graphical Processing Unit URI Uniform Resource Indicator GRID Global Resource Information Database VO Virtual Observatory GUI Graphical User Interface XML eXtensible Markup Language Tab 3 Abbreviations and acronyms 42 DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved i WARE A Ved A se Sees Ve Reference amp Applicable Documents ID RI R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 R13 R14 Title Code The Use of Multiple Measurements in Taxonomic Problems in Annals of Eugenics 7 p 179 188 Neural Networks University Press GB
49. rastructure based on HPC in the Cloud with NVIDIA GPU K20 Kepler series parallel execution capabilities The new FMLPGA algorithm now called FMLPGA where F stands for Fast offers the possibility to be executed on two optional user selected computing platforms in a transparent way to end user CPU normal serialized execution on a HPC host based on a multi processor multi core machinery 4 GPU accelerated parallel execution on a GPU device based on a NVIDIA K20 many core machinery The GPU option gives an average speedup acceleration of about 8x in respect of the CPU one It means that on average a same experiment is executed 8 times faster 6000 5500 7 5000 7 4500 7 4000 7 3500 7 3000 7 2500 4 2000 1500 1000 7 ieee sn 500 7 0 7 sec CPU H GPU 1000 2000 4000 8000 10000 30000 50000 iterations Fig 14 The computing time comparison between FMLPGA training execution on CPU and GPU platforms The experiment was based on 1000 input patterns each one composed by 10 parameters 17 DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved DAta Mining amp Exploration Program Regression Speedup GPU vs CPU 9 i gt o 7 4 n 7 roulette ranking fitting function Fig 15 The computing time speedup for FMLPGA on CPU and GPU platforms The function axis is referred to the differen
50. rget values are e For binary 0 1 targets the logistic sigmoid function is an excellent choice e For continuous valued targets with a bounded range the logistic and hyperbolic tangent functions can be used where you either scale the outputs to the range of the targets or scale the targets to the range of the output activation function scaling means multiplying by and adding appropriate constants e If the target values are positive but have no known upper bound you can use an exponential output activation function but you must beware of overflow e For continuous valued targets with no bounds use the identity or linear activation function which amounts to no activation function unless you have a very good reason to do otherwise The base of the MLP is the Perceptron a type of artificial neural network invented in 1957 at the Cornell Aeronautical Laboratory by Frank Rosenblatt It can be seen as the simplest kind of feedforward neural network a linear classifier The Perceptron is a binary classifier which maps its input x a real valued vector to an output value f x a single binary value across the matrix 1 ifw xr b gt 0 I x 0 else Despite of the above considerations it is well known that it is always possible to shift the center of the activation function by introducing the bias value associated to each neuron By this way it is always possible to obtain values in the 00 00 range But in this case it is sugge
51. roulette wheel selection and tournament selection The former mainly consists into the assignment of a probability to be selected for the reproduction for each chromosome This probability is directly proportional to its fitness The latter is alternatively based on the random choice of a number N of chromosomes from the current population and to compare their fitness the element with the best fitness is the winner and is selected for reproduction 10 DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved DAta Mining amp Exploration Program The tournament selection seems quite flexible because it permits to introduce more variability in the selection criterion higher N lower the possibility to select chromosomes candidates with worst fitness Anyway not always the elements with worst fitness must be considered bad choices for reproduction They in fact introduce more variety in the population exactly like the jumping out from a local minima for the gradient descendent learning algorithm in the BP and can speed up the convergence towards best solution But their use must be taken under control in the GA For example by monitoring population evolution where some chromosome has a static trend i e it has always associated a good fitness but not sufficient to become a winner during reproduction so remaining always the same through several generations In this cas
52. rst hidden layer of the network As suggestion this should be selected in a range between a minimum of 2N 1 where N is the number of input nodes e Ist activation function It is the choice of which activation function should be associated to neurons of the Ist hidden layer After this choice all neurons of the layer will use the same activation function type The options are 0 sigmoid 1 threshold 2 linear 3 hyperbolic tangent By default the hyperbolic tangent function is used 28 DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved WARE DAta Mining amp Exploration ni a Program e 2nd hidden layer nodes It is the number of neurons of the second hidden layer of the network As suggestion this should be selected smaller than the previous layer By default the second hidden layer is empty not used e 2nd activation function It is the choice of which activation function should be associated to neurons of the 2nd hidden layer After this choice all neurons of the layer will use the same activation function type The options are 0 sigmoid 1 threshold 2 linear 3 hyperbolic tangent By default if the 2nd layer is activated the hyperbolic tangent function is used e output activation function It 1s the choice of which activation function should be associated to neurons of the output layer After this choice all neurons of
53. rticipants This is the number of chromosomes in the population to be engaged in the so called Ranking Selection This is in practice used only in case of ranking selection function choice Among this number of participants the first two chromosomes with higher fitness value are chosen to generate childs 3 6 TEST Use case In the use case named Test the software provides the possibility to test the FMLPGA The user will be able to use already trained FMLPGA models their weight configurations to execute the testing experiments In the experiment configuration there is also the Help button redirecting to a web page dedicated to support the user with deep information about all parameters and their default values We remark that all parameters labeled by an asterisk are considered required In all other cases the fields can be left empty default values are used 3 6 1 Regression with FMLPGA Test Parameter Specifications In the case of Regression_FMLPGA with Test use case the help page is at the address http dame dsf unina it FMLPGA_help html regr_test e input dataset this parameter is a field required Dataset file as input It is a file containing input and target columns It must have the same number of input and target columns as for the training input file Zi DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved WARE DAta Mining a
54. sted to use a linear transfer function 7 DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved F WAGE i di DAta Mining amp Exploration R Program where wis a vector of real valued weights and W T is the dot product which computes a weighted sum b is the bias a constant term that does not depend on any input value The value of f x 0 or 1 is used to classify x as either a positive or a negative instance in the case of a binary classification problem If b is negative then the weighted combination of inputs must produce a positive value greater than b in order to push the classifier neuron over the 0 threshold Spatially the bias alters the position though not the orientation of the decision boundary The Perceptron learning algorithm does not terminate if the learning set is not linearly separable The Perceptron is considered the simplest kind of feed forward neural network The earliest kind of neural network is a Single Layer Perceptron SLP network which consists of a single layer of output nodes the inputs are fed directly to the outputs via a series of weights In this way it can be considered the simplest kind of feed forward network The sum of the products of the weights and the inputs is calculated in each node and if the value is above some threshold typically 0 the neuron fires and takes the activated value typically 1 oth
55. t By default if the 2nd layer is activated the hyperbolic tangent function is used e output nodes this parameter is a field required It 1s the number of neurons in the output layer of the network It must correspond to the number of target columns as contained in the dataset input file filed Training File e output activation function It 1s the choice of which activation function should be associated to neurons of the output layer After this choice all neurons of the layer will use the same activation function type The options are 0 sigmoid 1 threshold 2 linear 3 hyperbolic tangent By default the hyperbolic tangent function is used 35 DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved WARE amp DAta Mining amp Exploration eae Arie Program 4 Examples This section is dedicated to show some practical examples of the correct use of the web application Not all aspects and available options are reported but a significant sample of features useful for beginners of DAME suite and with a poor experience about data mining methodologies with machine learning algorithms In order to do so very simple and trivial problems will be described Further complex examples will be integrated here in the next releases of the documentation 4 1 Regression XOR problem The problem can be stated as follows we want to train a model to learn th
56. t chromosome selection type used during training evolution of genetic population The user has the opportunity to select the platform through a simple parameter during experiment setup For more instructions on parameter setup see below 3 2 Use Cases For the user the FMLPGA system offers four use cases e Train o Test e Run As described in A19 a supervised machine learning model like FMLPGA requires different use cases well ordered in terms of execution sequence A typical complete experiment with this kind of models consists in the following steps 1 Train the network with a dataset as input containing both input and target features then store as output the final weight matrix best configuration of network weights 2 Test the trained network in order to verify training quality it is also included the validation step available for some models The same training dataset or a mix with new patterns can be used as input 3 Run the trained and tested network with datasets containing ONLY input features without target ones In this case new or different input data are encouraged because the Run use case implies to simply execute the model like a generic static function 18 DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved WARE DAta Mining amp Exploration eel Program 3 3 Input We also remark that massive datasets to be used in
57. the second hidden layer is empty not used e 2nd activation function It is the choice of which activation function should be associated to neurons of the 2nd hidden layer After this choice all neurons of the layer will use the same activation function type The options are 30 DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved F WARE ea DAta Mining amp Exploration eel Program 0 sigmoid 1 threshold 2 linear 3 hyperbolic tangent By default if the 2nd layer is activated the hyperbolic tangent function is used e output nodes this parameter is a field required It is the number of neurons in the output layer of the network It must correspond to the number of target columns as contained in the dataset input file filed Training File It is mandatory to set 2 neurons at least e output activation function It 1s the choice of which activation function should be associated to neurons of the output layer After this choice all neurons of the layer will use the same activation function type The options are 0 sigmoid 1 threshold 2 linear 3 hyperbolic tangent By default the hyperbolic tangent function is used 3 7 Run Use case In the use case named Run the software provides the possibility to run the FMLPGA The user will be able to use already trained and tested FMLPGA models their weight configurations to execute the
58. tion obtaining the lower output error Moreover in this case to maintain the integrity of the MLP feed forward calculations the genes are not coded but leaved exactly as weight double values usually normalized between 1 and 1 The formulas showed in Fig 11 are cyclically repeated during training It is hence evident that the learning algorithm can be divided into two phases bottom up propagation and top down weight update The complete algorithm involves the following steps 1 Random generation of the initial genetic population of chromosomes population of different network weight vectors 2 Forward propagation of a training pattern s input through the neural network in order to generate the propagation s output activations This step is repeated for each network weight matrix a single chromosome of the genetic population 3 Calculation of training error batch mode as MSE of distances between network output and related target values 4 Evaluation of the output performance on all patterns against the genetic fitness function for each chromosome 13 DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved DAta Mining amp Exploration Program 5 Generation of a new population of chromosomes by applying genetic operators to the weight vectors of the network 6 Repeat steps from 2 to 5 until the number of training cycles or the chosen error threshol
59. ts Reserved a WARE DAta Mining amp Exploration A Program This class of networks consists of multiple layers of computational units usually interconnected in a feed forward way Each neuron in one layer has directed connections to the neurons of the subsequent layer In many applications the units of these networks apply a continuous activation function The number of hidden layers represents the degree of the complexity achieved for the energy solution space in which the network output moves looking for the best solution As an example in a typical classification problem the number of hidden layers indicates the number of hyper planes used to split the parameter space i e number of possible classes in order to classify each input pattern The universal approximation theorem R12 for neural networks states that every continuous function that maps intervals of real numbers to some output interval of real numbers can be approximated arbitrarily closely by a multi layer Perceptron with just one hidden layer This result holds only for restricted classes of activation functions e g for the sigmoidal functions An extension of the universal approximation theorem states that the two layers architecture is capable of universal approximation and a considerable number of papers have appeared in the literature discussing this property An important corollary of these results is that in the context of a classification proble
60. ultiple functionality domains in order to be used to perform practical scientific experiments with such techniques An overview of machine learning and functionality domains as intended in DAME Project can be found in A18 2 1 Multi Layer Perceptron The MLP architecture is one of the most typical feed forward neural network model The term feed forward is used to identify basic behavior of such neural models in which the impulse is propagated always in the same direction e g from neuron input layer towards output layer through one or more hidden layers the network brain by combining weighted sum of weights associated to all neurons except the input layer Output Units Summation Units a _ Pattern Units Input Units Fig 1 the MLP artificial and biologic brains As easy to understand the neurons are organized in layers with proper own role The input signal simply propagated throughout the neurons of the input layer is used to stimulate next hidden and output neuron layers The output of each neuron is obtained by means of an activation function applied to the weighted sum of its inputs Different shape of this activation function can be applied from the simplest linear one up to sigmoid or tanh or a customized function ad hoc for the specific application DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved DAta Mining amp Explorati
61. wo values 1 or 2 As suggestion this should be selected according the complexity of the problem usually 1 layer would be sufficient If left empty its default is 1 e 1st hidden layer nodes this parameter is a field required It is the number of neurons of the first hidden layer of the network As suggestion this should be selected in a range between a minimum of 2N 1 where N is the number of input nodes e Ist activation function It is the choice of which activation function should be associated to neurons of the Ist hidden layer After this choice all neurons of the layer will use the same activation function type The options are 0 sigmoid 1 threshold 2 linear 3 hyperbolic tangent By default the hyperbolic tangent function is used e 2nd hidden layer nodes It is the number of neurons of the second hidden layer of the network As suggestion this should be selected smaller than the previous layer By default the second hidden layer is empty not used e 2nd activation function It is the choice of which activation function should be associated to neurons of the 2nd hidden layer After this choice all neurons of the layer will use the same activation function type The options are 34 DAMEWARE FMLPGA Model User Manual This document contains proprietary information of DAME project Board All Rights Reserved DAta Mining amp Exploration Program 0 sigmoid 1 threshold 2 linear 3 hyperbolic tangen

Fast Multi Layer Perceptron with Genetic Algorithm User Manual

Contents

Download Pdf Manuals

Related Search

Related Contents