Bench4Q Tool 1.3.0 USER`S MANUAL
Contents
1. Bench4Q Tool 1 3 0 A QoS Oriented E commerce Benchmark USER S MANUAL Revision Sheet Release No Rev 1 0 09 24 2009 Rev 1 1 11 26 2009 Rev 1 2 05 10 2010 Rev 1 2 1 07 21 2010 Revision Description For Bench4Q Tool 1 0 0 For Bench4Q Tool 1 1 0 For Bench4Q Tool 1 2 X For Server side Resource Monitoring Contents 1 aia go Lo WERO maraton it atenieni sacii n dl la e an teii A 1 1 2 ASIC INE OMe A OM uan 1 1 2 Tarsetee Alldlente nenn ee 1 1 3 Structure of the document 2ccsssssesessnnnnnonnnnnnnnnnnnennnnnnnnnnnnnnnnnnnnnenn 1 2 Bench4Q A QoS Oriented E commerce Benchmark 3 2 OVERVIEW setea an i Genco one iii oa tt a ta 3 2 2 What is o pt a iza ag pt aa 3 ZI NM CGE da 5 3 BEICHAN TOO aenema aia did EE Sp tate nd 6 3 1 QUICK Introductions eee eee eee eee nennen 6 3 2 Downloading Bench4Q naan oneens oneven eee eee 6 3 3 Bench4Q Tool License eee eee eee eee eee nennen 6 3 4 Bench4Q Tool Desiena u alta aaa die du Dna a ba tei 7 32 COs ONG soars PAPES y o On e aj ab aa DA bau i 7 ALA in TN 8 A tri lie da s alle iti ba sali odaia abia 8 IAA Sever MON tan 12 O O y POP RE RER RTR OR RE EI RR AR A RI RI A RI SORTII 13 4 a e o ee aoleu ta i tt 15 4 1 Requirements dia 15 4 2 Bench4Q Tool Files Structure eee eee eee eee ae eee 15 A SON ee E RER E OR E INI ger 15 4 4 How do s2. www w3 org 2001 XMLSchema elementFormDefault qualified attributeFormDefault unqualified gt lt xs element name bench4Q gt lt xs annotation gt lt xs documentation gt lt xs documentation gt lt xs annotation gt lt xs complexType gt lt xs sequence gt lt xs element name testName type xs string gt lt xs element name testDescription type xs string gt lt xs element name rbe maxOccurs unbounded gt lt xs complexType gt lt xs sequence gt lt xs element name interval type xs double gt lt xs element name prepair type xs integer gt lt xs element name cooldown type xs integer gt lt xs element name out type xs string gt lt xs element name tolerance type xs double gt lt xs element name retry type xs integer gt lt xs element name thinktime type xs double gt lt xs element name mix gt lt xs simpleType gt lt xs restriction base xs string gt lt xs enumeration value shopping gt lt xs enumeration value ordering gt lt xs enumeration value browsing gt lt xs restriction gt lt xs simpleType gt lt xs element gt lt xs element name slow type xs double minOccurs 0 gt lt xs element name getlmage 29 Bench4Q Tool Manual type xs boolean gt lt xs element gt lt xs element name baseURL type xs string gt lt xs element name ebs maxOccurs unbounded gt lt xs complexType gt lt xs sequen
3. Bench4Q Tool you can start running tests 5 1 Test Configurations To run a test you need to know the actual meaning of Bench4Q parameters 5 1 1 RBE Type There are two RBE types closed and open CLOSEED Fixed number of user sit at the website forever Each user has the same behavior 1 N 4 N Y gt Get response OPEN Each user visit website make K request get response and then leave f Repeat Ktimes 5 1 2 Mix The percentage of each web interaction executed during each measured interval must following the mix as defined in following table 18 Bench4Q Tool Manual Web Interaction Browsing mix Shopping mix Ordering mix Browse related 95 00 80 00 50 00 Home 29 00 16 00 9 12 New products 11 00 5 00 0 46 Best sellers 11 00 5 00 0 46 Product detail 21 00 17 00 12 35 Search request 12 00 20 00 14 53 Search result 11 00 17 00 13 08 Order related 5 00 20 00 50 00 Shopping cart 2 00 11 60 13 53 Buy request 0 75 2 60 12 73 Buy confirm 0 69 1 20 10 18 Order inquiry 0 30 0 75 0 25 Admin request 0 10 0 10 0 12 Admin confirm 0 09 0 09 0 11 5 1 3 Warmup and Cooldown Warm up Seconds used to warm up the Agents The result of this phase of test is not considered The default warm up time is O Cool down Seconds used to cool down the Agents The result of this phase of test is not considered The default warm up time is O 5 1 4 Address and Port Web Application URL The address wher
4. the bench4Q jar file which can be found in the lib directory java org bench40 Console 4 Start the console on one of the test machines java org bench40 Agent 5 For each test machine do steps 1 and 2 and start an agent process 6 The agent will look for the bench40 properties file in the local directory If you like you can also specify an explicit properties file as the first argument For example 7 The console does not read the bench40 properties file It has its own options dialog choose the help Options menu option which you should use to set the java org bench40 Agent propertieFile communication addresses and ports to match those in the bench40 properties files 8 The console process controls can be used to trigger agent to start test Each agent process then generate load as the console configured 9 After the test done you can collect result of agent then some picture will be showed at the console 4 5 Console and Agent conmmunication Configuration AGENT In bench40 properties file the address of console is configured as follows 16 Bench4Q Tool Manual bench4Q consoleHost 133 133 133 123 bench4Q consolePort 6372 The agent will look for the bench4Q properties file and connect console using this address CONSOLE You can choose the help Options menu option to configure the address of console 217 Bench4Q Tool Manual 5 Bench4Q Tool Usage After you have deployed all parts of
5. CC_XACTS C_EXPIRATION f E C_DISCOUNT COUNTRY f CX_O_ID AVAIL C_BALANCE ii CX_TYPE STOCK C_YTD_PMT CX_CC_NUM ISBN C_BIRTHDATE CX_CC_NAME PAGE C_DATA CO_NAME CX_EXPIRY BACKING CO_CURRENCY CX_AUTH_ID DIMENSION CO_EXCHANGE ata CX_XACT_AMT ADDRESS ae CX_XACT_DATE CX_CO_ID AUTHOR ADDR _ID ADDR_STREET1 SHOPPING CART SHOPPING CART LINE ADDR STREET 7 ADDR_CITY SC_ID ADDR_STATE SC_TIME SCL_SC_ID ADDR_ZIP SCL QTY ADDR_CO_ID SCL ID For each table that composes the database the cardinality of the initial population is specified as follows Typical Row Typical Table Size Table Name Cardinality in rows Length in bytes in bytes CUSTOMER 2880 number of EB 760 2 188 888 k COUNTRY 92 70 6 44 k ADDRESS 2 CUSTOMER 154 887 040 k ORDERS 9 CUSTOMER 220 570 240 k ORDER_LINE 3 ORDERS 132 1 026 432 k AUTHOR 25 ITEM 630 1 575 k CC_XACTS 1 ORDERS 80 207 360 k ITEM 1k 10k 100k 1M 10M 860 8 600 k The following diagram shows the sequencing of web interactions During a User Session a sequence of web interactions will be performed That sequence corresponds to a traversal of this diagram Each node in the diagram contains the name of a web interaction type Home Best Seller Search Request etc An arrow between two nodes A and B i e A gt B indicates that after performing web interaction A it is possible for an Emulated Browser to next perform the web 10 Bench4Q Tool Manual inter
6. Tolerance LT measures the time a customer will wait for a response before change his behavior for example give up the ongoing session Once the request processing time is over its LT the Load Simulator will discard the current session Definition of tenacity for online behavior Web interaction tolerance Home New Products Best Sellers Product Detail Unimportant 8 seconds Search Request Search Results Shopping Cart Customer Registration Buy Request Important Buy Confirm Order Inquiry Order Display Admin 80 seconds Request Admin Confirm This parameter is used to increase gt 1 0 or decrease lt 1 0 think time While the parameter is set to 0 it means there s no tolerance time limitation 21 Bench4Q Tool Manual 5 2 Metrics Analyzer TPC W mainly focuses on performance metrics which include hit ratio per second WIPS and response time WIRT Although these metrics are important to evaluate the performance of middleware the QoS metrics about the B2C business should be more concerned The main QoS metrics should include Session based metrics Compared with performance metrics a B2C business concerns more that whether sessions especially profit sessions can be completed Theses metrics include total sessions failure sessions profit sessions failure profit sessions and the most possible requests leading to failure sessions Customer classes based metrics QoS differentiation is a popular re
7. action B When there are multiple arrows leaving a node the arc that is chosen for the next web interaction is determined probabilistically Start User Session Home lt Name gt Button name 3 Web Interaction transition via button gt Web Interaction transition via HREF link lt Search gt lt Search gt Admin Confirm lt Submit gt m Admin lt Search gt Request lt Search gt lt Submit uery gt Search Result a Query Best Seller NP CURL New Product lt Add to Cart gt Shopping Cart lt Confirm Updates gt lt Admin gt Product Detail lt Search gt Search Request lt Shopping Cart gt lt Shopping Cart gt lt Checkout gt lt Search gt Customer Registration lt Continue Buy gt Returning Customer Buy Request lt Confirm Buy gt lt Continue Buy gt Not Returning Customer lt Shopping Cart gt lt Search gt Order Inquiry lt Order Status gt lt Display Last Order gt Order Display lt Search gt 11 Bench4Q Tool Manual 3 4 4 ServerMon ServerMon is a server side resource monitoring tool which should be deployed on both web application server and database server It is written in JAVA and C The JAVA code uses JNI to invoke native methods written in C to collect the resource monitoring information and it exposes an RMI interfa
8. am shows the speed of disk write during the measurement interval The units of the speed are bytes per second Network Bytes Received sec 10 000 000 1 9 500 000 9 000 000 8 500 000 8 000 000 4 7 500 000 4 7 000 000 4 6 500 000 4 6 000 000 5 900 000 5 000 000 4 byte s 4 000 000 4 3 500 000 3 000 000 2 500 000 y 2 000 000 1 500 000 1 000 000 4 500 000 y 0 time s Basic test Max 9974858 1 Min 1 0 Average 4669574 3 StDev 3548880 3 Network Bytes Received sec this diagram shows the speed of server received data from the network during the measurement interval The units of the speed are bytes per second 27 Bench4Q Tool Manual Network Bytes Sent sec Y IDO ODO IPD ADO AUD VUY UUU RIO UU byte s DUO UU 2 UU VUN time s Basic test Max 1 07385629E7 Min 135341 4 Average 5089821 9 StDev 3901553 3 Network Bytes Sent sec this diagram show the speed of server sent data to the network during the measurement interval The units of speed are bytes per second 6 Getting Involved Any input or personal view for improving and or developing Bench4Q is welcome Bench4Q as an open source project also welcomes external contributions 28 Bench4Q Tool Manual APPENDIX The schema of Bench4Q Tool s parameters lt xml version 1 0 encoding UTF 8 gt lt xs schema xmins xs http
9. ce for the console to invoke and retrieve the monitoring data The types of data ServerMon monitors are CPU usages free physical memory disk bytes read per second disk bytes written per second network bytes received per second and network bytes sent per second The console invokes the methods of the ServerMon through RMI to collect the resource monitoring data ServerMon can be deployed under both Windows and Linux The following is the window of ServerMon under Windows es Cr WIRDOWS syst em32 cad exe E ServerMon java Dservermonregport 10724 jar ServerMon jar The server monitor has started successfully You can connect to port 18724 through RMI at the client side and lookup Server DataManager Type exit to quit 49 Bench4Q Tool Manual The following is the window of ServerMon under Linux usera fedoral3 AM SserverMon EU SF ARE SEV AT FPBA H user fedoral3 ServerMon run sh Below are the IP addresses of this machine B 155 135 155 227 1 127 0 0 1 Please choose the one the client will connect to Enter the number you choose e g 0 1 etc 0 You choose the IP address 133 133 133 227 The server monitor has started successfully You can connect to port 18724 through RMI at the client side and lookup Server DataManager pre exit to quit 3 4 5 ClusterMon ClusterMon is an extension of ServerMon which gives the resource monitoring information of a server clust
10. ce gt lt xs element name baseLoad type xs integer gt lt xs element name randomLoad type xs integer gt lt xs element name rate type xs integer gt lt xs element name triggerTime type xs integer gt lt xs element name stdyTime type xs integer gt lt xs sequence gt lt xs complexType gt lt xs element gt lt xs sequence gt lt xs attribute name rbetype gt lt xs simpleType gt lt xs restriction base xs string gt lt xs enumeration value open gt lt xs enumeration value closed gt lt xs restriction gt lt xs simpleType gt lt xs attribute gt lt xs complexType gt lt xs element gt lt xs sequence gt lt xs complexType gt lt xs element gt lt xs schema gt 30
11. cts Unlike traditional Web system e commerce system generates state full responses according to sequences of requests We refer to this class of systems as session based systems As a session based system e commerce system is sensitive to QoS guarantees The workload for these systems must be characterized with respect to sessions However the current benchmark specification such as TPC W is not competent to be the QoS oriented e commerce benchmark The reasons were as follows 1 TPC W can t simulate Internet customers which are unlimited unrelated to each other and sensitive to QoS 2 TPC W mainly concerns performance metrics such as hit rate and response times but incapable of analyzing QoS metrics of E commerce business such as session related metrics Bench4Q offers a better way to evaluate the QoS guarantees of e commerce system infrastructures Bench4Q Tool Manual 3 Bench4Q Tool 3 1 QUICK Introductions Bench4Q Tool is designed for Bench4Q benchmarking Bench4Q Tool offers a convenient way to configure the test and analysis the test result 3 2 Downloading Bench4Q Bench4Q is available on the Internet at http www trustie net projects files list bench4Q rm You can find latest version there Bench4Q is distributed as zip file which you should expand using unzip WinZip http www winzip com or similar 3 3 Bench4Q Tool License Bench4Q Tool is distributed according to the GNU Lesser General Public Licen
12. d Bench4Q Bench4Q as a QoS oriented e commerce benchmark has features to deduce a controllable and flexible representation of complex session based workloads and to simulate authentic customer behavior What s more we show that Bench4Q can be used to evaluate the system s performance and scalability Bench4Q has made many extensions of TPC W specification for measuring QoS guarantees of e commerce system infrastructures especially of load simulation and metrics analysis 2 2 What is TPC W TPC Benchmark W TPC W is a transactional web e Commerce benchmark introduced by the Transaction Processing Performance Council TPC W specifies an e Commerce workload that simulates the activities of a retail website which produces heavy load on the backend database The workload is performed in a controlled internet commerce environment that simulates the activities of a business oriented transactional web server The workload exercises a breadth of system components associated with such environments which are characterized by O Multiple on line browser sessions Bench4Q Tool Manual e Dynamic page generation with database access and update O Consistent web objects O The simultaneous execution of multiple transaction types that span a breadth of complexity O On line transaction execution modes O Databases consisting of many tables with a wide variety of sizes attributes and relationships O Transaction integrity ACID p
13. e deploy the SUT system under test project The IP address of web application server will be extracted from this address Web Application Server Monitoring Port The port of serverMon running on the web application server The default port is 10724 which usually need not to be changed unless the port has already been occupied Database Server Address The IP address of the database 19 Bench4Q Tool Manual Database Server Monitoring Port The port of serverMon running on the database server The default port is 10724 which usually need not to be changed unless the port has already been occupied 5 1 5Load Fluctuation Control The load fluctuation represents the concurrency feature of business and it has great effects on the choice of tuning policy of a B2C server Bench4Q simulate such load fluctuation by accumulating the simulated loads from multiple Load Workers Each Load Worker simulates a controlled load by the following parameters Y v v Y Y Base Load Fixed amount of EBs the Load Worker will be generated in each period during the test Radom Load Random amount of EBs will be generated with the bLoad Rate The rate of base load changes It can be positive negative or zero It s positive means the base load is increasing every second It s negative means the base load is decreasing every second While the rate is zero the load is fixed Trigger Time The time for the Load Worker to start generate its
14. eb application looks for datasource using JNDI so you need to configure database properties file in WEBAPPROOT WEB INF classes org bench4Q servlet Copy ServerMon or ClusterMon to the machines of IBM DB2 and JavaEE Application Server 15 ARE XZ1 TODO Add the ServerMon and EJB3 related content Bench4Q Tool Manual 4 4 How do I start Bench4Q It s easy 1 Start ServerMon or ClusterMon on the web application server cluster and the database server cluster Under Windows double click the run bat Under Linux start a shell type run sh Maybe you should first configure the firewall to let go of ServerMon s or ClusterMon s network interaction When using ClusterMon choose the node the client will connect to as the leader If both the application servers and the database servers are clusters and both clusters are on the same LAN then there should be only one leader you can do it this way make the node whose IP is in the Web Application URL textbox of the client tool the leader and every other application server node and database server node the subordinate and when benchmarking finishes all the nodes are treated as application server nodes Also you can easily make the client treat all the nodes as database server nodes It does not matter 2 Create a bench4Q properties file This file specifies general control information how the worker processes should contact the console 3 Set your CLASSPATH to include
15. er ClusterMon should be deployed on every node of the cluster And one of the ClusterMon instance should be designated as the leader with others being subordinates Subordinates send monitoring information of their own respective nodes to the leader at intervals and the client connects to the leader to retrieve monitoring information of the cluster in the same way as ServerMon The leader uses IP multicast to inform subordinates of its address so the cluster s underlying network must support IP multicast A cluster in which nodes are connected by a LAN certainly meets this need ClusterMon can be deployed under both Windows and Linux The following is the window of ClusterMon under Windows 13 Bench4Q Tool Manual ci Cz FINDOFS systen32 cad exe Mill this instance cluster server monitor be a leader please type Y or N for veg s or No n This instance will be a subordinate You can type exit to quit at any time Looking up a leader Succeed in finding a leader at 133 133 133 231 Now sending monitor data to the leader continually The following is the window of ClusterMon under Linux User redoral3 HF ABE EEV 35M AEH usertfedoral3 ClusterMon run sh Will this instance cluster server monitor be a leader please type Y or N for Ye s or No y Below are the IP addresses of this machine B 155 155 155 251 1 127 0 0 1 Please choose the one the client will connect to Enter the number you cho
16. es PicoContainer http picocontainer codehaus org See the file LICENSE PicoContainer for license and copyright details 3 4 Bench4Q Tool Design Bench4Q tool composed of three parts console agent and SUT system under test showed in the following picture Bench4Q Tool HTTP Request Application HTTP Server Database response Resource Monitor Data 3 4 1 Console The console configures the test collate and display result The main frame of console windows is showed in the following picture Bench4Q Tool Manual The Bench4Q Console Plan Result Action Help Deaa ana EF Global setting 9 T Load Simulator RBE type closed y FR LoadWorker ET User setting FX the load started 9 E Analysis Matrix mix Shopping FR Request i Different interaction mixes can be selected lali Session Closed means fixed number of users sit at the system forever like TPC W s workload generator Open means startiny some EB every interval each one visit web site just once E Ertor Web Application URL http 1 133 133 133 152 bench4Q o 7 Web App Serve Fi Database Send EE AGENT MICROSOF 11 Use EJB3 based SUT B Load Simulator EN LoadWorker warm up time 0 P User setting PR the load started 2 Analysis Matrix E PR Request lads Session E Error vl Monitor t
17. he Web Application Server The root URL for the Bench4Q SUT system under test Seconds used to warm up Bench40 Measured in second cool downtime 0 Seconds of steady state operation following measurment interval Measured in second Web Application Server Monitoring Port 10724 The port of the serverMon running on the web application server Monitor the Database Server Database Server Address Database Server Monitoring Port 3 4 2 Agent The agent is the real work part It follows the console s instrument and generates load as the console configured For heavy duty testing you can start an agent on each of several load injector machines The stats of agent can be controlled using the console There is little reason to run more than one agent on each load injector but you can if you wish 3 4 3 SUT E commerce must provide a system such as e commerce website in order to function Traditional e commerce system is structured on some infrastructures such as database web server and application server The common architecture shows as following picture Bench4Q Tool Manual Business Logic Tier Presentation Tier oy AS AS AS SUT System Under Test contains the business logic to simulate the tested scenario The SUT comprises all components which are part of the application bei
18. load Duration The duration for the Load Worker to generate its load By composite the simulated loads from multiple Load Works the following typical load fluctuations of B2C businesses can be simulated Load 1 Load 1 y f o Test Bda o Test Baraton a b Load Ba N A MAMA c d Fig 6 Some typical load fluctuations of B2C 5 1 6 Think Time The Think Time TT is defined by 20 Bench4Q Tool Manual TT T2 T1 where T1 and T2 are measured at the EB T1 time measured after the last byte of the last web interaction is received by the EB from the SUT and T2 time measured before the first byte of the first HTTP request of the next web interaction is sent by the EB to the SUT Each Think Time must be taken independently from a negative exponential distribution Each Think Time T must be computed from the following equation T In r u where In natural log base e r random number with at least 31 bits of precision from a uniform distribution such that O lt r 1 u 7 to 8 seconds inclusive The parameter is used to increase gt 1 0 or decrease lt 1 0 think time While the parameter is set to 0 it means there s no think time 5 1 7 Tolerance Time Patience customer tenacity and importance of transaction affect the behavior of a B2C customer Correspondingly we extend the TPC W by simulating QoS sensitive behavior of B2C customers as follows Latency
19. ng simulated This includes network connections Web Servers application servers database servers etc The SUT of Bench4Q is based on a distribution of the TPC W Benchmark Java Implementation originated of PHARM at the University of Wisconsin Madison The SUT of Bench4Q is an online bookstore which is a typical B2C application It simulates 14 business operations 8 of which involving database interactive The main features of this version are listed as follows 1 This version implements TPC W spec version 1 8 2 All VARCHAR database rows are limited to size lt 255 3 In the ITEMS table THUMBNAIL and _ IMAGE are varchars keeping the name of the image file 4 There are 1000 item images and thumbnails required 5 No Payment Getaway Emulator is used clause 2 7 3 3 so the CC_XACTS credit card processing record is created right away during the order creation 6 No SSL is used all connections are insecure The components of Bench4Q database are defined to consist of ten separate and Bench4Q Tool Manual individual base tables The relationships among these tables are defined in the entity relationship diagram shown below CUSTOMER ORDERS ORDER_LINE ITEM C_ID OL_ID IID C_UNAME OL_O ID TITLE C_PASSWD OL_I_ID LA ID C_FNAME OL_QTY PUB_DATE C_LNAME OL_DISCOUNT PUBLISHER C_ADDR_ID u OL_COMMENT SUBJECT C_PHONE O_SHIP_TYPE DESC C EMAIL O_SHIP_DATE i RELATED 1 5 C_SINCE THUMBNAIL nn Hl
20. nse Time WIRT showed in CDF From the CDF diagram user can get the basic Web Interaction Response Time situation The Web Interaction Response Time WIRT is defined by WIRT T2 T1 where T1 and T2 are measured at the EB T1 time measured before the first byte of the first HTTP request of the web interaction is sent by the EB to the SUT T2 time measured after the last byte of the last HTTP response that completes the web interaction is received by the EB from the SUT m ession 3 750 3 500 3 250 3 000 2 750 2 500 2 250 2 000 1 750 1 500 1 250 1 000 750 500 250 O Session number total Error Occur servlet Session type total Complete Session M total Error Session W init E admc M admr bess M buyc buyr Screg Bhome Mnewp Mordd Bord M prod Msreq Wsres M shop Session this diagram shows the basic information of session including total successful session error session benefit session etc 24 Bench4Q Tool Manual Error Error number o a Errors of Servlet Error type E init E admc admr M bess E buyc buyr creg home newp E ordd Mordi M prod E sreq Bsres M shop ERROR this diagram shows the error happened during the measurement interval The diagram catalogs the errors by web interaction 5 3 Server side Resource Monitor The bench4Q monitors the status of database server and web application server respectively CPU Usage time s Basic tes
21. ose e g 0 1 etc 0 You choose the IP address 133 133 133 231 This instance will be a leader The server monitor leader has started successfully The client may connect to port 18724 through RMI and look up ServerDataManager You can type exit to quit at any time 14 Bench4Q Tool Manual 4 Getting Started In order to run a Bench4Q test you will need to get some basic information of the Bench4Q Tool 4 1 Requirements As Bench4Q tool is written in Java to run Bench4Q tool a Java Virtual Machine JVM is needed A IBM DB2 and the JDBC driver are needed A JavaEE Application Server is needed 4 2 Bench4Q Tool Files Structure documentation steps amp tips Also some background Doc documentation The SUT System Under Test in this application it is a online ol book store web project Bench4Q Tool Bench4Q Tool it is the load generator DB2 Generator The jar used to generate DB2 data 4 3 Installation 1 2 Everything required to run The Bench4Q is in the zip file labelled grinder version zip all you need to do is unpack the package Then you need to install a IBM DB2 JDK should also be installed on the DB2 server and a JavaEE Application Server Initiate the database using DB2 Generator First you need to create a database and then configure the db properties file Then run the DBPopulate jar file This process may take several hours Deploy the SUT on your JavaEE Application Server The w
22. quirement of a B2C business which means that an important customer should receive better QoS than a less important customer especially when the B2C server has been overloaded So benchmark should provide QoS metrics according to the customer classes Profit based metrics This is the final target of a B2C business 11 so the B2C benchmark should provide the profit based metrics such as the total profits gained and the profits lost All the benchmarking result can be showed in picture format showed as follows WIPS WIPS 34 WIPS web interactions processed per second WIPS are computed as the total number of Web interactions requested and completed successfully within a measurement interval divided by the length of that interval in seconds 22 SPS Bench4Q Tool Manual Basic real SPS session completed per second SPS are computed as the total successful session within a measurement interval divided by the length of that interval in seconds Session number E Session 1 500 1 400 1 300 1 200 1 100 1 000 900 800 700 600 500 400 300 200 100 Session length Session type 38485868798 89 810 11 M12 Session length session length shows the session number divided by the session length 23 Bench4Q Tool Manual WIRT CDF percent 0 2 50 75 100 125 150 175 200 225 250 275 300 325 350 375 400 425 450 475 time ms WIRT Web Interaction Respo
23. roperties O Contention on data access and update The performance metric reported by TPC W is the number of web interactions processed per second Multiple web interactions are used to simulate the activity of a retail store and each interaction is subject to a response time constraint TPC W simulates three different profiles by varying the ratio of browse to buy primarily shopping WIPS browsing WIPSb and web based ordering WIPSo The primary metrics are the WIPS rate the associated price per WIPS S WIPS and the availability date of the priced configuration Users of the TPC W benchmark can browse and order products from the website In the case of TPC W the products are books The expected introduction or Home page is the first page user will see It includes the company logo promotional items and navigation options to the top best selling books a list of new books search pages client s shopping cart and order status pages User can browse pages containing a list of new or best selling books grouped by subject or perform searches against all books based upon a title author or subject A product page will give you detailed information for the book along with a picture of the book s front cover User may order books by entering the order web pages If user is a new customer s he will have to fill out a customer registration page while for returning customers their information will be retrieved from the database and filled in automa
24. s a typical and impartial way to evaluate the e commerce system is a key matter of people s concerns However the current e commerce benchmarks that mainly concerning on transaction is lack of cares about customer s behavior And they are lack of ability to measure the e commerce s characteristic Therefore the current e commerce Benchmark isn t an effective way to evaluate QoS guarantees of an e commerce system Bench4Q is a new methodology for QoS oriented e commerce benchmarking Bench4Q has features to deduce a controllable and flexible representation of complex session based workloads and to simulate authentic customer behavior What s more we show that Bench4Q that mainly concerns on session profit and QoS guarantee can provides an overall view of the system s performance Bench4Q Tool brings more conveniences in Bench4Q benchmarking With this tool we are offering a simply way for installation configuration and metrics analysis 1 2 Targeted Audience This document is targeting two types of audience O People who just want to use right away the Bench4Q Tool This is for those who will use the Bench4Q Tool to benchmarking the middleware O People who would like to modify Bench4Q Tool to fit their particular needs You may want to change a little bit our Bench4Q Tool to add some functionality or replace a component with another one 1 3 Structure of the document This document will walk you over O A brief introduc
25. se Bench4Q Tool is a free software you can redistribute it and or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation either version 2 1 of the License or any later version This source code is distributed as is in the hope that it will be useful It comes with no warranty and no author or distributor accepts any responsibility for the consequences of its use This version is a based on the implementation of TPC W from University of Wisconsin Madison See COPYRIGHT file of COPYRIGHT TPC W in Java distribution for license and copyright details This version used some source code of The Grinder See COPYRIGHT file of COPYRIGHT The Grinder for license and copyright details Redistribution and use in source and binary forms with or without modification are permitted provided that the following conditions are met 1 Redistributions of source code must retain the above copyright notice this list of conditions and the following disclaimer 2 Redistributions in binary form must reproduce the above copyright notice this list of conditions and the following disclaimer in the documentation and or other materials provided with the distribution 3 Neither the names of the copyright holders nor the names of the Bench4Q Tool Manual contributors may be used to endorse or promote products derived from this software without specific prior written permission Bench4Q Tool includ
26. t Max 53 9 Min 0 0 Average 24 8 StDev 16 8 CPU Usage this diagram shows the CPU Usage of the server during the measurement interval 25 Bench4Q Tool Manual Free Physical Memory 14 000 1 13 000 1 12 000 y 11 000 10 000 1 uuu 7 000 4 MBytes 6 000 1 5 000 4 4 000 4 3 000 2 000 4 1 000 4 DO time s Basic test Max 2750 0 Min 2740 0 Average 2746 6 StDev 1 8 Free Physical Memory this diagram shows the free physical memory of the server during the measurement interval DISK Read Bytes sec 19 000 000 18 000 000 17 000 000 16 000 000 15 000 000 14 000 000 13 000 000 2 000 000 11 000 000 y lt 10 000 000 4 S 9 000 000 A 7 000 0 8 000 000 7 000 000 6 000 000 5 000 000 4 000 000 3 000 000 2 000 000 1 000 000 time s Basic test Max 1 81621475E7 Min 0 0 Average 202506 5 StDev 1748989 7 DISK Read Bytes sec this diagram shows the speed of disk read during the measurement interval The units of the speed are bytes per second 26 Bench4Q Tool Manual DISK Write Bytes sec 260 000 4 240 000 1 200 000 y 180 000 4 160 000 y 140 000 7 byte s 120 000 4 100 000 4 80 000 60 000 4 40 000 20 000 1 O time s Basic test Max 260039 0 Min 0 0 Average 8660 2 StDev 29084 3 DISK Write Bytes sec this diagr
27. tart Bench4Q c eee eee eee eee eee eee aeeae eee 16 4 5 Console and Agent conmmunication Configuration 16 5 Bench4Q Tool Usage ccccooccccccnonccnncnnncnnnonanonnnnnnanonononnronnnonnronnnononannnnnnanos 18 Be Test CONT SULA ONS siria 18 SABE e PRR RR ARI RR eben IRI IRI 18 O an a E E tema le daia a ne 18 5 1 3 Warmup and Cooldown cccccccsssscccceessccceceeeceeeaeseceesaueeees 19 5 1 4 Address and Port une aaa 19 5 1 5Load Fluctuation Control ooooooccccccnonnncnnnnnononincnnnnannnononnnos 20 SLETE Te en 20 5 1 7 Tolerance Time aaan ee ee eee eee ee eee eee eee 21 5 2 Metrics Analyzer sonnet ea d iad aaa ara a id 5 3 Server side Resource Monitor aaneen ennen eneensnnensen 6 GELLING INVOIVEG sarna aia APPENDI OE ASEPE not done descaro ad Cena ee sali Daca de dl lo Za ee e ae bat Bench4Q Tool Manual 1 Introduction 1 1 basic information E commerce as a very popular business model has grown substantially with the development of Internet technology In the meantime the QoS of e commerce critical to the success of e commerce systems is a major factor in the success of competition While the infrastructures of an e commerce system such as application server play an important role in QoS guarantees how to evaluate the performance of these infrastructure techniques has become the focus of the issue Benchmark a
28. tically for them User can change the quantity of the order or delete a book from the shopping cart When user wishes to buy s he enters credit card information and submits the order The system will obtain credit card authorization from a Payment Gateway Emulator PGE and present the user with an order confirmation page At a later date user can view the status of his her last order Two additional web pages are provided for the system administrator to change book s front cover picture and price This change is reflected in the new product book list Essentially the TPC W benchmark provides basic functionality required of an Internet e Commerce website Bench4Q Tool Manual TPC W benchmark specifies the application database schema the set of 14 web interactions web pages and how their execution manipulates application data It also specifies data consistency and transactional requirements database scaling and population and various other aspects e g which web interactions must be covered by SSL However TPC W neither provides sample implementation of the application nor it prescribes any implementation methodic or limits it to a specific technology Besides the application structure TPC W specifies all aspects of the workload profile performance metrics of interest and required structure of the official disclosure report 2 3 Why Bench4Q Benchmarks have been the effective measures to evaluate performances of middleware produ
29. tion to Bench4Q in section 2 for people who have never 1 Bench4Q Tool Manual used Bench4Q before A brief introduction to Bench4Q Tool in section 3 Some basic information about Bench4Q Tool in section 4 Bench4Q Tool Usage in section 5 Bench4Q Tool Manual 2 Bench4Q A QoS Oriented E commerce Benchmark 2 1 Overview E commerce the use of the Internet to buy and sell goods and services asa very popular business model has grown substantially with the development of Internet technology In the meantime the competition of e commerce business has been more and more intense E commerce must provide a system such as e commerce website in order to function We have shown that in the context of web based systems characterized by highly variable workload it is difficult to find out how to make best use of current resource and how to find out the bottleneck Finding a simple way to help e commerce system provider with making decision has become the focus of the issue As a main way to evaluate the performance of e commerce system benchmark has become a key matter of people s concerns The current e commerce benchmark such as TPC W mainly concerns on transaction Without the ability of measuring QoS guarantee it may mislead the user By investigating the current e commerce Benchmark we find out the insufficient of them in measuring QoS guarantees Then we introduce a new methodology for QoS oriented e commerce benchmarking name
Download Pdf Manuals
Related Search