Home

Analysis, design and development of an automated testing platform

1. a List of tests AAA http 192 168 90 137 atp query manage_tests amp filter http Manage tests Manage test areas List executions Show statistics COTTON EE CON Manage campaigns http light page Simulates an user viewing a light sized page Services E http 1mb page Simulates an user viewing a 1MB sized page Services E http 10mb page Simulates an user viewing a 10MB sized page Services ra http 100mb page Simulates an user viewing a 100MB sized page Services TOR b Search tests ALFA 4 http 192 168 90 137 atp query manage_tests amp filter http Manage tests Manage test areas Manage campaigns List executions Show statistics EXE AAA re EA http light page Simulates an user viewing a light sized page Services ru http 1mb page Simulates an user viewing a 1MB sized page Services Remove http 10mb page Simulates an user viewing a 10MB sized page Services ne PP 4 Remov http 100mb page Simulates an user viewing a 100MB sized page Services E Test name New test here Description Test description Test area Backup and Restore Press here to add a new test c Add new test O O fa a W a Go http 192 168 90 137 atp index php query manage_tests8filter new test Manage tests Manage test areas List executions Show sta
2. admin login c Creation of a new campaign a a http 192 168 90 137 atpjindex php query manage_sets amp execute 10 Manage tests Manage test areas Manage campaigns List executions Show statistics Ol Login test Ci Backup and restore tests O pucr Olurtp Testing Certification and Benchmarking sl Execute Test set DHCP Target machine model Dell 830 3 Version 41 a IP Address Allow to execute campaigns in paral Allow to execute campaigns in paz Execution strategy Run campaign sequentially Number of executions Press here to execute the test set d Execute a campaign Figure 7 4 Manage campaigns interfaces 47 CHAPTER 7 DOCUMENTATION 48 Manage tests Manage test areas Manage campaigns Listexecutions Show statistics cription No test sets in execution Test set description Login test 41 192 168 90 135 Login test 41 192 168 90 135 DHCP 41 192 168 100 254 2006 08 09 14 53 55 141 938 5 DHCP r 192 168 100 254 2006 08 09 14 47 02 140 65 DHCP Em 192 168 100 254 2006 08 09 14 30 43 138 104 s HTTP 41 192 168 100 254 2006 08 09 14 27 08 13 2124 s HTTP 41 192 168 100 254 2006 08 09 14 27 08 13 3573 s
3. Bugzilla is used as a bug tracking tool originally developed and used by the Mozilla Foundation Figure 2 9d Sugar is a Costumer Relationship Management CRM system and is used to track communication with clients Figure 2 9e Enterprise Architect is used for UML design and construction including Use Case Logical Dynamic and Phys ical models Also allows process modeling and project documentation Figure 2 9f 12 CHAPTER 2 THE PRODUCT A NETWORK APPLIANCE RE Data lT Office Converged in a box office in a box edgebo 1100 2500 ENetD 15000 20000 m7 y nl ALCATEL networks 7000 3000 4800 1 0800 Linksys 3000 1 0000 400 300 Cisco SYSTEMS 1 400 683600 NORTEL AVAYA 1600 3900 1 900 5000 ADIRAN 4000 5000 ay zurrys 1800 3200 Multifunction router s IP PBX Voice Figure 2 10 Product comparison 2 33 Comparison with other products Although the market of converged offices in a box is extremely competitive the edeeBOX offers a very complete solution with several unique selling points The edgeBOX focus on service providers and is designed from scratch with end to end support for managed services through Items It has a software approach using COTS hardware enabling shorter period development and carrier certification cycles It is the most comprehensive and integrated IP I
4. Faculdade de Engenharia da Universidade do Porto Licenciatura em Engenharia Inform tica e Computac o Analysis design and development of an automated testing platform for a multi function network appliance at Critical Software LEIC curricular internship report 2006 Rui Andre Augusto Ferreira Supervisor at University of Porto Prof Jorge Barbosa Supervisor at Critical Software Alexandre Esper September 2006 To my family Abstract abstract aqui CONTENTS Contents Preface v Acknowledgements ix Guide to readers xi Acronyms xiii 1 Introduction 1 1 11 Theieompany s A ke AE A S SS tee 1 1 41 Background ii ds aS Qa hae eee dat ba h ue ha 1 112 Company profile as a A ae A A a A 1 11 37 Organization iii A A AE Seeks A A li 1 1 2 Overview of the product 2 020240 ds a ker na Q b de E 3 1 21 ed BOX MSA A eh ets Bt dr e A as PS IEA 3 1 22 A BE WR een dites 3 1 3 The Internship y ta se ee han kr 3 1 341 Global goal ren 22 gt eek ar ee A a wer A 3 1 3 2 Workplace oi en Br RD See klarer 4 13 37 Objeetvesa Ks enter hate 4 1 3 4 Framing of the internship in the product 4 13 50 Tasks 22 5 Bear BE DB A Bens 4 1 3 6 Planning A are Becher 5 2 The product a network appliance 7 2 1 Introduction a A BE eae Sow me A A q oa 7 2 1 1 Market opportunity 7 22 Market Size sn ss ee A de tatin NQ N 7 2 1 3
5. Manage test case Administrator Add new testing area Manage testing area Edit testing area Figure 4 3 Administrator s use cases The list of use cases available to the administrator Figure 4 3 are here defined This list has the requirements that have very high priority and are definitely to be implemented as the prototype of the automated testing platform Add new test case The stubs of a new test case can be added through the interface After the administrator has to complete the test through another mean e g command line text editor offline to the the platform 24 CHAPTER 4 REQUIREMENTS ANALYSIS Edit test case The name description and properties of a test case can be edited through the interface Configure the settings of the platform Configure the properties of the automated testing platform 4 5 2 Example automated test implementations required With the objective of demonstrating the application execution a small set of automated tests is to be developed Services Test Priority 1 Difficulty 1 Users Management Priority 1 Difficulty 2 Authentication Priority 1 Difficulty 3 QoS Priority 2 Difficulty 3 SMTP Priority 2 Difficulty 2 FireWall Priority 2 Difficulty 4 Backup Priority 1 Difficulty 4 4 5 3 Usage scenarios Add new testing area Add new test case Implement test c
6. HTTP or Windows file server with configurable disk quotas Web Caching Provides cached content for fast access and bandwidth saving Ethernet LAN WiFI 802 11b g Ethernet Cable USB External Storage ADSL modems RS232 console enterprise WAN Internal ADSL modem USB ADSL modem WAN UMTS Figure 2 Connects to the LAN WAN and acts as a VoIP Gateway Enterprise Grade Security edgeBOX contains a range of security features designed to protect the network from outside attack and internal misuse including Stateful Inspection Firewall for enterprise grade security for data and applications connected via physical links and VPNs Protects the network from all common types of threat such as suspicious packets and denial of service attacks Customisation options and detailed event logs included Authentication at Firewall enables user authentication at firewall level before granting access to services and resources Enables policy enforcing and selective access based on user machine group separately for the WAN LAN and EWAN Network Access Policies provide fully configurable and audited internet access policies for the WAN and EWAN by user machine group Each may have different QoS classes access restricted to specific time periods restrictions on service access e g VolP mail web Virtual Private Networks VPNs with IPSec allowing the secure connection of private networks through the internet ci
7. 35 CHAPTER 5 ARCHITECTURE AND DETAILED DESIGN Automated Test Case Testing logic System API Network Operations Configuration API _ Firewall Settings Figure 5 7 Components of an automated test case usr bin perl use ATP atp ATP gt new ARGV 0 start the communication with the server if atp login login as an administrator exit 1 failed to login this error is reported automagicaly my id atp gt startStep Turn on NAT declare a new test with a purpose atp gt setNatOn use the configuration API to turn the network transation on if atp natlsOn verify current status atp gt endStep id 0 Succeeded to turn NAT on report success atp gt end exit 0 atp gt endStep id 1 Failed_to_turn_NAT on report falure atp end exit 1 Figure 5 8 Example of an automated test 36 CHAPTER 6 TESTING AND SOFTWARE QUALITY ASSURANCE Chapter 6 Testing and software quality assurance An effective way to test code is to exercise it at its natural boundaries Brian Kernighan Program testing can be used to show the presence of bugs but never to show their absence Edsger Dijkstra Beware of bugs in the above code I have only proved it correct not tried it Donald Knuth 6 1 Scope of testing This chapter gives an overview of the software quality assurance used in all the development process the
8. software system Testing campaign A testing campaign is the execution of a set of the test suite with the intention of testing a functionality or a set of functionalities 3 2 Problem domain Figure 3 1 represents the domain of automated testing platforms and clears the relations among concepts explained in the previous section A test case is a member of a testing area Testing campaigns contain several test cases and can be executed in a machine with the intention of testing the machine and the version of a functionality A test case execution is composed by test steps that can have different proposes according to the result of previous test steps This is the most common domain but some variations also are usual Sometimes a test definition has a static set of tests that in contrast with the previous definition are not defined in run time It is also happens that sometimes a test case definition belongs to more than one testing areas 3 33 State of the art Considering the previous overview this section will explore what is the state of the art mainly but not only in open source tools that target the same global objectives The descriptions shown here consider the product s homepage public forums and if it was possible trials 17 CHAPTER 3 THE PROJECT AUTOMATED TESTING PLATFORM TestDirector TestDirector is a proprietary management tool for all aspects of software testing ranging from capturing require ments stori
9. EBOs With typically low Information Technology IT budgets and lack of specialized IT know how SMEs and EBOs need to have access to proper network services The use of the IT infrastructure to compete the need for security the growing number of home workers and the market shift to VoIP will be key instruments and challenges to the future of the world s companies 2 1 2 Market size InSTAT MDR estimates that the business gateways market will be worth 1 6bn by 2010 Figure 2 1 Worldwide broadband Costumer Premises Equipment CPE revenues grew 123 last year to reach 4 9bn in 2005 and it CHAPTER 2 THE PRODUCT A NETWORK APPLIANCE 2 000 Low End Business 1 500 Gateway 1 000 E High End Business Gateway 500 ii NT l 2005 2006 2007 2008 2009 2010 Source InStat MDR Report IN0501821LN Figure 2 1 Worldwide business gateway revenue forecast EM 2005 2010 25 000 Microfirms 20 000 m Small and midsize firms 15 000 Small large firms 10 000 Meddium large firms 5 000 0 2003 2004 2005 2006 2007 2008 Source Forrester Research Inc Figure 2 2 European managed services revenue forecast EM 2005 2010 is expected to reach 6 5bn in 2009 VoIP usage including wholesale retail consumer and retail business was estimated to top 2 4 trillion minutes in 2005 Critical is well positioned to have a important worldwide stake of this market that as figures 2 2 and 2 3 show has a huge dimension 2 1
10. Edit Execute pucr Remove Edit Execute Gurte Remove Edit Execute a List of campaigns ttp 192 168 90 137 atp index php query manage sets Ela h dex ph Manage tests Manage test areas Manage campaigns List executions Show statistics Remove O Login test Edit Execute Remove Backup and restore tests Edit Execute O pucr rest name Test Descriytlo UU alii dhcp range Changes dhep ip range es dhcp lease Asks five times for IP address and checks if it is in the right DHCP range Ourte admin login Tries to login in the edgebox as an administrator Remove http light page Simulates an user viewing a light sized page Edit http 1mb page Simulates an user viewing a 1MB sized page http 10mb page Simulates an user viewing a 10MB sized page imulates an user viewing a 100MB sized page b Expanded list of campaigns Manage tests Manage test areas Manage campaigns List executions Show statistics O Login test ecuti Backup and restore tests Remove Execute O pucr Remove Edit Execute Gurte Remove Edit Execute Description backup http 1mb page New test here Y dhcp lease dhcp range N R http light page 4 Press here to add a new test
11. HTTP 41 192 168 100 254 2006 08 09 14 27 08 13 3699 s a List of executions Manage tests Manage test areas Manage campaigns List executions Show statistics Testing Certification and Benchmarking Test set execution gt x s DHCP 4 1 dhcp range Changes dhcp ip range 16 479 s dhcp lease Asks five times for IP address and checks if it is in the right DHCP range 125 442 s b Add new area 9 http 192 168 90 137 atp index php query exec_details_end amp id 95 Manage tests Manage test a Manage campaigns List executions Show statistics Testing Certification and Benchmarking Test set execution dhcp range Changes dhcp ip range 16 479 5 dhcp le Asks five times for IP address and checks if it is in the right DHCP range Fam an Log in to the edgeBOX as admin Connected to the edgeBOX 0 299999999999 Getting LANIP and MASK 192 168 100 254 255 255 224 0 5 6 Setting the network down Done 0 199999999997 Done Current MAC 00 14 22 78 95 d3 unknown Faked MAC 00 14 22 78 95 d4 unknown Done Internet Systems Consortium DHCP Client V3 0 3 Copyright 2004 2005 Internet Systems Consortium Getting a dhcp lease All rights reserved 21 3 For info please visit http www isc org products DHCP Listening on LPF eth Changing the mac addr4 o Checking if 192 168 100 174 is in the Hatwor
12. access coupled with advances in networking technologies create a new opportunity for small and medium sized enterprises SMEs and organisations with branch offices to become more efficient and competitive by interconnecting sites deploying collaborative and productivity enhancing applications increasing access to data inside and outside the office teleworkers and integrating data and voice VoIP Unlike large enterprises these businesses typically don t have the necessary IT resources to deploy integrate and manage these services They need a pre configured fully integrated device that can be easily installed used and maintained by non IT experts edgeBOX is such a device with a comprehensive range of networking services including the latest in VolP technologies plug and play installation and remote management features it is the only device an SME or branch office needs for deployment of a fully converged enterprise grade network KS File Storage Sharing and Backup High speed internet access via ADSL Cable UMTS or other technology converted to Ethernet VPNs based on both Internet Protocol Security IPSec and Point to Point Tunnelling Protocol PPTP for site to site and tele workers respectively WiFi at each site with integrated security and authentication Authentication Authorisation and Accounting providing complete control and auditing of network resources
13. and analysis of unexpected testing behaviour 4 3 User roles and responsibilities There are two different actors kinds of users of the platform Figure 4 1 Tester The tester is the user that wants to test benchmark and see the results of one or more services of the software Administrator The administrator is a tester that can define develop and document new tests in the Automated Testing Platform 4 4 Interactions with other systems The ATP has an interface with the application to be tested In the case of the edgeBOX this interface was defined to be the PDP in contrast with the CLI that connects directly with the PEP server This is an important point on the definition of the requirements since it enables the ATP to simulate the interaction of the Graphical User Interface GUI and the Items configuration module 4 5 Functional requirements 4 5 1 Use cases In this section the use cases of the general automated testing platform are defined Tester The list of use cases available to the tester Figure 4 2 are here defined This list has the requirements that have very high priority and are definitely to be implemented as the prototype of the automated testing platform List available tests See a list and search available tests its description and documentation 22 CHAPTER 4 REQUIREMENTS ANALYSIS List test cases List testing areas List campaigns Create campaign Test
14. create a new campaign click on the link in the right menu A new form requiring information about the campaign will appear A description is to be added and a subset of the tests is to be selected using the horizontal arrows to select and the vertical arrows to sort Figure 7 4c After submitting the campaign is visible in the list of campaigns To execute a campaign the user presses the corresponding link in the operation column Then a form for execution of campaigns appears Figure 7 4d and the user introduces the several fields and the target machine version in test IP address of the target machine how many executions to perform and the execution strategy According to its strategy executions can run sequentially run in parallel or run in parallel just in executions of different campaigns After submitting the form the user is redirected to the List executions area 7 2 5 List and view executions The executions listing page is divided in the following three areas Figure 7 5a Running executions The list of executions that are currently running Pending executions The list of executions that are in the waiting queue Past executions The list of executions that were already completely executed Campaigns that are colored green 46 Manage tests Manage test areas Manage campaigns List executions Show statistics o Remove CHAPTER 7 DOCUMENTATION Login test Edit Execute Ci Backup and restore tests Remove
15. download and installation software updates and patches to ensure edgeBOX has the latest features and remains secure The edgeBOX support team monitors global security alerts on a 24x7 basis Tight monitoring and control over network service usage is provided through detailed event logging statistics and accumulated histories with remote logging support syslog Automatic Data Backup can be performed to a local USB disk or off site server e g Data Center Integrated Failover supports failover of edgeBOX to a backup unit with the possibly of using different connection types dual homing Available Q1 2006 edgeBOX GATEWAY Configurations based on standard COTS hardware MODEL HARDWARE Rec users Office Gateway appliance entry level server Business Gateway standard server Enterprise Gateway high end rack mountable server 100 500 WiFi PCI AP ADSL PCI Modem ISDN BRI and PRI single dual or quad span FXO FXS 2 2 Hardware Options technical data sheet Software Features Security Stateful Inspection Firewall e Filter on source and or destination IP address port value Filter on SYN ACK flags and ICMP Stateful inspection Provides enterprise grade firewall protection Full time Stateful Packet Inspection with built in support for most popular applications No predefined limit on the number of rules that can be created and applied All firewall messages can be logged to syslog serv
16. grouped by the model of the target machine and by the version tested This information is very useful to compare the performance of subsequent releases in the same machine compare the effect of better hardware in the same version or benchmark the execution of tests 49 CHAPTER 7 DOCUMENTATION 7 3 Installation manual Since almost completely Debian based Linux distributions are used in the project this installation manual as sumes the apt installation suite is used For non debian systems the used packages can be installed using the distribution s package manager For installing the platform execute the following commands install dependencies gt sudo apt get install apache2 php5 libapache2 mod php5 g perl autotools download the application from CVS gt cvs co user cvs critical pt cvs critical edge device implementation automated testing create a symbolic link in the HTTP directory to the user interface gt In s public_html automated testing atp ui compile the core of the application gt configure gt make gt sudo make install install the perl module of the testing API gt install sh Some automated testing scripts implemented use perl libraries that have to be installed Each of these tests include comments with the libraries used and the user shall install them by the perl command line 50 CHAPTER 8 FINAL CONSIDERATIONS AND RESULTS ANALYSIS Chapter 8 Final consider
17. know how in testing processes and results analysis 2 Analysis of the processes tools and technologies used in the product 3 Integrate the edgeBOX development cycle 4 Design and implement an automated testing platform 5 Implement some automated tests as a proof of concept 1 3 4 Framing of the internship in the product In the edgeBOX s development process after the development phase all functionalities should be tested regres sion testing This is necessary because posterior changes can cause already implemented functionalities to fail This task consumes a big quantity of resources and is unsustainable The solution is to have a set of automated tests that check if the core functionalities of the product still work Besides system testing this solution also has the advantage of being able to do diagnose tests in the costumers machines and in the development stage to do integration testing Having an automated test it is possible to do stress testing and benchmarking of the certificated hardware This creates added value to the product since beside certificating hardware it is possible to certificate hardware for a specific number of users 1 3 5 Tasks Product study This task includes the technical study of the product Architecture of the system technologies used open source packages used and tools used to coordinate the development were explored Participation in a product development cycle With the objective of
18. local RADIUS server allows for anonymous access or connection to a remote server allows for access by account holders System Network Services Security QoS Policies State Usors Wizarde EE Trame Control Configuration Panel Help information Service Information Services Upload Premium Service Description Hescricn premium Upload information Maximum Uprate 10 Total Bandwidthe Pipe Name Associated a 2 b a Log incormauon 7 SCP Markina ls Pan ntometon success amos Fai Download Information EBE Figure 1 edgeBOX 100 java and web based management GUI Collaboration Services Collaboration through sharing of data and network resources is provided via the inbuilt Web and FTP Server for easy deployment of intranet and internet pages including default user home pages Email Server maintains full internal email services in cases of WAN link failure Includes relay control LDAP authentication ban lists aliases anti spam POP IMAP Web mail anti virus and remote mail storage Print Server Enables installation and management of shared printers at the customer premises Seamless integration with Windows with single point log on via Windows Server using LDAP and can act as a primary domain controller Web Filtering Enables web site access blocking based on keywords File Sharing Enables sharing of files by acting as a FTP
19. new test case with the same name to one that already exists 4 Create a new test case with names bigger than the allowed 5 Try the previous tests in edition of an already existing test 6 Try several searches of existing test cases 7 Follow the create stub link Expected results 1 All existing areas must be listed in the option menu 2 The test must be listed in the list of test cases 3 An error must be presented to the user 4 It can t be possible to introduce more than the number of allowed characters in the text fields 5 The same results of the previous tests must occur 6 The list of the test cases must be narrowed to match the search 7 A sample automated test must be created in the file system and the create stub link must disappear Results archived The expected results were obtained 38 CHAPTER 6 TESTING AND SOFTWARE QUALITY ASSURANCE Campaigns Testing summary 1 2 3 4 5 Create a new campaign with valid name and description and using some testcases Test the addition and remotion of tests included in the campaign Create a new campaign with names bigger than the allowed Try the three previous tests in edition of an already existing campaign Try several searches of existing campaigns Expected results l 2 3 4 5 The campaign must be listed in the list of campaign and the test cases used must be listed under the campaign The tests included must disappear from the l
20. of test cases among projects and by multiple testers and versions It provides various at a glance views of the test suite for easy status determination and test suite navigation TCW also provides basic reporting capabilities and per project access control Tesly Tesly is a Web application written in PHP that helps you create execute and report on test plans QA leaders can track the progress of testing as testers use the interface to report completion of test cases Test Environment Toolkit Test planning software that is a commercial package but available for free to open source non profit and educa tional projects 18 CHAPTER 3 THE PROJECT AUTOMATED TESTING PLATFORM Testitool Testitool is a Web based application for QA test planning It creates a test plan and populates it with test cases maps test cases to functional requirements instantiates a test plan begins executing test cases and marks them as successful or failed generates reports on test plans copies test plans and test cases and tailors test plan instances by adding and removing test cases from them TestLink Web based test management and test execution system allowing QA teams to create manage execute and track test cases and organize them into test plans TestMaster A testcase management logging reporting and test automation tool similar to the commercial product TestDirec tor Features Progress stats reports test case import from CSV doc web
21. shell way 7 2 3 Manage areas The list of testing areas is available in the Manage areas menu Figure 7 3b By using the search box in the right the used can narrow the list To add a new area click on the link on the right side A form where the user can enter the name and description of the area appears and after submitting the form the new area is listed alongside the other areas The possibility of removing or editing an area is available in the operation column in the list of areas The scheme for editing is the same than for addition of a new area 44 CHAPTER 7 DOCUMENTATION O CA Z o http 192 168 90 137 atp query manage tests Manage tests Manage test areas Manage campaigns List executions Show statistics met name rest desertion NE CT ON backup This test perform a complete backup of the edgebox Backup and Restore Frag dhcp lease Asks five times for IP address and checks if it is in the right DHCP range _ Services ae dhcp range Changes dhcp ip range Services Kae http light page Simulates an user viewing a light sized page Services TRE http 1mb page Simulates an user viewing a 1MB sized page Services TE http 10mb page Simulates an user viewing a 10MB sized page Services FET http 100mb page Simulates an user viewing a 100MB sized page Services TEN admin login Tries to login in the edgebox as an administrator Simple functional examples ra
22. the end time of the execution 5 5 System decomposition This section explores the several modules of the platform in detail 5 5 1 Core module The core of the ATP Figure 5 5 follows an object oriented architecture implemented in C that uses the model described in figure 5 3 The architectures uses in an interesting way the bridge design pattern There are classes that implement the timing and execution functions this brings an excellent modularity to the system making it possible to change the way the timer and the executor works without changing the implementation of the objects The core module has a set of components that have crucial importance Timer This class has abstractions to OS timer functions and declares virtual functions to that are implemented in the internal data structures 33 CHAPTER 5 ARCHITECTURE AND DETAILED DESIGN Graphical User Interface Data Access Layer Add Operations CL Update Operations ____ Delete Operations ___ Select Operations Web Interface Presenter Test Manager Area Manager i __ Statistics calculator Figure 5 6 Components of the user interface Runner This class has abstractions to the OS execution functions and declares virtual functions that are imple mented in the internal data structures Data access ObjectFactory When the database is pooled this class generates the object to the internal structures according to the elements created by the user in the user i
23. to assure the possibility of future improvements that may include e Use of the platform to manage manual test cases areas and campaigns e Extension of the platform to a distributed model 4 6 4 Hardware considerations The platform shall be targeted for x86 architectures with recent but not necessarily top of the art characteristics 4 7 User interface prototypes Figure B 1 in appendix B includes prototypes of the user interfaces Although rough these prototypes are useful to understand the style of the interface that is provided to the user 4 8 Scope As a first prototype the ATP structure and the authentication will be implemented According to the internship plan this phase will be ready in the 21 of June 2006 The automated tests are going to be implemented by priority They are only being implemented as a proof of concept of the platform and are just provided as an add on to the internship project 26 CHAPTER 5 ARCHITECTURE AND DETAILED DESIGN Chapter 5 Architecture and detailed design Programming without an overall architecture or design in mind is like exploring a cave with only a flashlight You don t know where you ve been you don t know where you re going and you don t know quite where you are Danny Thorpe There are two ways of constructing a software design One way is to make it so simple that there are obviously no deficiencies and the other way is to make it so complicated that there ar
24. understanding the qualities and deficiencies of the product and also to provide a pair of working hands in a needful hour a considerable amount of time was spent actually testing the product in an effective old fashioned way Testing tools study Open source automated testing and functional testing tools were studied in order to develop a state of the art testing tool Complete development of an automated testing platform Full cycle development of an automated testing platform including 1 Requirements specification CHAPTER 1 INTRODUCTION 2 Architecture definition 3 Prototyping Complete implementation Testing N REA Documentation Development of some automated tests To prove that the platform is functional and useful some automated tests were developed and the results were analyzed 1 3 6 Planning The detailed task plan in the form of a gant chart is included in appendix 8 3 Table 1 1 sums up the global plan Task Duration Introduction 9 days Requirements specification 5 days Architecture specification 7 days Experience in a software development iteration 40 days Implementation 66 days Testing 8 days Internship report 17 days Total 152 days Table 1 1 Task planning 1 There is a gap between the days estimated in the the table 1 1 and the ones in the appendix 8 3 caused by consideration of week days instead of work days CHAPTER 2 THE PRODUCT A NETWORK APPLIANCE Ch
25. useless in other hardware e License key controls access to update server on update edgebox net IP Quality of Service IP QoS e DiffServ traffic prioritisation ToS byte marking Inbound and Outbound QoS CBQ traffic prioritisation e Configurable traffic prioritisation policies by addresses users groups port protocol High Availability Data backup to remote FPT site or USB disk Automatic configurable regular backups Manual restore of backed up data from restore list Failover Dual homing To be released in Q1 2006 Networking PPP over Ethernet and PPP over ATM Automatic IP and DNS assignment Static routing on the LAN and or WAN DHCP server and client DNS Server Master and Forward Dynamic DNS client Network Address Translation NAT Port Forwarding Supports public Web and e mail servers with NAT Collaboration Services HTTP Server with user home pages FTP Server Email server supporting POP3 IMAP SMTP Web mail LDAP Intranet Mail routing Email remote storage Email aliases management Email antivirus interfaces with Sophos Email antispam based on RBLs SMTP relay for road warriors Print Server for USB printers Windows file sharing SME Web Transparent Caching User Disk Quotas VoIP and IP PBX SIP and IAX support Analogue and Digital PSTN gateway Least Cost Routing Interactive Voice Response IVR Automatic Call Distribution ACD Telephone Number Mapping ENUM Vo
26. while providing improved functionality to end users Security patches Bug corrections edgeBOX Update Center update edgebox net Figure 4 Remote management and software update services mean edgeBOX is easy to support and maintain Scalable Standards Based A platform for the future Unlike other proprietary appliances and business gateways edgeBOX runs on Commercial off the shelf COTS hardware providing a platform that is easy to upgrade and expand as additional functions are used or the number of users grow Hardware reliability is guaranteed and warranty issues are simplified while ensuring the latest hardware technologies can always be utilised such as faster network interconnects and VolP gateway cards edgeBOX is certified to run on a number of hardware configurations from Dell IBM Advantech and Lexcom with other platforms available on request Teleworkers Remote Staff Soft Phone Staff aS Phone En Head Office VoIP Gatoway va Q ACD LCR 9 Figure 4 VoIP deployment scenario Simplified Management Administration of edgeBOX can be performed on site or off site via the simple to use web based interface If a service management company is contracted to support edgeBOX they will be able to remotely support and maintain the device without the need for on site intervention greatly reducing the effort and cost involved Critical Links provides an update service which can automatically
27. 3 The product and the services The edgeBOX delivers high speed internet access and enables a wealth of managed services out of a single integrated secure flexible reliable and easy to use platform Items installed at the Service Providers Network Operations Center NOC pairs with edgeBOX to provide an end to end solution than fulfils services providers needs to market provision and manage the services that SME and EBO require today 10 0 m Fixed Calls and Access 5 0 Internet amp Broadband o m 1999 2000 2001 2002 2003 m Corporate Data Services Source Ofcom Market Intelligence Figure 2 3 Telecom revenue B 1999 2003 CHAPTER 2 THE PRODUCT A NETWORK APPLIANCE IP Phone Firewall VPN amp gt Anti Spam Anti Virus Authenticason amp Authorizabon USB gt Atomatic amp remote backups DI Content Mgt System CMS HS ED Remote web Management Windows Mac Content filtering J Web and FTP server SMTP POP IMAP WebMail Linux Pie amp Printer Sharing WiFi iP PBX and VoIP Gateway q Network Services DNS DHCP L NAT basic router Figure 2 4 Features edgeBOX Gateway edgeBOX Software COTS Hardware Hardware The edgeBOX is installed in different Commercial Off The Shelf COTS hardware eg Dell IBM according to the performance needs of the final client Service providers or Value Added Reseller VAR s assemble the e
28. Firewall management either basic or authenticated stateful firewall for each external link and VPN VoIP Gateway for routing between PSTN and IP networks IP PBX with a comprehensive feature list for inbound and outbound call management QoS and Bandwidth Management for service and user prioritisation and ensuring voice quality Web server for intranet extranet or internet Intranet out of the box with standard collaborative services Email and associated services SMTP POP IMAP Webmail anti virus anti spam for email hosting or relaying File and printer sharing for office desktop user collaboration Storage with quota management and automatic backup to a local disk or remote server Remote configuration and support WAN capabilities between sites Simple Network Deployment Every feature of edgeBOX is controlled via a simple web based management interface that can be accessed on site or remotely Wizards are available for initial setup and there are several pre defined templates that reduce installation and setup time edgeBOX has a built in router ADSL modem optional Wireless access point card optional cable UMTS support and services such as DHCP NAT and DNS which enable it to integrate into any type of network with minimum administration effort WiFi support also enables instant deployment of a LAN at Greenfield sites and of hot spots that are integrated into the network A
29. ING AND SOFTWARE QUALITY ASSURANCE Other quality assurances not only focused in features include code inspections and module unit testing of the core of the application by feeding simulated user operations and executing simulated test cases 6 4 Test cases This section lists a summary of the set of test cases defined for the application including the purpose expected result and obtained result but omitting the test steps in order to provide useful readable and understandable information Testing areas Testing summary 1 Create a new testing area with valid name and description 2 Create a new testing area with the same name to one that already exists 3 Create a new testing area with names bigger than the allowed 4 Try the three previous tests in edition of an already existing area 5 Try several searches of existing testing areas Expected results 1 The area must be listed in the list of testing areas 2 An error must be presented to the user 3 It can t be possible to introduce more than the number of allowed characters in the text fields 4 The same results of the previous tests must occur 5 The list of the testing areas must be narrowed to match the search Results archived The expected results were obtained Test cases Testing summary 1 Check listed testing areas in the option menu when adding a new test 2 Create a new test case with valid name description and testing area 3 Create a
30. ML Policy Decision Point Unix Socket XML Policy Enforcement Point Server Execution XML PEP popup threads LAN System calls EWAN Figure 5 1 ATP integration with edgeBOX components 5 3 System overview In this section the architecture of the Automated Testing Platform is explained and the architectural and techno logical options taken are justified The Automated Testing Platform is divided in 3 main modules Figure 5 2 Different modules use different technologies and communicate with each other in different ways according to performance scalability and main tainability reasons that will be justified in the next section As an overview of the application organization and the interaction between modules is here defined Graphical user interface The GUI provides an interface to the user where she can execute all scenarios with the exception of the develop ment of the automated test case This module is implemented in PHP Core module The Core module manages the application s logic It is responsible for executing operations defined in the user interface e g execution of campaigns It also updates the information base according to the results of test cases benchmarks testing stages automated test cases or test steps This module is developed in C Automated test module The automated test case module may have several instances that share a common API for communication with oth
31. T and VoIP solution in the market With the use of open standards and open software has a great expandability and mainly because of that it has a very competitive price targeting SMEs Figure 2 10 shows a comparison with similar products comparing the price data capabilities and voice capabilities It is perceptible that edgeBOX offers the most comprehensive product with the best quality price relation 2 4 Product architecture 2 4 1 The operating system The edgeBOX Operating System eOS uses the linux kernel Some parts of the eOS are based in Debian and in Linux From Scratch LFS 2 4 2 Packages manager and licences authentication Each license has a server entry with the version it is running and an hardware key The installer during the installation checks if the license is valid comparing the local hardware key and the server key version 13 CHAPTER 2 THE PRODUCT A NETWORK APPLIANCE Configuration change edgeBOX Figure 2 11 Architecture and flow of the configuration system During the updates besides checking the hardware key the edgeBOX uses the information in the server to install the right packages of the appropriate version 2 4 3 Configuration architecture When there is a configuration change Figure 2 11 through the web interface that change is sent to the Policy Decision Point PDP After that the PDP sends the information to the Policy Enforcement Point PEP which
32. The product and the services 8 2 154 Business Models 4 2 2 a a ee Be 10 215 Critical NKS ss sus Sds ak aoa ele rer ihnen 11 Hi CONTENTS 2 2 2 3 2 4 Organization ui Yu a ee RE AEN nn BR A de 22 1 Group organization 2 2 Y Bee aia ES 2 2 2 Development model 2 2 5 a ee mas Ra AA ee eee ee Br Ain en Comparison with other products ee Product architecture 2254 2 so s ln a Bisa rag ne EE gone sU S ee ne 2 4 1 The operating system 24 2 Packages manager and licences authentication 2 4 3 Configuration architecture 24 4 Request handling e The project automated testing platform 3 1 3 2 3 3 3 4 Important concepts on testing Problem domain siise ci au ss me 2 a ns A EN sc Uh a We Er kuq State OMS a ee Bern a re ee U X COMPAFISON 2 e897 Ra s ne Saha Up AIR A Wi ee Rage sap ua o ba 3 Requirements analysis 4 1 4 2 4 3 44 4 5 4 6 4 7 4 8 Automated testing platform definition Objectives ci Ein a Sika Den rip teinte ht A aie User roles and responsibilities Interactions with other systems Functional requirements z ya p au ra ia iu ADA USE CASES ee ete we be 4 5 2 Example automated test implementation
33. ales cycle Another central duty is to collaborate with the Product Management including transfer of qualified information for future product development competitor information market trends product enhancements as well as the cooperation to prepare functional specifications The Support areas role is to supply assistance to clients triage and report problems found by costumers and provide user documentation of the product The Research and Development area develops the product features solves problems found in previous releases and evaluates the technical difficulty of costumers requests The Testing area tests new features the integration of new features on the previous features tests problems found by costumers benchmarks the performance of the product and certificates hardware 2 2 2 Development model The edgeBOX team uses an iterative and incremental development model with an average 3 month release cycle Given the time it takes to develop this large sophisticated software system it is not possible to define the problem and build the solution in a single step Requirements often change throughout a project s development due to architectural constraints customer s needs or a greater understanding of the original problem The edgeBOX update system is an excellent tool for the application of this development model since it allows the clients to have access to the new features It is also possible to use the update system to have b
34. all tests campaigns and test steps Content Operations separtion There is a clear separation of content and operation by the use of menus The users manual and a demonstration of the application is included in section 7 2 5 5 3 Automated test case A components diagram of this module is available in figure 5 7 An automated test case uses three APIs that were implemented using Perl Modules but that because of the mod ularity of the system can be easily implemented in other computer languages The use of these APIs brings great abstraction to the code promotes code reutilization and don t cause the system to be recompiled since an interpreted language is used The ATP protocol s layer implements an interface to the protocol so that the user can abstract it and use simple functions instead Functions to initialize the protocol start a test step or declare the result of a test step are provided The configuration API abstract the protocol of configuration causing that if the functionality changes the change in the automated test only has to be correct one time The system API is offered by the language itself and doesn t ties the code to any operating system or architecture as long there is an interpreter of Perl to that operating system or computer architecture The use of these APIs cause that the code of an automated test to be almost similar to pseudo code To illustrate a very simple example of this code is in figure 5 8
35. als can be used as tour of the application and show clearly why it is useful This document ends with chapter 8 Final considerations and results analysis where a review of the work produced and results archived is presented Some chapters refer to external information that is available in appendix xiii Acronyms CSW Critical Software SME Small Medium Enterprises SMB Small Medium Business EBO Enterprise Branch Offices CEO Chief Executive Officer CFO Chief Financial Officer ARPU Average Revenue Per User POTS Plain Old Telephone Service VoIP Voice over Internet Protocol IT Information Technology CPE Costumer Premises Equipment NOC Network Operations Center COTS Commercial Off The Shelf VAR Value Added Reseller CRM Costumer Relationship Management eOS edgeBOX Operating System LFS Linux From Scratch CLI Command Line Interface PDP Policy Decision Point PEP Policy Enforcement Point IPC Inter Process Communication ATP Automated Testing Platform GUI Graphical User Interface XV CHAPTER 1 INTRODUCTION Chapter 1 Introduction 1 1 The company 1 1 1 Background Critical Software CSW supplies solutions services and technologies for companies information systems an swering to the necessities of several markets as telecommunication public sector industry and the aerospace and defense sectors The company was founded in 1998 and currently employs more than 140 people in their offices in Coimbra Lisboa Porto in Por
36. although some tests failed during the implementation in the end all tests were producing satisfactory results 6 5 Unit testing Only the core module of the system was submitted to unit testing since it is the most critical module of the application These tests were produced by feeding defined testing orders to the database and by checking the results afterwards With the execution of these unit tests it was also possible to submit the platform to stress tests by executing them a massive amount of times and doing the tests in the previous section The results of this type of testing were positive an the platform proved to be prepared to deal with great quantities of data 6 6 Code inspections An inspection of a part of the code was performed by a member of the development team that evaluated the coding style the code organization the documentation and reviewed with particular attention the most crucial methods of the application s core The code reviewer comments were very positive and his suggestions were taken in account 40 CHAPTER 6 TESTING AND SOFTWARE QUALITY ASSURANCE 6 7 Beta testing During the edgeBOX s release in the begging of August the Automated Testing Platform was used and some automated tests were executed The test of backups and restores was very useful due to its implications in a great set of the functionalities and the amount of work and time involved in changing configurations and restoring them A test t
37. applies the changes by configuring the edgeBOX operating system The Command Line Interface CLI that allows command line editing of the edgeBOX connects directly to the PEP 2 4 4 Request handling The PEP scripts are written in perl and are objected oriented modules organized in packages The message requests are passed to the PEP already parsed as XML DOM elements The PEP changes these elements a return them to the requester Each PEP can implement some methods The pre method which is executed once when the PEP is executed the react method which is executed each time for each XML element the stat method to check the status of a service and the post method that is called after react is executed for each command node The PEP server is a daemon that receives requests for the PEPs executes them and replies with the result It uses different processes to handle with the requests To talk with those handlers the PEP server uses Inter Process Communication IPC 14 CHAPTER 3 THE PROJECT AUTOMATED TESTING PLATFORM Chapter 3 The project automated testing platform 3 1 Important concepts on testing In this section important concepts to understand posterior information are defined and explained For a more profound knowledge search there are several books and websites detailed in the bibliography Faults and failures Software engineers distinguish software faults from software failures In case of a failure the software d
38. apter 2 The product a network appliance How many IT services can you squeeze into a box for an SMB Juniper Networks Cisco Systems and 3Com have come up with some for security and Avaya Alcatel Nortel and Siemens for communications But how about 50 applications that do everything from being a wireless router has web e mail and DNS DHCP services IP telephony with Asterisk IP PBX and firewall with VPN authentication and anti spam anti virus capabilities The edgeBOX is what they call it and the company called Critical Software put it together Benjamin Koe in Security Tech Planet 2 1 Introduction 2 1 1 Market opportunity The market opportunity appears from the latest movements in service providers and from the new needs from SMEs and EBO Service providers In the last few years broadband prices have suffered a significant decrease Service providers need to increase the Average Revenue Per User ARPU through the sale of value added services Considering the market dimensions service providers need to easily and effectively manage these value added services in a wide number of clients Most broadband providers also provide Plain Old Telephone Service POTS and with the market shifting to Voice over Internet Protocol VoIP it is easier for them to keep the clients since they provide a similar service since long Nevertheless Skype Microsoft and Google are big names the will have a word to say in this market SMEs and
39. ase Administrator _W Create campaign List campaigns Execute campaign Tester y List campaign results See detailed campaign result Generate report Figure 4 4 Activity diagram with the most common scenarios In order to have a better understanding of the use flow of the application figure 4 4 has an activity diagram that includes the most common scenarios of use 25 CHAPTER 4 REQUIREMENTS ANALYSIS 4 6 Non functional requirements 4 6 1 Usability The Automated Testing Platform will follow the available standard interface guidelines on the several technologies adopted It is important that the system is easy to learn Regular software developers and testers with basic testing expe rience shall have an easy learning process An usability test will be performed with future users of the platform Their comments will be taken in account A particularly important point is error handling 4 6 2 Time requirements e The time bound of the execution will be in each test time and not by internal ATP processing e The tool will answer to user calls in an instantaneous way e The time measuring of executions can t count with internal operations of the testing platform 4 6 3 Architecture requirements The target operating system over which the testing platform will run is Linux But it is expected that with little or no effort porting the platform to other operating systems can be archived It is expected a modular architecture
40. ations and results analysis The single most important thing to remember about any enterprise is that there are no results inside its walls The result of a business is a satisfied customer Peter Drucker The truth is there are only two things in life reasons and results and reasons simply don t count Robert Anthony 8 1 Automated testing platform results The objectives proposed in the beginning of the internship were archived An automated testing platform with some innovative functionalities was developed as required Some automated tests were also implemented to show the use functionalities and value of the platform The results were very satisfying As an analysis of the platform the unique points in comparison with the other products mentioned in section 3 3 are e Architected to test network applications in particular the edgeBOX e Unifies functional testing and benckmarking e Architecture supports distributed automated testing 8 1 1 Improvements During the development and testing some ideas for new features were suggested by the future users of the appli cation These include e Better integration among test definition on the platform and test documentation e An interface to provide arguments to the automated tests e Easier distributed system testing platform functionalities 8 2 Automated tests results The automated test cases implemented were just a proof of concept to show that part of the testin
41. atp index php query show statistics Manage tests Manage test areas Manage campaigns List executions Show statistics Testing Certification and Benchmarking Login test This campaign was executed 89 times The average execution time is 0 339 seconds with the standard deviation of 0 489 seconds DHCP This campaign was executed 3 times The average execution time is 140 214 seconds with the standard deviation of 1 589 seconds HTTP This campaign was executed 11 times The average execution time is 12 588 seconds with the standard deviation of 3 866 seconds a Overview of the statistics 1 no http 192 168 90 137 atp index php query show_statistics SO O S Da Manage tests Manage test areas Manage campaigns List executions Show statistics Testing Certification and Benchmarking Login test This campaign was executed 89 times The average execution time is 0 339 seconds with the standard deviation of 0 489 seconds admin login This test was executed 89 times in this campaign The average execution time is 0 336 seconds with the standard deviation of 0 487 seconds O Twister This test was executed 10 times in this machine The average execution time is 0 288 seconds with the standard deviation of 0 251 seconds Dell 830 This test was executed 79 times in this machin The average execution time is 0 342 seconds with the standard deviation of 0 509 seconds 4 1 This test was executed 67 times in this version The average e
42. between the automated test and the core Bridge pattern The bridge pattern is used in the core of the application when all runnable tests have an abstraction and use the virtual execution and timer function which can have different implementations This is useful since measuring time or executing a campaign can be different of doing the same in a single test and in case of modification in one of these methods the modifications are encapsulated in one class 30 CHAPTER 5 ARCHITECTURE AND DETAILED DESIGN belongs he contains Test Case Definition name text description text execution path text description text name text description text is executed ash 7 Test execution aueuelten in J description text execution start date timestanp target ip address text execution start date timestamp executes in parallelization type int execution status int is composed by Test Step execution start date timestamp purpose text 4 contains containsp Test step result was successful bool description text execution time int Test execution result was successfull bool description text running time int Campaign r sult was successful bool description text execution end date timestamp T Version Figure 5 3 Class diagram of the ATP Extensibility pattern The automated
43. business gateway revenue forecast EM 2005 2010 8 European managed services revenue forecast EM 2005 2010 8 Telecom revenue EB 1999 2003 8 Features cd 2 SLR ke A A a a re a A 9 Business relations 2 a Ca ra A ae 10 Targetimarkets 2 E BAR PR ae A A 10 Group organizatim e 4 5 2 ee nn een 11 Grolip MEINES LS els uen rt Le w hu ed BK e SUYO Me se to 12 Coordination tools dn ses cow wR ORS RA Ee ee ee Ee ee Le ue 12 Product Comparison iscas au SR REE ee ee eS 13 Architecture and flow of the configuration system 14 Conceptual domain model 2 2 s ws a mess wre sobre mens 17 ACTOS y nr da a ee spe e sa er ne a Ya s 22 Testers USE CASES y sk wo SoS SL A ler ee 23 Administrator s use cases 24 Activity diagram with the most common scenarios 25 ATP integration with edgeBOX components 28 ATP decomposition in high level components 29 Class diapram Orme ATP sus 2 ace y we we Br deg un Dr bie mt RED 31 Physical model ofthe ATP s G Q U ah aW 8 D s e Ps q S 32 Components of the core module 000 000 0000 0000008 33 Components of the user interface 34 Components of an automated test case 36 Example of an automated
44. ccess layer has the following main components Add Operations In this component the operations that add records to the database are defined and implemented Update Operations In this component the operations that update records in the database are defined and imple mented 34 CHAPTER 5 ARCHITECTURE AND DETAILED DESIGN Delete Operations In this component the operations that remove records from the database are defined and implemented Select Operations In this component the operations that select records from the database are defined and imple mented The web interface has the following main components Test Manager This component shows information and provides operations over tests Area Manager This component shows information and provides operations over areas Campaign Manager This component shows information and provides operations over campaigns AJAX Campaign Run Time Presenter This component uses Asynchronous JavaScript and XML AJAX with the objective of providing an interactive real time presentation of the global state and test details of a cam paign execution Statistics calculator This component calculates the statistics defined in the requirements and shows them in a user friendly way In the architecture of the user interface some usability design patterns are to be followed Search do not sort This Gmail like usability design pattern was followed by keeping sorting to a minimum allowing the user to search
45. ch 2 2 0 si Dent 8 A E A A IRA AAA LEERE 37 OA MOST Cases a EA As Bee web a FP es A nr 38 6 3 Unittesung a oa fes a a a dai A 40 6 6 Code inspections 40 62 Beta testing s sets bas ee Br ee ely a SUN 41 Documentation 43 7 1 Approach to documentation 43 7 2 User manual and application tour 2 2 CC mn ee ee 43 22 Application front page asas baked de erster kuqe s h Gand seh 43 72 2 Manage tests 2 yuy bn ee Ihe een te ee 44 95233 Manage Areas ra Sea RE SER EN en A BEN RR SE es 44 4 2 4 Manage campaigns co unter re Der neh Pralinen el 46 12 3 List and view executions 4 2 4 rr ee e Nes ue s we 46 732 62 VIEw st tistics nur a un te ee IRS ee ee 49 7 3 Installation manual sus mi a SS el me a nee 50 Final considerations and results analysis 51 8 1 Automated testing platform results 51 8 1 1 improvements Reid adas A bee ele wee ee 51 8 2 Automated tests results x sm morar AAA 4 me ana en 51 8 3 Internship considerations 52 Gant chart 55 User interface prototypes 59 Edgebox Fleur 61 CD 65 LIST OF FIGURES List of Figures 1 1 1 2 2 1 2 2 2 3 2 4 2 5 2 6 2 7 2 8 2 9 2 10 2 11 3 1 4 1 42 4 3 4 4 5 1 5 2 5 3 5 4 5 5 5 6 5 7 5 8 COPTIC OS E a osa A AS A A Dee er Den AI Tes 2 Certifications su a6 bb etaitas as Ha bee eg e EE 2 Worldwide
46. days Mon 21 08 06 Task Milestone 4 External Tasks Project plan N Da Wed 28 06 08 Split Summary q external Milestone Progress ME Project Summary G Deadline JL Page 1 01 May 06 08 May 06 15 May 06 22 May 06 29 May 06 05 Jun 06 EISISIMITIWITIFISISIMITIWIT FISISIMITIWITIFISTSIMITIWITIFISISTM TIWITIFISTSIMITIWIT Project pl Date Wed 28 06 06 Task lan Split Progress Milestone Summary Wq External Milestone 4 Project Summary Egg External Tasks Deadline Y Page 3 20 Mar 06 27 Mar 06 03 Apr 06 10 Apr 06 7 Apr 06 24 Apr 06 FISISIMITIWITIF SISIM TIWITIFISIS MITIWITIFISISIMITIWIT 1 FISISIMITIWITIF SIS MIT WIT EEE SERRE ns Wi a _ _ e Task Milestone 4 External Tasks D en Split Summary RR External Milestone Progress ME Project Summary Ye Deadline Page 2 2 Jun 06 19 Jun 06 26 Jun 06 08 Jul 06 TO Jul 06 17 Jul 06 FISISIMITIWITIFIS ISIMITIWITIFISIS MITIWITIFISISTMITIWIT FISISIMITIWITIFISISIMITIWIT Project plan Date Wed 28 06 06 Task Split Progress Milestone 4 Summary gg Project Summary rg External Tasks External Milestone Deadline db Page 4 24 Ju
47. days Tue 07 03 06 EE 5 Y Elaboration of a document about the organization of thee 5days Mon 13 03 06 rn 6 Requirements specification 8days Mon 20 03 06 T Brainstorming about the requirements 3days Mon 20 03 06 8 Definition of a set of tests that can be performed automat 3days Tue 21 03 06 SY Elaboration of the requirements document 5 days Thu 23 03 06 10 Architecture specification 7days Thu 30 03 06 TV Brainstorming about the architecture 5days Thu 30 03 06 2 Y Elaboration of the architecture document 7days Thu 30 03 06 13 Experience in a software development iteration 40 days Mon 17 04 06 14 Implementation 86 days Mon 03 04 06 15 Elaboration of a vertical prototype for a specific usec 70days Mon 03 04 06 16 Eg Functionality prototype 30 days Mon 03 04 06 17 Stress test prototype 5 days Mon 26 06 06 18 3 Benchmark test prototype 5 days Mon 03 07 06 19 Testing of the prototype for feedback 13days Wed 21 06 06 20 ma Revision of the test cases to implement 2days Thu 29 06 06 a EM Implementation of the test cases by order of priority 21 days Mon 03 07 06 2 Eg Testing of use cases for feedback trial 10 days Tue 18 07 06 23 Testing 12 days Tue 01 08 06 a Ed Functional tests 5days Tue 01 08 06 25 Eg Unit testing 5days Tue 08 08 06 Erler Code inspection 2days Tue 15 08 06 27 Internship report 22 days Tue 01 08 06 28 ma Structure definition 3days Tue 01 08 06 29 Eg Elaboration 18days Thu 03 08 06 30 Ex Reviews from tutors and others 8
48. description written in prose The ideal scenario test has five key characteristics It is a a story that is b motivating c credible d complex and e easy to evaluate They are usually different from test cases in that test cases are single steps and scenarios cover a number of steps Test suites and scenarios can be used in concert for complete system tests Test plan A test plan is a systematic approach to testing a system such as a machine or software In software testing a test plan gives detailed testing information regarding an upcoming testing effort including scope of testing schedule test deliverables release criteria and risks and contingencies Test suite The most common term for a collection of test cases is a test suite The test suite often also contains more detailed instructions or goals for each collection of test cases It definitely contains a section where the tester identifies the system configuration used during testing A group of test cases may also contain prerequisite states or steps and descriptions of the following tests 16 CHAPTER 3 THE PROJECT AUTOMATED TESTING PLATFORM Test Area contains belongs Test Case Definition executes executes in executes in Execution is composed by Figure 3 1 Conceptual domain model Test script A test script is a short program written in a programming language used to test part of the functionality of a
49. dgeBOX and install the software The hardware is certified by Critical Figure 2 1 to assure the whole box runs properly Model Hardware platform Number of users Dell PowerEdge 400 Series IBM xSeries 206 DES p away Lexcom Twister 863A m Advantech FWA 3140 Dell PowerEdge 800 Series Business gateway IBM xSeries 226 20 100 Advantech FWA 3140 FWA 6280 RSA 200 Dell PowerEdge 1800 2800 Enterprise gateway IBM xSeries 236 100 500 Advantech FWA 6280 RSA 200 Table 2 1 Certified hardware Software The edgeBOX uses the linux kernel and several other open source packages to provide its operating system The software developed by Critical includes providing to the user an easy way of configuring services Other software developed in house includes an update system and licenses enforcement Features The highlights of edgeBOX features are in figure 2 4 The complete list of features can be found in the edgeBOX technical flyer Appendix C CHAPTER 2 THE PRODUCT A NETWORK APPLIANCE Partners End Users Custom developments custom dev ses gt Value added services Internet SMEs Service CLK Providers Ena mall an Medi Critical Entep FEBOS Links Sublicensing A Installation Enterprise Maintenance Branch 1 amp 2 tier support Offices Hw integration Licensing edgeBOX Licensing Items 3 tier support On site support Hw certification Figure 2 5 Business relations 2 1 4 Busin
50. e applied Model view controller The model view controller MVC design pattern is the model that shapes the architecture of the Automated Testing Platform This software architecture separates the application s data model user interface and control logic in three distinct components causing that modifications in one component have minimal impact in others The model component includes the database storage mechanism and data access layers that interact with the data storage and include domain logic to add some meaning to the raw data and object orientation The view component includes the web interface and presents information to the user The controller component includes the core of the application and the automated test cases Client server The controller component of the MVC design pattern implements another architectural design pattern the client server model The core of the application is considered the server and executes several instances of automated test cases waits for requests and processes them Each test case is considered a client that connects to the server and places notifications of the status of the test Interface pattern The interface pattern is used in communication among different modules When two modules communicate abstraction class or interfaces are used This is clear in the database abstraction classes in user interface and application s core modules Another use of this pattern is in the communication class
51. e no obvious deficiencies The first method is far more difficult C A R Hoare 5 1 Design standards In order to have an high homogeneity level use good software practices and maximize code reuse Critical Soft ware has a common set of standards used in its software development process Among this set of standards the following are of particular importance e The use of UML 1 4 for specification of all components of the system e The use of Enterprise Architect as a tool for project management and specification e The use of Critical Software s coding standards 5 2 Integration in the system The Automated Testing Platform connects to the edgeBOX through the HTTP or HTTPS protocols using post procedures server and client side cookies for authentication and XML for passing parameters This procedure is the same that is used by the GUI and therefore enables to do the same configuration changes that must be validated in the same way The ethernet links allow the ATP to connect to the edgeBOX as regular LAN client WAN provider or EWAN DMZ server With these connections all services provided can be tested making the ATP an useful and complete network testing suite A diagram of the connection of the ATP to the edgeBOX s components is provided in figure 5 1 27 CHAPTER 5 ARCHITECTURE AND DETAILED DESIGN Automated Testing Platform Ethernet HTTP HTTPS Post XML edqeB0 HTTP Server Execution X
52. e spread across Europe the US and Asia in the Telecom Aerospace and Defense Sectors The experience acquired working with customers in these sectors provides extensive business knowledge that Critical Links is using in developing innovative products and services such as edgeBOX and ITEMS 1 1 3 Organization One way of describing the positioning and operations of Critical Software is to look at its horizontal organization CHAPTER 1 INTRODUCTION a Coimbra aa b Lisboa c Porto d California USA e United Kingdom Figure 1 1 Offices P m o gt CMMI 2 _ j Y SPYCE I a ISO 9001 2000 TickIT b CMMI Level 3 c Nato Secret Level d ISO 15504 Spice Figure 1 2 Certifications Enterprise application integration Deals with complex problems of integration and development of applica tions Dependability Focus on dependability and trustability of software and computers Experimental and analytical methods are applied Ground segment software Offers solutions to the communication area mainly to the spacial aeronautical and defense sectors Networking and communication Focus on planning design and development of communication solutions High performance computing Dedication to performance problems in companies and organizations information systems Products and services include optimization tuning the parametrization of processes development of parallel code and applications Manufacturing ex
53. ecution systems Focus on industrial information systems and in its integration with assembly lines Critical has experience and know how in the following vertical markets Aerospace with Jet Propulsion Laboratory NASA Eumetsat ESA EADS Astrium Alcatel Telecommunication with Vodafone TMN Portugal Telecom Public sector with the Portuguese Government Industry with Infineon Technologies Soporcel Defense with NATO Portuguese Marines CHAPTER 1 INTRODUCTION The administration is composed by Goncalo Quadros Chief Executive Officer CEO Joao Carreira VP business development Diamantino Costa VP business development Abel Pinto VP finance and human resources Pedro Murtinho Chief Financial Officer CFO Rui Cordeiro Engineering director 1 2 Overview of the product 1 2 1 edgeBOX Installed at the edge of the WAN and the LAN the edgeBOX delivers high speed internet access and enables a wealth of managed services out of a single integrated secure flexible reliable and easy to use platform The edgeBOX replaces 5 8 traditional costumer premise equipments Router Forwards data packets across a network toward their destinations Security Controls traffic between different zones of trust VoIP IP PBX Allows voice over internet protocol with call switching WiFi Access Point Offers a wireless local area network access point Network Access Controller Enforces security policy and restrict prohibited traffic types Quali
54. er Execute campaign Generate Reports View executed campaigns View campaign result Figure 4 2 Tester s use cases List testing areas See a list and search testing areas and its description See the list of tests included in each testing area List campaigns See a list and search testing campaigns including the lists of tests included in each campaign Create a new campaign Select tests and write a description to include in a new campaign Execute a campaign Execute a campaign Execute multiple times a campaign Execute multiple times a campaign in a concurrent manner 23 CHAPTER 4 REQUIREMENTS ANALYSIS See statistics about executed tests See time statistics about the tests executed including per campaign per test per machine and per version Generate reports about executed campaigns Generate a printer friendly report of an executed campaign See the result of one campaign See the results of a campaign in run time or of previously executed campaigns Results of each campaign include results of each automated test and respective test steps See list of executed campaigns The tester can see through the interface a list of the executed campaigns in the past including related informa tion and the campaign result Admin Add new test case Pr Edit test case Implement an automated test
55. er modules Its task is to perform tests in an automated way and report results to the core of the application 28 CHAPTER 5 ARCHITECTURE AND DETAILED DESIGN Automated Testing Platform Automated Test Case Object Oriented Timer Object Oriented Test Runner Testing logic System API t Test Communication Layer Be N Ethernet Object Oritented Data Access Layer Configuration API edgeBOX Graphical User Interface Data Access Layer Web Interface Presenter HTTP server Figure 5 2 ATP decomposition in high level components This module uses the language Perl User interface and application s core communication The user interface and the core module use a mysql database to process information All data created and actions performed in the user interface are added to the database The core frequently polls the database for new requests These requests are processed and the results are written back in the database Automated tests execution The automated tests are executed by the core module using command line execution This is important since it is intended that the languages in which the tests are executed is not specified Application s core and automated tests communication When the core executes a new automated test it opens a TCP socket connection with the test to communicate useful information including when a test starts or finishes what is the propose of a test step or what was the result of a test s
56. ers AAA Authentication authorisation and accounting external and internal e Group based access control Virtual Private Networking e IPSec no pre defined limit on VPN tunnels Internet Key Exchange IKE including Aggressive Mode 3DES 168 bit encryption Supports Perfect Forward Secrecy DH Group 2 and 5 Provides protection from replay attacks IPSec VPN Firewall to control tunnel access PPTP Server with local or remote authentication User Authentication e Local RADIUS server or LDAP server e g W2K Configuration and Management Configuration Easy setup through a browser based user interface Wizard based setup Default basic configuration Configuration and management using HTTP serial console and SSH Syslog server support Secure Management Local and remote management RADIUS management authentication support RADIUS support for accounting time amp traffic SSH secure management channel SNMP support Software Update Management Automatic update from central update server update edgebox net Manual configuration option which downloads available updates and waits for user confirmation before install e Notifications for service shutdown and service restart for planned maintenance e Update service access based on yearly subscription Anti piracy protection Proprietary software is encrypted AES 256 bit e Software key lock tied to hardware signature to deter piracy renders edgeBOX software
57. ess model The supply chain is represented in figure 2 5 Target markets The main target markets Figure 2 6 are emerging market where the IT infrastructure has not been build yet Brasil India Russia and Chine are big targets and developed markets will follow fan 4 t Ma Figure 2 6 Target markets 10 CHAPTER 2 THE PRODUCT A NETWORK APPLIANCE edgeBOX Product Helder Sousa PreSales R amp D Support Test Valentina Garcia Ant nio Alves Philip Mediam Alexandre Esper Jo o Belo Alvaro Bruno Andr Andr Cla dia Jayme Rui Hugo Nelson Rui Lopes Ramos Lemos Barbosa Serafim Bulh es Portugal Almeida Vale Ferreira Figure 2 7 Group organization 2 1 5 Critical Links In order for Critical Software be able to finance such a huge project a new company was founded With this company Critical Links 1t is being negocied the inclusion of venture capital Critical Links uses all Critical Software infrastructure and support areas 2 2 Organization 2 2 1 Group organization The project is coordinated by Helder Sousa and is organized in 4 areas Figure 2 7 a Business Development department and a Marketing and Communication department The PreSales area s is responsible for preparation and accomplishment of presentations of edgeBOX s solutions to prospect customers partners and analysts This area offers the technical contact to the partner and prospect cus tomer within the whole s
58. eta clients that provide feedback about each modification bug fix or even the usability of the product Requirements analysis There is a medium long term road map with planned features for future releases This plan is highly flexible Before a release cycle there is a discussion within the product board Figure 2 8 that merges the engineering 11 CHAPTER 2 THE PRODUCT A NETWORK APPLIANCE Meetings Engineering Board Product Board Mondays 11 30 12 30 Mondays 15 00 17 00 Ant nio Valentina Alves Garcia Figure 2 8 Group meetings pe EE 7 w E Crhical Sofiare CVS Repasizary En me orae a Wiki b CVS c Salome uses pres EE pn nn ns d Bugzilla e Sugar f Enterprise Architect Figure 2 9 Coordination tools sales clients and management views A special attention in given to the client s view represented by the PreSales area Tools used for coordination Wiki is used as an information management system Documents organization tasks are accessible through the Wiki SnipSnap Figure 2 9a CVS Concurrent Versioning System implements a version control system It keeps track of all work and all changes in the implementation and documentation It is used for collaborative work in the same modules of code Figure 2 9b Salome TMF is an independent Test Management Tool used to manage the entire testing process by creating tests executing manual tests and tracking results Figure 2 9c
59. g of the edgeBOX can be done using the automated testing platform This objective was archived during the edgeBOX s 51 CHAPTER 8 FINAL CONSIDERATIONS AND RESULTS ANALYSIS release in the beginning of August by using the platform for testing of sensitive and difficult functionalities as the backup or the DHCP server package 8 3 Internship considerations During this internship an inside knowledge of the edgeBOX was gained It s was a valuable experience for me as a student to work in a real world team s project with its time money and quality constrains Atatechnical level new technologies were learned my networking knowledge has increase since it was something in which I didn t had great interest during my degree and that was a gap in my formation as a computer engineer I learned a great deal about organization of software projects how to contribute as a member of a team to a big project 10 members in the engineering team and 10 members in supporting areas when there are multiple goals and short amount of time and resources Pve also gained life experience that for sure is going to be helpful in the rest of my professional life by moving to a different city meeting different people from different nationalities and by working in one of the most dynamic and innovative Portuguese companies I feel that Pve archived my objectives and that it was a very good choice to be an intern at Critical Software In my opinion and from t
60. he feedback I ve received Critical s managers were also very happy with my performance I was evaluated according to the standards of the company having a final classification of superior 4 2 out of 5 a excellent classification considering the exigence of the evaluation I was invited to keep working at Critical Software a propose that I m going to ponder and decide after the deliver and presentation of this report 52 BIBLIOGRAPHY Bibliography Tanenbaum A Computer Networks Fourth edition Prentice Hall 2003 Tanenbaum A and Steen M Distributed Systems Prentice Hall 2002 Srinivasan S Advanced Perl Programming O Reilly 1997 Beck k Test Driven Development Addison Wesley 2003 Brooks F The Mythical Man Month Addison Wesley 1975 Gamma E et al Design Patterns Elements of Reusable Object Oriented Software Addison Wesley 1997 Stroustrup B The C Programming Language Addison Wesley 1991 53 APPENDIX A GANT CHART Appendix A Gant chart 55 D Task Name Duration Start eb 06 06 Mar 06 13 Mar 06 o TIWITIFISISIMITIWITIFISIS MITIWIT T Y introduction 13 days Wed 01 03 06 Iso Introduction to the company workplace and work mates 3days Wed 01 03 06 gt Global planning 4 days Fri 03 03 06 Ty Meeting with tutors and planning aprovment 5
61. hrough any inter process communications mechanism with the assurance that these events are routed through the same code paths as real keystrokes and mouse clicks System testing System testing is testing conducted on a complete integrated system to evaluate the system s compliance with its specified requirements System testing falls within the scope of Black box testing and as such should require no knowledge of the inner design of the code or logic Alpha testing and Beta testing are sub categories of System testing 15 CHAPTER 3 THE PROJECT AUTOMATED TESTING PLATFORM Alfa and Beta testing In the first phase of alpha testing developers test the software using white box techniques Additional inspection is then performed using black box or grey box techniques This is usually done by a dedicated testing team This is often known as the second stage of alpha testing Once the alpha phase is complete development enters the beta phase Versions of the software known as beta versions are released to a limited audience outside of the company The software is released to groups of people so that further testing can ensure the product has few faults or bugs Sometimes beta versions are made available to the open public to increase the feedback field to a maximal number of future users Testing during the beta phase informally called beta testing is generally constrained to black box techniques although a core of test engineers are li
62. ice Mail with optional email forwarding of voice messages Call routing between remote edgeBOX switches branch offices Digital Receptionist Incoming Call Rules DID Call Routing Sound Manager Follow me feature for user extensions Call Parking Call Forwarding Conference Bridging Music on Hold Call Queuing Hunt Groups User Profiles with call permissions Support for all common codecs Hardware Features edgeBOX runs on standard hardware and supports a range of hardware features listed below For support of specific hardware configurations please contact us WAN Interface Cable Modems ADSL modems internal or USB UMTS Ethernet based modems LAN Interface Primary Ethernet Secondary Ethernet for DMZ or bridging e WiFi 802 11b g AP mode EWAN Interface Ethernet port for Enterprise WAN Serial Interface One asynchronous serial console port Telephony For interconnection with digital and analog telephony equipment edgeBOX supports and auto detects a number of hardware devices from various manufacturers including BRI boards 1 4 and 8 ports PRI up to 124 channels and POTS 2 2 boards from Digium and Beronet No additional hardware is needed for VoIP Open Source Software The edgeBOX software distribution includes a number of open source packages that are distributed according to their specific licenses including Linux 2 6 kernel Openldap Apache php sendmail OpenSSL bind dhcpd OpenSwa
63. ipts developed using Perl were documented using the excellent Perl documentation system that produces code information similar to unix man pages 7 2 User manual and application tour This section explains the how to use the Automated Testing Platform and is included in this report mainly as a tour of the application so that the final work can be analyzed and evaluated A movie is included in the CD Appendix D so a real example of the application in use can be observed 7 2 1 Application front page In the application s front page the user has access to the several conceptual areas in which the Automated Testing Platform is divided Manage test cases In this menu it is possible to create edit and remove tests from the test list There is also documentation about the tests and automatic stub generation tools Manage test areas Testing areas allow a user to group tests In this menu it is possible to create edit and remove testing areas 43 CHAPTER 7 DOCUMENTATION gt A a E a G http 192 168 90 137 atp Y G See Manage tests Manage test areas Manage campaigns List executions Show statistics Testing Certification and Benchmarking Figure 7 1 Main page interface Manage campaigns A test campaign is a set of tests to be executed In this menu it is possible to view create edit remove and execute a campaign List executions In this menu it is possible to view a summary and the detail
64. ist of tests to include It can t be possible to introduce more than the number of allowed characters in the text fields The same results of the previous tests must occur The list of campaigns must be narrowed to match the search Results archived The expected results were obtained Campaign executions Testing summary 1 2 3 4 5 Try executing a campaign with an old target machine and with a new target machine Try executing a campaign targeting a old version and a new version Try executing a campaign using a valid IP address and an invalid IP address Execute a campaign using the several execution strategies Try several executions of a campaign Expected results 1 5 A list of target machines already tested must be presented in a option menu If a new target machine is selected this machine must appear in the menu the next time a campaign is executed A list of versions already tested must be presented in a option menu If a new version is selected this version must appear in the menu the next time a campaign is executed If the IP address is valid the test must be executed If the IP address in not valid the test must be executed fail and stop in the beginning of the execution and output the proper error message The campaigns must be executed according to the selected execution strategy If there are several execution strategies in queue the one in the first campaign must be selected and
65. k IP belongs to the network 192 168 100 254 255 255 224 0 O Setting the network down Done 0 0999999999985 Done Current MAC 00 14 22 78 95 d4 unknown Faked MAC 00 14 22 78 95 d5 unknown 9109000090002 Changing the mac address Done Internet Systems Consortium DHCP Client V3 0 3 Copyright 2004 2005 Internet Systems Consortium Getting a dhcp lease All rights reserved 217 For info please visit http www isc org products DHCP Listening on LPF eth Checking if 192 168 100 173 is in the network IP belongs to the network 192 168 100 254 255 255 224 0 O Setting the network down Done 0 0999999999985 Done Current MAI 78 95 d5 unknown 125 442 s Changing the mac address RE S s un knew 0 100000000002 Done Internet Systems Consortium DHCP Client V3 0 3 Copyright 2004 2005 Internet Systems Consortium Getting a dhcp lease All rights reserved 23 7 For info please visit http www isc org products DHCP Listening on LPF eth Checking if 192 168 100 172 is in the IP belongs to the network 192 168 100 254 255 255 224 0 O network Setting the network down Done 0 0999999999985 Done Current MAC 00 14 22 78 95 d6 unknown Changing the mac address DER NA an en 0 100000000002 s c Add new area Figure 7 5 View executions interfaces CHAPTER 7 DOCUMENTATION RE o A a m http 192 168 90 137
66. kely to continue with white box testing in parallel to the beta tests Thus the term beta test can refer to the stage of the software closer to release than being in alpha or it can refer to the particular group and process being done at that stage So a tester might be continuing to work in white box testing while the software is in beta a stage but he or she would then not be part of the beta test group activity Regression testing Regression testing is any type of software testing which seeks to uncover regression bugs Regression bugs occur whenever software functionality that previously worked as desired stops working or no longer works in the same way that was previously planned Typically regression bugs occur as an unintended consequence of program changes Test case In software engineering a test case is a setof conditions or variables test steps under which a tester will determine if a requirement upon an application is partially or fully satisfied Written test cases include a description of the functionality to be tested taken from either the requirements or use cases and the preparation required to ensure that the test can be conducted Written test cases are usually collected into Test suites Scenario test A scenario test is a test based on a hypothetical story used to help a person think through a complex problem or system They can be as simple as a diagram for a testing environment or they could be a
67. l 06 31 Jul 06 07 Aug 06 14 Aug 06 21 Aug 06 28 Aug 06 FISISIMITIWITIFISISIMITIWITIFISISIMITIWITIFISISIMITIWITIFISISIMITIWITIFISISIMITIWIT y yn 4 ms _d Task Milestone 4 External Tasks Summary q External Milestone 4 Progress ME Project Summary G Deadline JL Project plan Date Wed 28 06 06 Split Page 5 APPENDIX B USER INTERFACE PROTOTYPES Appendix B User interface prototypes 59 APPENDIX B USER INTERFACE PROTOTYPES 60 a Front page interface NAME 1 TESTI DESC TESTA Desc TESTS DESC NAME2 TESTI Desc m f View campaign details interface e View campaign executions Figure B 1 User interface prototypes APPENDIX C EDGEBOX FLEUR Appendix C Edgebox Fleur 61 Web FTP and Proxy Server o VPN QO ADSL Modem Gans a O IP PBX p VolP Gateway Authentication Server WiFI AP e A Integrated Security Management Router Business Gateway for SMEs and Enterprise Branch Offices edgeBOX is a fully integrated network device with comprehensive networking security collaboration storage built in QoS and a feature rich IP PBX providing a complete converged voice and data solution for Small and Medium sized Enterprises SMEs Best Bicadhand Appliance 2005 Today affordable high speed internet
68. n Squid Samba Iptables Iproute2 Asterisk qpopper imapd postgreSQL tomcat proftpd net snmp freeradius OpenSSH mrtg mailscanner Copyright 2006 Critical Links S A All rights reserved Critical edgeBOX TEMS and their respective logos are trademarks of Critical Software SA All other trademarks are held by their respective companies Critical reserves the right to make changes to the product specifications at any time without notice European Headquarters Critical Links S A Parque Industrial de Taveiro Lote 48 3045 504 Coimbra Portugal Tel 351239989100 Fax 351 239 989 119 Lisbon Office Critical Software S A Polo Tecnol gico de Lisboa USA Office Critical Software Limited San Jose California USA UK Office Critical Software Technologies Ltd 72 New Bond Street London W1S IRR Critical links www critical links com edgebox critical links com APPENDIX D CD Appendix D CD 65
69. ng test scripts test execution and defect management Salome TMF Salome TMF is an independent Test Management Tool which manages the entire testing process by creating tests executing manual or automatic tests tracking results managing requirements and defects and producing HTML documentation Salome TMF is compatible with Junit Abbot and Beanshell to define automatic tests and with Bugzilla and Mantis to manage your defects Salome TMF can also be extended by plug in according to your requirements Bugzilla Test Runner Bugzilla Test Runner is a test case management system that works as an add on over the Bugzilla bug tracking system FitNesse FitNesse is a collaborative testing and documentation tool It provides a very simple way for teams to collabora tively create documents specify tests and run those tests QATraq Covers everything from defining test plans to writing test cases and recording results Rth A web based tool designed to manage requirements tests test results and defects throughout the application life cycle The tool provides a structured approach to software testing and increases the visibility of the testing process by creating a common repository for all test assets including requirements test cases test plans and test results Test case Web Test Case Web is an online TCM system built with PHP and a SQL backend It provides an efficient means for generation organization and execution reporting
70. nmn 3 x 1s 1 uognDaxaoubjedueo Te D _ OS 4eyouen yred uoTpNIaxa 20 O0Z 1e42J8A UOTIATIISAP 00Z JeyDJeA UOTIATIDSAP E lt A4 gt TT IUT pT ubtedueo lt j gt TT PUT PT 358 09 JeyaJen aueu lt Ad gt TI IUT PT oz J4eyoJea aueu lt j gt 3 TT UT pr ease O0Z 1e42J8A UOTIATIDSAP CHAPTER 5 ARCHITECTURE AND DETAILED DESIGN Test Communication Layer ObjectFactory __ Runner ol Figure 5 5 Components of the core module tables have keys that represent the logical connection among them and descriptive attributes like the name of the entity and sometimes the description of the class The class execution type is used to represent information about the execution of the class definition Tables campaignexecution testexecution and teststep have information about the objective of the execution the target of the execution the way that the execution in going to be performed e g parallel or sequential and time information about the start of the execution The results of execution type is used to store the information about the results of an execution The tables cam paignresult testexecutionresult and teststepresult include information about to which execution they belong an integer result of the execution using the posix returning results convention a textual description of the result in order to include some automatic error information retrieval and information about
71. nterface Data access ObjectRecorder When important changes occur in the internal data objects the ObjectRecorder saves the information back in the database so it can be accessible to the user Test communication ObjectFactory Once that the creation of test steps is dynamic and can happen in runtime when the test reports a new test step the communication layer constructs the object automatically and appends it to the internal data structures Testing script communication As explained before the communication with the testing script uses sockets abstracted in a multi operating system library The messages passed using XML as defined in the following document type definition DTD lt DOCTYPE test lt ELEMENT test test logfile description status gt lt ATTLIST test name CDATA REQUIRED gt lt ELEMENT description PCDATA gt lt ELEMENT logfile PCDATA gt lt ELEMENT status PASS FAIL gt gt 5 5 2 User interface The web interface chosen platform is PHP4 producing a web accessible HTTP interface This interface is very flexible because the HTTP protocol is a wide spread standard and can be accessed from a remote computer The user interface has two main layers The data access layer produces an abstraction to the database communi cation The web interface layer shows information and allows users to pass actions to the core application The data a
72. ntroduction starts by presenting the company Critical Software where the internship occurred The product to which the software developed is applied is also introduced in this chapter The introduction is finished by presenting the internship goals tasks and planning Chapter 2 The product a network appliance includes information about the edgeBOX A business overview the organization the market competition and product architecture are the topics explored Chapter 3 The project automated testing platform introduces the area of testing by giving an overview of the modern concepts in automated testing the domain of the problem and the state of the art Chapter 4 Requirements analysis specifies the requirements for the automated testing application to be developed and tries to provide the reader information to understand the application by the means of use cases and usage scenarios Chapter 5 Architecture and detailed design explains the architecture and technologies by reviewing in detail the system each of its modules and the data structures used Chapter 6 Testing and software quality assurance sums up the testing done to the application by including a summary of the test cases specified and by explaining the unit testing code inspection and beta testing approaches Chapter 7 Documentation describes the approach and tools used to document the system and provides the reader with manuals to use and install the platform These manu
73. o the DHCP server was also performed in an automated way since the simulation of several users requesting IP addresses is difficult to do manually During this phase the requests comments and observations of tester and developers were taken in account and later integrated in the application or left in the future improvements list Most of these comments were usability issues and particular uses that were unforeseen in the planification of the application In spite of non planning of these functionalities considering the modular design of the application the changes won t affect a significant amount of code 41 CHAPTER 7 DOCUMENTATION Chapter 7 Documentation Every technology really needs to be shipped with a special manual not how to use it but why when and for what Alan Kay The idea is to try to give all the information to help others to judge the value of your contribution not just the information that leads to judgment in one particular direction or another Richard Feynman Hungarians assume that everything will soon break And they usually want to fix it themselves So good Hungarian manuals read more like machine shop specifications than user guides J nos Kosztol nyi 7 1 Approach to documentation The implementation of the Automated Testing Platform was documented using Doxygen Doxygen generates several kinds of outputs e g LaTeX and HTML that can give an insight to the details of the code The tests scr
74. oes not do what the user expects A fault is a programming error that may or may not actually manifest as a failure A fault can also be described as an error in the correctness of the semantic of a computer program A fault will become a failure if the exact computation conditions are met White box and black box testing In the terminology of testing professionals software and some hardware the phrases white box or glass box clear box and black box testing refer to whether the test case developer has access to the source code of the software under test and whether the testing is done through simulated user interfaces or through the application programming interfaces either exposed by published or internal to the target In white box testing the test developer has access to the source code and can write code that links into the libraries which are linked into the target software This is typical of unit tests which only test parts of a software system They ensure that components used in the construction are functional and robust to some degree In black box testing the test engineer only accesses the software through the same interfaces that the customer or user would or possibly through remotely controllable automation interfaces that connect another computer or another process into the target of the test For example a test harness might push virtual keystrokes and mouse or other pointer operations into a program t
75. or SQL STAF plugin 3 4 Comparison Many of the tools mentioned above are non commercial and have a clear lack of quality simplicity and good architecture design The exception is Test Director Test Director is a commercial expensive and over complete tool Therefore it is not an option on the short term The advantages of the development of an testing platform from scratch are A tool tailored to the networking product that is supposed to test The merge of functional testing load and stress testing and benchmarks on one tool e An architecture suitable for network distribution Ease of developing collaboration with other development tools used in the project More on this topic will be discussed in the results analysis Chapter 8 19 CHAPTER 4 REQUIREMENTS ANALYSIS Chapter 4 Requirements analysis Not everything that can be counted counts and not everything that counts can be counted Albert Einstein Requirements are like water They re easier to build on when they re frozen Anonymous The hardest part of the software task is arriving at a complete and consistent specification and much of the essence of building a program is in fact the debugging of the specification Frederick P Brooks 4 1 Automated testing platform definition An automated testing platform is a software tool that allows an automatic execution of previously defined auto mated test cases An automated test case can be an automa
76. s of running past and pending executions Show statistics In this menu it is possible to view execution time statistics of campaigns tests and versions An image of the application s front page is available in figure 7 1 7 2 2 Manage tests When opening the Manage test menu the user is presented with the list of tests in the system Figure 7 2a Then using the search box on the right it s possible to filter the list by keywords Figure 7 2b To create a new automated test case click on the link on the right side A new form will appear where the test name a descriptive text are inserted and the area in which the test in included is selected from the list of existing areas Figure 7 2c By submitting the form the page is reloaded a confirmation that the test was included in the application is presented and the test is listed in the list of tests A test can be deleted or edited by selecting the proper operation in operation column in the list of test To edit a similar form as to add a new test is presented with the current test definitions After adding a new test it is possible to create the skeleton of the test script by pressing the Create stub link Figure 7 2d This stub is created in the tests directory in the base folder of the application Then the user have to edit the test by programming in the language Perl Other languages can be used just by declaring the language in the begging of the script in the standard unix
77. s required 4 5 3 Usage SC NATOS nony A a Dee A A ar wens Non functional requirements 4 6 1 Usability su pue mada Bu ee we ee ee ee 4 6 2 Time requiiements u suas vo a ne Die e es 4 6 3 Architecture requirements 4 6 4 Hardware considerations r rr r rr sss rs eee Us rinterface prototypes acaso un aa re Rukun UN S TREE O US RRA SCOPES sua BY Sie il w range up a BEC O Riga rag a Su S p eds ia a Architecture and detailed design 5 1 3 2 5 3 Design standards co pu sn b pa a ee A EA Integration in the system a na U S Q SYSIEM OVEIVIEW El sed BA A a Gee web E A ee 5 3 1 Architecture justification r r rr res ee 5 3 2 Design Patterns casu ssa en le ph Lego tbe dits 15 15 17 17 19 21 21 21 22 22 22 22 25 25 26 26 26 26 26 26 26 CONTENTS 24 Data model ss iia et RE de she le EB RN a 31 34 1 Class model o 2 A a ne a a Meee She Seles 31 942 Physical model sus ana k a ee s 2 re ke E 31 3 3 System decomposition cus sara C a a nie N Eden 33 3931 Coteimodule ss 554 4 re date bei ha RS dd tae p N es 33 99 2 US rintertac rs s gears us be A ee Re At 34 3 3 3 Automated test case amis vr a se re ei 35 Testing and software quality assurance 37 6 1 SCOpE OL ests 2 2645 VR RR a E ach en Dh 37 6 27 Schedules rs mise awa e Bee a He RA RE ee Bach 37 6 3 Approa
78. ted functionality test an automated load test or a benchmark The focus of the tool is the management and operations over test cases testing areas and testing campaigns and the respective results 4 2 Objectives In every release of developed software new functionalities added can interfere with the core functions imple mented previously As mentioned it is a resources consuming task to in each release cycle test all features again Therefore it is extremely useful to have a tool that tests in an automated way the core functionalities Having an automated testing platform it can be used to benchmark the performance of the functionalities and to test them under high usage load The objectives expected to reach are e To develop an Automated Testing Platform ATP to manage and execute tests of a set of functionalities of the edgeBOX project e To make a general automated testing and benchmarking platform easily scalable and with the flexibility to be ported to different but with similar architecture projects e To have a simple graphical interface to define which tests are going to be executed 1 The definition of concepts used is provided in section 3 1 21 CHAPTER 4 REQUIREMENTS ANALYSIS lt lt Tester Administrator Figure 4 1 Actors e To have a simple graphical interface to execute the desired actions e To have a simple graphical interface to see the results of the executed tests e To allow inspection
79. tep 5 3 1 Architecture justification This architecture was selected considering the several requirements defined The requirement of scalability for a distributed system is of particular importance when the objective is to stress the target machine In order to fulfill 29 CHAPTER 5 ARCHITECTURE AND DETAILED DESIGN this requirement having a central server to schedule where and when the programs are going to be executed is fun damental That was also the reason of the option for standard tested and efficient TCP sockets for communication between the core application and the automated test cases The reasons for the use of Perl to develop the test APIs was that it already exists a broad code base using that uses Perl and that can be reused Anyway if there is interest in developing an APl in other language for instance Python this should be a straight forward operation and no modifications in the other modules are required The language PHP was the option to use in the user interface because of the ease of use and the short development effort needed The database choice was the open source complete and efficient mysql This option is not of most importance since all modules that have interaction with the database have data access layers that use standard and generic database abstraction libraries 5 3 2 Design Patterns With the objective of having a very modular scalable and maintainable platform the following design patterns wer
80. ternship protocol and the non disclosure agreement signed Acknowledgements No one who achieves success does so without acknowledging the help of others The wise and confident acknowledge this help with gratitude Alfred Whitehead My acknowledgement to Critical Software S A for the opportunity offered of a great internship the warm welcome and outstanding work conditions To my university tutor in the Engineering University of the University of Porto Jorge Barbosa for his constant availability and support To my company tutor in Critical Software S A Alexandre Esper for his successful efforts for my integration within the team the development of a interesting project and above all the opportunity of learning To all Critical Software workers and interns for with no exceptions their companionship and support A special thanks to Nelson Vale Bruno Ramos Andre Barbosa Andre Lemos Helder Sousa Goncalo Jesus Claudia Serafim and Rui Portugal Being this the end of an important cycle in my life and the beginning of a new one I thank my family and my friends who always helped me in the course of my life xi Guide to readers It is well to read everything of something and something of everything Lord Henry P Brougham This documents tries to describe the work done at Critical Software between March and August of 2006 in a chronological order and trying to give insights of the software development stages Chapter 1 I
81. test 36 vii LIST OF FIGURES viii 7 1 T2 7 3 7 4 7 5 7 6 B 1 Main page interfaces sr A ne ea af rs 44 Manage tests Interfaces os a aus ans a a ea SU s a eng aa ge 45 Manag ar as Interfaces us ou dde a a tn pros kak ES rie ma 46 Manage campaigns interfaces a ss so 47 View executions interfaces 48 VIEW Statistics gt interfaces ar un en UE ee ee DS PS S Ree ne 49 User Interface prototypes ss ama s Hans sinn as Ben Boe ne ng Be Ben 60 Preface This document intents do describe the work developed at Critical Software between March 2006 and August 2006 in the scope of the internship project Analysis design and development of an automated testing platform for a multi function network appliance by Rui Ferreira The language of choice was English since it is the standard language for the production of documentation at Critical Software and it allows this document to reach a wider audience The target audience is the jury appointed to the evaluation of the internship but this can be considered an introduc tion to the automated testing platform described here or even to automated testing in general The project and all associated information was developed and producted under contractual agreement among Critical Software S A and University of Porto Therefore the access is restricted according to the defined in the in
82. test specification and the test execution The software quality assurance used follows the Critical s Quality Management System QMS and has the ob jective of producing high quality standards software Most of the process described here uses the tool Enterprise Architect that simplifies the organization of testing plans and software quality assurance 6 2 Schedule The testing process was distributed along the development of the software Although this temporal distribution there were three periods where there was a bigger focus on testing From 21st to 29th of June the testing was targeted to the functionalities offered by the prototype From the 18th of July to the Ist of August the platform was tested in the release 4 1 of the edgeBOX by beta usage of the implemented automated tests e g test of the backup and restore feature In the first fifteen days of August a more methodical approach to tests was taken including the documentation of unit tests performed during the development specified functional tests and code inspection 6 3 Approach The test of features is going to be based almost completely on tests of the use cases Another crucial point is in the proof of user introduced errors on the use of library of communication with the application s core with the test scripts This last test has particular importance since it could cause testing data to be badly organized or the application to run unpredictably 37 CHAPTER 6 TEST
83. testing platform allows users to add extensions to the platform in form of new tests New test can be added through the user interface without the modification or compilation of the platform s code This is archived by using the client server model and the interface pattern 5 4 Data model Considering the domain model represented in figure 3 1 data models were constructed and refined with the ob jective to have appropriate data structure and model with efficient performance and expansion capabilities 5 4 1 Class model Figure 5 3 includes the class diagram used in the architecture of the system This figure is a refinement of the domain model Figure 3 1 The most significant change is the introduction of a class to represent the campaign execution with the objective of simplifying the implementation of the requirements that include operations with campaigns On this diagram the level of detail was also increased by the specification of the attributes in each class An in depth view of the attributes is presented in the next section 5 4 2 Physical model The database model used in the ATP is in the third normal form and is represented in the figure 5 4 It was an option taken not to go in much detail since the much of the data and relationships were already explained The tables defined can be classified on three different types The class definition type the class execution type and the results of execution The class definition t
84. th 3DES encryption and MD5 authentication PPTP Connections enable the establishment of VPN tunnels across the internet for secure remote access by teleworkers Figure 3 Illustrative edgeBOX deployment scenario Integrated Voice and Data A Software PBX and VoIP Gateway edgeBOX includes a complete soft PBX based on Asterisk providing all of the features expected from a PBX such as IVR Interactive Voice Response ACD Automatic Call Distribution digital receptionist incoming call rules sound manager call routing forwarding queuing and parking LCR tables conference bridging hunt groups telephone number mapping ENUM music on hold email forwarding and user management with call permissions To ensure the quality of VoIP calls edgeBOX provides QoS Quality of Service and bandwidth management which allows for reservation and prioritisation of a percentage of available bandwidth for voice traffic Branch offices can route calls between edgeBOX units at each site over secure IPSec VPNs and teleworkers can access their extensions over secure PPTP VPNs This allows free calls to be made between all staff members regardless of where they are located Calls can also be routed through an ITSP Internet Telephony Service Provider that provides free calls to other IP Telephony users and reduced rate calls to land line numbers and mobile phones The combined effect of these cost savings can greatly reduce a company s phone bill
85. the other campaigns only can run after all the tests within the execution strategy are finished The several campaigns must be inserted in the campaigns waiting to run queue Results archived The expected results were obtained 39 CHAPTER 6 TESTING AND SOFTWARE QUALITY ASSURANCE Campaign lists and campaign details Testing summary 1 View list of campaigns 2 View details of a campaign Expected results 1 The campaigns must be in three different lists The campaign can be in queue for execution can be executing or with its execution complete The campaigns finished executing must appear with different colors if they fail or pass 2 Ifa campaign is running the details must appear dynamically without refresh of the page If the campaign execution is completed all details must appear Also the result must be differentiated with colors and the running time of test steps must be available to the user Results archived The expected results were obtained Execution statistics Testing summary 1 View execution statistics page Expected results 1 The metrics shown must only consider the tests executed with success The metrics are to be organized by campaign test case target machine and version tested Results archived The expected results were obtained Some but not all of these tests were produced before the implementation of the application They were used as acceptance tests and
86. tistics Create stub R ve Manage campaigns New test here Description Backup and Restore d Create a test stub Figure 7 2 Manage tests interfaces 45 CHAPTER 7 DOCUMENTATION gt O A a WF nc http 192 168 90 137 atp index php query manage_areas Manage tests Manage testareas Manage campaigns List executions Show statistics Simple functional examples The tests in this area are simple functional tests of the edgeBOX Boasts Backup and Restore The tests related to Backup and Restore operations Kane Services The tests related to edgeBOX s services Remove Edit a List of areas gt O As a WF a G http 192 168 90 137 atp index php query manage_areas amp filter services Y G 0086 Manage tests Manage testareas Manage campaigns List executions Show statistics Services The tests related to edgeBOX s services FET services Add new test area Area name New area Description Area description Press here to add edit a new test area b Add new area Figure 7 3 Manage areas interfaces 7 2 4 Manage campaigns The manage campaign menu lists the campaigns in the system Figure 7 4a and allows searching by writing the keywords in the search field in the right To view the tests included in each campaign it is possible to click in the folder s icon and the new more detailed list will appear Figure 7 4b To
87. tugal Southapton in the United Kingdom and San Jose California in the USA Figure 1 1 Critical Links is a wholly owned subsidiary of Critical Software that develops and markets edgeBOX a multi function server appliance or business gateway targeted at Small Medium Enterprises SME and Enterprise Branch Offices EBO The management and product team behind Critical Links builds on the success and reputation of the parent com pany Critical Software and leverages its technical talent and market experience 1 1 2 Company profile Critical Software success is based upon the use of high quality levels and technological innovation as agents in the introduction competitive advantages in clients and partners The company currently commercializes in the international market innovative products in areas as dependability and high performance computing Accordingly the company invests heavily on R amp D around 12 of total revenues as well as in its quality system that have been certified to demanding international standards Critical is flexible and agile and has proved that it can respond quickly and efficiently to customers rapidly changing business environments From its foundation in 1998 Critical has received several awards and distinc tions Among them it has been listed in Business Week s Top 300 fastest growing companies and has been awarded by INSEAD for Management excellence Critical has a strong international customer bas
88. ty of Service Offers bandwidth constraining by class of user Storage Allows users to create personal network storage public storage and group storage Collaboration Includes mail server webmail web server CMS server The edgeBOX uses community open source software but brings the ease of use of professional software by offering an user friendly and idiot proof web interface It ends with manual edition of configuration files but still has the flexibility to be used in small medium enterprises or enterprise branch offices Another advantage is that it is certified for specific hardware appropriate to different company sizes A business organizational and technical analysis of the product and its organization can be found in chapter 2 1 2 2 Items Items is a tool for the batch configuration of several edgeBOX This tool makes business sense to companies that want to offer edgeBOX s services as a added value service to their core products and services Once again in chapter 2 more information can be found on this topic 1 3 The internship 1 3 1 Global goal The global goal of the internship is the development and integration of open source based technologies and tools to the construction of an automated testing platform for edgeBOX s testbed CHAPTER 1 INTRODUCTION 1 3 2 Workplace The internship took place in Coimbra s premises Figure 1 1a where the edgeBOX development team is based 1 3 3 Objectives 1 Acquire
89. xecution time is 0 316 seconds with the standard deviation of 0 53 seconds 4 0 3 This test was executed 12 times in this version The average execution time is 0 491 seconds with the standard deviation of 0 331 seconds DHCP This campaign was executed 3 times The average execution time is 140 214 seconds with the standard deviation of 1 589 seconds HTTP This campaign was executed 11 times The average execution time is 12 588 seconds with the standard deviation of 3 866 seconds b Statistics by campaign test machine and version Figure 7 6 View statistics interfaces were successfully executed red campaigns failed to execute or the result of the execution was not the expected To view the execution details follow the link in the execution list If the execution is running the results will appear dynamically using AJAX technology If the execution is over all information is available Figure 7 5b and further details can be analyzed by expanding each execution Figure 7 5c This interface also provides a way to see reports by clicking in the pdf icon or to manually change the result status of the execution 7 2 6 View statistics The statistics page has a list of the average running time and standard deviation of the successful executions of each campaign If the information is expanded by pressing the folder link this information is grouped by the tests included in the campaign If further expansion is requested the information is
90. ype represents the tables that define the concepts used in the domain These concepts definition includes the tables version machine testarea testcasedefinition campaign and the table campaign test that is used to represent the n to n connection between a campaign and a testcasedefinition The attributes of these 31 CHAPTER 5 ARCHITECTURE AND DETAILED DESIGN TT AUT auT UOTN28X8 O0Z 1ey2J8A UOTIATIDSAP TT 1UT 1njssaDons sem duejsaut ep pus UOTINIAXA oz 4eyoJea aueu 0Z JeyoJea aueu lt xd gt TT UT PT TI AUT uty butuuns 002 JeyJen uotidtJDsap lt yd gt TT UT pr lt Ad gt TT pI 00Z 4eyoJea uoTIdTJIsap TT JUT 1nyssaoans sem cuypew uosa yns21d9 s 59 TT UT 11NJSS829NS sem lt Ad gt TI AUT tpt H a lt Ad gt _ TT 1UT pr yinsaiubiedwes E 3 ns3 140 1N99X3 59 lt o 002 Jey gt Jea asodund S due ssut ep 31815 UOTINIAXA x Y lt j gt TI UT pr pinsau aa Kato oo atea TT UT d y uotjezT 31 e4ed lt J gt TT UT PY uorynaaxejsa dwejsauT 8 8p 118 5 UOTINIAXA 3 lt Ad gt i TT IUT PE sT ueyoJea sseuppe dt jabuey 007 JeyoJea UOT AT1258p E lt j gt TT UT pT qnsau q dwejsawt ajep 11815 UOTINDaxa lt 44 gt IT AUT pt auTyoeuw 9 lt 4 gt TI IUT PT UOTITUTL8P S8 lt AJ gt TI IUT PT UOTSI8A 2 i lt j gt TT 1UT pT y nsau lt g gt TI IUT pT ubredues xd ID YuT pt lt xd gt ID YuT pr pal uo

Analysis, design and development of an automated testing platform

Contents

Download Pdf Manuals

Related Search

Related Contents