Wiley Optimize Quality for Business Outcomes: A Practical Approach to Software Testing
Contents
1. 1994 0 20 40 60 80 100 Succeeded Challenged _ Failed Figure 1 IT project success rate has changed over the past 10 years To learn why the success rate is so low we wondered Why were the successful projects successful The Standish group also collected data on success factors ask ing respondents what contributed to their project s success Executive support and user involvement were high on the list Here are the other factors that figured in project success m Executive support 18 a User involvement 16 m Experienced project manager 14 Clear business objectives 12 m Minimized scope 10 a Standard software infrastructure 8 m Firm basic requirements 6 4 4 A MQM book Page 4 Wednesday May 7 2008 11 29 AM 4 sm Optimize Quality for Business Outcomes a Reliable estimates 5 a Other criteria 5 E While we expected to see executive support and experienced project management we were interested in other noteworthy elements like user involvement clear business objectives and firm basic requirements on the list All three of those factors have to do with the end user or the business which leads us to the next question Why is user involvement so important The answer lies in the following statistics Figure 2 based on a 2002 study conducted by the National Institute of Science and Technology NIST Understanding Defects Introduction of Defects 802. Majority of defects are 60 introduced during the ens requirements and design phase 20 i E Requirement Coding amp User Production amp Design Unit test Acceptance Test Detection of Defects s However majority of 80 defects are actually detected during user acceptance testing and 40 in production o A Requirement Coding amp User Production amp Design Unittest Acceptance Test 60 Source NIST 2002 RTI Project 7007 011 Figure 2 Comparison of where defects are introduced against where they are discovered Figure 2 shows that most defects are introduced in the require ments phase of the traditional software development lifecycle SDLC 4 4 A MQM book Page 5 Wednesday May 7 2008 11 29 AM A What Is the Big Deal About Testing m 5 Defects in this phase happen mostly because the requirements themselves are incomplete ambiguous or contradictory When unclear requirements are translated into developers technical specifications misalignment of the original ambigu ous business requirement and misinterpreted technical specifi cation results in significant rework Interestingly the same people who created the requirements and introduced problems in the requirements phase couldn t see their mistakes until the user acceptance phase when they could spot them So we recommend as a best practice for those people creating requirements double check i
3. A MQM book Page 1 Wednesday May 7 2008 11 29 AM MIB d 7 Chapter 1 What Is the Big Deal About Testing Time waste differs from material waste in that there can be no salvage The easiest of all wastes and the hardest to correct is the waste of time because wasted time does not litter the floor like wasted material Henry Ford This book is about saving time The work we ve done in the IT industry for several years has taught us that professional test ing can help a business save a significant amount of money on product development reduce time to market and gain satis fied customers Finding defects early in the development process makes for a product development cycle that satisfies the business stake holders and the customers we can vouch for that In manufacturing for instance a perfect example of the approach we favor is Toyota s In a huge departure from the hallowed big three U S automakers Toyota aligned its busi ness requirements with its users m They test a cup holder like a user would not like a designer 9 4 A MQM book Page 2 Wednesday May 7 2008 11 29 AM 2 Optimize Quality for Business Outcomes m Any person on the manufacturing floor can stop the entire manufacturing process if they notice something is not right Everybody on the assembly line is a quality checker m When a problem is spotted early it s a lot cheaper to fix at that point i
4. Page 9 Wednesday May 7 2008 11 29 AM A What Is the Big Deal About Testing m 9 The answer lies in the cost of fixing defects in various stages of the SDLC Figure 3 Assessing Cost of Defects Costs of Correcting Defects Source B Boehm and V Basil Software Defect Reduction Top 10 List EEE Computer Design Coding Testing Maint Software Development Lifecycle Phase o Defect Corrected This industry average is used as a baseline for arriving at cost savings industry References 3 8 Boehm and V Basili Software Defect Reduction Top 10 List EEE Computer IEEE Computer Society Vol 34 No 1 January 2001 pp 135 137 Figure 3 The cost of correcting defects increases dramatically as we move toward the end of the SDLC If we are complacent neglecting to identify defects introduced in the requirements phase until we get to user acceptance test ing we lose approximately 7 000 per defect This quickly adds up to significant numbers It also results in the failure of many IT projects when most of the budget for the project is already spent and a huge additional monetary outlay is needed toward the end of the project In our bicycle car example all the extra work and investment was for something essentially useless to the business In a busi ness situation if we consider other indirect costs that crop up from added time to market and increased project risk the cost can really break a business This l
5. data flow testing branch condition testing branch condition combination testing and modified condition deci sion testing For detailed descriptions of these test techniques see Appendix A Black Box Testing In white box testing we saw that the tester usually the developer had insight into how the application was devel oped However when a third party or impartial tester does not know how the programs were developed they can conduct black box testing In computer programming software engineering and software testing we use black box testing also known as concrete box or functional testing to check that the outputs of a program given certain inputs conform to the functional specification of the program In electrical hardware testing we black box test the specifications of the interface between the device and appli cation circuit The term black box indicates that the tester does not examine the internal implementation of the program being executed For this reason black box testing is not normally carried out by the programmer In most engineering firms one group does design work and a separate group does the testing 4 4 is A MQM book Page 27 Wednesday May 7 2008 11 29 AM A What Is the Big Deal About Testing m 27 The most common techniques and approaches used in black box testing are the smoke test equivalence partitioning boundary value analysis and user input validation For det
6. ailed descriptions of these tests techniques see Appendix A d Wrapping Up By now you can see how testing can help your company s bot tom line in a new product release saving you considerable money by catching flaws early in the process and reducing rework for your team We ve showed you how our approach works in the software field but you can apply this method in most any industry and it s an excellent way to troubleshoot internal business pro cesses By setting out clear requirements that align with your business goals you ll save yourself some headaches and preserve good customer relations Our experiences with other test methodologies have led us to adopt an approach to test design that departs somewhat from traditional testing It s aligned with business outcomes and is focused on the end user experience testing to make sure the customer will have a satisfactory experience not just that developers see the results they expect But before we start testing we have to better understand what to test A ia MQM book Page 28 Wednesday May 7 2008 11 29 AM 28 m Optimize Quality for Business Outcomes
7. are nor mally defined by the operations team or if the operations task is outsourced these quality gates can be defined by a combina tion of the service providers that host the application and the internal operations team Test Concepts and Techniques Many test techniques have been developed over the years for the different test phases but there are two basic types of test The white box test and the black box test White Box Testing White box testing is also known as clear box testing glass box testing or structural testing The term white box or glass box indicates that the tester knows the code used to execute certain functionality 4 4 is A MQM book Page 26 Wednesday May 7 2008 11 29 AM A 26 m Optimize Quality for Business Outcomes In software testing it s usually a programmer that performs white box tests Often multiple programmers will write tests based on certain code to gain various perspectives on possible outcomes Up In the fields of computer programming software engineering and software testing white box testing is used to check that the outputs of a program given certain inputs conform to the structural specifications of the program In electrical hardware testing every node in a circuit may be probed and measured in circuit test or ICT The most common techniques and approaches used in white box testing are syntax testing statement testing branch deci sion testing
8. before they can use the system The customer related information is stored in the centralized cus tomer relationship management CRM system along with users passwords ol mi a oda A MQM book Page 18 Wednesday May 7 2008 11 29 AM 18 m Optimize Quality for Business Outcomes The following functions are needed m Login m Log out Register m Search flight Book flight Requirements Verification The prerequisite for the Requirements Verification phase is broken down into logical business functions Figure 6 Functional Decomposition example I iit nan fi i Figure 6 The business process is broken down into business functions The Requirements Verification phase documents the accep tance criteria for each requirement of each separate business function Suppose one requirement for the business function LOGIN is The user has to enter a username and a password By pressing gt NW Alls 4 4 is A MQM book Page 19 Wednesday May 7 2008 11 29 AM A What Is the Big Deal About Testing m 19 a lt LOG IN gt button or ENTER on the keyboard the validation of username and password should start d The acceptance criteria for that requirement could be m fa username without a password is entered an error message is displayed m Ifthe username and the password are a correct combi nation the system shows the flight search scree
9. eads to the final question Why can t we make use of this knowledge to improve the testing process gt lt ii 4 4 is A MQM book Page 10 Wednesday May 7 2008 11 29 AM A 10 m Optimize Quality for Business Outcomes To answer that question let s first look at some testing defini tions up Testing is about finding defects bugs or flaws This is true for software as well as for all other products built to suit specific needs Testing is done by everyone all the time so why bother If we start looking at the testing process more carefully we find that in most cases testing is not just about finding defects but also about knowing whether a product meets business expecta tions Testing is to determine if a product meets business expectations Although this sounds pretty much the same there is a funda mental difference A defect is any flaw or imperfection in a software product or process This implies that a definition of the perfect working product has been made and is clear to all stakeholders It s not a defect just because the expectations of an individual tester have not been met Unfortunately intuitive testing based on undocumented expec tations is a common approach in the industry today This test ing approach is limited because it only works with the completed product and does not allow testing to be done early enough in the SDLC or in parallel with development The answer to
10. ed previous testing phases Unit System Inte gration Performance and UAT Operational Readiness testing ensures that the system we are about to deliver has the appropriate production data loaded is ready to communicate with required external systems and is ready to run the necessary regular batch processes while meet ing any specific security and compliance requirements For the flight application we might check to make sure that the flight data is being populated correctly that billing processes that use an external system are working and that the ticket purchase process is done through secure channels so users are not worried about sharing their private information 4 4 is A MQM book Page 25 Wednesday May 7 2008 11 29 AM A What Is the Big Deal About Testing m 25 This test phase is the last check of the production system for operational readiness before the system go live point This phase should be a part of the Project Implementation Plan d Key elements of Operational Readiness testing are m End user setup m Role definitions and requirements Security allocations and passwords End to end connectivity verification Production data verification m Core business functionality verification Removable or read only system interaction Application monitoring tool verification m Call Service center support setup Support documentation The quality gates for the Operational Readiness Test
11. h a system or compo nent is operational when required for use Scalability How easily a system or component can handle growing amounts of work in a graceful manner or the criteria to readily enlarge the capacity of a system vertically or horizontally Obviously it is not possible to test against all the different goals in one step So testing is usually conducted using test phases in a phased approach Each test phase concentrates on phase spe cific quality goals and requires a different level of readiness of the product PA d ki A MQM book Page 13 Wednesday May 7 2008 11 29 AM What Is the Big Deal About Testing m 13 This phased approach is a common practice in engineering and is done to assure the quality of a single unit before it is verified as an integrated unit to other units The positive effects are obvious When you discover an error before getting too far down the production line it pays off in tremendous cost sav ings It s easy to troubleshoot a product or process or do root cause analysis when each element has been tested For instance if you re building a brick wall each brick has been tested before being placed in the wall so you know the problem is in the wall structure you don t have to test the bricks again You enter every integration step with a known quality of the components the bricks Testing is easier because the functionality to be tested is limited to
12. h evaluation point such as a true false decision been executed and tested m Path Coverage Has every possible route through a given part of the code been executed and tested Integration Test During the Integration Test we first look at the implementation of each business function After we validate that the business function is functioning correctly we can move to the next phase the various combinations of business functions Business Function Test We re testing these business functions in our example applica tions m Login Log out Register m Flight search m Flight booking We test each function independently as soon as it has passed the Unit Test using various approaches to complete the func tion test We suggest using equivalence partitioning for the test case development we ll explain more in Chapters 4 and 5 Regardless of the applied methodology the quality gates need to be defined and met Possible quality gates for the Business Function Test are m All planned test cases have been executed m All incidents have been logged as defects a ag a A MQM book Page 21 Wednesday May 7 2008 11 29 AM A What Is the Big Deal About Testing m 21 a All identified defects have been resolved Resolved means they re either closed or a decision has been made to delay the closing Business Process Test When it comes to the Business Process Test we want to make sure to test the var
13. he question What would make me sign off on this require ment The acceptance criteria needs to m Validate the completeness and correctness of the requirements m Associate the requirements with the business functions a Eliminate ambiguities m Validate that the result of each requirement is testable m Identify the business functions that are changed during this project maintenance projects only f gt J als is A MQM book Page 15 Wednesday May 7 2008 11 29 AM A What Is the Big Deal About Testing m 15 Unit Test The Unit Test is a white box test conducted by the developers to find out whether the application is stable and completely implemented from a developer s perspective We look into white box and black box testing later in this chapter d Unit testing validates that a particular module of source code is working as the developer intended A developer conducts the testing not an end user A unit has to be executable and the developer uses a debugger or self developed tools to find out whether the code meets his expectations and the technical specification Integration Test The Integration Test phase has two stages m Business Function Test Business Function testing is the first black box test stage where the functionality of each business function is tested against the require ments m Business Process Test Business Process testing checks the business
14. ious possible business function sequences In our flight booking example the possible process is shown in Figure 7 Business Process example To be tested Flight booking Figure 7 Sample business process for a flights application It s important to test each possible connection between the business functions once The focus is the transition from one business function to the next and not the functional correctness of the single business function addressed during the Business Function Test Possible quality gates for the Business Process Test might be m All end to end processes can be executed a A workaround exists for all defects found e MP iS A i 4 4 A MQM book Page 22 Wednesday May 7 2008 11 29 AM 22 m Optimize Quality for Business Outcomes E System Test During the System Test each individual application should have passed the quality gates for the Integration Test The Sys tem Test often involves large and complex environments with multiple independent applications working together to sup port a business process Figure 8 Flight System Architecture CRM Application Flight Planning MI System Flight Booking Over the Web ERP ail Application Print Facility Figure 8 A flight booking system is supported by multiple independent applications The System Test makes sure various applications work together as the business process re
15. lls co MQM book Page 7 Wednesday May 7 2008 11 29 AM What Is the Big Deal About Testing m 7 At this point people rarely go back to their initial assumption in this case that the customer needs a bicycle to find out whether the additional information the customer doesn t want to get wet and needs to carry a briefcase means the whole design should be reconsidered The reason seems to be that we make decisions based on our own perspective of the situation rather than use a mechanism that would help obtain a more complete picture As a result we attempt to solve the problem without solving the problem Aa Q ls A MQM book Page 8 Wednesday May 7 2008 11 29 AM 8 m Optimize Quality for Business Outcomes E Despite all efforts to the contrary we ve got an unhappy customer along with a wasted invest ment into something that doesn t solve the problem After more iterations the solution that the contractor comes up with will be more what the customer originally wanted and the problem is solved But taking the time to sort that out in today s dynamic marketplace means we have compromised m Time to market m Cost efficiency m Efficiency of the solution Applying this common example to the concepts of software development we come to the next question Why is the late discovery of defects a major concern as long as defects are detected and fixed a ls A MQM book
16. n m Ifthe username and the password are not a valid combi nation an error message is displayed Ideally the business analyst or business user writing the accep tance criteria identifies missing information and ambiguities in the requirements documents those as defects and hands them over to the development team Possible quality gates for the Requirements Verification phase are m One hundred percent of the requirements have at least one documented acceptance criterion such as the online flight booking m We have divided the application into business functions and prioritized each business function according to the quality needed to meet the business goal of allowing the user to book a flight online Unit Test Unit testing is part of the development lifecycle so we don t make specific recommendations for this phase in our airline booking example However we need to agree on the quality gates for the Unit Test because it defines the quality level we can expect when the software is handed over In other words if the software does not meet the defined criteria the test team will not accept it for the Integration Test 4 4 is A MQM book Page 20 Wednesday May 7 2008 11 29 AM 20 m Optimize Quality for Business Outcomes Possible quality gates for the Unit Test are d E m Statement Coverage Has each line of the source code been executed and tested m Condition Coverage Has eac
17. nitial requirements for ambigu ity contradictions and incompleteness This effect is not limited to software The following example shows what can happen in situations where people omit neces sary details they have a clear idea of what they want and have made preconceived assumptions but neglect to commu nicate the specifics of what they require I WANT SOMETHING TO GETME ACROSS TOWN IN THE SHORTEST TIME 4 4 Je Ais 7 A MQM book Page 6 Wednesday May 7 2008 11 29 AM 6 m Optimize Quality for Business Outcomes The whole process begins when somebody needs something in this case a means of transport the product and expresses an expectation the requirement The contractor asks no questions because he has a picture in his mind of what the customer wants Now the contractor has made an assumption without checking with the customer We re clearly missing some communication a requirement proof point here The building process starts and soon the contractor hands the product over to the customer for acceptance BUT I DON T WANT To GET Wer AND HOW AM t GONG TO CARRY MY BRIEFCASE During the acceptance phase the customer rejects the product because important features are missing and more require ments are added Instead of starting from scratch the devel oper tries to incorporate the new features into the existing construction NV E e Y A
18. nstead of cranking out a bunch of flawed cars and later having to rework the design or possibly even do a recall The HP Quality Model was born out of the question How can we save IT costs for testing and quality management To find the answer we followed the approach made popular by Taiichi Ohno inventor of the Toyota Quality System and others The 5 Why method asks five questions that help lead us down the right path First we asked Why is IT so expensive when hardware costs are dropping and development technologies have improved developer productivity We found that despite the advancements in IT process and technology IT projects are becoming more complex and they almost always run out of time When time is short people tend to sacrifice the quality of the product The Standish Report over the past 15 years shows the rate of successful projects that is under budget and on time is below 30 percent Figure 1 The advancements haven t lowered the cost On the contrary with more complexity the need for proper planning and project management is greater When budget and time constraints dictate the project parame ters functionality and quality get the short end of the stick a Zig A MQM book Page 3 Wednesday May 7 2008 11 29 AM A What Is the Big Deal About Testing m 3 Success of IT Projects 29 53 18 28 23 M 749 2004 2000 1998 1996
19. of the com ponent Portability How easily a system or component can be transferred from one hardware or software environment to another Reliability The ability of a component to perform its required functions under stated conditions for a speci fied period of time Correctness How free a component is from faults in its specification design and implementation It is also o e a is A MQM book Page 12 Wednesday May 7 2008 11 29 AM A 12 mm Optimize Quality for Business Outcomes how well a component meets specified requirements or user needs and expectations And it s the ability of a component to produce specified outputs when given specified inputs and the extent to which they match or satisfy the requirements Up m Completeness How well the component implements all required capabilities Efficiency How well a component performs its desig nated functions using minimal resources Understandability How clear the meaning of a soft ware component is to the user a Performance How well the article under test AUT meets the performance needs of the user Performance may be a measure of throughput response time trans action or data volume or any other related measure Security How well the system as a whole is protected from unauthorized access disclosure use disruption modification and destruction m Availability The degree to whic
20. plan It is to be executed after all other cutover activities are complete and prior to any go no go considerations The Oper ational Readiness Test Summary Report is the final business acceptance signoff Each test phase requires a particular completeness of the com ponents to be tested These components have to be executed in a certain sequence which is synchronized with the progress of the development team The test phases are repeated for every release of the software and they define the software test lifecy cle a parallel process with the SDLC but the two are also integrated Figure 5 E a 7 A MQM book Page 17 Wednesday May 7 2008 11 29 AM What Is the Big Deal About Testing m 17 Software Test Lifecycle Application Demand Service Desk Ee 4 Quality Gate at Figure 5 The software test lifecycle is shown running parallel to and integrated with the SDLC Case Study Flight Application We ll illustrate how the phased approach is applied by examin ing the operation of a web based portal that allows customers to book airline flights The portal is an easy to use straightforward solution that allows customers to search select and book flights in the flight database The system interfaces with the centralized booking system based on the Systems Applications and Products in the Data Processing enterprise resource planning ERP financial module Customers have to register
21. process end to end inside the appli cation and focuses on the correct implementation of the interfaces between the business functions System Test The System Test is conducted on a complete integrated system to evaluate the entire system s compliance with its specified business requirements System testing falls within the scope of black box testing and as such should require no knowledge of the inner design of the code or logic Performance Test The Performance Test determines the performance and scal ability of some aspect of a system how it performs under a real world workload This test may measure throughput vol ume limits transaction time or any other similar metrics 4 4 is d ki A MQM book Page 16 Wednesday May 7 2008 11 29 AM 16 m Optimize Quality for Business Outcomes User Acceptance Test The User Acceptance Test UAT is done by users or their rep resentatives on a new or changed information system If the system behaves according to their expectations the user accep tance testers approve deploying the system Cosmetic and other small changes may still be required as a result of the test but the system is considered stable and behaving according to requirements Operational Readiness Test The Operational Readiness Test is the final phase of testing per formed on the prospective production system This test is scheduled and conducted as a part of the production go live
22. quires flight planning ticketing customer relations The focus of the System Test is Functional correctness All interfacing applications are in place and the application functions correctly in the defined environment m Reliability The system can perform and maintain its functions in routine circumstances as well as in hostile or unexpected circumstances AWA Zs 4 4 A MQM book Page 23 Wednesday May 7 2008 11 29 AM A What Is the Big Deal About Testing m 23 m Accessibility A system is usable by as many people as possible with modification Possible quality gates for the System Test are m All end to end processes can be executed No severe defects exist Performance Test The Performance Test helps us m Demonstrate that the system meets performance criteria Compare two systems to find which performs better m Measure which parts of the system or workload cause the system to perform badly When troubleshooting software engineers use tools such as profilers to measure which parts of a device or software con tribute most to the poor performance or to establish through put levels and thresholds to maintain acceptable response time The later a performance defect is detected the higher the cost of remediation When Performance testing begins at the inception of the devel opment project and extends through to deployment you have a better chance of catching defect
23. s early and thereby saving money and time that would otherwise have to be budgeted to fix problems down the line In Chapter 6 we focus on the different aspects of the Perfor mance Test and describe how it can be performed throughout the product lifecycle is d ki A MQM book Page 24 Wednesday May 7 2008 11 29 AM A 24 m Optimize Quality for Business Outcomes User Acceptance Test When users are satisfied that the business functional specifica tions and business requirements have been met system sign off and release can take place The User Acceptance Test UAT phase involves turning loose a number of real end users to try the system using it as they would in real life to accomplish whatever task the system is designed for The goal is to make sure that the end users are comfortable with the system they will be working with and that all their requirements are met Often a team will leverage training of the end users as a means to perform User Acceptance testing In our flight example we might have reservation agents go through the system and make flight reservations and help identify any issues in the flight system The quality gate for the UAT is the end user sign off Operational Readiness Test The project requirements and design specification documents clearly define the expectations for systems operational readi ness We ve been aware of operational readiness levels while we ve conduct
24. the component to be tested The test phases build on each other and are defined in such a way that testing can be done as early as possible in the SDLC Because cost efficiency is crucial it typically does not make sense to enter a test phase before the previous phase has met its exit criteria For example it may not be effective to do Perfor mance testing before Unit testing is complete The prerequisite for each test phase is that the preceding test phase was executed successfully to a certain extent The entry and exit criteria used for each test phase are called quality gates They are introduced at the beginning and end of each phase and determine the success of the test phases Quality goals are requirements that describe the quality char acteristics of a software product Because different people may be involved and the timing and the nature of the tests to be conducted are different we divide up the test phases as you can see in Figure 4 E a 7 A MQM book Page 14 Wednesday May 7 2008 11 29 AM 14 m Optimize Quality for Business Outcomes HP Quality Methodology STLC Ml Roquiromont 3 i Deployed o_o gt Test Execution Time Figure 4 HP Quality Model In the HP Quality Model we recommend the following test phases Requirements Verification In the Requirements Verification phase the end user adds acceptance criteria to each requirement in order to answer t
25. the last question is that we can t improve the testing process using this knowledge because in today s mar ket IT often dominates discussions around product expecta tions and defines the desired functionality around technological aspects rather than business needs Therefore the testing process is often misaligned with the needs of the business or ideal business outcomes 4 4 is d ki A MQM book Page 11 Wednesday May 7 2008 11 29 AM What Is the Big Deal About Testing m 11 The HP Quality Model is a pragmatic approach to software testing that aligns IT with business outcomes Quality Goals and Test Phases To build meaningful and concise tests we need to define requirements that describe the system behavior we anticipate But that alone would not be enough to specify the intensity or the coverage needed for the tests to meet the end users quality goals Quality goals are non functional requirements Typically in IT projects we use these quality goals Adaptability How easily software can be modified to meet new requirements Maintainability How easily a component can be modified to correct faults improve performance or other attributes or adapt to a changed environment Modularity How much of a system or computer pro gram is composed of discrete components and a change to one component has minimal impact on other compo nents Generality The breadth of applicability
Download Pdf Manuals
Related Search