Test Tool Validation Guidelines
Contents
1. rorum Requirements for Test Tool Validation NFC Forum Reqs_for_NFCForum_TestTool_Validation_v1 1 2010 09 23 RESTRICTIONS ON USE This document is copyright O 2010 by the NFC Forum and is made available subject to the following terms 1 You may without charge copy for internal purposes only and share this document with your members employees and to the extent related to the use of this document on your behalf consultants You may not modify or create derivative works of this document for external distribution THIS DOCUMENT IS PROVIDED AS IS WITHOUT WARRANTY OF ANY KIND EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE ACCURACY COMPLETENESS AND NONINFRINGEMENT OF THIRD PARTY RIGHTS IN NO EVENT SHALL NFC FORUM ITS MEMBERS OR ITS CONTRIBUTORS BE LIABLE FOR ANY CLAIM OR ANY DIRECT SPECIAL INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE DATA OR PROFITS WHETHER IN AN ACTION OF CONTRACT NEGLIGENCE OR OTHER TORTIOUS ACTION ARISING OUT OF OR IN CONNECTION WITH THE USE OF THIS DOCUMENT NFC Forum Inc 401 Edgewater Place Suite 600 Wakefield MA USA 01880 Contents Contents A io A A NS 1 1 1 Audience nnen ean a EU da ds 1 1 2 Applicable Documents or References 0 0 0 eee eeeessecsseceecsseceseeeseeeseeeseeseneeeaeeeaeeenaeenaes 1 1 3 AMISTAD a ee 1 1 4 Special Word Us a2e 0 A ned ae 2 A O O NOTO 2 22. A Validation Process Flows A 1 Figure 1 illustrates the workflow for each test suite Desk Study Test Scripts Review N Select Test Suite Select Test le A Validation Process Flows Case j Select Test le Sub case f The Commands and the Parameters are well configured to achieve the test objective YES The Pass Criteria of the Script is implemented according to the Test Plan and there are no ambiguities in the set up requirements or user guidance YES NO Are there any ambiguities in the Test Script description NO Are there more Sub cases for Test case j Are there more Test Cases for YES f f 1 Test Suite NO Figure 1 Test Script Review Flow gt j j 1 Note that a test script is considered successfully validated only if the test case and all its sub cases were successfully executed Requirements for Test Tool Validation Page 8 Validation Process Flows Before sending a report to the tool manufacturer the laboratory will cross verify any errors as shown in Figure 2 DETECTED BY ENGINEER 2 ERRORS i DETECTED BY ENGINEER 1 ERRORS FOUND BY ENGINEER 1 ERRORS FOUND BY ENGINEER 2 REVIEW ALL THE ENGINEER 2 ENGINEER 1 REVIEW ALL THE Figure 2 Error Cross verification Flow After all errors hav
3. 3 the laboratory shall verify that the correct number of required Test Cases are implemented in the test tool and note the number of additional test scripts not required for the certification wave 2 The laboratory or test house shall review the test scripts for the certification wave or phase for correctness and efficiency against the test specifications and note any issues Requirements for Test Tool Validation Page 3 Test Tool Validation Process 3 The laboratory or test house shall conduct and observe execution of the test cases by the test tool using prototype devices engineering samples provided by the NFC Forum and compare the test results with the expected outcome for any given test case 4 After the validation process is completed the laboratory or test house shall compile a full validation report and fill out the test tool evaluation sheet provided by the NFC Forum 2 4 Validation Process Steps To achieve the validation objectives the laboratory shall perform the following three processes 2 4 1 Desk Study 1 Review the literature documentation provided by the tool manufacturer which is typically the material the tool vendor would supply to a potential customer This documentation provides all relevant information about the tool as well as instructions to set it up for operation 2 The laboratory should familiarize itself with the tool technology specifically the scripting of the Test Cases based on documentation
4. Test Tool Validation ProcesSsS mcccocncccncnnccnnaccnnancnonancconanoconancconananonanenananenan 3 2 1 Validation ODJECUVES societarios 3 22 Validation Process Overview oconoccnonocncncononononinacicnnonononanicacoconononnnnnnoronncnononnnncciccnononos 3 2 3 Test Tool Evaluation Requirements ooooocccnoncccnonanononcnonononnnnnonnnacnnnnononnnrnnnn anno nc cnnnccnnnnnss 3 2 4 Validation Process EPS cita lid deeds Leds aiii Le Uk ee 4 3 Test Tool Requiem iS iio 6 A Validation Process FlOWS ccoocccccnnncconanocananocananonanancnannr conan rnnnanaronanarananenanana 8 ATI Desk Study Test Scripts RevieW ooooooccnoncccnonccononacnnonccnnoconnnncnnonocnnnconnnnn ronca anar na cnnnccnnnes 8 A 2 Validation Testing Test Cases Validation oooonoccnnnnccnonccnnonaconnncnonoconnnonononacnnnccnnncnnnns 10 B Revision History Arial ed 11 Figures Figur 1 Test Script Review Flowcrete hai nine heii A AR 8 Figure 2 Error Cross verification FlOW scroscio ieie nn a a rA eaa A EE aa 9 Figure 3 Validation Testing FlOW cccecccesssecesseceecceceeeesaeceeseeceeneeesaeceesaecseaeeceeeeenaeeneaaeceeeees 10 Tables Table 1 Revision History il a dt et 11 Requirements for Test Tool Validation Page i Introduction 1 Introduction The NFC Forum Certification Program provides consistency in the behavior of NFC Forum devices by confirming that products conform to the NFC Forum device requirements Confirmation will be achieved throu
5. operator executing the certification IUT test plan according to the ICS removes or adds one or more test scripts the tool must record it clearly in the log file e g TC nnn NOT RUN etc 13 The test tool must be able to accept data entered for ICS and IXTT settings for use by the tool for an IUT test plan Requirements for Test Tool Validation Page 6 Test Tool Requirements 14 All NFC Forum certification test scripts must be locked so that the test tool operator cannot change them 15 The test tool must secure the final conformance results so that the test tool operator cannot make any changes 16 The test tool must be able to check that the sequences of command response in IXIT related to the indicated Type Tag Operation test cases perform the related Type Tag Operation functions such as NDEF Read In addition to the above test tool requirements the following recommendations should be considered for easier results analysis 1 Generate test logs with the basic information about the test case in the header name identifier objective basic configuration and tool verdict 2 Map each command and response in the communication to the corresponding step in the test plan Identify any requirement of the pass criteria that has to be checked during log generation 4 Highlight any failure detected during test case execution and include error messages where possible Requirements for Test Tool Validation Page 7
6. atability of test scripts e Re execute selected tests scripts on other engineering sample s with different features present or absent and verify that the results match 2 If timing is explicitly mentioned in a test case verify that the timing is correctly implemented 3 Evaluate the test results per test script and identify any incorrect outcomes Provide comments and suggest improvements 4 Evaluate the overall reporting features of the tool 5 Evaluate the operator interface of the tool 6 Verify that all NFC Forum Test tool requirements are fulfilled NOTE Appendix A shows the validation house work flows to validate test cases 2 4 3 Reporting Results 1 ol Gy E O E de Summarize the results of both the Desk Study and Validation Testing Provide assessment and rate all issues that are found Comment on the general suitability of the test tool for certification testing Provide suggestions for improving the test tool Present the final results to the NFC Forum Document all the steps followed in validating the test tool Provide a list of all correctly implemented test cases that have been validated against at least two engineering samples Testing against additional engineering samples may be required to cover all required test cases Requirements for Test Tool Validation Page 5 Test Tool Requirements 3 Test Tool Requirements All test tools must meet the requirements defined below A competent third party va
7. e been cross verified the laboratory will inform the tool manufacturer of any errors for resolution while continuing with the next phase of the validation process Requirements for Test Tool Validation Page 9 Validation Process Flows A 2 Validation Testing Test Cases Validation Figure 3 depicts the procedure to be followed during this step It is important to note that each test suite will be run using no fewer than two NFC Forum devices with two test engineers analyzing the results Select Test A Suite Run Test Suite Time mentioned in the Test Case Analyze Time 1 Analyze Results for each test case Test case also FAIL in the Desk Study Test case also NO PASS in the Desk Study Further analysis to il YES determine the source of the difference 2 The error is due to the NFC device NOT CLEAR Further analysis to determine the source of the difference 2 1 Analyze time based on timing measurement functionality of the test tool 2 The results of two NFC devices will be compared to facilitate better analysis If possible the results will be compared against other validated test tools Figure 3 Validation Testing Flow Requirements for Test Tool Validation Page 10 Revision History B Revision History The following table
8. ess should occur automatically with minimal operator intervention 5 The NFC Forum Protocol interactions and transactions must be displayed in human readable format e g not in binary format The tool must have the ability to store and retrieve any such content 6 The test tool must generate a summary report of the test results and have the ability to store the report in a well known format such as PDF The results document must be secured so that 1t cannot be altered The report should also include verdicts and comments mainly in case of inconclusive or failed tests A basic configuration menu could be provided to allow a user to configure the result report The reporting menu should let users add laboratory information such as the laboratory name technician environmental conditions etc 7 The test tool must highlight test case failures and inconclusive results and must include the faulty results in a report 8 The test tool log files must be exportable to and retrievable by a computer such as a PC 9 The test tool must have a function to run applicable test seripts according to the provided ICS 10 When a certification UT Implementation Under Test test plan is executed according to the ICS the test results and the settings including ICS and IXIT must be stored in the test tool s results file 11 The test tool operator should be able to select the order of the test cases in an IUT test plan 12 If a test tool
9. gh formal NFC Forum certification testing for which the development of specialized test tools will be required To ensure that test tools are suitable for testing NFC Forum functionality the tools will be subjected to a formal assessment or validation process conducted by an independent third party expert such as a laboratory or test house selected by the Forum for this purpose Protocol tests are assumed to include not just data formatting but also timing The tool validation must include the ability to verify requirements such as Frame Response time with a high degree of accuracy Once successfully validated a tool will be deemed an NFC Forum approved test tool and appear on the Forum s list of approved test tools 1 1 Audience This document is intended for use by e A laboratory or test house when validating a test tool for use by NFC Forum recognized laboratories e Members for testing NFC Forum Device implementations 1 2 Applicable Documents or References ICS Implementation Conformance Statement ICS Version 1 0 July 2010 NFC Forum IXIT Implementation eXtra Information for Testing IXIT July 2010 NFC Forum PROCESS Administrative Process for NFC Forum Test Tool Validation July 2010 NFC Forum RFC2119 Key words for use in RFCs to Indicate Requirement Levels RFC 2119 S Bradner March 1997 Internet Engineering Task Force 1 3 Administration The Requirements for Test Tool Validation is a document s
10. lidation house selected by the NFC Forum will verify that all these requirements are met and provide the NFC Forum with the evaluation results 1 The test tool vendor must supply an installation guide with instructions for the operating environment If the test tool can be installed in different environments installation instructions for all environments must be provided The installation of any test tool software should be automatic by simply running an executable file 2 The vendor must supply an operations user manual containing instructions to operate the tool This must include a description of the operator interface and any command line executables with appropriate parameters Files to be manually configured during testing should be kept to a minimum to help avoid human error 3 The vendor must provide a description of the timing calibration process that was followed for the test tool and where possible include both a calibration certificate and report indicating the most recent calibration event Where possible tool schematics with details of timing outputs should also be provided 4 The test tool must have the ability to be upgraded to support additional test cases test scripts as new versions of the NFC Forum specifications and or new versions of test cases are released The test tool vendor must provide information on how scripts firmware and platform updates are achieved When test scripts need to be updated the upgrade proc
11. ls going forward e The test tool implements and complies with the NFC Forum test tool requirements described in Section 3 For more details about the Validation Objectives see PROCESS Section 2 2 2 2 Validation Process Overview For a test tool to be approved it must comply with the minimum requirements described in Section 3 In addition test tools must have implemented all test cases required for a particular certification wave or phase After the vendor confirms that the tool complies with the NFC Forum test tool requirements and that all required test cases have been implemented correctly the vendor works with the NFC Forum s selected laboratory to evaluate the test tool The laboratory evaluates the test tool based on the requirements defined in Section 2 3 After completing the process the laboratory compiles a full evaluation report and also completes a results worksheet provided by the NFC Forum The NFC Forum will analyze the validation results and make an assessment as to whether the test tool can be used for testing functionality in laboratories recognized by the NFC Forum 2 3 Test Tool Evaluation Requirements 1 The laboratory shall make a full assessment of a vendor s test tool against the requirements described in Section 3 All deviations from the NFC Forum test tool requirements including where the test tool exceeds the requirements must be clearly noted 1 In addition to the requirements listed in Section
12. provided by the tool vendor The test tool vendor may be required to provide training to the laboratory at a mutually acceptable date and location The test tool vendor should provide the laboratory with telephone or email contact information if there are any questions during the evaluation Compare the Test Script with the Test Case in particular the test objective 4 Identify deviations of the Test Script from the Test Case Identify ambiguities in the Test Script In particular the laboratory should look at the set up requirements operator guidance and interpretation of test results 6 Identify ambiguities in the Test Case description wording parameters compared to the test tool script 7 Rate all instances where test tool does not yield the expected results e Severe Wrong test result test script needs to be improved e Moderate Potentially wrong test result caused by ambiguous descriptions varying engineering sample features etc Further research required e Minor Clarifications and recommendations required NOTE Appendix A shows the validation house work flow to review test scripts Requirements for Test Tool Validation Page 4 Test Tool Validation Process 2 4 2 Validation Testing 1 Execute a complete set of test scripts applicable to the first certification wave or phase test cases on the engineering samples supplied by the NFC Forum e Re execute selected tests on same engineering sample to ensure repe
13. provides the revision history of Requirements for Test Tool Validation Table 1 Revision History Document Name Revision amp Status Change Supersedes Release Date Notice Requirements for Test Version 1 0 Final Tool Validation July 2010 Requirements for Test Version 1 1 Final Version 1 0 Tool Validation September 2010 Requirements for Test Tool Validation Page 11
14. specifications the test cases are Poll NFC A Poll NFC B Poll NFC F Poll T1T Poll T2T Poll T3T Poll T4AT Poll T4TB Poll P2P NFC A Poll P2P NFC F Listen NFC A Listen NFC B Listen NFC F Listen TIT Listen T2T Listen T3T Listen T4AT Listen T4TB Listen P2P NFC A Listen P2P NFC B T1T Operation T2T Operation T3T Operation and T4T Operation Requirements for Test Tool Validation Page 2 Test Tool Validation Process 2 Test Tool Validation Process For a test tool to be used for NFC Forum certification testing of NFC Forum functionality by a test laboratory the tool must successfully complete the validation process defined by the NFC Forum This process includes a thorough evaluation of the test tool s features functionality and accuracy of measurement by an independent laboratory selected by the NFC Forum for this purpose 2 1 Validation Objectives The primary objectives of the Test Tool Validation Process are to verify that e All Test Cases and Sub cases required by the NFC Forum have been implemented in the test tool e The Test Cases selected by the NFC Forum have been correctly implemented on the test tool being validated based on a Test Script review e Test scripts are implemented in a way that ensures that the tests can be repeated consistently e Executing the same set of tests on the same device yields the same test results regardless of the test tool being used ensuring consistency across validated test too
15. upported by the Near Field Communication Forum Inc located at 401 Edgewater Place Suite 600 Wakefield MA 01880 Tel 1 781 876 6216 Fax 1 781 610 9864 Requirements for Test Tool Validation Page 1 Introduction http www nfc forum org The Testing Working Group maintains this document 1 4 Special Word Usage The key words MUST MUST NOT REQUIRED SHALL SHALL NOT SHOULD SHOULD NOT RECOMMENDED MAY and OPTIONAL in this document are to be interpreted as described in RFC2119 1 5 Glossary Engineering Sample A prototype commercial or certified NFC Forum Device In the context of test tool validation it is a sample used to verify that the tool correctly executes a test case Sub cases A set of scenarios within a test case Test Case A description of the testing of one or more requirements derived from a specification Test Plan A collection of Test Cases contained in one document Test Script The implementation of a Test Case into a specific test tool using a scripting language Depending on the test tool the Test Script can be contained in a single file distributed over several files or be parameter driven or it can be in text form giving the operator test instructions Test Suite A set of test cases that checks a specific protocol mode operation of the Engineering Sample being tested In the context of the NFC Forum Digital Protocol
Download Pdf Manuals
Related Search