Compatibility Test Suite (CTS)
Contents
1. Test execution Test Arrests Store and view results The CTS is an automated testing harness that includes two major software components e The CTS test harness runs on your desktop machine and manages test execution e Individual test cases are executed on attached mobile devices or on an emulator The test cases are written in Java as JUnit tests and packaged as Android apk files to run on the actual device target 3 1 Workflow 1 Use the bundled CTS release or download the CTS from the Android Open Source Project onto your desktop machine Install and configure the CTS Attach at least one device or emulator to your machine Launch the CTS The CTS test harness loads the test plan onto the attached devices For each test in the test harness The test harness pushes a apk file to each device executes the test through instrumentation and records test results o The test harness removes the apk file from each device 5 Once all the tests are executed you can view the test results in your browser and use the results to adjust your design You can continue to run the CTS throughout your development process nae ae ics Google Confidential compatibility program When you are ready you can submit the report generated by the CTS to cts android com The report is a zip archived file that contains XML results and supplemental information such as screen captures 3 2 Types of test cases The CTS includes2. resolution keypad screen type The details of the executed test plan are present in the test summary section which provides the CTS plan name and execution start and end times It also presents an aggregate summary of the number of tests that passed failed time out or could not be executed The next section also provides a summary of tests passed per package Google Confidential CINDROID Compatibility Test Package android widget Test Result Failure Details Suite android widget cts AbsSeekBarTest testConstructor pass testAccessThumbOffset pass testSetThumb pass testOnTouchEvent pass testDrawableStateChanged pass testOnDraw pass testOnMeasure pass testVerifyDrawable fail _junit framework AssertionFailedError at android widget cts AbsSeekBarTest testVerifyDrawable AbsSeekBarTest java 327 testOnSizeChanged pass testAndroidTestCaseSetupProperly pass ButtonTest testConstructor pass testAndroidTestCaseSetupProperly pass ChronometerTest testConstructor pass testAccessBase pass testAccessFormat pass testOnDetachedFromWindow pass testOnWindowVisibilityChanged pass testStartAndStop pass CompoundButtonTest testConstructor pass testAccessChecked pass testSetOnCheckedChangeListener pass testToggle pass testPerformClick pass testDrawableStateChanged pass testSetButtonDrawableByDrawable pass testSetButtonDrawableByld pass te
3. T General yeon ccreidit 13 6 2 Known Framework iSSUeS 2 ssseeeeeeeeeeeeeeeeeeeeeeeeeeenenaneeeeess 13 6 3 KNOWN TSENG SUC Src voces cccsscccc iennet eeek ea aiar aoreet EEE 0 7 Appendix CTS Console Command Reference 00 ss08 14 compatibility program 1 Why be compatible More devices The Android Ecosystem More users More developers amp content 1 Give your users the best possible experience with the applications they run When a device is compatible with Android users can choose from among many high quality applications Applications that take full advantage of Android s features are likely to perform best on compatible devices 2 Make it easy for developers to write top quality applications for your device Developers want to streamline their applications for Android and this is easiest for them when they are writing for a predictable platform 3 Take advantage of the Android Market Compatible handsets can give users access to the Android Market Android compatibility is free and it s easy Google Confidential compatibility program 2 How can become compatible 1 Comply with Android 2 Pass the Compatibility 3 Submit CTS Report Compatibility Definition Test Suite CTS 2 1 Comply with Android Compatibility Definition document To start read the Android compatibility definition for the Android platform version that you want Thi
4. the console 4 4 Selecting CTS Plans For this release the following 7 test plans are available 1 Or e S CTS contains all tests and will run 21 000 tests on your device These tests are required for compatibility At this point performance tests are not part of this plan this will change for future CTS releases Signature contains the signature verification of all public APIs Android contains tests for the android APIs Java contains tests for the Java core library VM contains tests for the Dalvik virtual machine Google Confidential compatibility program 6 RefApp contains reference application tests more coming in future CTS release 7 Performance contains performance tests for your implementation more coming in future CTS releases These can be executed with the start command as mentioned earlier Google Confidential CiINDAROID compatibility program 5 Interpreting the Test Results The test results are placed in the file CTS_ROOT repository results lt start time gt zip Inside the zip the testResult xm1 file contains the actual results open this file in any web browser Firefox 3 x recommended to view the test results compatibility program Test Report for dream HT851LZ01986 Test Summary by Package The device information section provides details about the device and the firmware make model firmware build platform and the hardware on the device screen
5. the following types of test cases e Unit tests test atomic units of code within the Android platform e g a single class such as java util HashMap e Functional tests test a combination of APIs together in a higher level use case e Reference application tests instrument a complete sample application to exercise a full set of APIs and Android runtime services Future versions of the CTS will include the following types of test cases e Robustness tests test the durability of the system under stress e Performance tests test the performance of the system against defined benchmarks for example rendering frames per second 3 3 Areas Covered The unit test cases cover the following areas to ensure compatibility Description For each Android release there are XML files describing the signatures of all public APIs contained in the release The CTS Signature tests contains a utility to check those API signatures against the APIs available on the device The results from signature checking are recorded in the test result XML file Test the platform core libraries and Android Application Framework APIs as documented in the SDK Class Index to ensure API correctness Platform API Tests e correct class attribute and method signatures e correct method behavior negative tests to ensure expected behavior for incorrect parameter handling Dalvik VM Tests The tests focus on testing the Dalvik VM Google Confidential compatibility progra
6. compatibility program Compatibility Test Suite CTS Framework User Manual Android 1 6 CTS r4 Open Handset Alliance Google Confidential Google Confidential compatibility program Contents 1 Why be compatible cis cesctse ce cesses ca sece doeeawtteuens corssacteateiareeeteeved annie 3 2 How can I become Compatible ccccccccccceeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeees 4 2 1 Comply with Android Compatibility Definition document 4 2 2 Pass the Compatibility Test Suite CTS eee eeeeeeeeeettteees 4 2 3 SUDMIt EPON aisiacieestecsosseatvaeveds ottiaddeiade sateen eaaa aE 4 3 How does the CTS Work cccssseeeeeseeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeaneeaes 5 3 1 VVOURMIOW gt casiesscslasentyeepuncsscvvhex nanana yess didases ais Paaa aa aap a adna Ea 5 Dec YPES Of LOSE CASES cnoiar ea araea Erraiak 6 3 3 Areas COVETEQ seisesuserncsausiclesvaccetesessseesasendciseadhvayncedienssadeeersunmmibernces 6 4 Setting up and using the CTS cece eeeeeeeeeeeeeneeeeeeeeeeeeeeeeeeneenenees 8 4 1 Contigunng the C TS secatetacessie ret ecsccacveedvececedeaaceueguedaveteres seebeauscierere 8 4 2 Setting up your device 5 csccssscenccedcncscicetencesdencededessuencccusutedebeonssnentces 8 43 USMY iNe O lS ooie aeaiee eee EE RARE EE EEE GEE E 9 4 4 Selecting C LS Plas sseni nines 9 5 Interpreting the Test Results sssseseeeeeeeeeeeeeeeeeeeeeeeseeeeeeseeneeees 11 6 Release NOES ieii eee ee eee 13 O
7. d new packages to the case repository Remove the specified package from the case repository all removes all packages List the results for all available sessions If session_id is specified then lists results for that specific session If pass fail etc is specified then filters test results based on the specified results List all commands in history If count is specified last count commands in history are shown e allows the command with number num to be executed directly from history List all attached devices anD compatibility program Google Confidential
8. gs gt Sound amp Display gt Screen Timeout should be set to Never Timeout Make sure the Stay Awake development option is checked Settings gt Applications gt Development gt Stay awake Make sure Settings gt Application gt Development gt Allow mock locations is set to true Make sure the device is at the home screen at the start of CTS Press the home button While a device is running tests it must not be used for any other tasks Do not press any keys on the device while CTS is running Pressing keys or touching the screen of a test device will interfere with the running tests and may lead to test failures 4 3 Using the CTS To run a test plan 1 2 3 4 Make sure you have at least one device connected or the emulator running Launch the CTS console by running the startcts script which you modified to match your environment e g bash android cts tools startcts You may start the default test plan containing all of the test packages by typing start plan CTS This will kick off all the CTS tests required for compatibility Type 1s p to see a list of test packages in the repository Type 1s plan to see a list of test plans in the repository See the CTS command reference or type help for a complete list of supported commands Alternately you can just run a CTS plan from the command line using Starlets start plan lt plan name gt You should test progress and results reported on
9. m The CTS tests the core platform data model as exposed to application developers through content providers as documented in the SDK android provider package Platform Data Model contacts browser settings more The CTS tests the core platform intents as documented in the SDK Platform Intents Available Intents The CTS tests the core platform permissions as documented in the Platform Permissions SDK Available Permissions The CTS tests for correct handling of the core platform resource types as documented in the SDK Available Resource Types This includes tests for simple values drawables nine patch animations layouts styles and themes loading alternate resources Platform Resources Google Confidential compatibility program 4 Setting up and using the CTS 4 1 Configuring the CTS To run CTS make sure you have atleast the Android 1 6 r4 SDK installed on your machine There are changes to adb in 1 6 that will cause CTS to not work correctly with older versions of adb To configure CTS extract the contents of the zip file and edit the android cts tools startcts script modify the variable SDK_ROOT to match your environment Example SDK_ROOT home myuser android sdk linux x86 1 6 rl This should point to the top level directory where you unzipped the Android 1 6 SDK to 4 2 Setting up your device CTS can be executed only on consumer device since Android 1 6 you can run CTS only on develo
10. per builds for Android 1 0 and 1 5 This section is important as not following these instructions will lead to test timeouts failures 1 Please download and install the Android 1 6 SDK on your machine 2 Your phone should be running a user build Android 1 6 and later from source android com 3 Please refer to this link on the Android developer site and set up your device accordingly 4 Make sure that your device has been flashed with a user build Android 1 6 and later before you run CTS 5 You need to download the TTS files via Settings gt Speech synthesis gt Install voice data before running CTS tests Note that this assumes you have Android Market installed on the device if not you will need to install the files manually via adb 6 It is advisable to log in to the device with a test Google account not an account that you actually use 7 Make sure the device has a SD card plugged in and the card is empty Warning CTS may modify erase data on the SD card plugged in to the device 8 Do a factory data reset on the device Settings gt SD Card amp phone storage gt Factory data reset Warning This will erase all user data from the phone 9 Make sure no lock pattern is set on the device Settings gt Security amp location gt Require Pattern should be unchecked Google Confidential 10 11 12 13 14 15 compatibility program Make sure the Screen Timeout is set to Never Timeout Settin
11. s document enumerates the software and the hardware features in a compatible Android device Except where noted the features are all required for Android compliance To learn more about Android compatibility definition in general and to locate and download a particular definitions document see the current Compatibility Definition Archived versions of older Compatibility Definitions may be found on the Downloads page 2 2 Pass the Compatibility Test Suite CTS The Compatibility Test Suite CTS is a downloadable open source testing harness that you can use in any way you like as you develop your handset for example you could use the CTS to do continuous self testing during your development work For more about the CTS and the compatibility report that it generates see the Compatibility Test Suite page For instructions on using the CTS see the CTS User Guide 2 3 Submit report When you are ready to claim compatibility for your device you can submit the CTS generated report to cts android com When you submit a CTS report you can also request access to the Android Market This is an early preview of CTS The compatibility site and the service to certify your compatibility reports are work in progress we will update you when these are ready Google Confidential compatibility program 3 How does the CTS work On your machine Download and install the CTS Run the CTS Devices you attach to your machine
12. stOnCreateDrawableState pass testOnDraw pass tactArrace nctanraCtata nace This is followed by details of the the actual tests that were executed The report lists the test package test suite test case and the executed tests It shows the result of the test execution pass fail timed out or not executed In the event of a test failure details are provided to help diagnose the cause Further the stack trace of the failure is available in the XML file but is not included in the report to ensure brevity viewing the XML file with a text editor should provide details of the test failure search for the lt Test gt tag corresponding to the failed test and look within it for the lt StackTrace gt tag Google Confidential compatibility program 6 Release Notes 6 1 General e This CTS release contains approximately 21 000 tests that you can execute on the device Please make sure all steps in section 4 2 Setting up your device have been followed before you kick off CTS Not following these instructions may cause tests to timeout or fail 6 2 Known Issues The framework restarts the device periodically this is expected behavior Concurrent devices are not supported in this release CTS can be executed on only one device at a given time The CTS console allows the user to derive a new test plan based on previous results This is useful for re running tests that did not pass in a previous run Successive derivation of
13. test plans i e deriving a test plan from test results of an already derived test plan may result in the plan including extra tests this is a known issue for this release Occasionally while running the tests a system dialog may pop up informing the user that process android process acore is not responding The user is given the option to kill the process or wait for it to respond This alert dialog interferes with some tests by grabbing all key and pointer events causing the tests to fail Re running the tests usually fixes the problem Google Confidential compatibility program 7 Appendix CTS Console Command Reference Host help exit Test Plan Is plan lt test_plan_name gt add plan lt new_plan_name gt add derivedplan lt new_plan_name gt s lt session_id gt Fr pass fail timeout notExecuted rm plan lt test_plan_name all gt start plan lt test_plan_name gt Google Confidential Display the list of available commands Exit the CTS console Displays the contents of the specified test plan If no plan is specified a list of all plans is displayed Create a new test plan The console will guide you through the test packages to select the tests you want to include in your plan Note that the plan name must be unique Create a new test plan from an existing result This test plan will consist of all test with the specified result type in the specified session If no res
14. ult type is given all but the passed tests are included If no session is given the latest results are used Remove the specified plan from the plan repository all removes all test plans Start the specified test plan and displays progress information The console will only prompt for further commands when the plan has run to completion If there are available test sessions for the specified test plan the CTS console will prompt user to choose between two options 1 Choose a session from the existing sessions compatibility program d device lt device_id gt t test lt test_name gt p package lt java_package_name gt Test Package Is p package lt package name gt add p package lt zip_file_path gt rm p package lt package_name gt all Test Result Is r result pass fail timeout notExecuted s lt session_id gt History history h lt count gt e lt num gt Device Is d device Google Confidential 2 Create a new session If more than one device connected CTS host will prompt user to choose one device Start the specified test plan using the specified device Start to run the specified test contained in the specified test plan Start to run the specified Java package contained in the specified plan List all available test packages in the repository If package name is specified then lists all its test suites test cases Ad
Download Pdf Manuals
Related Search