4.3. Simulated Drive Test Cases
Contents
1. UE Manual Interface UE Control Method Specifies how to control the UE under test Figure 3 2 UE Parameters Window UE Interface Parameters 3 3 2 UE Manual Interface Parameters If the UE Control Method parameter of the UE Interface tab is set to Manual a command fails or is not provided by the selected UE Control Method the parameters of the UE Manual Interface are used In this case a window displays prompting you to perform the necessary action on the UE Chapter Three Using Simulated Drive 9 e Timeout Specify how long the system waits before automatically selecting the defined Timeout Button A value of 0 indicates the function will wait indefinitely Note that even if Manual control is not selected it may be used in case of failure If the timeout value is set to 0 the system will wait indefinitely which may not be desirable e Timeout Button If a non zero timeout is specified these parameters specify if the OK or Cancel button is selected when the timeout expires m Test Manager Ef File Edit View Execute Report Help z da a re gt a S Sony Ericsson ue lay Suites B 1 Timeout steve Connect SuitesFile1 is n Disconnect 2 DYT Suites Ensure Idle Originate Voice Call Answer Voice Call End Voice Call Reset E 2 Timeout Button Answer Voice Call Reset Suite Files 8100 4300 Module Files o uy 0 u o Pa c lt w Result Files UE Inte2. Change Windows P Wireless Network Connection Disabled DW1530 Wireless N WLAN Half Mini Card Firewall settings Figure 3 7 UE Spirent Data Client Parameters Dial Up Networking 3 Forthe Connection window select Disabled from the drop down menu If this option is left enabled for any period of time it prevents the continuous run required for Simulated Drive testing and may affect performance and results 4 Set the Device Interface Name a Enable USB Tethering on the UE that is connected to the client laptop b Make sure the name for the Local Area Connection Device Name established by the device in the Network Connections window on the client laptop matches the Device Interface Name that you set in the UE file Capitalization and Spaces all need to be exact 14 UMTS Simulated Drive User Manual 3 4 Specifying UE and Session Parameters As indicated in the 7est Manager User Manual you cannot run a suite successfully unless it undergoes validation Both UE and Session parameter files are required for execution of UMTS Simulated Drive test cases Refer to the Test Manager Operations Manual for more information on creating UE and Session Files UC Miu lt 2 aa Results Folder Selection C Program Files Spirent Com Results Cd v Session File Selection C Program Files Sp SessionsFile1 ses G v UE File Selection C Program Files Spirent C UEFile1 ue Ga v Platform
3. The in call percentage gt Fini Percentage should be 65 for all the complete PASS centage oreater than or equal to iterations 65 Basic Voice Call Statistics I Mum Call Num Call 11 can Average Iteration status InCall Setup setup Num Call Attempts success TPS Drops Overall Pass 79 49 4 54 Complete 66 50 35 15 14 2 Complete 89 20 10 6 5 3 Complete 88 40 8 7 6 4 Complete 74 90 5 4 3 5 Complete 98 60 2 2 1 6 Complete 81 70 13 10 g 7 Complete 100 00 d 1 0 8 Complete 87 60 5 4 3 Handover Statistics Num Num Handover Iteration Handover Handover Failure Success Failure Rate Overall 745 2 0 30 174 1 0 60 2 154 1 0 60 3 98 L 0 00 4 15 0 0 00 5 173 o 0 00 6 92 0 0 00 a g 0 0 00 8 39 L 0 00 Figure 5 9 Consolidated Report Chapter Five Reports 31 5 3 Pass Fail Criteria Figure 5 10 shows a portion of the spreadsheet created in the previous section Simulated Drive Test Results Pass Pass Fail Criteria Actual Value Average The average number of Number of call drops should be less call drops than or equal to 5 5 The number of complete iterations should be at least 8 For all the complete in Call iterations the in call The in call percentage Percentage Percentage should be 65 for all the complete greater than or equal to iterations Number of Iterations 65 Figure 5 10
4. Answer Enabled Chapter Six Appendix A 37 Run Test Manager and pause the test when the wait 60 seconds for registration message displays Put UE in flight mode If the UE does not register reboot the UE and make sure it makes a data call automatically Tethering Settings Wireless amp Networks gt Tethering and Mobile Hotspot gt USB tethering On Make sure that the device displays as Motorola USB remote NDIS Network device under the Data Clienttab in UE file It should match the settings under WIFI Connections on the client laptop If it does not change the UE file name save the file and stop the test Rerun the tests repeating steps 4 5 and 6 6 4 7 Samsung Infuse 4G Android ICS Mode 1 2 Software Drivers Samsung Kies Simcard ATT Sim and Spirent PLMN Phone Settings a Settings Wireless amp Networks gt Mobile Networks gt Data Roaming On b Settings Wireless amp Networks gt Mobile Networks gt Use Packet Data On Automatic Call Answer Enabled Run Test Manager and pause the test when the wait 60 seconds for registration message displays Put UE in flight mode If the UE does not register reboot the UE and make sure it makes a data call automatically Tethering Settings Wireless amp Networks gt Tethering and portable Hotspot gt USB tethering On Make sure that the device displays as Samsung Android USB remote NDIS Network device under the
5. File Selection C Program Files PlatformsFile1 plm G v Figure 3 8 Selecting UE and Session Files for UMTS Simulated Drive 3 5 Locating the Test Suite and Test Cases 3 5 1 Locating the Test Suite 1 In Test Manager under the Suite Filestab open the Suites folder 2 Open the A7 amp 7 16815 Simulated Drive Test Specification vx x x subfolder 3 The UMTS SDT Suite is labeled Simulated Drive Test MRAB Vx x X as shown in Figure 3 9 File Cabinet yy Suites AT amp T 16815 Simulated Drive Test Specification v1 6 6 f Simulated Drive Test MRAB v1 6 6 Development Library Interference Cancellation Performance Test Suites v1 0 1 UMTS Call Reliability UMTS Data Performance Wideband AMR Performance Test Suites v1 0 1 cease aaa YY Test Folders _ Result Files _ Figure 3 9 Locating UMTS Simulated Drive Suite Chapter Three Using Simulated Drive 15 3 5 2 Running the Test Suite To run the test suite 1 In Test Manager open the test suite from the Suite Files tab of the File Cabinet as explained in section 3 4 2 From the Test Manager menu select Execute Start Session Run All Test Cases to start executing the entire Test Suite You also have the option of executing only the currently selected test case The Execute Session window displays 3 To run the UMTS Simulated Drive Test Suite ensure that you have the correct platform selected If you want to run the test with th
6. SDT Results Table The overall result of the Test is shown as Pass in green or Fail in red This result depends on all should be Pass to have overall Test as Pass the Status column values in Figure 5 10 All iterations are Complete and the Average Number of Call Drops and the In Call Percentage is below the pre determined threshold 5 3 1 Number of Iterations The number of complete iterations must be 8 for the Test Status to be Pass For example if we had created report from 2012 5 11 1226 which had only 7 complete iterations results display as shown in Figure 5 11 the individual Status and Overall Results display as Fail Simulated Drive Test Results Fail Pass Fail Criteria Actual Value Ea Average The average number of Number of call drops should be less call drops than or equal to 5 5 The number of complete or iterations should be at ri FAIL Iterations least 8 For all the complete iterations the in call The in call percentage is In Call not greater than or equal tage Percentage should be to 65 for all the FAIL greater than or equal to R i 65 complete iterations Figure 5 11 Fail Status in Table 32 UMTS Simulated Drive User Manual 5 3 2 In Call Percentage The parameter which captures the essence of the test results is In Call Percentage Even though average number of call drops is used to determine the test result In Call percentage must be used to de
7. True Incomplete Figure 4 1 Test Suite Execution Flowchart 18 UMTS Simulated Drive User Manual 4 3 Simulated Drive Test Cases This section describes the Test Cases and what may be different or unique to the individual test case Functionality that is common to all is described in the next section 4 3 1 Boston Drive Route Leg 1 e Test case duration is approximately 60 minutes start to finish e 3 Full WCDMA Cells Single Frequency e 50 In Call Percentage required for statistical verification e The RF conditions presented in this route are more difficult than any of the other routes 4 3 2 Boston Drive Route Leg 2 e Test case duration is approximately 60 minutes start to finish e 3 Full WCDMA Cells Single Frequency e 65 In Call Percentage required for statistical verification 4 3 3 Boston Drive Route Leg 4 e Test case duration is approximately 50 minutes start to finish e 3 Full WCDMA Cells Single Frequency e 65 In Call Percentage required for statistical verification 4 3 4 New York New Jersey Drive Route Leg 1 e Test case duration is approximately 15 minutes start to finish e 3 Full WCDMA Cells Two Frequencies e 65 In Call Percentage required for statistical verification e Hard Handover possible on this route 4 3 5 New York New Jersey Drive Route Leg 2 e Test case duration is approximately 70 minutes start to finish e 3 Full WCDMA Cells Single Frequency e 65 In Call Percentage requ
8. UE s behavior rather than forcing scenarios and expecting a well defined outcome by the UE Because this test is reactive and provides only analysis call drops are expected The frequency of call drops UE recovery capability etc are analyzed collated and averaged across all routes to provide and accurate analysis of the UE s performance Pass and Fail criteria are described in the reporting section of this document Part of the Simulated Drive functionality is pseudo random FTP file transfers to emulate a data push to the UE similar to an email update etc Because this data push is randomized a rerun of the exact same test case with the exact same UE may produce different results This mostly comes into play during bad RF conditions If the UE is only responsible for handovers to keep the voice call active during the bad RF conditions it may make it through without dropping the call However if one of the data transfers occurs in conjunction with the handovers needed etc and RLC retransmission occurs we may have to drop the call due to RLC Unrecoverable error RLC reset or RLC reset due to RLC retransmission exhaustion This explains why some results may be different even with the same UE on the same test case Simulated Drive runs may also differ due to rapidly changing RF Conditions While in most cases soft handovers etc will be relatively consistent there may be cases when the UE attempts to register and is also performi
9. frequent Measurement Reports from the UE which in turn causes lots of handovers than another leg leading to more call drops than any other leg This is expected behavior 7 3 2 Test Case says Other Incomplete and stops at step 2 UE could not register with the Network on all domains even after LAC RAC change within 60 seconds The solution is to rerun the test case making sure that UE is has been turned off and on and is well shielded 42 UMTS Simulated Drive User Manual 7 3 3 Test Case says Other Incomplete and stops at step 9 Network was not able to set up Mobile Terminated 3 G voice call Check to make sure Auto Answerer App is running for Android or Auto answer is set other platforms or UE file contains incorrect COM port 7 3 4 Test Case says Other Incomplete and stops at step 10 Network was not able to set up 3 G data call Verify that tethering is enabled UE is not powered off USB cable is connected to client laptop For some devices make sure that Desktop management software is running 7 3 5 Test Case says Other Incomplete and stops at step 26 In Call Percentage for certain Test case iteration was less than the criteria set up for that case This is due to call not getting re established after the call drop Testcase may need to be rerun 7 4 Test Manager Error Events This section explains the RED lines seen on Test Manager window with error messages 7 4 1 Failed to Initiate Transfer Because t
10. makes a data call automatically Enable Tethering Make sure that the device displays as Samsung Android USB remote NDIS Network device under the Data Clienttab in UE file It should match the settings under the WIFI connections on client laptop If it does not change the UE file name save the file and stop the test Rerun the tests repeating steps 5 6 and 7 6 4 4 HTC Inspire Android ICS Mode 1 2 3 Software Drivers HTC Sync Simcard ATT Sim and Spirent PLMN Phone Settings a Settings Wireless amp Networks Mobile Networks Access Point Names gt Spirent MCC 001 b Settings Wireless amp Networks gt Mobile Networks Access Point Names gt Spirent gt MNC 01 c Settings Wireless amp Networks gt Mobile Networks gt Data Roaming On d Settings Wireless amp Networks gt Mobile Networks gt Enable always on mobile data On Automatic Call Answer Enabled Run Test Manager and pause the test when the wait 60 seconds for registration message displays 36 UMTS Simulated Drive User Manual 6 7 8 9 Put UE in flight mode If the UE does not register reboot the UE and make sure it makes a data call automatically Settings Wireless amp Networks gt Mobile Network On Settings gt Wireless amp Networks gt USB Tethering On Make sure that device displays as HTC Remote NDI based device under the Data Clienttab in UE file It should
11. match the settings under WIFI connections on the client laptop If it does not change the UE file name save the file and stop the test Rerun the tests repeating steps 5 6 7 and 8 6 4 5 HTC Aria Android ICS Mode 1 2 Software Drivers HTC Sync Simcard ATT Sim and Spirent PLMN Phone Settings a Settings Development Allow Mock Locations On b Settings Wireless amp Networks gt Mobile Network On Settings Wireless amp Networks gt Mobile Networks gt Data Enabled On 2 9 Settings gt Wireless amp Networks gt Mobile Networks gt Data Roaming On e Settings Wireless amp Networks gt Mobile Networks gt Enable always on Mobile Data On Automatic Call Answer Enabled Run Test Manager and pause the test when the wait 60 seconds for registration message displays Put UE in flight mode If the UE does not register reboot the UE and make sure it makes a data call automatically Settings Wireless amp Networks gt USB Tethering On Make sure that the device displays as HTC Remote NDIS based device under the Data Clienttab in UE file It should match the settings under WIFI connections on the client laptop If it does not change the UE file name save the file and stop the test Rerun the tests repeating steps 5 6 and 7 6 4 6 Motorola Atrix Android ICS Mode 1 2 3 Software Drivers MotoHelper Simcard ATT Sim and Spirent PLMN Automatic Call
12. message to the UE and call drops around this time network may not be able to send message to the UE Hence the above message is printed 7 4 7 Setup Data Call Failed Because the RasDial Error If a UE is tested with dialup connection mode Spirent Data Client controls the dialup COM port on data client laptop When data client cannot control the COM port it prints the above message When COM port frees up the situation corrects itself 7 4 8 Data Client Download Error HResult Oxffffffff Error Message Download Transfer Timeout after 15 Seconds of No Data Transfer If a call drops during active FTP transfer Data Client times out due to lack of pending data transfer and above message is printed 7 4 9 RLC Unrecoverable Error RLC Reset This error occurs when after RF loss network does RLC retransmission which does not complete or the UE times out 7 4 10 RF Loss Fading conditions cause the RF loss 44 UMTS Simulated Drive User Manual 7 4 11 The NET Method Download Transfer threw an exception The Process Cannot Access the File X When a call drops during active FTP transfer the file being transferred is sometimes locked out by windows and the lock is not removed for few minutes After this Windows garbage collection does not clear any locks on File X so that when the next test starts file is locked and cannot be accessed 7 4 12 Release 3G Data Call Data Call Not Terminated Timeout Waiting f
13. the event or action This is due to artificial latency in the messaging between the network emulation and the PC notifying the PC that action is complete or the event received and when it can be printed on the graph Some things may appear slightly out of order etc However this is a very good representation of the activity and a reasonable timeline of when events occur 4 4 2 1 RF Level Lines Cell Activity Lines The Horizontal lines on the Procedure Graph represent the different cells that are configured for the Drive Route The height of the horizontal line represents the cell s power level at that point in time If the horizontal line is solid then that cell is part of the current active set in soft handover If the horizontal line is dotted then the cell is NOT part of the current active set 4 4 2 2 Event Lines Every event that is received from the UE is logged in the Procedure graph and is shown with a dotted vertical line Not every event is acted on If the network is currently acting on an event it is unable to process the next event until that action is complete In order for the network to remain current to the UE events are discarded until the current action is complete and then the next event received from the UE will be processed The only exception to this rule is Compressed Mode All of the following events are supported by the system This does not mean all events will happen on every leg or on any leg in particular Cha
14. the following platforms 8100 A304 8100 A604 3 2 2 Platform Selection As indicated in the Test Manager Operations Manual you cannot run a suite successfully unless it undergoes validation using the specified Platform Session and UE Parameter files In the case of UMTS Simulated Drive installed with the Test Manager the platform file used with this module is already selected In the event of multiple platforms you are required to specify the correct platform from the Platform Selection dropdown field in the Execute Session window as shown in Figure 3 1 3 2 3 Specifying a Platform File In addition to the Platform Selection you must specify a Platform File Refer to the Test Manager Operations Manual for more information on creating a Platform File Chapter Three Using Simulated Drive 7 Execute Session Tester Name Tester 1 UE Manufacturer Manufacturer 4 UE Model 1234 Results Folder Selection C Program Files Spirent Com Results C J v Session File Selection C Program Files Sp SessionsFilel ses UE File Selection z C Program Files Spirent C UEFile1 ue GJ v Platform File Selection C Program Files S PlatformsFilel pim Platform Selection 8100 4300 v Perform Validation iyi Verify the settings are correct for each Test the instrument connections for test case each test case Indicates Required Field Figur
15. to page requests UEreselection during registration due to rapidly changing RF conditions Other Rerun This is a test that is rerun due to a previous Incomplete run 5 Simulated Drive Reports I 5 1 Report Generation 5 1 1 Requirements These following are the requirements that must be met before generating a SD report e The test should show all 8 test cases legs as Complete under Status column within Summary section of the Test Manager e For Other Incomplete Ifthe test had exited out due to UE not registering a tethering issue or hardware problems with test equipment the Test step number should be low i e Step 2 or Step 9 The test may also be Incomplete due to low In Call Percentage explained later in the document This usually stops at step 26 or 29 Regardless of the reason for the Incomplete Status the Simulated Drive Test will then automatically rerun the test once So If any of the test results show Incomplete as the test status they are incomplete after one rerun Ifthe test fails on the retry you need to rerun the iterations as a new test and then combine the previous and new tests into a consolidated test as described in Section 5 1 1 1 e There should be results mdb file in the folder that contains the test results For example the Results Path Selection is C Program Files Spirent Communications AirAccess WCDMA HS Test Session Logs Sim Drive Devel
16. 10 scceeeeeeeeees 42 7 3 5 Test Case says Other Incomplete and Stops at step 26 cccseeeeeeees 42 7 4 Test Manager Error Events seesssosessrossesesecesosesseosesersesososesssoeseso 42 7 4 1 Failed to Initiate Transfer Because the PS RAB Session Could Not be Restarted vicccsancercssactscensaracdecsascinesasnceeesscceewsencabsecsenseccecnaneeasseoaubeeebesteseees 42 7 4 2 Start FTP Transfer FTP Transfer Ended Unexpected ly s sscssecsseeseee 42 7 4 3 UE Failed to Respond to PAGE veces ceccscs coves ch ccpeveasscoashsccderessccentseaccpevearectees 42 Fete VOICE Cal Not ALIVE csesecaesioscnaceesecasvepdeesearesucenye eucaeavopdesaverssesearoseneseeceses 42 7 4 5 Compressed Mode Physical Channel Reconfig Complete Not Received by the NetWOtK ceccuxecnceisetineresscausesieatstceuasaneesuesamenss Mannaeaavanctusbiucelnenneetenianss 43 7 4 6 Compressed Mode Error in Sending Physical Channel Reconfig Message 43 7 4 7 Setup Data Call Failed Because the RasDial Errof cccccccsssssssssssscccees 43 7 4 8 Data Client Download Error HResult oxffffffff Error Message Download Transfer Timeout after 15 Seconds of No Data Transfer s ssssss0000 43 7 4 9 RLC Unrecoverable Errot RLC ReSet scssisosesscsssssscesessesseesassvadessesecsencseoeves 43 DOT LOS 5 n E E lec ausuina EEA 43 iv UMTS Simulated Drive User Manual 7 4 11 The NET Method Download Transfer threw an exception The Pr
17. Data Clienttab in UE file It should match the settings under WIFI Connections on the client laptop If it does not change the UE file name save the file and stop the test Rerun the tests repeating steps 5 6 and 7 6 4 8 Blackberry Bold Dialup Mode 1 2 Software Drivers Blackberry software Registry settings Client Laptop a Open RegEdit and go to HKEY_LOCAL_MACHINE SYSTEM CurrentControlSet Enum Root PORTS 38 UMTS Simulated Drive User Manual b Alist of sub keys displays Each sub key represents a port c Select a subkey if the DeviceDesc value is not set to RIM Virtual Serial Port v2 move on to the next sub key and loop If it does not proceed with the next step Right click the sub key and select Permissions from the menu Under Full Control make sure the Allow option is enabled Create a new DWORD value called DeviceType and set its value to Ox1b Loop za moa Reboot Computer Simcard Spirent Sim Phone Settings Default No Change Tethering Settings Default No Change APN Settings Default No Change Special Steps Make sure that the Modem Com port in Client Laptop for this UE matches the COM Port in the UE file Make sure BlackBerry Desktop software is running on data client laptop Create a Dial up Connection with the following settings a Connection Name TestDrive Data Connection b Username test Password test Phone Number 99 e Make Com Port the d
18. at E Lib za Voice Call Drop Rate WCDMA 27KB TestStand Seq Network Configurati E Voice Call Drop Rate WCDMA 4KB TLC File E Result Panes Voice Call Drop Rate WCDMA 25KB XSD File E utility Voice Call Drop Rate While Pe 46KB TestStand Seq Platforms E Voice Call Drop Rate While Pe 4KB TLC File E E Results Voice Call Drop Rate While Pe 60KB X5D File ml uaina Call wan Naka Whila Na AA VD TaebkChand Cam lt lt ill i Figure 1 3 Accessing the Manual from Windows Explorer 1 5 2 Accessing Documentation from Test Manager Access this Manual from the Test Manager menu by selecting Help Modules UMTS Simulated Drive Simulated Drive User Manual pdf as shown in Figure 1 4 Platforms Report Engines 3G HSDPA DL Mobility 3G R99 Mobility CS PS 3G 2G Hierarchical Cell Selection 3G 2G Inter RAT Call Reliability 3G 2G Reselection About Figure 1 4 Accessing the Manual from Test Manager 2 Simulated Drive Description SO 2 1 2 2 2 3 Overview This chapter provides a high level overview of the key components necessary to configure and use UMTS Simulated Drive Description UMTS Simulated Drive is an integrated software component that allows you to connect a device and analyze its behavior and performance against the same RF conditions as if it was being drive tested in the corresponding city routes All routes are run under t
19. call up However if the UE requests RLC Reset or the network exhausts all retransmissions a RLC Reset occurs and the call is dropped UE Latency in Add Delete Update Requests At times during runs the RF Conditions between cells change so rapidly that by the time a cell is added it immediately needs to be deleted which may cause sync loss Also the RF may change radically before the request can be sent in time from the UE for the network to react This may lead to a cell not being added to SHO in time resulting in eventual RF loss and Call drop Definition of Individual Test Case Results Individual Test Cases have the possibility of reporting any one of the 3 result values listed below Although these result values give an indication of whether or not the phone is having difficulty with a particular Drive Route they do not represent the overall Pass or Fail of the Test Suite As described in the next section the Pass Fail criteria is determined through analysis of the data from ALL routes combined Other Complete Test Case ran to completion with adequate In Call Percentage that is considered statistically relevant and usable in the reports generated Other Incomplete Any failure for any reason is considered Incomplete This includes but is not limited to In Call Percentage was below the required threshold to be considered Statistically relevant UE Failed to register UE Failed to tether UE Failed to respond
20. check if the UE responds correctly and reliably before testing Chapter Three Using Simulated Drive 11 1 UEAT Interface Pot COM1 El Commands 5 El Commands 0 Name Ensureldle AT String AT E Commands 1 Name OriginateVoiceCall AT String ATD555555555 E Commands 2 Name AnswerVoiceCall AT String ATA E Commands 3 Name EndVoiceCall AT String AT CHUP Commands 4 Name Reset AT String AT CFUN 1 1 COM Port Settings Bits Per Second 19200 wy Data Bits 8 Parity None Stop Bits 1 Flow Control None Test AT Commands Click here to test the AT commands UE AT Interface Figure 3 4 UE Parameters Window UE AT Interface Parameters 3 3 4 UE Spirent Data Client Parameters In general there are two connection modes the UE uses to connect to the data client while running Simulated Drive 1 Dial up Networking DN 2 Internet Connection Sharing ICS These modes are described in detail in the following sections 12 UMTS Simulated Drive User Manual 3 3 4 1 Dial up Networking When in Dial up Networking Mode default the UE connects to the client laptop via USB as a standard modem On the client laptop under network connections the Test Drive Data Connection should have Standard Modem chosen with the corresponding correct com port that is also selected for the AT commands specified above Connection Mode Dial up Networking Connection Verification Type Ping MTU Size 1500 E 5 Shel
21. daaensdavesears 40 7 1 1 Fail to Remote Desktop into the Application Server essnee 40 7 4 2 Tests Halt Due to Autoset failure 508 scccssscccsccesccsscensesdeccaceascccsssssseceses 40 7 1 3 UE Fails Hard Handover 100 Of the Time cssccssesssssssscsssssstescceseesees 40 7 1 4 Fails to Start DEE Mode sessccoceccsscaccevscassecavsencvcascasaccesacaswsvansenseensenescancseses 40 Feds UE F ilUreS sccssixisiscensainfeostiuassineniaabageesasbiusssineniiateindecstonsaiverwiadsensviys 40 7 2 1 UE Fails to Register After Installing Drivers and Connecting to Instruments 40 7 2 2 While making a data call in ICS mode Data call cannot be established since the UE IP address has not been assigned message is displayed 41 7 2 3 UE device drive is failed to show up on client laptop Windows Device MANAGOT 1 acco scncinasaassuinnsstanalecnsecaankopnsaaalstnraanutethaasansevaecsanaassensaeaeaseinnadias 41 7 2 4 SIM Not Provisioned 3 irs issiccceateccsctiess saciarteuceararacaneherialdatesssatianaeunesaseneats 41 Foe WOSUCASS RESINS sivas cleresceveoensersaavecesbeesieoecsietxsanGeovevsadee taueni andouneenee 41 7 3 1 Boston Leg 1 has more call drops than the other l GS s ccsceseeesseeees 41 7 3 2 Test Case says Other Incomplete and Stops At step 2 csseseeeeeeeceees 41 7 3 3 Test Case says Other Incomplete and stops At step 9 sseeeeenneeeceees 42 7 3 4 Test Case says Other Incomplete and Stops at step
22. dure Samples 00000 33 Gili OVERVIEW soe sirieesies urere storke no seekos ri ETE ENOCE EEE Ta EiS 33 6 2 Common Proced TeS sessssccississsrrrirrirrerseserressssiriissererr oeren riire 33 6 3 Installing Auto Answer App for Android Phones ssccccsssssesceeees 33 6 4 Specific UE InstructionS ssseessssseeesssseeeesssseecesssseeeessseeeessseeeessseeeee 33 6 4 1 Samsung FOCUS WiNdOWS7 Dialup MOE cccceesessssnnsscccceceesssesseees 33 6 4 2 Samsung SkyRocket Android ICS MOdE 1 cccccccessessssnsnscccceccesesessnees 34 6 4 3 Samsung Captivate Android ICS MOe 1 cccccccessessssnsnscccceccesseessnees 34 6 4 4 HTC Inspire Android ICS Mode u cceeeeenssseccceccceesessssnsnsccceeecesssssenees 35 6 4 5 HTC Aria Android ICS MOde cccssssccccesssseccecsssscceccessnsceccssseeeceeeeneees 36 6 4 6 Motorola Atrix Android ICS MOde sssseesecccecccessensssnsnneccceeceessessees 36 6 4 7 Samsung Infuse GG Android ICS MOdE 0ccccccccessessssnsnseccceccesssenseees 37 6 4 8 Blackberry Bold Dialup Mode sicisssscsazecsasacnsssiccaverecsasseusccceciasicecerssavever 37 Table of Contents iii 6 4 9 Samsung Galaxy S2 Android ICS Mode cssssscecesssssccccssssrcccessssenees 38 7 Appendix B Frequently Asked Questions ccccccccccsscssssssssssreee 40 7 4 System FANN ES scseieess nessarercesenssnetessnnarecvasacyiatvenessigsauveussuwe
23. e 3 1 Execute Session Window Platform Selection for UMTS Simulated Drive 3 3 UEParameters The level of automation and reliability of the system is heavily dependent on the ability of the system to control the UE UE parameters are configured to facilitate these activities 3 3 1 UE Interface Parameters The UE interface parameters specify how the UE is controlled during test case execution The options for the UE Control Method are e Manual The system prompts the operator to perform actions on the UE This is the least desirable with regards to running Simulated Drive Tests due to the asynchronous events from the UE s and the length of the tests being performed e AT The system issues the specified AT commands to control the UE e Custom Advanced Allows custom UE control If Custom is selected the UE Control Sequence File parameter specifies the Control Sequence file used by the system 8 UMTS Simulated Drive User Manual The ability of the system to return the UE to a known state before execution of a test case is important If the UE supports a reliable reset command set the Reset for Ensure Idle parameter to True y Test Manager Sees Sony Ericsson ue Ley Suites El Misc sey ao UE Control Method Manual 2 pine l UE Control Sequence File C Program Files Spirent Communications T es Eai DVT Suites Reset for Ensure Idle True 8100 4300 Module Files o pa o Pa Cc a lt wW
24. e 5 7 Combined Tests 1 Right click on 2012 05 14 0523 and select Create Report Simulated Drive Test Report Detailed Report as shown in Figure 5 8 30 UMTS Simulated Drive User Manual er 201205111226 201205131722 201205 140523 Results Dashboard X Summary x 2012 05 11 1226 l x 2012 05 13 1722 Indicator Values Test Description Test Case 2 i eat Statii Percentage gt GSM BTR 1 54 Developer Template Close 0 of 8 0 2 GSM BTR 1 54 Developer Template X Delete 0 of 8 0 3 GSM BTR 1 54_ Developer Template Combine Results 0 of 8 0 4 GSM BTR 1 54 Developer Template Add to Zip file 8 of 8 100 5 GSM BTR 1 54_ Developer Template ___HH MM SS M BTR 1 54 Developer Template Create Report Custom Report gt Properties Detailed Report Spreadsheet Report gt gt 40 ASAT NUTT SRE EEE ane ee ae WE RRS el Figure 5 8 Creating a Spreadsheet from Combined Tests 2 The final consolidated report is shown in Figure 5 9 Parameter Pass Fail Criteria Actual Value Average The average number of Number of call drops should be less 51 PASS calldrops than or equal to 5 5 The number of complete Number of iterations should be at 8 Pass least 8 For all the complete een iterations the in call
25. e default fields as shown in the Execute Session window skip steps 5 and 6 For more information on configuring Session and UE parameter files refer to Section 3 2 4 If desired specify the Tester Name UE Manufacturer and UE Model These fields display on all reports you create based on the results of this session 5 Click OK to begin validation If successful the test session begins as shown in Figure 3 10 E Test Manager DER Simulated Drive Test MRAB v1 6 6 sui 20120427 1624 Bj Suites S AT amp T 16815 Simulated Drive Test Simulated Drive Test MRAB v Indicator Values Test Description Development Library Test Status Percentage 1 GSN 1 5 r Template ent Library Other Complete 39 4 B Interference Cancellation Performa Fail 0 of 72 0 Developer Template i Other Complete 8 Ly UMTS Call Reliability Pass Oof 72 0 GSM BTR 1 54_ Developer Template i Other Complete E ma va Aien iorn te Pending Oo 72 0 GSM BTR 1 54 Developer Template i Other Complete cis Other 72 of 72 100 GSM BTR 1 54_ Developer Template Other Complete Elapsed Time HH MM SS GSM BTR 1 54 Developer Template Development Library Other Complete Session 2 04 56 45 J m j View tteration V Auto Scroll z za JE WCDMA Cell 1 Data Series WCDMA Cell 3 WCDMA Cell 1 WCDMA Cell 5 WCDMA Cell 3 WCDMA Cell 5 M Show Annotations T i Call Drops Cell P
26. ed A dienCE s icuccevescsvcnicsshinzasseoassagsnnsossesdtocsonengaessnseeneaxeaneiveaas 1 1 3 Before You Get Started issos aseesinaine as 1 1 4 Security Information sesesssssessssesesssesesessseeeeesssseeeesssseeessseeeessseeeessseee 1 1 5 Accessing Documentation ssssseesesesseseceseeeseesecesecsseeseessecoreeseeeseeseeee 3 1 5 1 Accessing Documentation from Windows EXDPIOFel csssceeeceeeeeseeeeeeeees 3 1 5 2 Accessing Documentation from Test MANAGET ccccccccssnsssesecescceceneees 4 Simulated Drive Description sscsoooooosoooossosossosssssscsssssssssossssssesse D Zaks OVERVICW E E E A ET 5 2 2 DESCUDUON sincurisricrinstvancnsbeictsvie i neei eenen Tanos EEEE EEEE ik 5 2 3 Simulated Drive Testing es sseseessssseesseccsscccsseesseecsssscossccoseeceseeessseess 5 Using Simulated Drive eeeoossececcoseececcsseeeeccsseeeecsseeceecoseeccsseseesssee Bile QVEMEW snsitersioniidcascinasriimnsmedmitiiaiasieepmiamemaiae 6 3 2 Platform Parameters jis ceicssecasesavassatunccdescsenssseeasscteescsanenomvtedetecesssseonnce 6 3 21 Supported Platforms osere ee E E ects 6 9 2 2 Platform Selection sissssssereievesiseirsreirsiirestt cser ess tvert e nokost EVERE NERE TEES 6 3 2 3 Specifying a Platform File ooooeeooennoeenseenseenneesseeeseesseeeseeeseeereeeseeeseeeseeee 6 3 3 UE Parameters iiissicsicudapstesariniosninaceiiecs cries 7 3 3 1 UE Interface ParametefS sseee
27. efault connection in the Device Manager when the BlackBerry is connected a9 6 4 9 Samsung Galaxy S2 Android ICS Mode 1 2 3 Software Drivers Samsung Kies Simcard ATT Sim and Spirent PLMN Phone Settings a Settings Wireless amp Networks gt Mobile Networks gt Data Roaming On b Settings Wireless amp Networks gt Mobile Networks gt Use Packet Data On Automatic Call Answer Enabled Run Test Manager and pause the test when the wait 60 seconds for registration message displays Put UE in flight mode If the UE does not register reboot the UE and make sure it makes a data call automatically Chapter Six Appendix A 39 7 Tethering Settings Wireless amp Networks gt Tethering and Portable Hotspot USB Tethering On 8 Make sure that device displays as Samsung Mobile USB remote NDIS Network device under the Data Clienttab in UE file It should match the settings under WIFI Connections on the client laptop If it does not change the UE file name save the file and stop the test Rerun the tests repeating steps 5 6 and 7 7 Appendix B Frequently Asked Questions A 7 1 System Failures 7 1 1 Fail to Remote Desktop into the Application Server Connect mouse keyboard and to the application server Restart the Server and make sure you can now login remotely 7 1 2 Tests Halt Due to Autoset failure 508 Check if the RF cables in the SR5500 are tightened using a tor
28. eight test cases as shown in Figure 5 6 The test cases have been combined and the resulting eight test cases correspond to the original eight in terms of Boston New York and Seattle routes 28 UMTS Simulated Drive User Manual Program Files Spirent Communications AirAccess WCDMA HS Test Session Logs 1 Developer Template Other Complete 2 Developer Template Other Complete 3 Developer Template Other Complete 4 Developer Template Other Complete 5 Developer Template Other Complete 6 Developer Template Other Complete lt es i r Source Constructed B C Program Files Spirent Communications Airdccess WCDMA HS Test Session Logs 1 Developer Template Other Complete 1 ete 2 Developer Template Other Complete 3 Developer Template Other Complete 4 Developer Template Other Complete 5 Developer Template Other Complete Figure 5 4 Selecting Test 1 5 from 2012 5 11 1226 B C Program Files Spirent Communications AirAccess WCDMA HS Test Session Logs 1 Developer Template Other Complete 2 Developer Template Other Complete 3 Developer Template Other Complete 4 Developer Template Other Complete 5 Developer Template Other Complete 6 Developer Template Other Incomplete 7 Developer Template Other Complete 8 Developer Template Other Complete B C Program Files Spirent Communications AirAccess WCDMA HSi Test Session Logs 1 De
29. ent Library Other Complete Other 6 of 6 100 5 GSM BTR 1 54_ Developer Template Development Library Other Complete Elapsed Time HH MM SS 6 GSM BTR 1 54 Developer Template Development Library Other Complete Session 06 56 07 J lt Figure 5 2 New Test with 6 Runs of the Previous Incomplete Run The next step is to create a consolidated test containing 8 test cases The steps are shown below 1 Open Test Manager and select Result Files from the File Cabinet 2 Select the 2012 5 11 1226 and 2012 05 13 1722 folders Right click and select Combine Results from the menu as shown in Figure 5 3 Chapter Five Reports 27 File Edit View Execute Report Help D e aJa aal n m Close Alt F4 X Delete Combine Results Add to Zip file Suite Files Create Report Properties Figure 5 3 Combining Results 3 The Results Combinerwindow displays as shown in Figure 5 4 Under Source select the desired test cases For example select Test case 1 5 from 2012 5 11 1226 and click the arrow button to move them to Constructed window area 4 Select test case 1 from 2012 05 13 1722 and repeat as shown in Figure 5 5 This test case corresponds to the Test case 6 Seattle Drive Leg 1 which can be found under the Deta section of 2012 5 11 1226 5 Repeat the procedure for remaining two test cases 7 and 8 of test 2012 511 1226 The Constructed window now displays
30. es not register reboot the UE and make sure it makes a data call automatically Tethering Settings Wireless amp Networks gt Tethering and portable Hotspot gt USB tethering On Make sure that device shows as Samsung Mobile USB remote NDIS Network device data client tab in UE file This should match the settings under WIFI connections on client laptop If it does not change the UE file name save the file and stop the test Rerun the tests repeating steps 5 6 and 7 6 4 3 Samsung Captivate Android ICS Mode 1 2 3 Software Drivers Samsung Kies Simcard ATT Sim and Spirent PLMN Phone Settings a Onthe UE dial 9090 Chapter Six Appendix A 35 b Service Model gt Log via USB c Dial 7284 d Phone Utility gt UART modem and USB PDA e Switch off and onthe UE connect to client laptop and verify that it shows as Samsung Android USB modem or 1 2 etc under device manager gt modem f Settings Wireless amp Networks gt Mobile Networks gt Data Roaming On g Settings gt Wireless amp Networks gt Mobile Networks gt Use Packet Data On h Special Steps Make sure that the Modem Com ports in Client Laptop for this UE matches the COM Port in the UE file Automatic Call Answer Enabled Run Test Manager and pause the test when the wait 60 seconds for registration message displays Put UE in flight mode If the UE does not register reboot the UE and make sure it
31. f 8 100 5 GSM BTR 1 54 Developer Template Development Library Other Complete 39 Elapsed Time HH MM SS 6 GSM BTR 1 54 Developer Template Development Library Other Incomplet 2 Session 05 49 17 Procedure Graph a View taaten D Zoom m Data Series Events X Type PC Timestamp Description Debug 5 11 2012 5 14 17 PM Start Call Processing Debug 5 11 2012 5 14 21 PM ReSetCellPower Eror 5 11 2012 5 14 21 PM Action Step Force Registration UE Failed to register with the network on all required domains Reverting LAC RAC PSC Cell Eror 5 11 2012 5 14 21 PM Step 2 Eror handling exiting the test case Figure 5 1 Test Run with Test 6 Incomplete As shown in this example out of eight Test cases Test case 6 is Incomplete even after one automatic retry as UE failed to register A new test is created 2012 05 13 1722 which runs the failed Test case six times as shown in Figure 5 2 Simulated Drive Test MRAB v1 6 8 sui 201205111226 201205 13 1722 Dashboard XxX Summary Indicator Values Test Folder Test Status Percentage GSM BTR 1 54 Developer Template Development Library Other Complete Fail 0 of 6 0 2 GSM BTR 1 54 Developer Template Development Library Other Complete Pass 0 of 6 0 ITE GSM BTR 1 54_ Developer Template Development Library Other Complete Pending 0 of 6 0 nE GSM BTR 1 54__ Developer Template Developm
32. g tests provided by this module Passwords are tied to security information provided by the USB hard lock dongle that comes with the module installation If you have any questions or concerns contact Spirent Technical Support at support spirent com or by phone at 1 800 SPIRENT 1 5 Accessing Documentation There are two ways to access this document from the Controller PC 1 Windows Explorer 2 Test Manager These methods are described in detail in the sections below 1 5 1 Accessing Documentation from Windows Explorer Access this Manual offline by navigating to C Program Files Spirent Communications Test Manager Modules UMTS Simulated Drive Simulated Drive User Manual pdf using Windows Explorer as shown in Figure 1 3 4 UMTS Simulated Drive User Manual UMTS Call Reliability File Edit Yiew Favorites Tools Help Q Bak wi P Search le Folders Gi Address C C Program Files Spirent Communications Test Manager Modules UMTS Call Reliability Go Folders x Name Size Type a gt SRSSOO Testkit Mobile Originated Voice Call 5 4KB TLC File O Test Manager Mobile Originated Voice Call 5 24KB XSD File E Common a Mobile Terminated Yoice Call 5 23KB TestStand Seq E Manager Mobile Terminated Voice Call 5 4KB TLC File 2 Modules E Mobile Terminated Voice Call 5 24KB X5D File 2 UMTS Call Reliability 9 module xml 9KB Mozilla XML File C Channel Configurati Fjuser Manual pdF 47 KB Adobe Acrob
33. he PS RAB Session Could Not be Restarted Before every FTP transfer starts network pings the UE and waits for 4 successful ping After 10 consecutive pings with not 4 consecutive responses above message is printed 7 4 2 Start FTP Transfer FTP Transfer Ended Unexpectedly If a call drops during active FTP transfer above message is printed 7 4 3 UE Failed to Respond to Page If UE does not respond to network pages due to RF condition or due to constantly re registering on different cells due to RF condition above message is printed When RF condition improves and UE stabilizes on one cell this situation is automatically corrected 7 4 4 Voice Call Not Active Due to RF loss or poor RF conditions attempt to setup voice call fails and this message is printed Chapter Seven Appendix B 43 7 4 5 Compressed Mode Physical Channel Reconfig Complete Not Received by the Network When RF condition is not good and deteriorating UE may want to switch to a better cell OR enable Compressed Mode If Network sends Physical Channel Reconfiguration message to the UE and call drops around this time UE will not be able to send response message back to Network Hence the above message is printed 7 4 6 Compressed Mode Error in Sending Physical Channel Reconfig Message When RF condition is not good and deteriorating UE may want to switch to a better cell OR enable Compressed Mode If Network sends Physical Channel Reconfiguration
34. he RF conditions reported for that route with the standard Network settings described in the sections corresponding to the individual test case for the route The UMTS Simulated Drive Test Pack leverages the state machine implementation of the network emulator component to handle rapidly changing RF conditions and asynchronous events from the UE to simulate a drive route through a wireless network The UMTS Simulated Drive test cases run with the Test Manager software This provides an easy way to automate test sessions with analysis and reporting capabilities Simulated Drive Testing UMTS Simulated Drive offers a complete platform to evaluate UE behavior and performance similar to that of a live network The platform plays back recorded RF conditions from the field in an emulated environment to allow a UE to be tested for a drive route in a lab environment rather than having to drive the actual route This allows carriers to gain a realistic view of how a UE performs on a live network without requiring them to leave the lab 3 Using Simulated Drive 3 1 Overview This chapter provides information on how to use UMTS Simulated Drive from Test Manager This includes the following steps Configuring the Platform Parameters Configuring the UE Parameters Configuring the Session Parameters Locating the Test Cases Running the Test Suite 3 2 Platform Parameters 3 2 1 Supported Platforms The UMTS Simulated Drive Test suite supports
35. i SPIRENT 8100 Mobile Device System UMTS Simulated Drive User Manual Spirent 541 Industrial Way West Eatontown NJ 07724 USA Email sales spirent com Web http www spirent com AMERICAS 1 800 SPIRENT 1 818 676 2683 sales spirent com EUROPE AND THE MIDDLE EAST 44 0 1293 767979 emeainfo spirent com ASIA AND THE PACIFIC 86 10 8518 2539 salesasia spirent com This manual applies to UMTS Simulated Drive Test Packs Version 1 6 5 or higher Page Part Number 71 006823 Version AO Copyright 2012 Spirent All Rights Reserved All of the company names and or brand names and or product names referred to in this document in particular the name Spirent and its logo device are either registered trademarks or trademarks of Spirent plc and its subsidiaries pending registration in accordance with relevant national laws All other registered trademarks or trademarks are the property of their respective owners The information contained in this document is subject to change without notice and does not represent a commitment on the part of Spirent The information in this document is believed to be accurate and reliable however Spirent assumes no responsibility or liability for any errors or inaccuracies that may appear in the document Table of Contents 1 MEPEPOOUCHION iscccccescausceseenseroberstesseueeessensbesserehesssedceccccccesepecenseeeceeeosses I 14 OVENI EWN gia eE E as eae aaa E EEES 1 1 2 Intend
36. ired for statistical verification Chapter Four Reference 19 4 3 6 Seattle Drive Route Leg 1 e Test case duration is approximately 50 minutes start to finish e 3 Full WCDMA Cells Single Frequency e 65 In Call Percentage required for statistical verification e 1GSM Cell at a constant 80 dBm Inter rat handover possible but not expected 4 3 7 Seattle Drive Route Leg 2 e Test case duration is approximately 20 minutes start to finish e 3 Full WCDMA Cells Two Frequencies e 65 In Call Percentage required for statistical verification e Hard Handover possible on this route 4 3 8 Seattle Drive Route Leg 3 e Test case duration is approximately 30 minutes start to finish e 3 Full WCDMA Cells Single Frequency e 65 In Call Percentage required for statistical verification e 1GSM Cell at a constant 80 dBm Inter rat handover possible but not expected 20 UMTS Simulated Drive User Manual 4 4 Description of UMTS Simulated Drive Functionality The functionality used in the Simulated Drive Test Cases is a sub set of the overall capability of the UMTS 8100 platforms However within the context of Simulated Drive the functionality that is supported is fully integrated and works together seamlessly The main differentiator between Simulated Drive and other Spirent 8100 Platform products is that Simulated Drive leverages state machine functionality to REACT to real world real time events and provide an analysis of the
37. l Commands Pre Registration Pre Call Post Call E 6 FTP Options FTP Mode Active Agilent Power Supply Insertion Loss erered Figure 3 5 UE Spirent Data Client Parameters Dial Up Networking 3 3 4 2 Internet Connection Sharing Most new phones are smart phones which require USB tethering this requires you to utilize Internet Connection Sharing Mode on the client laptop To set up the UE file to use ICS Mode 1 Set the Connection Mode to Internet Connection Sharing 2 Set the MTU Size to 1410 Chapter Three Using Simulated Drive 13 y E 1 General Connection Mode Internet Connection Sharing gt z AT Connection Verification Type Ping A MTU Size 1410 5 E 3 Internet Connection Sharing amp Connection Dialog Box Disabled t Device Interface Name Motorola USB RNDIS Device TE 5 Shell Commands Pre Registration Pre Call Post Call E 6 FTP Options FTP Mode Active Insertion Loss Figure 3 6 UE Spirent Data Client Parameters Dial Up Networking s Network Connections File Edit view Favorites Tools Advanced Help O Ba gt B O Search Folders E Address Network Connections Name Status Device Name Network Tasks LAN or High Speed Internet SEE A 21394 Connection Disabled 1394 Net Adapter 4 Set up a home or small Local Area Connection Connected Intel R 82579LM Gigabit Network Connection office network L Local Area Connection 11 Connected Motorola USB RNDIS Device
38. led on the UE and verify on the network connections tab that connected can be seen for that data connection Moreover in some devices require their desktop management software to be running on the client laptop for making a data call Make sure the Device Interface name matches in the UE file and the client laptop Check on the network connection tab if the LAN has limited connectivity and in this case remove the USB cable from the client laptop restart the test and verify that the LAN is showing connected this time 7 2 3 UE device drive is failed to show up on client laptop Windows Device manager Make sure the USB cable is not too long max 2 ft recommended 1 ft from UE to client laptop 7 2 4 SIM Not Provisioned 7 3 This error is usually seen on the UE as a result of the UE attempting to attach to a live network instead of the network emulator Ensure the UE is properly shielded from all outside RF signals other than the network emulator signal Power off the UE and then power back on If the UE continues to report SIM Not Provisioned it may be remembering the last network it attempted to attach to instead of attaching to the network emulator Power off the UE pull the battery power source from the UE reapply the power source battery to the UE and power the UE back on Test Case Results 7 3 1 Boston Leg 1 has more call drops than the other legs Boston Leg 1 has more challenging RF conditions resulting in more
39. leted after an Event 2f OR due to a Stop Compressed Mode Postponed retry Start FTP Transfer Start one of the Pseudo Randomized FTP Transfers Hard Handover Hard Handover to another frequency WCDMA Cell successful Failure Actions Most Failures listed below are usually a result of rapidly changing RF conditions where the request comes in a little too late Call Drop Current Call Dropped common causes are described in the next section Add Soft Handover Failure Attempt to add cell to soft handover failed Delete Soft Handover Failure Attempt to delete cell from soft handover failed Start Compressed Mode Failure Attempt to put the UE into compressed mode failed 24 UMTS Simulated Drive User Manual 4 5 Stop Compressed Mode Failure Attempt to bring the UE out of compressed mode failed Hard Handover Failure Hard Handover to another frequency failed 4 4 2 4 Common Causes for Call Drops All of the Call Drop causes below are expected and considered normal behavior The frequency of the drops is what determines whether or not the behavior of the UE is acceptable RF Loss RF Loss on one of the cells before it can be deleted from the active set causes the cells to go out of sync and drop the call RLC Reset Due to changing RF conditions the UE may go into RLC retransmission In this case the RF conditions may recover and the call stays up The first attempt moves the receive window in an attempt to keep the
40. ll Statistics Table In the above table Average number of call drops is 4 6 which is the average of 8 values of column named Num Call Drops This is also shown as the value under average number of call drops in the Simulated Drive Test Results table There are other columns in the table that deserve attention Let us consider iteration1 corresponding to Boston Drive Route Leg1 Here an attempt was made to setup voice call 35 times and 15 of those attempts were successful In these successful attempts the call dropped 14 times 6 Appendix A UE Configuration Procedure Samples 6 1 Overview The following are a few examples with various UE s User should download the latest UE software s and use the latest firmware as it was observed that doing so does improve the In call percentage and give better results 6 2 Common Procedures Wifi Bluetooth GPS Location Services should be off Create UE files for the ICS mode or Dial Up mode as per the method described earlier in Chapter 3 6 3 Installing Auto Answer App for Android Phones 1 Download Google AutoAnswer apk with GPL license and place it in the Client Laptop under C Program Files Spirent Communications Spirent Data Client 2 Enable USB debugging by selecting Settings Applications Development gt USB Debugging On 3 Execute on command prompt to make sure that android USB device shows up with device number and a message 1 device connected b
41. nd Ciphering Up to 3 Full WCDMA Cells plus 1 Full GSM Cell Up to 2 Full WCDMA Cells on one frequency and 1 Full WCDMA Cell on a different frequency Rate Shift to from R99 UL HSDPA DL to from R99 UL R99 DL Voice Data dormancy to PS Idle RAB Release Dormancy CS PS Supplementary Service with multi RAB call Pseudo Randomized FTP Transfers Pre R8 SCRI Handling 4 4 2 Procedure Graph For the purposes of the explanations in this section Horizontal wavy lines on the graph are referred to as the RF Level Cell Activity Lines Vertical dotted lines are Event Lines Vertical Solid Lines are Activity lines 22 UMTS Simulated Drive User Manual pa a al Ez D FADD 407 5 f j g t i 4 i H 4 50a a H 7 wu we al A Ze Y pe a T FA UA 60 r ve gt Zs Start Compressed hdode Pag L 70 5 o 5 oOo 3 5 2 E B WCDMA Cell 1 PP m ps Pe WCDMA Cell 3 a 5 8 a A a 2 WCDMA Cell 5 k T a gt s 7 Delete Sott edeleehel tet 7 17 15 41 677 i 17 14 52 602 17 15 02 417 17 15 12 232 17 15 22 047 H sit 17 15 31 862 17 15 51 492 17 16 01 307 17 16 11 122 17 16 20 937 17 16 30 752 17 16 40 567 17 16 50 382 17 17 00 197 17 17 10 012 17 17 19 827 Time Figure 4 2 Procedure Graph The events and actions that are printed on the graph may not be the EXACT time of the occurrence of
42. ng cell reselection which will force another registration Also the network can only process one handover request at a time Therefore if another request comes in while a previous one is being processed it must be ignored until the first request is complete This may cause the UE to lose the best cell due to the rapidly changing RF conditions which can result in a call drop This is expected and accounted for in the number of call drops expected SDT Inter RAT handover to GSM is supported only in one way direction Data RAB is dropped before handing over to the GSM network Only voice RAB hands over to GSM GSM gt WCDMA inter RAT handovers are not supported After successful completion of inter RAT handover the voice call is dropped and drive simulation continues after the UE reselects the WCDMA network SDT requires the UE to be tethered to enable ICS mode Proper UE drivers are also required to install NIC interface to enable tethering If a UE does not support tethering a non preferred dial up mode can be used Chapter Four Reference 21 4 4 1 Supported General Functions The following functionality is fully integrated and supported in conjunction with all the features listed for SDT Voice Data R99 UL HSDPA DL HSDPA Serving Cell Change Multi RAB 3 Way Soft Handover Inter Frequency Hard Handover 3G gt 3G Voice and Data Inter RAT Hard Handover 3G lt gt 2G Voice Only Compressed Mode Voice Playback Authentication a
43. nssssssseeeeoseesssssesecocosssssssseceeosossssseeeeeseoss 7 3 3 2 UE Manual Interface Parameter cccccecccesesesceeeseeeceseseececeeseeeceeseceeeeeees 8 3 3 3 UEAT Interface Parameters sisicsseirorisereirereiirecarironinncinni aii 10 3 3 4 UE Spirent Data Client Parameters sssscsciscsscceccsdtonscsessasssoresseceacasesverctuvavics 11 3 4 Specifying UE and Session Parameters sesssseesssssseesssseeeessseeeessssse 14 3 5 Locating the Test Suite and Test Cases ssseeeesssseeeessseceessseeeessseeeee 14 351 Locating the TeStSU Eoinn E a 14 3 5 2 RUNNING te Test Suite sorser e a A a ae 15 Simulated Drive ReferenCe eeeecccccccsssssssececcccccssssseeeceececsssseeeees LO AT OVENVIEW e aea E AA EEA EEEE EEE AAE 16 4 2 Test Suite Execution ssssssssssesesesesesescsessseececesecesecsssosssoescsseeseeceee 16 ii UMTS Simulated Drive User Manual 4 3 Simulated Drive Test Cases ssc siwccc cccsessxessemmeicccseeeasssamueninicceesewiduas 18 4 3 1 Boston Drive Route LOG To cssescevcsssazasccdandcaasassexscsdesecwsbausdvasnveseaseansvenceiean 18 4 3 2 Boston Drive Route Leg 2 siscccsssivecosececdonseavergsensaecsssaubeasdeoseueieeseresieseoeaanes 18 4 3 3 Boston Drive Route Leg 4 esessssssssesssssesssssseseeoeosssssssecoeeosossssseeecoseesssssese 18 4 3 4 New York New Jersey Drive Route Leg 1 sssssssseesesssessnssssssenssssesens 18 4 3 5 New York New Jersey Drive Route Leg 2 1s
44. ocess 7 4 12 Release 3G Data Call Data Call Not Terminated Timeout Waiting for ReleaseCall in Air ACCESSHS ccccccsesssnesesecceccceceeneesssccecccecceeeceneeeeeeeeceees 1 Lil 1 2 1 3 1 4 introduction Overview This manual provides information on the performance testing features of the UMTS Simulated Drive Test Packs SDT You will become familiar with the Suites Test Cases and setting up UMTS Simulated Drive testing scenarios in the Test Manager environment by following the step by step procedures and Test Reference documentation Intended Audience This manual is intended for those who have a working knowledge of wireless communication equipment and are familiar with the automated testing of mobile devices It is assumed that you are familiar with the Test Manager GUI environment If you are unfamiliar with Test Manager refer to the 7est Manager User Manual before proceeding Before You Get Started Before getting started install all software and power on the test system The controller PC is shipped with the Test Manager Test Executive software and UMTS Simulated Drive Test Packs installed Security Information UMTS Simulated Drive is shipped with the appropriate dongle and software hardware security passwords configured To verify the security information 1 Open Test Manager and select Help About The About Test Manager window displays as shown in Figure 1 1 2 UMTS Simulated Drive Use
45. om 2012 5 11 1226 Chapter Five Reports 29 6 Click Save As and save the results file in the same folder as the combined test files The Combiner automatically creates a new consolidated test with the updated date and time Close the Combiner 201205111226 201205131722 201205 140523 s Lt r 2012 05 13 1722 Indicator Values Test Folder l 2012 05 14 0523 Test Status Percentage GSM BTR 1 54 Developer Template Development Library Other Complete 3 Fail 0 of 8 0 GSM BTR 1 54 Developer Template Development Library Other Complete 3 Pass 0 of 8 0 GSM BTR 1 54 Developer Template Development Library Other Complete 3 Pending Oof 8 0 GSM BTR 1 54_ Developer Template Development Library Other Complete 3 Other 8 of 8 100 GSM BTR 1 54_ Developer Template Development Library Other Complete 3 Elapsed Time HH MM SS GSM BTR 1 54 Developer Template Development Library Other Complete 3 Senin seats SS Procedure Graph v Results z 2012 05 11 1226 5 2 Generating a Report This section explains how to create the excel report from the 8 test cases we just created This would apply to the test cases obtained standalone on first run or created through combining various tests as in the previous section Cabinet TIEI X Summary Cell Power dBm Figur
46. opment Samsung Galaxy S2 In this case the Samsung Galaxy S2 folder contains a subfolder with the test case name containing eight all Complete first time to sixteen all rerun subfolders for each of the test cases iterations Results mdb file Spreadsheet report and other folders 5 1 1 1 Combining Results Normally all the legs will complete and there is no need to rerun tests However out of the 8 Test cases some of the test cases might show status as other Incomplete not due to genuine lower In Call percentage but due to UE not registering or Internet Connection Sharing establishment problems etc So we might want to rerun the problematic test cases as a new test s For example consider the following run from Samsung Captivate 2012 5 11 1226 26 UMTS Simulated Drive User Manual y Simulated Drive Test MRAB v1 6 8 sui 2012 05 11 1226 Dashboard X Summary Indicator Values Description Test Case Test Folder Test Status Percentage 1 GSM BTR 1 54 Developer Template Development Library Other Complete 39 Fail 0 of 8 0 2 GSM BTR 1 54 Developer Template Development Library Other Complete 39 Pass 0 of 8 0 3 GSM BTR 1 54_ Developer Template Development Library Other Complete 39 Pending 0 of 8 0 4 GSM BTR 1 54_ Developer Template Development Library Other Complete 39 Other 8 o
47. or ReleaseCall in AirAccessHS This error is seen with the smartphones As Test Manager prepares to tear down the data and voice calls at the end of the test some UE s due to always on feature try to set up data call which leads to this race condition and the error
48. ower dBm Compressed Mode Events FA Type PC Timestamp Inform 4 27 2012 4 24 40 PM Connecting to AirAccessHS inform 4 27 20124 24 56 PM Connecting to SR5500 1 Inform 4 27 2012 4 25 39 PM Connecting to Agilent Power Supply Inform 4 27 2012 4 25 39 PM Started lt Developer Template gt infom 4 27 2012 4 25 43 PM Ensuring AirAccessHS is idle Inform 4 27 2012 4 25 48 PM Ensuring SR5500 is idle Inform 4 27 2012 4 25 49 PM Ensuring Agilent Power Supply is idle 22012 4 25 50 Dit pp pecting to Soani Daalen Sof mk i er Start Compresseq Moda 8 pa o a 5 ra Delete Sot Mandya etb elb ahid Figure 3 10 Test Session in Progress 4 Simulated Drive Reference 4 1 Overview This chapter provides detailed descriptions of the UMTS Simulated Drive test cases It includes the algorithms implemented during test execution 4 2 Test Suite Execution This section describes the sequence of operations performed during the execution of the SDT suites from Test Manager Chapter Four Reference 17 Failure Validate Suite Pass False Connectto the Instruments Start Call Processing and Wait for Initial Call Establish Pass Wait for Dynamic Environment Emulation DEE ie RF Playback to start Stop DEE Wait for Call Teardown and Stop Call Processing Disconnect from the Instruments False Complete
49. pter Four Reference 23 Event 1a Add Cell to Soft Handover Event 1b Delete Cell from Soft Handover Event 1d Update Best Cell May result in Physical Channel Reconfiguration to change the HSDPA Serving Cell If not currently in a HSDPA call this event is handled internally and will not show on the procedure graph Event 2b Hard Handover Event 2d Enable Compressed Mode In the case of this event if the network is busy processing a different event Event 2d will be saved and retried until either successful Call Drop or Event 2f is received Event 2f Disable Compressed Mode Event 3a Inter RAT Handover 4 4 2 3 Action Lines Events that are processed and actually take place in the state machine are referred to as Actions Other significant actions may also occur such as FTP transfers Call Drops etc All of these actions are denoted with solid vertical lines in the procedure graph and labeled accordingly Successful Actions Add Soft Handover Action completed after an Event 1a Delete Soft Handover Action completed after an Event 1b Start Compressed Mode Postponed Action postponed due to Event 2d received and another action is currently being processed Start Compressed Mode Action completed after an Event 2d OR due to a Start Compressed Mode Postponed retry Stop Compressed Mode Postponed Action postponed due to Event 2f received and another action is currently being processed Stop Compressed Mode Action comp
50. que wrench and DBB cables at the rear are locked in place and restart the test suite 7 1 3 UE Fails Hard Handover 100 of the Time One of the things that may be checked is For A304 and A604 make sure that the rack is not cabled as 2 4 channel Rx diversity 7 1 4 Fails to Start DEE Mode Open the Control Panel and make sure that the Firewall Settings are disabled Then select Administrative Tools Services and ensure that the FTP Windows service is running Open the Command Prompt and type ftp 192 168 0 35 If you get a response prompting you to enter a User Log in and Password the FTP server is running Otherwise under Administrative Tools Services start and then restart the FTP service or FileZilla service Note that only one FTP server can be running at a time Check to make sure SR5500 sync cables are properly connected 7 2 UE Failures 7 2 1 UE Fails to Register After Installing Drivers and Connecting to Instruments Check that a proper RF connection is made on the UE and ensure that the primary and secondary connections or GPS port is not connected to RF are not swapped Also ensure that the UE is powered on not in sleep mode and the USB is connected to the client laptop Hard reset the phone Chapter Seven Appendix B 41 7 2 2 While making a data call in ICS mode Data call cannot be established since the UE IP address has not been assigned message is displayed Make sure that tethering is enab
51. r Manual About Test Manager Test Manager Version 1 6 0 Build 148 Copyright 2008 2009 Spirent Communications Inc Test Folders Building Blocks Platforms Passwords 34 Test Folders Ea ae 2G Call Reliability CS amp PS S P l R E N j 3G Call Reliability CS amp PS Communications 3G HSDPA DL Mobility 3G R99 Mobility CS amp PS 3G 2G Hierarchical Cell Selection 3G 2G Inter RAT Call Reliability 3G 2G Reselection Version Figure 1 1 About Test Manager Window 2 Select the Passwords tab 3 Under nstalled Features confirm the UMTS Simulated Drive Test Packs are installed as shown in Figure 1 2 Chapter One Introduction 3 About Test Manager Test Manager Version 1 6 0 Build 148 Copyright 2008 2009 Spirent Communications Inc Test Folders Building Blocks Platforms Passwords Installed Features Installed Features a Password SPIRENT TM UCR 3GCR MDTTM UCI e lapsien TM UCR HS DL MOBILITY t Update Password Delete Password TM UCR 2G3GICR MDTTM ASA Trial Expiration Date TM UCR 2GCR MDTTM UCI TM UCR 2G3GRES MDTTM THAICR R OGRE ITY hn lt iii Feature Version Add New Password Password Hard Lock Serial ID o Em Figure 1 2 About Test Manager Window Installed Features Password authentication is a prerequisite to runnin
52. rface 1 Timeout Figure 3 3 UE Parameters Window UE Manual Interface Parameters Configuring these parameters correctly can be effective in allowing some level of automation even if AT commands are not available or supported for the UE under Test For example if the UE can be configured to auto answer an incoming call you can set the timeout for Answer Voice Call This allows the UE time to auto answer the call and the Timeout Button for Answer Voice Call can be set to OK This configuration allows the device to reliably answer network initiated voice calls 10 UMTS Simulated Drive User Manual 3 3 3 UE AT Interface Parameters If the UE reliably supports AT commands AT control can be very useful The UE is connected to the Data Client PC for AT control after the manufacturer s driver software is installed The following parameters are provided Port Set this parameter to the port on the Data Client PC assigned to the modem when the UE is connected Commands Configure the available AT command strings Bits Per Second Configure the baud rate Data Bits Configure the number of data bits Parity Configure the parity Stop Bits Configure the stop bits Flow Control Configure the flow control Enable DTR Enables Data Terminal Ready Signaling Enable RTS Enables Request To Send Signaling New Line Specifies the new line terminator to use Test AT Commands Use this wizard to test AT commands to
53. sssssssseesssssssenscenssensesseeens 18 4 3 6 Seattle Drive Route Leg 1 s sssseseeeeeeoeeeeeeeeeeeeeeeoeeeeeeeereeeeeeoreeeeeeereeeeeeeee 19 4 3 7 Seattle Drive Route Leg 2 sssssseeseeesseeeeeeeseeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee 19 4 3 8 Seattle Drive Route Leg 3 ssssessesesssessesensessesesssssssssssesesenssssssensssseeens 19 4 4 Description of UMTS Simulated Drive Functionality sessssenessseeee 20 4 4 1 Supported General FunctionS esssssseeesseeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee 21 4 4 2 Procedur Graph scessisissrrisrestereseesrerisssrtoerdstsrerisreser torei etrassts setasion uae 21 4 5 Definition of Individual Test Case ReSUItS cccccsssssccceeeeeeeeeceeees 24 Simulated Drive RepottS ssccsoocsoooossoosoossooosscsssssssossssossssssssssesee ZD 5 1 Report Gene ratiONn ersecsiecsssoesrsecarsscsirss tors re sipoti YE Ki STERNE r EEKE S 25 5 1 1 R eg ireMentS ccsssiisiesececrsieesi scenens ineert iese EE EEEE RE 25 5 2 Gen rating a Reporterescsesisserersierseriiicserrsire iieii 29 5 3 Pass Fail Criteria esessessessesssessesssssssscescosscsscessosesossesescsscesecsesese 31 5 3 1 Number Of MCrQuons sccscscsscessasveaticscisiicieneisiavbevsdatedoecataosevasiasessdicaenenadouses 31 532 ICQI Percentage nna a easiness eae 32 5 3 3 Average Number of Call Drops sevccsanjccnsescanascsncdascesxsesteavantadeinasarvevsantnsce 32 Appendix A UE Configuration Proce
54. termine if the Call Drop Number is statistically relevant This is important for the Simulated Drive Test due to rapidly fading conditions under which UE operates leading to call drops and potential failure to re establish the call For example in the Boston Route Leg 1 in about 50 minute test there may be a legitimate call drop after 10 minutes and we may not be able to set the call back up for the remaining 40 minutes We report the single call drop in this iteration which is very good however in reality out of the 50 minute test the UE was in call for only 10 minutes A better representation of the test is the percentage of time UE stayed in Call which is In Call Percentage For this example it would be 20 10 50 Therefore in order for the Call Drop number in the test to be considered statistically relevant it must also meet the In Call Percentage criteria 5 3 3 Average Number of Call Drops This is the average of the Number of Voice call Drops averaged over 8 iterations Basic Voice Call Statistics Num Call Num Call N Call Average Iteration InCall Setup Setup s a Num Call Attempts Success non overen Pass id n efa 1 Soome es ss as 2 conte o2 6 s _ 3 compute sam 6 7 6 _ compete toon s 3 Ts compu es 2 2 1 _ 6 compre orm os _ 7 compute i000 1 0 _ 8 oome orem s 3 _ Figure 5 12 Basic Voice Ca
55. veloper Template Other Complete 2 Developer Template Other Complete 3 Developer Template Other Complete 4 Developer Template Other Complete 5 Developer Template Other Complete 6 Developer Template Other Complete 1 Developer Template Other Complete 2 Developer Template Other Complete 3 Developer Template Other Complete 4 Developer Template Other Complete 5 Developer Template Other Complete 1 Developer Template Other Complete Figure 5 5 Selecting Test 1 from 2012 05 13 1722 B C Program Files Spirent Communications Airdccess WCDMA HS Test Session Logs 1 Developer Template Other Complete 2 Developer Template Other Complete 3 Developer Template Other Complete 4 Developer Template Other Complete 5 Developer Template Other Complete 6 Developer Template Other Incomplete veloper Template Other Complete 8 Developer Template Other Complete B C Program Files Spirent Communications AirAccess WCDMA HSi Test Session Logs 1 Developer Template Other Complete Constructed 1 Developer Template Other Complete 2 Developer Template Other Complete 3 Developer Template Other Complete 4 Developer Template Other Complete 5 Developer Template Other Complete 1 Developer Template Other Complete veloper Template Other Complete loper Template Other Complete Figure 5 6 Selecting Test 7 8 fr
56. y typing the command adb devices If not make sure that step 1 is executed C Program Files Spirent Communications Spirent Data Client adb devices 4 Execute on command prompt C Program Files Spirent Communications Spirent Data Client adb install AutoAnswer apk 6 4 Specific UE Instructions 6 4 1 Samsung Focus Windows7 Dialup Mode 1 Software Drivers Samsung Kies 2 Simcard Spirent Sim 3 Phone Settings Default No Change gt Tethering Settings Default No Change 34 UMTS Simulated Drive User Manual 5 6 APN Settings Default No Change Special Steps Make sure that the Modem Com port in Client Laptop for this UE matches the COM Port in the UE file Create a Dial up Connection with the following settings a Connection Name TestDrive Data Connection b Username test Password test d Phone Number 99 e Set Com Port as the default connection in the device manager when the BlackBerry is connected o 6 4 2 Samsung SkyRocket Android ICS Mode 1 2 3 Software Drivers Samsung Kies Simcard ATT Sim and Spirent PLMN Phone Settings a Settings Wireless amp Networks gt Mobile Networks gt Data Roaming On b Settings Wireless amp Networks gt Mobile Networks gt Use Packet Data On Automatic Call Answer Enabled Run Test Manager and pause the test when the wait 60 seconds for registration message displays Put UE in flight mode If the UE do
Download Pdf Manuals
Related Search