Home

ARIB STD-T64-C.S0018-E v2.0 Minimum Performance Specification

1. Label Operating Point Condition Enc Dec Connection b05 1S 96 C Clean Nominal 22 dB R R b06 EVRC A Car Noise IRS at 15 dB S N M M b07 EVRC A Car Noise IRS at 15 dB S N M T b08 EVRC A Car Noise IRS at 15 dB S N T M b09 EVRC A Car Noise IRS at 15 dB S N T T b10 15 96 Car Noise IRS at 15 dB S N R R b11 EVRC A Street Noise Flat at 12 dB S N b12 EVRC A Street Noise Flat at 12 dB S N M T Car Noise IRS at 12 dB S N b13 EVRC A Street Noise Flat at 12 dB S N T M Car Noise IRS at 12 dB S N b14 EVRC A Street Noise Flat at 12 dB S N T T Car Noise IRS at 12 dB S N b15 15 96 Street Noise Flat at 12 dB S N R R Car Noise IRS at 12 dB S N b16 EVRC A Office Noise Flat at 20 dB S N Car Noise IRS at 15 dB S N b17 EVRC A Office Noise Flat at 20 dB S N M T Car Noise IRS at 15 dB S N b18 EVRC A Office Noise Flat at 20 dB S N T M Car Noise IRS at 15 dB S N b19 EVRC A Office Noise Flat at 20 dB S N T T Car Noise IRS at 15 dB S N b20 IS 96 C Office Noise Flat at 20 dB S N R R Car Noise IRS at 15 dB S N b21 EVRC A Tandem Nominal 22 dB M M M M b22 EVRC A Tandem Nominal 22 dB M M T T b23 EVRC A Tandem Nominal 22 dB T T M M b24 EVRC A Tandem Nominal 22 dB T T T T b25 1S 96 C Tandem Nominal 22 dB R R b26 Reference MNRU 5aB b27 Reference MNRU 15aB b28 Reference MNRU 20aB b29 Reference MNR
2. 6 1 Nominal Level and Noise Processing input speech file Master Test encoder in a output m packet Master Test decoder gt src s22 c15 b20 s15 given operating point p speech file operating point 6 2 FER Processing input speech file Master Test encoder in a packet file output sre s22 c15 b20 s15 given operating point le packet file fersig Ba with Erasure Master Test decoder speech file A A A L fer 396 bin 6 3 Low level and Signaling Processing packet file input speech file packet file Go gt tae ando Master est output scaldemo 5 532 operating point signalling level decoder speech file 10 dB output file dimming 1 A A dim 196 pls bin operating dim_1 bin pkt level point source level dim file dim file 6 4 High level Processing level input speech file Master Test encoder in a output adjusted src s12 gt given operating point tp packet file Master Test decoder speech file scaldemo 10 dB gt output speech file A operating point 6 1
3. 2 28 2 2 8 1 ACR Listening Test Procedures Experiment 2 28 2 2 8 2 P 835 Listening Test Procedures Experiment ll teers 2 29 2 2 9 Analysis of Results 200ereretererrteronononononononononsosososososososooesnonononononononouonoosososososososononononoaonon 2 32 2 2 9 1 Basic Results for the SO 68 Listening tests 2 32 2 2 9 2 Minimum Subjective Requirement for SO 68 Listening Tests 2 32 2210 Expected Results for Reference Conditions nee 2 33 2 2 10 1 Experiment Reference Conditions ssssssssseeeeee 2 33 2 2 10 2 Experiment Il Reference Conditions ssssssseeeenene 2 33 2 3 Performance Testing for SO 70 entrent enne nnns nnns entrent 2 34 2 3 1 Objective Performance Testing for SO 70 eme 2 34 2 3 1 1 Average Data Rate Test t evrererererererereooososososososososooosoonsoonononononononsososososososononononononon 2 34 2 3 1 1 1 Average Data Rate Computation for SO 70 2 34 2 3 1 1 2 Average Data Rate Requirement for SO 70 2 35 2 3 1 2 Unity Gain Requirement 6steteereeorersossorosonooosoonsonsososonnooosanoooasoresonnsasoonoossnsososnnosnns 2 35 2 3 1 3 End to end Algorithmic Delay Recommendation 2 36 2 3 2 Subjective Performance Testing for S
4. 2 53 2 3 9 Randomlizatlon niea ep cipe prie UD Re EORR DR ERROR ey 2 55 24336 Presentation ee tette the etn utens tati uinea tasas i casae tin tetas aceti enin 2 57 2 3 7 1 the tee ete ee eee dre etate ea enia atate eaten e 2 57 2 3 8 Listening Test Procedures 2 57 2 3 8 1 ACR Listening Test Procedures Experiments 1 3 and 5 2 57 2 3 8 2 P 835 Listening Test Procedures Experiments 2 4 6 2 58 2 3 9 Analysis of E 2 61 2 3 9 1 Basic Results for the SO 70 Listening tests mnn 2 61 2 3 9 2 Minimum Subjective Requirement for SO 70 Listening Tests sree 2 61 2 3 10 Expected Results for Reference Conditions eee 2 62 2 3 10 1 Reference Conditions for Experiments 1 3 and 5 2 62 2 3 10 2 Reference Conditions for Experiments 2 4 and 6 2 62 2 4 Performance Testing for SO 73 enne enne nnns sitne nns 2 63 2 4 1 Objective Performance Testing for SO 73 nennen 2 63 2 4 1 1 Average Data Rate 2 63 2 4 1 1 1 Average Data Rate Computation for SO 73 2 63 2 4 1 1 2 Average Data
5. 4 3 xvii 1 2 3GPP2 50018 v2 0 Table 4 5 Description of EVRC NW2K Software Distribution Contents Table 5 1 1 Variance Source Table for the ANOVA xviii 1 20 21 22 23 24 25 26 27 28 29 30 31 32 33 3GPP2 C S0018 E v2 0 1 INTRODUCTION This standard details definitions methods of measurement verification of bit exactness and minimum performance characteristics of the EVRC A EVRC B EVRC WB EVRC NW and EVRC NW2K enhanced variable rate speech codecs for digital cellular spread spectrum mobile stations and base stations specified in 1 This standard shares the purpose of 14 and 15 This is to ensure that a mobile station can obtain service in any cellular system that meets the compatibility requirements of 16 This standard consists of this document and an associated software distribution The Software Distribution contains e Audio source material e Clear channel packets produced from the master codec e Impaired channel packets produced from the master codec and degraded by a channel model simulation e Output audio files produced from the master encoded packets decoded by the master decoder e Calibration source material e C C language source files for the compilation of bit exact fixed point codec e C C language source files for a number of software data analysis tools e Modulated Noise Reference Unit MNRU
6. 2 75 Table 2 4 2 3 6 2 SO 73 Listening Experiment 6 Test 2 76 Table 2 4 2 3 7 1 Numerical Parameters for the SO 73 Listening Experiments 2 76 Table 2 4 4 5 1 Cutting Points for the astrip Software Tool for the SO 73 Experiments 1 and 3 ACR XV 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 3GPP2 C S0018 E v2 0 Table 2 4 5 1 Example Randomization for the Experiments 1 and 3 ACR 2 81 Table 2 5 1 1 1 1 Target ADR vs Capacity Operating Point 2 89 Table 2 5 2 1 Test Suites for SO 77 eee 2 91 Table 2 5 2 2 Experiments for SO 77 compliance sss 2 91 Table 2 5 2 3 1 1 SO 77 Listening Experiment 1 Test 2 92 Table 2 5 2 3 1 2 SO 77 Listening Experiment 1 Test Conditions 2 93 Table 2 5 2 3 2 1 SO 77 Listening Experiment 2 Test 2 94 Table 2 5 2 3 2 2 SO 77 Listening Experiment 2 Test 2 94 Table 2 5 2 3 3 1 SO 77 Listening Experiment 3 Test 2 95 Table 2 5 2 3 3 2 SO 77 Listening Experiment 3 Test
7. Exp 6 Narrowband P 835 14 Babble 20dB SNR 2 FER Mode 0 LB portion of Wideband mode decoder test only M T 15 Car 20dB SNR 2 d amp b Mode 0 LB portion of Wideband mode decoder test only M M 16 Car 20dB SNR 2 d amp b Mode 0 LB portion of Wideband mode decoder test only M T f15 Car 20dB SNR 1 pls Mode 0 LB portion of Wideband mode decoder test only M M 16 Car 20dB SNR 1 pls Mode 0 LB portion of Wideband mode decoder test only M T 2 2 3 2 3 7 Subjective Experiment 7 for SO 70 The Test Parameters for Listening Experiment 7 are presented in Table 2 3 2 3 7 1 Table 2 3 2 3 7 1 SO 70 Listening Experiment 7 Test Parameters peed Audio Input Level 22 dB Filter characteristics P 341 refer Section 3 3 2 4 Reference conditions 4 Specified reference conditions Test conditions 0 FER and 3 FER Encoder Decoder Combinations 2 M M M T e The Test Conditions for Listening Experiment 7 are presented in Table 2 3 2 3 7 2 Table 2 3 2 3 7 2 SO 70 Listening Experiment 7 Test Conditions Exp 7 Wideband Music File Reference Condition g01 MNRU 15aB Reference g02 MNRU 25dB Reference g03 MNRU 35aB Reference 004 Source Reference File Test Condition Enc Dec 005 0 FER 006 0 M T 007 396 FER M M 008 3 FER M T 2 49 3GPP2 C S0018 E v2 0 2 2 3 2 3 8 Subjective Experiment 8 for SO 70 3 The Test Parameters for Listening
8. seen 2 5 Table 2 1 2 3 2 2 SO Listening Experiment II 2 5 Table 2 1 2 3 3 1 Numerical Parameters for SO 3 Listening 2 7 Table 2 1 10 1 Multipliers for Equations 2 1 10 4 and 2 1 10 5 0 eee cece esee 2 14 Table 2 2 1 1 1 1 Target ADR vs Capacity Operating Point 2 16 Table 2 2 2 3 1 1 SO 68 Listening Experiment Test 2 18 Table 2 2 2 3 1 2 SO 68 Listening Experiment Test 2 19 Table 2 2 2 3 2 1 SO 68 Listening Experiment II Test 2 20 Table 2 2 2 3 2 2 SO 68 Listening Experiment Il Test 2 21 Table 2 2 2 3 3 1 Numerical Parameters for the SO 68 Listening Experiments 2 22 Table 2 2 4 5 1 Cutting Points for the astrip Software Tool for the Experiment ACR Test 2 25 Table 2 2 4 5 2 Cutting Points for the astrip Software Tool for the Experiment II P 835 Test 2 26 Table 2 2 4 5 3 Composition of the Sentence Triad Samples for the Experiment Il P 835 Test 2 26 Table 2 2 5 1 Example Randomization for the Experiment ACR 2 27 Ta
9. 2 1 2 1 1 1 2 Average Data Rate Requirement sssssseeeeneeeneen 2 2 2 1 1 2 Unity Gain Requirement 2 2 2 1 1 8 End to end Algorithmic Delay 2 2 2 1 2 Subjective Performance Testing for SO 3 eene nnne 2 2 24 2 1 Definition et oi aia edet d 2 3 2 1 2 2 Method of eene eee eren nnnm nnne nnns 2 3 2 1 2 3 Test Conditions and Test Design for SO 3 Listening Experiments 2 3 2 1 2 3 1 Subjective Experiment for SO 3 2 3 2 1 2 3 2 Subjective Experiment Il for SO 3 2 5 2 1 2 3 3 Numerical Parameters for SO 3 Listening 2 7 2 1 8 Source Speech Material for SO 3 Testing sse 2 7 2 1 3 1 Source Speech Material for Experiment 1 2 7 2 1 3 2 Source Speech Material for Experiment ll 2 8 2 1 4 Processing of Speech Material for SO Testing 2 8 2 1 4 1 Encoding by the Test Codec 2 9 2 1 4 2 Decoding by the Master Test 4 tenentes 2 9 2 1 4 3 Introduction of 1 lt tenete nnne tnn 2 9 21 4 4 Tandem Conditions eene n
10. 3GPP2 C S0018 E v2 0 Reference output Input Packet File Operating Point Condition speech files for bit exact compliance evrc nw opl1 p12 EVRC NWZ2K High 12 dB evrc nw 0op1 012 8k operating point 1 8 kHz sampling evrc nw op1 p32 EVRC NW2kK Low 32 dB evrc_nw_op1 032 8k operating point 1 8 kHz sampling evrc nw op6 fer 396 p EVRC NWZ2K Nominal 22 dB evrc nw op6 fer 396 22 operating point 6 8 kHz FER 3 022 8k sampling evrc nw op6 p12 EVRC NWZ2K High 12 dB evrc nw op6 012 8k operating point 6 8 kHz sampling evrc nw op6 p32 EVRC NWZ2K Low 32 dB evrc nw op6 032 8k operating point 6 8 kHz sampling evrc nw op7 p22 EVRC NWZ2K Nominal 22 dB evrc nw 7 022 8 operating point 7 8 kHz sampling evrc nw opti dim 196 p EVRC NWZ2K Nominal 22 dB evrc nw opl1 dim 196 ls 196 p22 operating point 1 8 kHz 1 d amp b 1 pls pls 196 022 8k sampling evrc nw opi pc EVRC NWZ2K Nominal 22 dB evrc nw op1 oc 8k operating point 1 8 kHz 15 dB car noise sampling evrc nw op6 dim 196 p EVRC NWZ2K Nominal 22 dB evrc nw op6 dim 196 Is 196 p22 operating point 6 8 kHz 1 d amp b 1 pls pls 196 022 8k sampling evrc nw op6 pc EVRC NW2kK Nominal 22 dB evrc nw op6 oc 8k operating point 6 8 kHz 15 dB car noise sampling evrc nw 0op7 pc EVRC NWZ2K Nominal 22 dB evrc nw op7 oc 8k operating point 7 8 kHz 15 dB car noise sampling evrc nw opi ps EVRC NW2kK Nominal 22
11. Meee triad Randomization 2973697 2999007 3029505 3055744 3083777 3111278 3147009 3167927 25310 3049 2623 2803 27501 35731 2091 2541 For each of the two subjective experiments each presentation sample consists of a speech sample processed under a condition of the test For the ACR Experiment the sample consists of a pair of concatenated sentences of approximately 8 sec duration For the P 835 Experiment 11 the sample consists of three sub samples where each sub sample is a single sentence of approximately 4 sec duration The samples shall be presented to the listeners in a randomized presentation order The listeners for each file set shall be presented with practice trials for subjective Experiments and Il The randomization of the test samples has been accomplished with the following constraints for the two experiments 2 26 3GPP2 C S0018 E v2 0 1 Atrial i e a test sample for the combination of each test condition and each talker shall be presented exactly once to each listening panel i e trials panel conditions x talkers Randomization is in blocks such that one sample of each test condition is presented once with a randomly selected talker in each block This ensures that listeners rate each test condition equally often in the initial middle and final parts of the block and controls for the effects of time and order of presentation A block contains the same number of samp
12. 2 95 Table 2 5 2 3 4 1 SO 77 Listening Experiment 4 Test 2 97 Table 2 5 2 3 4 2 SO 77 Listening Experiment 4 Test Conditions 2 98 Table 2 5 2 3 5 1 SO 77 Listening Experiment 5 Test 2 99 Table 2 5 2 3 5 2 SO 77 Listening Experiment 5 Test Conditions sess 2 100 Table 2 5 2 3 6 1 SO 77 Listening Experiment 6 Test 2 100 Table 2 5 2 3 6 2 SO 77 Listening Experiment 6 Test 2 101 Table 2 5 2 3 7 1 SO 77 Listening Experiment 7 Test 2 101 Table 2 5 2 3 7 2 SO 77 Listening Experiment 7 Test 2 101 Table 2 5 2 3 8 1 SO 77 Listening Experiment 8 Test 2 102 Table 2 5 2 3 8 2 SO 77 Listening Experiment 8 Test 2 103 Table 2 5 2 3 9 1 Numerical Parameters for the SO 77 Listening Experiments 2 104 Table 2 5 4 5 1 Cutting Points for the astrip Software Tool for the SO 77 Experiments 1 and 3 ACR Table 2 5 4 5 3 Cutting Points for the astrip Software Tool for the SO 77 Experiment 7 ACR Test
13. 3 23 Table 3 3 4 5 1 Test Suites of input test vectors for SO 70 compliance 3 27 Table 3 3 4 5 2 SO 70 Encoder Suite A Bit exact Test Conditions 3 27 Table 3 3 4 5 3 SO 70 Suite A Decoder Bit exact Test Conditions 3 28 Table 3 3 4 5 4 SO 70 Encoder Suite B Bit exact Test Conditions 3 28 Table 3 3 4 5 5 SO 70 Suite B Decoder Bit exact Test Conditions 3 29 Table 3 3 4 5 6 SO 70 Encoder Suite C Bit exact Test Conditions 3 31 Table 3 3 4 5 7 SO 70 Suite C Decoder Bit exact Test 3 31 Table 3 3 4 5 8 SO 70 Encoder Suite D Bit exact Test Conditions 3 32 Table 3 3 4 5 9 SO 70 Suite D Decoder Bit exact Test 3 32 Table 3 4 3 3 1 Packet File Structure from Master Codec Channel Error Model 3 37 Table 3 4 4 5 1 Test Suites of input test vectors for SO 73 compliance 3 40 Table 3 4 4 5 2 SO 73 Encoder Suite A Bit exact Test Conditions 3 41 Table 3 4 4 5 3 SO 73 Su
14. sse 3 37 3 4 3 3 Formats for SQ 793 niei Ue to teet preti ese 3 37 344 Fixed Point Bit Exact Codec for SO 73 eee 3 38 3 4 4 1 Fixed Point Codec Program tenentes 3 38 3 4 4 2 Compiling the Fixed Point Codec Simulation tenner 3 38 3 4 4 3 Running the Fixed Point Codec Simulation ssssssseeese 3 38 3 4 4 4 3 39 3 4 4 5 Verifying Bit Exact Performance of the Fixed Point Test 3 39 3 5 Specific Standard Test Conditions for SO 77 sss 3 46 3 5 4 Audio Path and Calibration for SO 77 emen 3 46 Audio 3 46 3 54 2 ree eer ede a daye de cete ee tenia pw de aeo be pana lev ek chia caved ve 3 46 3 5 2 Software Test Tools for SO 77 3 47 3 5 2 1 Channel Model Utilities exsig29 exe ene rentent 3 47 3 5 2 2 Channel Error and Signaling Masks sssssseeeeneen 3 47 3 5 2 3 EVRC NW Interworking Function 3 47 35 24 PANT EIE en nnne Peg rentes era eie en 3 48 353 Master Codec for te uo TNR Ee T 3 48 3 5 3 1 Compiling the Master Codec Simulation nnn 3 49 3 5 3 2 Running the Master Codec Simulation sse 3 49 3 5 3 3 File Formats for SO 77 3
15. o 3 h of gt gt a06 a04 al4 17 a05 a01 a07 al0 a22 a20 a23 a02 al6 al2 a08 al a21 all a24 al9 al3 a09 al5 256 m4s4 f2s2 mis8 fls6 mis6 357 1251 254 1457 358 1255 457 1455 452 1252 fls8 m3s4 456 1354 152 1156 358 al2m1s8 m4s7 a21m4s3 a02f3s5 a01f4s4 a22m3s8 a05m4s8 18 a04m1s3 a24f1s2 a04m1s3 a05m2s6 a19f4s1 al9 F4s1 147156 a21m1s2 a03m3s4 a21m2s6 a17m3s4 a12m3s3 The randomization lists for each of the eight listening panels for each experiment are provided in so70 subjctv exp data play Ist 2 56 20 21 22 23 24 25 26 27 28 29 3GPP2 C S0018 E v2 0 2 3 6 Presentation Presentation of speech materials for the SO 70 codec listening tests shall be made with one side of high fidelity supra aural headphones with the other ear uncovered The speech material delivery system shall meet the requirements of Section 3 3 1 1 The listeners should be seated in a quiet room with an ambient noise level of 30 dBA or below 2 3 7 Listeners The listener sample is intended to represent the population of telephone users with normal hearing acuity The listeners should be naive with respect to telephony technology issues that is they should not be experts in telephone design digital voice encoding algorit
16. 122664 1164 1274 10956 10880 118850 097 NO NO 6957 112149 01539 8876 8866 116617 130938 123975 115326 12153 10445 10290 sf po sf po W 3GPP2 C S0018 E v2 0 Table 2 5 4 5 2 Cutting Points for the astrip Software Tool for the SO 77 Experiments 2 and 4 Start sample Length samples 1 57758 Sentence a 5 _ 2 02 288424 55912 2s02 3404062 59225 9 mso 3463288 51884 10 m3s02 515172 59593 2503 1041844 54821 2502 1096665 50042 3503 1156707 51558 1 1 1 misos 1388174 59172 15051494915 55981 12505 1734792 57721 P 835 Test Experiment II P 835 Start Length sample samples 60 3s10 3404618 58012 2 108 Start Length Sentence sample samples 66 2 12 3734538 55821 68 22512 3852015 60190 69 mssiip 3912115 55432 80 festa 4567054 55052 86 misis 4900324 51588 89 mzsi5 5080387 65752 90 mzsis 5146139 63251 95 3518 5508241 51792 3GPP2 C S0018 E v2 0 Table 2 5 4 5 3 Cutting Points for the astrip Software Tool for the SO 77 Experiment 7 ACR Sentence pair m m Eur ar 1 m f f f f f m 714241 51712 m Ol 101299 106700
17. 2 109 Table 2 5 4 5 4 Cutting Points for the astrip Software Tool for the SO 77 Experiment 8 P 835 Test 2 110 Table 2 5 4 5 5 Composition of the Sentence Triad Samples for the Experiments 2 and 4 P 835 Test Table 2 5 4 5 6 Composition of the Sentence Triad Samples for the Experiment 8 P 835 Test 2 111 Table 2 5 5 1 Example Randomization for the Experiments 1 3 and 7 ACR Test 2 112 Table 3 1 3 3 1 Packet File Structure From Master Codec Channel Error Model 3 6 xvi 20 21 22 23 24 25 26 27 28 29 30 31 32 33 3GPP2 C S0018 E v2 0 Table 3 1 4 6 2 1 Source and Bit exact Default Mode Test Vector 3 11 Table 3 1 4 6 2 2 Source and Bit exact Rate 1 2 Max Test Vector 3 12 Table 3 1 4 6 2 3 Source and Bit exact Full Rate Only Test Vector Files 3 12 Table 3 1 4 6 2 4 Decoder Output Test Vector Files sse 3 13 Table 3 2 3 3 1 Packet File Structure From Master Codec Channel Error Model 3 16 Table 3 2 4 5 1 SO 68 Encoder Bit exact Test Conditions sse 3 18 Table 3 2 4 5 2 SO 68 Decoder Bit exact Test Conditions sess 3 19 Table 3 3 3 3 1 Packet File Structure From Master Codec Channel Error Model
18. 2 3 5 Randomization For each of the first six subjective experiments each presentation sample consists of a speech sample processed under a condition of the test For the ACR Experiments 1 3 and 5 the sample consists of a pair of concatenated sentences of approximately 8 sec duration For the P 835 Experiments 2 4 and 6 the sample consists of three sub samples where each sub sample is a 2 55 20 21 22 23 3GPP2 50018 v2 0 single sentence of approximately 4 sec duration The samples shall be presented to the listeners in a randomized presentation order The listeners for each file set shall be presented with practice trials for subjective Experiments 1 3 and 5 and Experiments 2 4 and 6 The randomization of the test samples has been accomplished with the following constraints for each of the six experiments 1 A trial i e a test sample for the combination of each test condition and each talker shall be presented exactly once to each listening panel i e trials panel conditions x talkers Randomization is in blocks such that one sample of each test condition is presented once with a randomly selected talker in each block This ensures that listeners rate each test condition equally often in the initial middle and final parts of the block and controls for the effects of time and order of presentation A block contains the same number of samples as there are test conditions involved in t
19. 11 EVRC B 9 3 kbps Nominal 22 dB T T a12 EVRC B 9 3 kbps Nominal 22 dB T M a13 EVRC B 5 8 kbps Nominal 22 dB M M 14 EVRC B 5 8 kbps Nominal 22 dB M T a15 EVRC B 5 8 kbps Nominal 22 dB T T a16 EVRC B 5 8 kbps Nominal 22 dB T M ai EVRC B 4 8 kbps Nominal 22 dB M M 18 EVRC B 4 8 kbps Nominal 22 dB M T 19 EVRC B 4 8 kbps Nominal 22 dB T T a20 EVRC B 4 8 kbps Nominal 22 dB T M a21 EVRC B 9 3 kbps Low 32 dB 196 d amp b 196 pls M M a22 EVRC B 9 3 kbps Low 32 dB 1 d amp b 1 pls M T a23 EVRC B 9 3 kbps Low 32 dB 1 d amp b 1 pls T T a24 EVRC B 9 3 kbps Low 32 dB 1 d amp b 1 pls T M a25 EVRC B 5 8 kbps Low 32 dB 1 d amp b 1 pls M M a26 EVRC B 5 8 kbps Low 32 dB 1 d amp b 1 pls M T 2 19 2 3 3GPP2 C S0018 E v2 0 Label Operating Point Condition Encoder Decoder Combinations a27 EVRC B 5 8 kbps Low 32 dB 1 d amp b 1 pls T T a28 EVRC B 5 8 kbps Low 32 dB 196 d amp b 1 pls T M a29 EVRC B 9 3 kbps High 12 dB M M a30 EVRC B 9 3 kbps High 12 dB M T a31 EVRC B 9 3 kbps High 12 dB T T a32 EVRC B 9 3 kbps High 12 dB T M a33 EVRC B 5 8 kbps High 12 dB M M a34 EVRC B 5 8 kbps High 12 dB M T a35 EVRC B 5 8 kbps High 12 dB T T a36 EVRC B 5 8 kbps High 12 dB T M a37 EVRC B 9 3 kbps Nominal 22 dB 3 FER M M a38 EVRC B 9 3 kbps Nominal 22 dB 396 FER M T a39 EVRC B 5 8 kbps Nominal 22 dB 396 FER M M a
20. 21 al5f4s7 07 1 7 27 abin2s6 aa CES ae l 0294 20 LS pue Table 2 4 5 1 Example Randomization for the Experiments 1 and 3 ACR Test allm4s5 a20f a04f1s4 al3 a03m4s3 a22f3s3 18 a04 a23 2091252 a05n3s5 al mdsi 2092337 2182136 a17m2s5 alinis2 2011453 219 454 a06m4s8 2151253 256 242234 a13 4s6 24 a02m3s1 a2 f4s8 1251 a03f2s1 a08m3s7 156 a05 al8m3s1 a07m3s1 f4s8 fls7 a02f1s4 al7m4s1 18 156 a01m4s3 352 al5m2s8 al3f4s6 F3sl1 F352 a22m2s6 10 251 a06m2s4 al9f3s6 a20 a al f4s5 1152 f2s8 m4s6 s4 s2 s7 m2s6 s6 s7 fls5 m3s5 f4s1 a23m3s8 a03f3s4 a05m1s5 a09f1s7 olejnej gt HS of HS ofw Fh Phy S h a n o 3 h of gt gt a06 a04 al4 17 a05 a01 a07 al0 a22 a20 a23 a02 al6 al2 a08 al a21 all a24 al9 al3 a09 al5 256 m4s4 f2s2 mis8 fls6 mis6 357 1251 254 1457 358 1255 457 1455 452 1252 fls8 m3s4 456 1354 152 1156 358 al2m1s8 m4s7 a21m4s3 a02f3s5 a01f4s4 a22m3s8 a05m4s8 18 a04m1s3 a24f1s2 a04m1s3 a05m2s6 a19f4s1 al9 F4s1 147156 a21m1s2 a03m3s4 a21m2s6 a17m3s4 a12m3s3 The randomiz
21. 39 40 3GPP2 C S0018 E v2 0 2 4 1 1 2 Average Data Rate Requirement for SO 73 The total average data rate Rayg for each operating point shall not exceed the target average data rate by more than the tolerance level in Table 2 4 1 1 1 1 otherwise the test codec fails the compliance test 2 4 1 2 Unity Gain Requirement The specific EVRC NW test codec shall output speech with unity gain when compared with the input speech The unity gain measurement output active speech level input active speech level will be performed over the entire input speech database for the clean nominal level source conditions for each mode The measurement should be made using the STL 2000 tool 6 6a actlev and must not show more than 0 5 dB deviation between input and output active speech levels This procedure is fully described in 9 2 4 1 3 End to end Algorithmic Delay Recommendation The algorithmic delay for the specific EVRC NW test codec should be calculated analytically by the codec manufacturer In considering the algorithmic delay it can be assumed that all transmission channels have infinite bandwidth and that all processing elements have infinite throughput Algorithmic delay is defined as the sum of all sequential filter delays and buffering delays in the encode decode path The maximum end to end algorithmic delay should be no greater than that of the master codec For the master codecs defined in 1 the algorithmic delay is giv
22. 3GPP2 C S0018 E v2 0 Table 2 4 2 3 6 2 SO 73 Listening Experiment 6 Test Conditions Exp 6 Narrowband Music File Reference Condition f01 MNRU 10dB Reference 02 MNRU 20dB Reference f03 MNRU 30dB Reference 04 Source Reference File Test Condition Enc Dec 05 0 FER 06 096 FER M T f07 3 FER M M 08 396 FER M T 2 4 2 8 7 Numerical Parameters for the SO 73 Listening Experiments Table 2 4 2 3 7 1 describes the resultant numerology that is used for the six SO 73 listening experiments The first column is the description of the parameter and columns 2 to 7 show the numerical value for each of the parameters for the six listening experiments For each listening experiment the different Encode Decode Test conditions include various interconnections between the Master and Test Encoders and the Master and Test Decoders There are eight reference conditions in each of the experiments 1 through 4 and four reference conditions in experiments 5 and Table 2 4 2 3 7 1 Numerical Parameters for the SO 73 Listening Experiments Parameter Expi Exp2 Exp 3 Exp4 Exp5 Exp 6 Reference Conditions s e a Total Gonditons 1 ow de e e fal Takers 8 o e e wy Stimuli per Taker tgene 8 a e e e a Total Stimuli per Experiment 1536 1152 3072 2016 FEN Listening Panels Stimuli per Listening Panel NECS 2 76 2
23. Sentence m3s05 m3s06 3505 3506 m1s07 m1s08 f1s07 f1s08 m2s07 m2s08 2s07 f2s08 Start Sample Length samples 23995 30021 20723 28685 21654 254 23163 23941 21946 28998 27136 27136 au 70 76 m2s m2s f2s f2s m3s m3s mis fis n oO J S o Q A Go Co 254204 Start Sample 9 9 Length samples 2379 3252 2371 2799 2372 3284 2068 2513 2976 2325 m1s03 m1s04 f1s03 f1s04 m2s03 409345 432594 460033 480352 0209 91 voe 2324 2743 203 29857 24265 m3s07 m3s08 3507 3508 m1s09 394433 421569 447808 474817 498939 26239 27009 24122 26310 21087 w m2s m2s f2s 25 m3s B 2567937 2591323 2615297 2643664 2672385 m2s04 f2s03 f2s04 m3s03 m3s04 3s03 f3s04 mls05 mls06 f1s05 f1s06 m2s05 NP NI NE NI NI NI NI DT DT NI 335211 34474 637 888 465 3698 661 8659 587 3884 6528 88395 0849 w Oo gt Co Oo w Oo gt oo Co 3 Fh W N 0 0 29239 4 258 25286 3 o o f1s09 fls m2s09 m2s f2s09 m3s09 R95 f3s mis mis fls 636353 28571 25 25320 22289 27887 26 29 25367 25577 21843 26541 22924 27508 23930 w N w Bc m3s f3s f3s mis mis ffs fis m2s
24. i infn required Specifies the name of the input speech file or the name of the input packet file if only decoding is being performed see the d option below o outf required Specifies the name of the output speech file or the name of the output packet file if only encoding is being performed see the e option below d Instructs the simulation to perform only the decoding function The input file must contain packets of compressed data e Instructs the simulation to perform only the encoding function The output file will contain packets of compressed data If neither the d or the e option is invoked the coder performs both the encoding and decoding functions by default M max Sets the maximum allowable data rate to max where max is element of 4 3 2 1 using the codes specified in the first column of Table 3 5 3 3 1 m min Sets the minimum allowable data rate to min where min is element of 4 3 2 1 using the codes specified in the first column of Table 3 5 3 3 1 If neither the M nor m option is invoked the coder allows the data rate to vary between Rate 1 and Rate 1 8 3 5 8 8 File Formats for SO 77 Files of speech contain 2 s complement 16 bit samples with the least significant byte first The packet file contains twelve 16 bit words with the low byte ordered first followed by the high byte The first word in the packet contains the data rate while the remaining 11 words contain the encoded speech data pa
25. of oy o un 2 25 m2p5 2p5 3p5 3p5 4p5 4p5 1p6 1p6 2p6 2p6 3p6 3p6 4p6 4p6 1p7 1p7 2p7 2p7 3p7 3p7 4p7 4p7 1p8 1p8 2p8 2p8 3p8 3p8 m4p8 f4p8 m m m D Fy N N oO 1 1 Oy CO CO OY NO oo BI OD C oo A YP BP BED oO m m m m m 5017 5529 5094 4838 5478 5401 50432 5068 5632 51712 5657 4582 5248 210048 215116 5094 5376 19157 17360 57088 5 a 5 52736 19108 17616 5580 Ne E o 63232 4633 5504 5017 3GPP2 C S0018 E v2 0 Table 2 2 4 5 2 Cutting Points for the astrip Software Tool for the Experiment Il P 835 Test Experiment II P 835 ar eng ar eng ar eng isanpies sentence samole sanples sentence sameie samples 1 5 Pi 25242 33 34 3 06 68 2 12 2174965 32524 3505 57 11502 3506 m2s0 98817 27194 m1s07 m3s 2250 f1s07 8 12202 179276 30645 fIs08 5 mso 209921 21939 2 07 1 13 10 m3s02 231860 25677 mZs08 mis 22507 fis 2207489 2231208 225920 2282930 5777 2336464 2465025 2489017 8785 2371 2799 23729 067009 087732 222145 NIN o of of N OW w 21654 254 23 239 21946 289 27 so 2068 2513 2399 2976 2325 au 26924 29241 31635 33829 36729 OF D N 3 v 3 B u w 91 39443 42156 44780
26. 0 5 dB deviation between input and output active speech levels This procedure is fully described in 9 2 5 1 3 End to end Algorithmic Delay Recommendation The algorithmic delay for the specific EVRC NW2K test codec should be calculated analytically by the codec manufacturer In considering the algorithmic delay it can be assumed that all transmission channels have infinite bandwidth and that all processing elements have infinite throughput Algorithmic delay is defined as the sum of all sequential filter delays and buffering delays in the encode decode path The maximum end to end algorithmic delay should be no greater than that of the master codec For the master codecs defined in 1 the algorithmic delay is given as Delay Source Delay ms Signal Preprocessing Delay 0 0 Filterbank Analysis 0 8 LPC Analysis Look ahead 10 0 LPC Analysis Window 20 0 Highband excitation generation delay 1 5 Highband synthesis overlap and add delay 2 0 Filterbank Synthesis Delay 1 1 Total 35 4 Therefore the total algorithmic delay imposed by a SO 77 test codec should not exceed 35 4 milliseconds 2 5 2 Subjective Performance Testing for SO 77 This section outlines the subjective testing methodology of the subjective performance test The purpose of this testing is to evaluate the quality of the test codec under a variety of conditions To accomplish this suites of listening experiments have been designed to test speech codec quality
27. 22dB R 20 dB SNR babble noise segment 22dB R 15 dB SNR car noise segment 22dB R 15 dB SNR street noise segment 22dB The above files are to be processed with EVRC NW encoder at various capacity operating points defined by the active speech average channel rate shown in Table 2 4 1 1 1 1 Table 2 4 1 1 1 1 Target ADR vs Capacity Operating Point Capacity Operating Point Target Average Channel Data active speech average channel data rate Hale kbps EVRC NW RATE REDUC 000 5 641 596 EVRC NW RATE REDUC OO1 5 92 1 5 EVRC NW RATE REDUC 010 4 82 1 5 EVRC NW RATE 011 4 57 1 5 EVRC NW RATE REDUC 100 4 259 1 5 EVRC NW RATE REDUC 101 4 079 1 5 EVRC NW RATE 110 3 954 1 5 EVRC NW RATE_REDUC 111 3 29 1 5 The above table provides the maximum allowable average channel rate including full half quarter and eighth rate for the different operating points These maximum allowable average channel rates were obtained by processing the 7 wide band benchmark files for the 16 kHz case and 6 narrow band benchmark files for the 8kHz case through the master floating point software See Section 3 4 2 1 for details in using the provided software tool that can be used to aid in making this calculation 2 64 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38
28. 29 30 31 32 33 34 35 36 37 38 39 40 41 3GPP2 C S0018 E v2 0 2 5 8 2 P 835 Listening Test Procedures Experiments 2 4 and 8 Experiments 2 4 and 8 use the P 835 test methodology described in 13 The P 835 methodology is specifically designed to evaluate the quality of speech in background noise It yields a measure of Signal Quality SIG a measure of Background Quality BAK and a measure of Overall Quality OVRL In general OVRL scores are highly correlated with MOS but the OVRL score provides greater sensitivity and precision in test conditions involving background noise While the OVRL score is of most interest here the SIG and BAK scores also provide valuable diagnostic information For each trial in a P 835 test listeners are presented with three sub samples where each sub sample is a single sentence approx 4 sec duration processed through the same test condition In one of the first two sub samples listeners rate the Signal Quality on a five point rating scale with the points labeled Very natural no distortion Fairly natural little distortion Somewhat natural some distortion Fairly unnatural fairly distorted Very unnatural very distorted PC For the other of the first two sub samples listeners rate the Background Quality on a five point rating scale with the points labeled Not noticeable Fairly noticeable Noticeable but not intrusive Fairly conspicuous some
29. 111641 1170177 1269249 1367297 m w N m N N w m a ds w a w 142156 1 m 147481 50432 m 152524 158156 E Test Experiment I SS Got Go GW woj Co O Of CO o o P m n N Co Af uv P P 47104 47104 50944 54272 53248 56320 54784 ooo Of OF OF OF Ort Oa Or oS wi N rR W CO SI o BS GO DJF CO oO Ds 2 109 ACR eng Sentence ar eng samples pair Sample samples 19664 1636353 19664 19152 m2p5 5017 2 5 5529 3p5 5094 3p5 1838 1 5 5478 1 5 5401 1p6 50432 Ip 5068 2p6 5632 2p6 51712 3p6 5657 3p6 1582 1 6 5248 1 5094 1 7 5376 Ip 19157 2p7 17360 2p7 57088 3p7 5 3p7 a 1 7 5 api 5 Ip8 19408 Ip8 17616 2 8 5580 2p8 3p8 maps m D Fy Ops N N Of Nn Oy CO AQA COC NO oo ojoj oo A AP AP BR a m m m 2736 m m 63232 4633 5504 5017 m 3GPP2 C S0018 E v2 0 Table 2 5 4 5 4 Cutting Points for the astrip Software Tool for the SO 77 Experiment 8 P 835 Test Experiment II P 835 Sentence mis0 mls02 f1s0 f1s02 m2s0 m2s02 f2s0 f2s02 m3s0 m3s02 f3s0 3502 Start Sample 1 25243 49665 70737 98817 126011 152321 179276 209921 231860 257537 280483 Length samples 25242 24422 21072 28080 27194 263 2695 3064 2193 2567 2294 244 e
30. 196 FER M T a24 2kbps Nominal 22 dB 3 FER M T 1 2 5 2 3 8 Subjective Experiment 8 for SO 77 2 The Test Parameters for Listening Experiment 8 are presented in Table 2 5 2 3 8 1 Table 2 5 2 3 8 1 SO 77 Listening Experiment 8 Test Parameters Type of test Number of talkers Specified test conditions Audio Input Level 22 dB Filter characteristics MIRS P NSA P 835 Narrowband 3 males 3 females Reference conditions 10 Specified reference conditions Test conditions Car Noise 15 dB SNR Street Noise 15 dB SNR Babble Noise 20 dB SNR Car Noise 15 dB SNR 2 FER Street Noise 15 dB SNR 2 FER Babble Noise 20 dB SNR 2 FER 2 102 2 3GPP2 C S0018 E v2 0 Encoder Decoder Combinations 4 M M M T T T T M The Test Conditions for Listening Experiment 8 are presented in Table 2 5 2 3 8 2 Table 2 5 2 3 8 2 SO 77 Listening Experiment 8 Test Conditions Exp 4 Narrowband P 835 Reference Conditions File MNRU c01 MNRU 40aB SNR 40dB Reference c02 MNRU 20aB SNR 40dB Reference c03 MNRU 10aB SNR 40dB Reference c04 MNRU 0aB SNR 40dB Reference c05 MNRU 40aB SNR 20dB Reference c06 MNRU 40dB SNR 10dB Reference c07 MNRU 40aB SNR 0dB Reference c08 MNRU 30aB SNR 30dB Reference c09 MNRU 20aB SNR 20dB Reference c10 MNRU 10aB SNR 10dB Reference Test Conditions Fil
31. 20 21 22 23 24 25 26 27 3GPP2 C S0018 E v2 0 N4 number of frames encoded at Rate 1 4 Ng number of frames encoded at Rate 1 8 and N1 Ng For the 16 kHz input The total average channel data rate for the test codec is then given by Ravg 1 7 R ambient background segment 12dB R ambient background segment 32dB R ambient background segment 22dB R 20 dB SNR babble noise segment 22dB R 10 dB SNR car noise segment 22dB R 20 dB SNR car noise segment 22dB R 15 dB SNR street noise segment 22aB For the 8 kHz input The total average channel data rate for the test codec is then given by Ravg 1 6 R ambient background segment 12dB R ambient background segment 32dB R ambient background segment 22dB R 20 dB SNR babble noise segment 22dB R 15 dB SNR car noise segment 9 22dB R 15 dB SNR street noise segment 22dB The above files are to be processed with EVRC NW encoder at various capacity operating points defined by the active speech average channel rate shown in Table 2 4 1 1 1 1 Table 2 5 1 1 1 1 Target ADR vs Capacity Operating Point Capacity Operating Point Target Average Channel Data active speech average channel data rate Rata EVRC NW RATE REDUC OOO 5 641 596 EVRC NW RATE REDUC 001 5 92 1 5 EVRC NW RATE REDUC 010 4 82 1 5
32. 21 22 23 24 25 26 27 28 29 30 31 32 33 34 3GPP2 C S0018 E v2 0 For the other of the first two sub samples listeners rate the Background Quality on a five point rating scale with the points labeled Not noticeable Fairly noticeable Noticeable but not intrusive Fairly conspicuous somewhat intrusive Very conspicuous very intrusive A C For the third sub sample listeners rate the Overall quality on a five point rating scale with the points labeled Excellent Good Fair Poor Bad ou Data from 32 listeners shall be used for Experiment Il four listeners for each listening panel where each listening panel uses a different randomization Before starting the test the listeners should be given instructions for performing the subjective test An example set of instructions for the P 835 test are presented below The instructions may be modified to allow for variations in laboratory data gathering apparatus Instructions for P 835 Speech Rating Experiment In this speech rating experiment each trial will involve three sentences and you will give a rating for each sentence For the first sentence in each trial you will be asked to attend only to the speech signal and rate how natural or conversely how degraded the speech signal sounds to you You will use the rating scale shown in the figure below to register your ratings of the speech signal Your task will be to choose th
33. 8 kHz sampling evrc nw op3 p32 EVRC NW2K operating Low 32 dB evrc_nw_0p3 032 8k point 3 8 kHz sampling amp 19523 pon 3 8 ez samecing 3 Ex pis 9 5 027 8k evrc nw op3 pc EVRC NW2K operating Nominal 22 dB evrc nw op3 oc 8k point 3 8 kHz sampling 15 dB car noise evrc nw op3 ps EVRC NW2K operating Nominal 22 dB evrc nw op3 os 8k point 3 8 kHz sampling 15 dB street noise evrc nw op3 fer 2 EVRC NW2K operating Nominal 22 dB evrc nw op3 fer 296 point 3 8 kHz sampling 15 dB babble ob 8k 3 56 noise 2 FER 3GPP2 50018 v2 0 Table 3 5 4 5 4 SO 77 Encoder Suite B Bit exact Test Conditions Input File Operating Point Condition ko ee src s22 8k EVRC NWZK operating Nominal 22 dB evrc nw op1 p22 point 1 8 kHz sampling src s12 8k EVRC NW2K operating High 12 dB evrc_nw_op1 p12 point 1 8 kHz sampling src s32 8k EVRC NW2K operating Low 32 dB evrc nw op1 p32 point 1 8 kHz sampling src s22 8k EVRC NWZK operating Nominal 22 dB evrc nw opt dim 196 p22 point 1 8 kHz sampling 1 d amp b src s22 8k EVRC NW2K operating Nominal 22 dB evrc nw op6 p22 point 6 8 kHz sampling src s12 8k EVRC NW2K operating High 12 dB evrc nw op6 p12 point 6 8 kHz sampling src s32 8k EVRC NW2K operating Low 32 dB evrc nw op6 p32 point 6 8 kHz sampling src s22 8k EVRC NW2K operating Nominal 22 dB evrc nw op6 dim 196 p22 point 6 8 kHz sa
34. Audio Input Level 22 Encoder Decoder Combinations 4 M M M T T T T M 2 92 2 3GPP2 C S0018 E v2 0 The Test Conditions for Listening Experiment 1 are presented in Table 2 5 2 3 1 2 Table 2 5 2 3 1 2 SO 77 Listening Experiment 1 Test Conditions Exp 1 Wideband ACR Reference Conditions File MNRU 01 ZdB MNRU Reference 02 14dB MNRU Reference a03 21dB MNRU Reference 04 28dB MNRU Reference 05 35dB MNRU Reference a06 42dB MNRU Reference a07 49dB MNRU Reference 08 Direct Source Reference Test Conditions File Condition Enc Dec 09 Nominal level M M a10 Nominal level M T a11 Nominal level T T 12 Nominal level T M a13 Low level 196 d amp b M M 14 Low level 1 d amp b M T 15 Low level 1 d amp b T T a16 Low level 196 d amp b T M 17 High level M M a18 High level M T 19 High level T T a20 High level T M a21 196 FER 196 PLS M M a22 196 FER 196 PLS M T a23 396 FER M M a24 396 FER M T 2 93 3GPP2 C S0018 E v2 0 2 5 2 3 2 Subjective Experiment 2 for SO 77 The Test Parameters for Listening Experiment 2 are presented in Table 2 5 2 3 2 1 Table 2 5 2 3 2 1 SO 77 Listening Experiment 2 Test Parameters Number of talkers 3 males 3 females Test conditions o Car Noise 10 dB SNR o Noise 20 dB SNR 2 FER o Street Noise 15 dB SNR o Babble noise 20 dB S N Encoder Decoder Com
35. EVRC NW RATE 100 4 259 1 5 EVRC NW RATE REDUC 101 4 079 1 5 EVRC NW RATE REDUC 110 3 954 1 5 EVRC NW RATE_REDUC 111 3 291 596 The above table provides the maximum allowable average channel rate including full half quarter and eighth rate for the different operating points These maximum allowable average channel rates were obtained by processing the 7 wide band benchmark files for the 16 kHz case and 6 narrow band benchmark files for the 8kHz case through the master floating point software See Section 3 5 2 1 for details in using the provided software tool that can be used to aid in making this calculation 2 5 1 1 2 Average Data Rate Requirement for SO 77 The total average data rate Rayg for each operating point shall not exceed the target average data rate by more than the tolerance level in Table 2 5 1 1 1 1 otherwise the test codec fails the compliance test 2 89 25 26 27 28 29 30 31 32 33 34 35 36 3GPP2 50018 v2 0 2 5 1 2 Unity Gain Requirement The specific EVRC NW2K test codec shall output speech with unity gain when compared with the input speech The unity gain measurement output active speech level input active speech level will be performed over the entire input speech database for the clean nominal level source conditions for each mode The measurement should be made using the STL 2000 tool 6 6a actlev and must not show more than
36. GOOD FAIR POOR BAD 2 116 20 21 22 23 24 25 26 27 28 3GPP2 C S0018 E v2 0 2 5 9 Analysis of Results The response data from the practice blocks shall be discarded Data sets with missing responses from listeners shall not be used i e a complete set of data is required for 24 listeners four for each of six listening panels Responses from the different listening panels for the corresponding test conditions shall be treated as equivalent in the analysis 2 5 9 1 Basic Results for the SO 77 Listening tests The votes for each of the test conditions for SO 77 Experiments and II shall be averaged to produce an associated mean score M as shown in Equation 2 4 9 1 1 and a Standard Deviation SD as shown in Equation 2 4 9 1 2 where L is the number of listeners and T is the number of talkers involved in the experiment pL LxT LxT 1 lt 2 5 9 1 1 SD 2 5 9 1 2 2 5 9 2 Minimum Subjective Requirement for SO 77 Listening Tests The Terms of Reference for the MPS tests state that the mean score for each of the Test Encoder Decoder Combinations E DC should be not worse than the mean score for the Reference E DC For most of the test conditions involved in the subjective experiments there are three Test E DC s M T T M and T T which means there are three statistical tests against the Reference E DC M M The three statistical tests are n
37. Hz Figure 3 5 2 4 1 SO 77 ITU T P 311 P 341 Transmit Mask and Filter responses 3 5 3 Master Codec for SO 77 This section describes the C simulation of the speech codec specified by 1 The master codec C simulation used for verifying the performance of a non bit exact EVRC NW2K implementation shall be the floating point master C simulation included in the associated Software Distribution 1a 3 48 20 21 22 23 24 26 27 28 29 30 31 32 33 34 35 3GPP2 C S0018 E v2 0 3 5 3 1 Compiling the Master Codec Simulation The source code for floating point simulation can be compiled using the GNU G compiler and make utility A G compatible makefile has been included in the appropriate sub directory in 1a Typing make this directory will compile and link the code and create the executable file called Evrc nw Evrc nw exe on Win32 systems which will be placed in the same directory The included makefile may require some user modification for a particular hardware platform and or operating system 3 5 3 2 Running the Master Codec Simulation The EVRC NW2K floating point executable Evrc nw files use command line arguments to receive all information regarding input and output files and various parameters used during execution Executing Evrc nw with no command line arguments will display a brief description of the required and optional command line arguments The options are described below
38. evrc nw opti dim 196 p EVRC NW operating Nominal 22 dB evrc nw opl1 dim 196 ls 196 p22 point 1 8 kHz sampling 1 d amp b 1 pls pls 196 022 8k evrc nw opl1 pc EVRC NW operating Nominal 22 dB evrc nw op1 oc 8k point 1 8 kHz sampling 15 dB car noise evrc nw op6 dim 196 p EVRC NW operating Nominal 22 dB evrc nw op6 dim 196 ls 196 p22 point 6 8 kHz sampling 1 d amp b 1 pls pls 196 022 8k evrc nw op6 pc EVRC NW operating Nominal 22 dB evrc nw op6 oc 8k point 6 8 kHz sampling 15 dB car noise evrc nw op7 pc EVRC NW operating Nominal 22 dB evrc nw op7 oc 8k point 7 8 kHz sampling 15 dB car noise evrc nw opi ps EVRC NW operating Nominal 22 aB evrc nw op1 os 8k point 1 8 kHz sampling 15 dB street noise evrc nw opi fer 296 p EVRC NW operating Nominal 22 dB evrc nw opi fer 296 b point 1 8 kHz sampling 15 dB babble noise ob 8k evrc_nw_op6 ps EVRC NW operating Nominal 22 dB evrc nw op6 os 8k point 6 8 kHz sampling 15 dB street noise evrc nw op6 fer 296 p EVRC NW operating Nominal 22 dB evrc nw op6 fer 296 b point 6 8 kHz sampling 3 45 15 dB babble noise ob 8k 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 3GPP2 C S0018 E v2 0 3 5 Specific Standard Test Conditions for SO 77 3 5 1 Audio Path and Calibration for SO 77 3 5 1 1 Audio Path The audio path for wideband test conditions Experiments 1 and 2 must meet the following
39. evrc_wb_op0 pls_1 pc EVRC WB Nominal 22 dB evrc_wb_op0 pls_1 0c 8k operating point 0 15 dB car noise 8 kHz sampling 1 pls evrc_wb_op0 ps EVRC WB Nominal 22 dB evrc_wb_op0 os 8k operating point 0 8 kHz sampling 15 dB street noise evrc wb opO0 fer 296 pb EVRC WB operating point 0 8 kHz sampling Nominal 22 dB 20 dB babble noise 296 FER evrc wb opO fer 296 0b 8k evrc wb 3 EVRC WB operating point 0 8 kHz samplin 3 31 Generic audio signal fer 396 evrc wb opO fer 3 8 3GPP2 C S0018 E v2 0 Table 3 3 4 5 8 SO 70 Encoder Suite D Bit exact Test Conditions Input File Operating Point Condition E src s22 8k EVRC WB operating point 4 Nominal 22 dB evrc wb op4 p22 8 kHz sampling src s12 8k EVRC WB operating point 4 High 12 dB evrc wb op4 pi2 8 kHz sampling src s32 8k EVRC WB operating point 4 Low 32 dB evrc wb op4 p32 8 kHz sampling src s22 8k EVRC WB operating point 4 Nominal 22 dB evrc wb op4 dim 196 p22 8 kHz sampling 196 d amp b src s22 8k EVRC WB operating point 7 Nominal 22 dB evrc wb op7 p22 8 kHz sampling Src c15 8k EVRC WB operating point 4 Nominal 22 dB evrc wb op4 pc 8 kHz sampling 15 dB car noise src s15 8k EVRC WB operating point 4 Nominal 22 dB evrc wb op4 ps 8 kHz sampling 15 dB street noise src b20 8k EVRC WB operating point 4 Nominal 22 dB evrc wb op4 pb 8
40. rate to half rate Details of these operations are given in Section 6 The signaling file and other utilities are provided in so73 tools directory 2 4 44 Ensuring Proper Encoded Frame Packet Files All encoded frame packet files shall be examined to ensure that the files only contain data in those file locations where data should exist for a given data rate The examination of the encoded frame packet files should indicate the occurrence of any improper data in the files but the examination must not alter the encoded frame packet files in any way 2 4 4 5 Post processing of test condition output files In order to build the play sets to be presented to the listening panels the output files for the various test conditions must be processed to provide the appropriate listening conditions In addition the concatenated output files must be partitioned into the samples representing the combination of test condition and talker The listening conditions for Narrowband experiments are provided by filtering the output files using the STL software tool filter with the MIRS receive filter mask The listening conditions for Wideband experiments are provided by mixing STL tool oper the output files with Psophometrically filtered noise STL tool filter PSO filter mask at 74dBov STL tool astrip is also used to split the concatenated files into the individual samples appropriate for the experiment Table 2 4 4 5 1 shows the cutting points to be used wi
41. requirements for electro acoustic performance measured between the output of the D A converter and the output of the headphone 1 Frequency response shall be flat to within 2 dB between 50 Hz and 7000 Hz and below 50 Hz the response shall roll off at a minimum of 12 dB per octave Equalization may be used in the audio path to achieve this A suitable reconstruction filter shall be used for playback 2 Total harmonic distortion shall be less than 1 for signals between 50 Hz and 8000 Hz 3 Noise over the audio path shall be less than 35 dBA measured at the ear reference plane of the headphone 4 Signal shall be delivered to the headphone on the listener s preferred telephone listening ear and the other ear shall be uncovered No signal shall be delivered to the other headphone The audio path for narrowband test conditions Experiments 3 4 7 and 8 must meet the following requirements for electro acoustic performance measured between the output of the D A converter and the output of the headphone 1 Frequency response shall be flat to within 2 dB between 200 Hz and 3400 Hz and below 200 Hz the response shall roll off at a minimum of 12 dB per octave Equalization may be used in the audio path to achieve this A suitable reconstruction filter shall be used for playback 2 Total harmonic distortion shall be less than 1 for signals between 100 Hz and 4000 Hz 3 Noise over the audio path shall be less than 35 dBA measured at the e
42. the test codec in response to the average rate benchmark files referred to in Section 2 1 1 1 and located in the so3 objctv directory of the associated Software Distribution The program is invoked as follows avg rate filename 1 filename 2 filename 3 filename n 3 1 2 2 Scaling speech files sv56 c This program is used to scale each sample in a linearly quantized speech file by a factor that renders the file s root mean square RMS level equal to a user specified value The program is intended to be used on the test codec s speech output files to ensure that their RMS level is consistent with the requirements of Section 2 1 2 3 of this document The source code sv56 c is available from 6 and 6a The inputs to the program are the optional desired RMS value in dB the input speech file name and the optional output speech file name The outputs are the initial prior to scaling maximum sample RMS and average DC values in the speech file the final after scaling maximum RMS and DC values in the output file the number of samples that were clipped the scale factor applied and an output speech file appropriately scaled If no target RMS value is specified the program calculates and prints the initial statistics mentioned above and copies the input file to the output file unmodified The program is invoked as follows sv56 Desired RMS Level File In File Out Sample Rate Resolution Note The desired level specifi
43. 1 Channel Model Utilities fersig27 exe This utility program provides a the ability to introduce Frame Erasure channel impairment b the ability to verify use of half rate or lesser frame rate during dim and burst and packet level signaling c the ability to measure the Average Data Rate from an encoded packet file A log output of ersig27 provides detail on the ADR performance of the preceding encoder In these applications the utility is invoked as in following examples for 3 FER and 1 signaling fersig27 c EVRC B e fer 3 bin infile outfile fersig27 c EVRC B s dim 15 e fer 3 bin infile outfile 3 2 22 Channel Error and Signaling Masks These binary Frame Error Rate and Signaling masks source level and packet level 1 byte of either 0 or 1 per frame are used with the fersig27 channel impairment and inter working simulation functions for the various conditions fer 3 bin dim 1 bin 9 The GNU C compiler G and software development tools including documentation are available without charge from the Free Software Foundation They can be contacted at Free Software Foundation Voice 1 617 542 5942 59 Temple Place Suite 330 Fax 1 617 542 2652 Boston MA 02111 1307 USA gnu gnu org or on the World Wide Web at http www fsf org 3 14 20 21 22 23 24 25 26 28 29 30 31 32 33 34 35 36 38 3GPP2 C S0018 E v2 0 dim 1 pls bin 3 2 2 3 EVRC B
44. 1 Variance Source Table for the ANOVA Source Degrees of Freedom df Sum of Squares SoS Mean Square MS F Ratio X E DC E DC s d 1 SoS SIX X a Ms SoS df MS MS Subjects Subjects s 21 505 y dx o gt x4 Residual df df dfs SoS SoS SoS SoS MS SoS df TER 2 Total c x s 1 SoS Y Y x 5 2 Stage 2 Dunnett s Multiple Means Test Test CC s vs the Reference CC In Stage 2 of the Dunnett s Test the Mean score for each of the Test E DC s X is compared statistically to the Mean for the reference codec X ref as shown in Equation 5 2 1 The value for the Standard Error of the Mean Difference SEyp is computed using the estimate of Mean Square Error MSg derived from the Stage 1 ANOVA The equation for computing SEyp is shown in Equation 5 2 2 where MS is the Residual Mean Square from the ANOVA MS in Table 5 1 1 p X a SE yp J 2x MS subjects 5 2 1 5 2 2 For each Test CC the computed value of Dc is compared to critical values of the Dunnett s statistic where the parameters are criterion probability lt 05 total number of CC s 4 degrees of freedom for the df 93 Dunnett 2 09 5 2 3GPP2 50018 v2 0 6 PROCESSING BLOCKS FOR SO 68 SO 70 SO 73 AND SO 77
45. 47481 49893 63635 f2s08 m3s07 m3s08 3507 3508 71509 1509 686017 27 262 27009 24 21087 91 w 60 W co o Bo lt gt TS WO DD CO f w 25 um fis m2s f2s f2s m3s 3 3 WI N 0 as cof cof sf as f3s 2567937 2591323 5297 2643664 267 269 2727169 N Oyo w I wo N NI W o o e Coy 2589 2338 23974 2836 28721 26883 27901 19206 w N n OVD J MY lt A UW 3s03 f2s03 63713 2s04 88817 oo 711129 736449 758738 786625 812788 fls m2s09 m2s f2s09 f2s j e 25320 22289 27887 63 29133 A f3s fis Oo 2746375 2876673 2899795 2926081 2946101 26362 23122 26286 20020 27596 3s04 17 13 16 18 9 24 2 2 5 841921 867288 892865 914708 2050049 2072973 2100481 2124411 m3s09 f3s OY N 91 gt w D w Of WL Of Of Cf amp J Of GL DO oO Mm 88 3 ee eu w 0 NJ 25367 25571 22924 NI ho NI oy w or On oo apwo jin ooo Fy e ok of oo CO CO CO oo a 60 N W of Cp Ss Gof WO CO 1 OD OT ST Sui Ke m2s f2s m3s mima Www ujuja Oy UO ON KSI
46. 49 354 Fixed Point Bit Exact Codec for SO 77 eee enn 3 50 3 5 4 1 Fixed Point Codec Program Files rennes 3 50 3 5 4 2 Compiling the Fixed Point Codec Simulation m 3 50 3 5 4 3 Running the Fixed Point Codec Simulation sees 3 50 3 5 4 4 3 51 3 5 45 Verifying Bit Exact Performance of the Fixed Point Test 3 51 4 CONTENTS OF SOFTWARE DISTRIBUTION essent entere 4 1 5b DUNNET TIS FEST 5 1 5 1 Stage 1 Analysis of 1 enne enne nnns enne 5 1 xi 1 3GPP2 50018 v2 0 5 2 Stage 2 Dunnett s Multiple Means Test Test CC s vs the Reference 5 2 6 Processing BLOCKs FOR SO 68 SO 70 SO 73 and SO 77 6 1 6 1 Nominal Level and Noise 6 1 6 2 FER Processing ect ted dtu ge diat tudo d eia tue 6 1 6 3 Low level and Signaling Processing essen 6 1 6 4 High level Processing de kn ret tea tee E Ree 6 1 xii 20 21 22 23 24 25 26 27 28 29 30 3GPP2 C S0018 E v2 0 LIST OF FIGURES Figure 125 4 Test Model ettet iet tuit 1 8 Figure 2 1 8 1 Instructions for Listeners cessionis iie annin 2 12 Figur
47. 5 3 3 1 Unused bits are set to 0 For example in a Rate 1 8 frame the packet file will contain the word 0x0100 byte swapped 0x0001 followed by one 16 bit word containing the 16 data bits for the frame in byte swapped form followed by ten 16 bit words containing all zero bits 3 5 4 5 Verifying Bit Exact Performance of the Fixed Point Test Codec This section outlines the methodology of verifying whether a Fixed point Test codec is bit exact to the Fixed point reference software The purpose of this testing is to evaluate the bit exactness of the test codec under a variety of conditions which may occur To accomplish this suites of test vectors have been designed to test for bit exactness of the Test Codec under a variety of conditions depending on a number of parameters These conditions include channel impairments audio background noise and different input levels Figure 3 5 4 5 1 illustrates a decision tree to arrive at the suite of test vectors that are needed to demonstrate Minimum Performance Spec compliance through bit exactness of a Test implementation of SO 77 for different profiles of equipments that support SO 77 3 51 3GPP2 C S0018 E v2 0 Is 16kHz Sampling Rate Supported Yes No Run Test Suite A Run Test Suite B Figure 3 5 4 5 1 SO 77 Fixed point bit exact test suite decision flowchart An implementation may support SO 77 only for 8 kHz sample rate input output for example a Base station transcode
48. B 5 8 kbps Car Noise 15 dB M M b14 EVRC B 5 8 kbps Car Noise 15 dB M T b15 EVRC B 5 8 kbps Car Noise 15 dB T T b16 EVRC B 5 8 kbps Car Noise 15 dB T M b17 EVRC B 4 8 kbps Car Noise 15 dB M M b18 EVRC B 4 8 kbps Car Noise 15 dB M T b19 EVRC B 4 8 kbps Car Noise 15 dB T T b20 EVRC B 4 8 kbps Car Noise 15 dB T M b21 EVRC B 9 3 kbps Street Noise 15 dB M M b22 EVRC B 9 3 kbps Street Noise 15 dB M T b23 EVRC B 9 3 kbps Street Noise 15 dB T T b24 EVRC B 9 3 kbps Street Noise 15 dB T M b25 EVRC B 5 8 kbps Street Noise 15 dB M M b26 EVRC B 5 8 kbps Street Noise 15 dB M T b27 EVRC B 5 8 kbps Street Noise 15 dB T T b28 EVRC B 5 8 kbps Street Noise 15 dB T M b29 EVRC B 9 3 kbps Office Noise 20 dB M M b30 EVRC B 9 3 kbps Office Noise 20 dB M T b31 EVRC B 9 3 kbps Office Noise 20 dB T T b32 EVRC B 9 3 kbps Office Noise 20 dB T M b33 EVRC B 5 8 kbps Office Noise 20 dB M M b34 EVRC B 5 8 kbps Office Noise 20 dB M T b35 EVRC B 5 8 kbps Office Noise 20 dB T T b36 EVRC B 5 8 kbps Office Noise 20 dB T M 2 2 2 8 8 Numerical Parameters for the SO 68 Listening Experiments Table 2 2 2 3 3 1 describes the resultant numerology that is used for the two SO 68 listening experiments The first column is a variable name given to each of the parameters the second column is the description of the parameter the third column shows the required calculation for determining th
49. Bit Exact Codec for SO 3 3 7 3 1 4 1 Fixed Point Codec Program Files tnnt nna 3 7 3 1 4 2 Compiling the Fixed Point Codec Simulation 3 7 3 1 4 3 Running the Fixed Point Codec Simulation n 3 8 3444 Eile Formalism e aaa aea 3 9 3 1 4 5 Verifying Proper Operation of the Fixed Point Codec 3 9 3 1 4 6 Verifying Bit Exact Performance of the Fixed Point Test Codec 3 9 3 1 4 6 1 Description of Bit Exact Source Files sse 3 10 3 1 4 6 2 Instructions for Processing Bit Exact Test Vectors 3 11 3 2 Specific Standard Test Conditions for SO 68 3 13 3 24 Audio Path and Calibration for SO 68 emen 3 13 3 21 Audio 3 13 324 2 Calibtatjona scieti e EE 3 13 3 2 Standard Software Test Tools for SO 68 eee eene 3 14 3 2 2 1 Channel Model Utilities exsig27 exe seem Hm 3 14 3 2 2 2 Channel Error and Signaling Masks ssssssseeeeenenes 3 14 3 2 2 3 EVRC B Interworking Function IWF 3 15 3 23 Master Codec for SO 68 esee eene 3 15 3 2 3 1 Compiling the Master Codec Simulation nnn 3 15 3 2 8 2 Running the Master Codec Simulation 3 15 3 2 8 3 File Formats for SO 68dan praeesse entente nnns tenens 3 16 3 24 Fixed Point Bit Exact Codec
50. E DC should be not worse than the mean score for the Reference E DC For most of the test conditions involved in the subjective experiments there are three Test E DC s M T T M and T T which means there are three statistical tests against the Reference E DC M M The three statistical tests are not independent however Since they all involve the same ratings for the Reference E DC t tests are not appropriate The appropriate statistical test for multiple Test conditions against a common Reference condition is Dunnett s Test A complete description of Dunnett s Test is contained in Appendix B The critical value for the Dunnett s testis 2 09 one sided test p 05 4 E DC s df 93 For those test conditions where a single Test E DC T T is compared against the Reference E DC the appropriate statistical test is Student s t test The critical value for the Student s t test is 1 70 one sided test lt 05 df 31 In both the Dunnett s Test and the t test the MPS test is evaluated by dividing the difference between the mean score for the Test E DC and the mean score for the Reference ED C by the Standard Error of the Mean Difference SEyp as shown in Equation 2 2 9 2 1 If the resultant Test value is less than 4 The appropriate t test is a matched groups t test and the is based on the differences between individual listener s average ratings where the average is over talkers Therefore the SEwp is based o
51. Mode 1 interoperable with Mode 0 of SO 68 support T M di5 Car 15dB SNR Mode 7 interoperable with Mode 0 of SO 68 support T T di6 Car 15dB SNR Mode 7 interoperable with Mode 0 of SO 68 support T M di7 Car 15dB SNR Mode 7 interoperable with Mode 0 of SO 68 support M M 918 Car 15dB SNR Mode 7 interoperable with Mode 0 of SO 68 support M T d19 Street 15dB SNR Mode 0 LB portion of Wideband mode decoder test M M only d20 Street 15dB SNR Mode 0 LB portion of Wideband mode decoder test M T only d21 Street 15dB SNR Mode 1 interoperable with Mode 0 of SO 68 support M M d22 Street 15dB SNR Mode 1 interoperable with Mode 0 of SO 68 support M T d23 Street 15dB SNR Mode 1 interoperable with Mode 0 of SO 68 support T T 2 98 1 2 3GPP2 C S0018 E v2 0 Exp 4 Narrowband P 835 d24 Street 15dB SNR Mode 1 interoperable with Mode 0 of SO 68 support T M d25 Babble 20dB SNR 2 FER Mode 0 LB portion of Wideband mode decoder test M M only d26 Babble 20dB SNR 2 FER Mode 0 LB portion of Wideband mode decoder test M T only d27 Babble 20dB SNR 2 FER Mode 1 interoperable with Mode 0 of SO 68 support M M d28 Babble 20dB SNR 296 FER Mode 1 interoperable with Mode 0 of SO 68 support M T d29 Babble 20dB SNR 2 FER Mode 1 interoperable with Mode 0 of SO 68 support T T d30 Babble 20dB SNR 2 FER Mode 1 interoperable with Mode 0 of SO
52. Reference e07 B5dB MNRU Reference e08 Direct Source Reference Test Conditions File Condition Enc Dec e09 Nominal Mode 0 LB portion of Wideband mode decoder test only M M e10 Nominal Mode 0 LB portion of Wideband mode decoder test only M T e11 Low Mode 0 LB portion of Wideband mode decoder test only M M e12 Low Mode 0 LB portion of Wideband mode decoder test only M T e13 High Mode 0 LB portion of Wideband mode decoder test only M M e14 High Mode 0 LB portion of Wideband mode decoder test only M T e15 Nominal Mode 0 1 D amp BS LB portion of Wideband mode decoder test only M M e16 Nominal Mode 0 1 D amp BS LB portion of Wideband mode decoder test only M T e17 Nominal Mode 0 10 D amp BS interoperable with Mode 0 of SO 68 support M M e18 Nominal Mode 0 10 D BS interoperable with Mode 0 of SO 68 support M T e19 FER 296 Mode 0 196 D amp BS LB portion of Wideband mode decoder test only M M e20 FER 296 Mode 0 196 D amp BS LB portion of Wideband mode decoder test only M T e21 FER 6 Mode 0 10 D amp BS interoperable with Mode 0 of SO 68 support M M e22 FER 6 Mode 0 10 DYoBS interoperable with Mode 0 of SO 68 support M T e23 Nominal Mode 0 196 PLS LB portion of Wideband mode decoder test only M M 24 Nominal Mode 0 1 PLS LB portion of Wideband mode decoder test only M T 2 3 2 8 Subjective Experiment 6 for SO 70 The Test Parameters for Listening Experiment 6 are presented in Table 2 3 2
53. Test Conditions Reference output Input Packet File Operating Point Condition speech files for bit exact compliance evrc nw opO fer 396 p2 EVRC NW operating Nominal 22 dB evrc nw opO fer 3 2 point 0 16 kHz sampling 396 FER 022 evrc nw opO fer 196 pls EVRC NW Nominal 22 dB evrc nw 196 p22 operating point 0 1 FER 1 pls fer 196 pls 196 022 16 kHz sampling evrc nw opO0 p12 EVRC NW operating High 12 dB evrc nw op0 o12 point 0 16 kHz sampling evrc nw opO dim 196 p EVRC NW operating Low 32 dB evrc_nw_op0 dim_1 32 point 0 16 kHz sampling 1 d amp B 032 evrc_nw_op0 pc1 EVRC NW operating Nominal 22 dB evrc nw opO oc1 point 0 16 kHz sampling 10 dB car noise evrc nw fer 396 p EVRC NW Nominal 22 dB evrc nw opO fer 3 c2 operating point 0 20 dB car noise oc2 16 kHz sampling fer 396 evrc nw opO ps EVRC NW operating Nominal 22 dB evrc nw opO os point 0 16 kHz sampling 15 dB street noise evrc nw opoO pb EVRC NW operating Nominal 22 dB evrc nw op0 ob point 0 16 kHz sampling 20 dB babble noise evrc nw opO fer 3 EVRC NW operating point 0 16 kHz sampling Generic audio evrc nw opO fer 3 signal fer 396 om evrc nw opt fer 396 p2 EVRC NW operating Nominal 22 dB evrc nw op1 fer 396 2 point 1 8 kHz sampling FER 3 022 8k evrc nw opi pi2 EVRC NW operating High 12 aB evrc nw opt o12 8k point 1 8 kHz sampling e
54. Wideband mode decoder test M T only d27 Babble 20dB SNR 2 FER Mode 1 interoperable with Mode 0 of SO 68 support M M d28 Babble 20dB SNR 2 FER Mode 1 interoperable with Mode 0 of SO 68 support M T d29 Babble 20dB SNR 2 FER Mode 1 interoperable with Mode 0 of SO 68 support TT d30 Babble 20dB SNR 296 FER Mode 1 interoperable with Mode 0 of SO 68 support T M d31 Car 15dB SNR Mode 6 interoperable with Mode 6 of SO 68 support M M d32 Car 15dB SNR Mode 6 interoperable with Mode 6 of SO 68 support M T d33 Car 15dB SNR Mode 6 interoperable with Mode 6 of SO 68 support TT d34 Car 15dB SNR Mode 6 interoperable with Mode 6 of SO 68 support T M d35 Street 15dB SNR Mode 6 interoperable with Mode 6 of SO 68 support M M d36 Street 15dB SNR Mode 6 interoperable with Mode 6 of SO 68 support M T d37 Street 15dB SNR Mode 6 interoperable with Mode 6 of SO 68 support TT d38 Street 15dB SNR Mode 6 interoperable with Mode 6 of SO 68 support T M d39 Babble 20dB SNR 2 FER Mode 6 interoperable with Mode 6 of SO 68 support M M d40 Babble 20dB SNR 296 FER Mode 6 interoperable with Mode 6 of SO 68 support M T d41 Babble 20dB SNR 2 FER Mode 6 interoperable with Mode 6 of SO 68 support TT d42 Babble 20dB SNR 296 FER Mode 6 interoperable with Mode 6 of SO 68 support T M 2 4 2 3 5 Subjective Experiment 5 for SO 73 The Test Parameters for Listening Experiment 5 are presented in Table 2 4 2 3 5 1 Table 2
55. a small shift in the means in either direction or a slight stretching or compression of the scale near the extremes may imply a problem in the execution of the evaluation test In particular MOS should be monotonic with MNRU within the limits of statistical resolution and the contour of the relation should show a similar slope MOS 6 10 20 30 40 50 Figure 2 3 10 1 1 Typical Plot of MOS versus MNRU 2 3 10 2 Reference Conditions for Experiments 2 4 and 6 Reference conditions for P 835 tests are constructed as a combination of SNR and MNRU processing to provide degradation in overall speech quality in two dimensions signal distortion and background noise intrusiveness Table 2 3 2 3 2 2 shows the eight reference conditions 601 608 involved in the 2 62 20 21 22 23 24 25 26 27 28 29 30 31 32 3GPP2 C S0018 E v2 0 P 835 Experiments 2 4 and 6 In general results are expected for these reference conditions such that the obtained score profiles are similar to those shown in Figure 2 3 10 2 1 SNR 40dB Car Noise MNRU 40dB MNRU SNR E a a SG SG SG B BAK _ OVRL _ BAK amp OVAL 835 Scores w P 835 Scores w P 835 Scores w OVRL 10 10 dB 20 20 dB 30 30 dB 40 40 dB SNR Car Noise oaB 20aB 40aB
56. and then enter the corresponding number on your keyboard Attending ONLY to the BACKGROUND select the category which best describes the sample you just heard the BACKGROUND in this sample was 5 NOT NOTICEABLE SOMEWHAT NOTICEABLE NOTICEABLE BUT NOT INTRUSIVE FAIRLY CONSPICUOUS SOMEWHAT INTRUSIV VERY CONSPICUOUS VERY INTRUSIV For the third and final sentence in each trial you will be asked to attend to the entire sample both the speech signal and the background and rate your opinion of the sample for purposes of everyday speech communication Select the category which best describes the sample you just heard for purposes of everyday speech communication the OVERALL SPEECH SAMPLE was EXCELLENT GOOD FAIR POOR BAD 2 60 20 21 22 23 24 25 26 27 28 3GPP2 C S0018 E v2 0 2 3 9 Analysis of Results The response data from the practice blocks shall be discarded Data sets with missing responses from listeners shall not be used i e a complete set of data is required for 32 listeners four for each of eight listening panels Responses from the different listening panels for the corresponding test conditions shall be treated as equivalent in the analysis 2 3 9 1 Basic Results for the SO 70 Listening tests The votes for each of the test conditions for SO 70 Experiments 1 and 3 and Experiments 2 a
57. be used for Experiments 1 3 and 7 four listeners for each listening panel 1 where each listening panel uses a different randomization Before starting the test the listeners 12 should be given instructions for performing the subjective test An example set of instructions for the 18 ACR are presented in Figure 2 5 8 1 1 The instructions may be modified to allow for variations in 14 laboratory data gathering apparatus This is an experiment to determine the perceived quality of speech over the telephone You will be listening to a number of recorded speech samples spoken by several different talkers and you will be rating how good you think they sound Use the single headphone on the ear you normally use for the telephone On each trial a two sentence sample will be played After you have listened to the sample determine the category from the list below which best describes the overall quality of the sample Press the numeric key on your keyboard corresponding to your rating for how good or bad that particular passage sounded The quality of the speech should be rated according to the scale below 5 Excellent Good 4 3 Fair 2 Poor 1 Bad During the session you will hear samples varying in different aspects of quality Please take into account your total impression of each sample rather than concentrating on any particular aspect 16 Figure 2 5 8 1 1 Instructions for Listeners 2 113 21 22 23 24 25 26 27
58. contains a 22 dB 1004 Hz reference signal The audio circuit shall be calibrated so that the test signal has a level of 16 dBPa at the ear reference plane while maintaining compliance with Section 3 1 1 1 3 1 2 Standard Software Test Tools for SO 3 This section describes a set of software tools useful for performing the tests specified in Section 2 1 Where possible code is written in C code 19 and has been developed and compiled using the GNU GCC C language compiler and software maintenance utilities The tools have been verified under various representative operating systems on a number of different hardware platforms The 3GPP2 supplied tools are all located in the so3 tools directory in the associated Software Distribution and 8 The GNU C compiler GCC and software development tools including documentation are available without charge from the Free Software Foundation They can be contacted at Free Software Foundation Voice 1 617 542 5942 59 Temple Place Suite 330 Fax 1 617 542 2652 Boston MA 02111 1307 USA gnu gnu org or on the World Wide Web at http www fsf org 3 3 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 3GPP2 50018 v2 0 can be built using the GNU make utility using static libraries and no special optimizations by copying the contents of the so3 tools directory to a new directory on a writeable disk and typing m
59. evrc wb opoO pi2 16 kHz sampling src s32 EVRC WB operating point 0 Low 32 dB 196 d amp b evrc wb opO dim 196 p32 16 kHz sampling src c10 EVRC WB operating point 0 Nominal 22 dB evrc wb opO pc1 16 kHz sampling 10 dB car noise src c20 EVRC WB operating point 0 Nominal 22 dB evrc wb op0 pc2 16 kHz sampling 20 dB car noise src s15 EVRC WB operating point 0 Nominal 22 dB evrc wb opO ps 16 kHz sampling 15 dB street noise src b20 EVRC WB operating point 0 Nominal 22 dB evrc wb opo pb 16 kHz samplin 20 dB babble noise 3 27 3GPP2 C S0018 E v2 0 Table 3 3 4 5 3 SO 70 Suite A Decoder Bit exact Test Conditions operating point 0 16 kHz sampling 10 dB car noise Reference output Input Packet File Operating Point Condition speech files for bit exact compliance evrc wb opO0 fer 396 p22 EVRC WB Nominal 22 dB evrc wb opO fer 396 022 operating point 0 3 FER 16 kHz sampling evrc wb opO fer 196 pls 196 p22 EVRC WB Nominal 22 dB evrc wb opo operating point 0 3 FER fer 196 pls 196 022 16 kHz sampling evrc wb p12 EVRC WB High 12 dB evrc wb op0 012 operating point 0 16 kHz sampling evrc wb opO dim 196 p32 EVRC WB Low 32 dB evrc wb op0 dim 196 032 operating point 0 1 d amp B 16 kHz sampling evrc_wb_op0 pc1 EVRC WB Nominal 22 dB evrc_wb_op0 oc1 evrc wb fer 3Yo pc2 EVRC WB Nominal 22 dB evrc wb opO fer 396 0c2 operating point 0 20 dB car
60. execution Executing EvrcB fx with no command line arguments will display a brief description of the required and optional command line arguments The options are described below i infn required Specifies the name of the input speech file or the name of the input packet file if only decoding is being performed see the d option below o outf required Specifies the name of the output speech file or the name of the output packet file if only encoding is being performed see the e option below d Instructs the simulation to perform only the decoding function The input file must contain packets of compressed data e Instructs the simulation to perform only the encoding function The output file will contain packets of compressed data If neither the d or the e option is invoked the coder performs both the encoding and decoding functions by default M max Sets the maximum allowable data rate to max where max is element of 4 3 2 1 using the codes specified in the first column of Table 3 2 3 3 1 m min Sets the minimum allowable data rate to min where min is element of 4 3 2 1 using the codes specified in the first column of Table 3 2 3 3 1 If neither the M nor m option is invoked the coder allows the data rate to vary between Rate 1 and Rate 1 8 In addition if max min the data rate varies between max and min using the same rate decision algorithm where the data rate is set to max if the selected dat
61. for Experiments 1 3 5 and 7 2 118 2 5 10 2 Reference Conditions for Experiments 2 4 and 8 2 118 3 Codec Standard Test Conditions nnne nes 3 1 3 1 Specific Standard Test Conditions for SO 3 sse 3 3 3 1 1 Audio Path and Calibration for SO 3 ee eem een 3 3 20 21 22 23 24 25 26 27 28 29 30 31 32 33 3GPP2 C S0018 E v2 0 aiti VIERTEL OL body cy ak a da kipa l DEAR A da DATE AY RAY po 3 3 2 1 AD GallbratiOriz oe ie a k m eie kk ee kn ok an a m va med bre taped deed 3 3 3 1 2 Standard Software Test Tools for SO 3 seen 3 3 3 1 2 1 Average Data Rate Determination Utility _ 3 4 3 1 2 2 Scaling speech files SV56 C nennen nennen tntntn tatnen 3 4 3 4 23 Law Companding mu Q tttetesesesessssososososososoooooooonononononononononsososososososononononononon 3 5 3353 Master Godec for SO e reir e reri kd ranp Ei dde 3 5 3 1 3 1 Compiling the Master Codec Simulation nnne 3 5 3 1 3 2 Running the Master Codec Simulation 3 5 3 4 3 3 File BONM ALE et aaa ya loa ava ka aa tk aaa 3 6 3 1 3 4 Verifying Proper Operation of the Master Codec AMA 3 6 3 1 4 Fixed Point
62. for Listening Experiment 1 are presented in Table 2 4 2 3 1 1 Table 2 4 2 3 1 1 SO 73 Listening Experiment 1 Test Parameters Type of test MOS P 800 Wideband Number of talkers 4 males 4 females Background noise none ambient Audio Input Level 22 dB 32 dB 12 dB Filter characteristics P 341 refer Section 3 3 2 4 Reference conditions 8 Specified reference conditions Test conditions Low Audio Input Level 32 dB 196 d amp b Nominal Audio Input Level 22 dB High Audio Input Level 12 dB 3 FER and 1 FER 2 pls at Nominal Audio Input Level 22 Encoder Decoder Combinations 4 M M M T T T T M The Test Conditions for Listening Experiment 1 are presented in Table 2 3 2 3 1 2 2 67 3GPP2 50018 v2 0 Table 2 4 2 3 1 2 SO 73 Listening Experiment 1 Test Conditions Exp 1 Wideband ACR Reference Conditions File MNRU 01 ZdB MNRU Reference a02 14dB MNRU Reference a03 21dB MNRU Reference 04 28dB MNRU Reference a05 35dB MNRU Reference a06 42dB MNRU Reference a07 49dB MNRU Reference a08 Direct Source Reference Test Conditions File Condition Enc Dec a09 Nominal level M M a10 Nominal level M T a11 Nominal level T T 12 Nominal level T M a13 Low level 196 d amp b M M 14 Low level 1 d amp b M T a15 Low level 196 d amp b T T a16 Low level 196 d amp b T M 17 High level M M 18 High level M
63. for the SO 77 codec listening tests shall be made with one side of high fidelity supra aural headphones with the other ear uncovered The speech material delivery system shall meet the requirements of Section 3 5 1 1 The listeners should be seated in a quiet room with an ambient noise level of 30 dBA or below 2 5 7 Listeners The listener sample is intended to represent the population of telephone users with normal hearing acuity The listeners should be naive with respect to telephony technology issues that is they should not be experts in telephone design digital voice encoding algorithms and so on They should not be trained listeners that is they should not have been trained in these or previous listening studies using feedback trials Age distribution and gender should be nominally balanced across listening panels Each listener shall provide data only once for a particular evaluation A listener may participate in different evaluations but test sessions performed with the same listener should be at least two months apart so as to reduce the cumulative effects of experience 2 112 3GPP2 C S0018 E v2 0 1 2 5 8 Listening Test Procedures 2 5 81 ACR Listening Test Procedures Experiments 1 3 and 7 s The listeners shall listen to each sample and rate the quality of the test sample using a five point 4 Scale with the points labeled 5 5 X Excellent 6 4 Good 7 3 Fair 8 2 Poor 9 1 Bad 10 Data from 32 listeners shall
64. function The input file must contain packets of compressed data e Instructs the simulation to perform only the encoding function The output file will contain packets of compressed data If neither the d or the e option is invoked the coder performs both the encoding and decoding functions by default M max Sets the maximum allowable data rate to max where max is element of 4 3 2 1 using the codes specified in the first column of Table 3 4 3 3 1 m min Sets the minimum allowable data rate to min where min is element of 4 3 2 1 using the codes specified in the first column of Table 3 4 3 3 1 If neither the M nor m option is invoked the coder allows the data rate to vary between Rate 1 and Rate 1 8 In addition if max min the data rate varies between max and min using the same rate decision algorithm where the data rate is set to max if the selected data rate is gt max and the data rate is set to min if the selected data rate is lt min 3 4 4 4 File Formats Files of speech contain 2 s complement 16 bit samples with the least significant byte first The packet file contains twelve 16 bit words with the low byte ordered first followed by the high byte The first word in the packet contains the data rate while the remaining 11 words contain the encoded speech data packed in accordance with the tables specified in 1 The packet file value for each data rate is shown in Table 3 4 3 3 1 Unused bits are set
65. in detail by Table 2 5 2 2 Table 2 5 2 2 Experiments for SO 77 compliance Experiment Individual tests Notes 1 WB clean level FER signaling ACR Mobile supporting 16 kHz Fs 2 WB noise FER P 835 Mobile supporting 16 kHz Fs 3 NB clean level FER signaling including BS supporting 8 kHz and MS supporting SO 68 interoperable mode tests ACR 8 16 kHz 4 NB noise FER including SO 68 BS supporting 8 kHz and MS supporting interoperable mode tests P 835 8 16 kHz 5 WB music decoder test ACR Mobile supporting 16 kHz Fs 6 NB music decoder test ACR BS supporting 8 kHz Fs 7 2kbps NB clean level FER BS supporting 8 kHz and MS supporting 8 16 kHz 2 91 21 3GPP2 C S0018 E v2 0 Experiment Individual tests Notes 8 2kbps NB noise FER P 835 BS supporting 8 kHz and MS supporting 8 16 kHz 2 5 2 4 Definition The codec subjective test is intended to validate the implementation of the speech codec being tested using the master codec defined in 3 5 3 as a reference Experiments 1 3 and 7 are based on the Absolute Category Rating ACR method which yields the Mean Opinion Score MOS as described in 10 Experiments 2 4 and 8 are based on the ITU T Recommendation P 835 described in 13 2 5 2 2 Method of Measurement The subjective tests involve a listening only assessment of the quality of the codec being tested using the master codec as a reference Subjects from the general population of telephone
66. m2s f2s f2s m3s m3s f3s 2699268 RN NOP N N NI N a who w oO Co 1 w oo of N j FR 5 o o o oo t KES BY 2531 3049 2623 2803 3t f2s05 961025 32 f2s06 986311 26682 NI I fls 2100481 2124411 26758 wa ES w co N to L Co oo oo ap aji oy Co coy 91 w N 5 1 ou f3s 3147009 3167927 9 NI oy 4 Ko CO Table 2 5 4 5 5 Composition of the Sentence Triad Samples for the Experiments 2 and 4 P 835 Test n Ieee n Tee en n Me 2 1 10 20 21 22 23 24 25 26 3GPP2 C S0018 E v2 0 Table 2 5 4 5 6 Composition of the Sentence Triad Samples for the Experiment 8 P 835 Test n CS n ime TA TOY n TAKE 2 5 5 Randomization For each of the first four and the 7 and 8 subjective experiments each presentation sample consists of a speech sample processed under a condition of the test For the ACR Experiments 1 3 and 7 the sample consists of a pair of concatenated sentences of approximately 8 sec duration For the P 835 Experiments 2 4 and 8 the sample consists of three sub samples where each sub sample is a single sentence of approximately 4 sec duration The samples shall be presented to the listeners in a randomized presentation order The listeners for each file set shall be presented w
67. n2p1 2 1 n3p1 3p n4p1 4p 4 p2 n4p2 f4p2 n1p3 fes p3 n4p3 f4p3 nip4 41 p4 f2p4 f4p OY Ke Ko Of Of O17 OTF on OIF Of OTF Bol A GOT Fe CO Go 1817228 1920188 2038580 2160485 2281 2385937 2494010 2591520 2698895 2815196 2920207 304491 3160678 3280125 339051 3502478 rc Length samples n 60 GD GW GI rn KO COF oy KOL S 2 107 NA Sentence Start mips 3615102 fips 3728548 2 1204461 43223 4427918 Oo e Jo N N p n n I Co oy W n Oy CO Co B m p5 4 5 mipo 4539257 fipo m2p 1 2 6 866256 rape rip 5570050 6142186 6243725 mips 6362601 fips mp8 20 1200 2 7 932 SELE 5 2 8 maps 7089854 Fape_ 7194312 1 5 4 1 p5 n4p5 f4p5 1 TA n4p6 5232827 f4p6 5342393 1 7 4 1 p7 n4p7 4 7 n1p8 4 4 p8 n4p8 f4p8 Length samples W ws n WO OY 11506 gt I 129258 7851 05606 339 723 10581
68. noise 16 kHz sampling fer 396 evrc wb op0 ps EVRC WB Nominal 22 dB evrc wb op0 os operating point 0 16 kHz sampling 15 dB street noise evrc wb opO0 pb EVRC WB operating point 0 16 kHz sampling Nominal 22 dB 20 dB babble noise evrc_wb_op0 ob evrc_wb_op0 fer_3 pm EVRC WB operating point 0 16 kHz sampling Generic audio signal fer 396 evrc_wb_op0 fer_3 om Table 3 3 4 5 4 SO 70 Encoder Suite B Bit exact Test Conditions Input File Operating Point Condition neverence Packet mes koke exact compliance src s22 EVRC WB operating point Nominal 22 dB evrc wb opoO p22 0 16 kHz sampling src s12 EVRC WB operating point High 12 dB evrc_wb_op0 p12 0 16 KHz sampling src s32 EVRC WB operating point Low 32 dB evrc wb op0 p32 0 16 kHz sampling 196 d amp b src c10 EVRC WB operating point Nominal 22 dB evrc wb opO0 pc1 0 16 kHz sampling 10 dB car noise 3 28 3GPP2 C S0018 E v2 0 Input File Operating Point Condition paetos nee exact compliance src c20 EVRC WB operating point Nominal 22 dB evrc_wb_op0 pc2 0 16 kHz sampling 20 dB car noise src s15 EVRC WB operating point Nominal 22 dB evrc wb opO ps 0 16 kHz sampling 15 dB street noise src b20 EVRC WB operating point Nominal 22 dB evrc wb opoO pb 0 16 kHz sampling 20 dB babble noise src s22 8k EVRC WB operating point No
69. of quality Please take into account your total impression of each sample rather than concentrating on any particular aspect Figure 2 4 8 1 1 Instructions for Listeners 2 4 8 2 P 835 Listening Test Procedures Experiments 2 and 4 Experiments 2 and 4 use the P 835 test methodology described in 13 The P 835 methodology is specifically designed to evaluate the quality of speech in background noise It yields a measure of Signal Quality SIG a measure of Background Quality BAK and a measure of Overall Quality OVRL In general OVRL scores are highly correlated with MOS but the OVRL score provides greater sensitivity and precision in test conditions involving background noise While the OVRL score is of most interest here the SIG and BAK scores also provide valuable diagnostic information For each trial in a P 835 test listeners are presented with three sub samples where each sub sample is a single sentence approx 4 sec duration processed through the same test condition In one of the first two sub samples listeners rate the Signal Quality on a five point rating scale with the points labeled Very natural no distortion Fairly natural little distortion Somewhat natural some distortion Fairly unnatural fairly distorted Very unnatural very distorted A CI 2 83 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 3GPP2 C S0018 E v2 0 For the other of the first two sub
70. only assessment of the quality of the codec being tested using the master codec as a reference Subjects from the general population of telephone users will rate the various conditions of the test Material supplied with this standard for use with this test includes source speech impaired packet files from the master codec encoder and source speech processed by various Modulated Noise Reference Unit MNRU conditions and other references The basic Absolute Category Rating test procedure involves rating all conditions using a five point scale describing the opinion of the test condition This procedure is fully described in 10 2 1 2 3 Test Conditions and Test Design for SO 3 Listening Experiments The two listening experiments for SO 3 are similar in design and are performed as MOS listening tests Each experiment will test the same number of codecs and the number of test conditions for each experiment is five There will be one condition typifying CDMA channels 396 FER a clear channel condition and a clear channel tandem condition All tandem conditions shall be asynchronous where asynchronous implies the introduction of a partial frame offset between encoding operations A nominal input level of 22 dB shall be used for these conditions Additional test conditions include background noise and audio input level variation For reference u law 4 MNRU conditions 5 15 20 and 25 dBQ values and G 728 8 will be included in each experiment
71. operating point 3 8 kHz sampling src 2kbps s12 8k EVRC NW2K High 12 dB evrc nw op3 pi2 operating point 3 8 kHz sampling src 2kbps s32 8k EVRC NW2K Low 32 dB evrc nw op3 p32 operating point 3 8 kHz sampling src 2kbps c15 8k EVRC NWZ2K Nominal 22 dB evrc nw op3 pc operating point 3 8 kHz 15 dB car noise samplin 3 54 3GPP2 C S0018 E v2 0 Input File Operating Point Condition iine aes DID src_2kbps s15 8k EVRC NWZ2K Nominal 22 dB evrc nw op3 ps operating point 3 8 kHz 15 aB street sampling noise src 2kbps b20 8k EVRC NW2K Nominal 22 dB evrc nw op3 pb operating point 3 8 KHz 20 dB babble samplin noise Note 9 3 kbps mode is generated using anchor operating point 0 and 5 8 kbps mode is generated using anchor operating point 2 Table 3 5 4 5 3 SO 77 Suite A Decoder Bit exact Test Conditions Reference output Input Packet File Operating Point Condition speech files for bit exact compliance evrc nw opO fer 2 EVRC NW2K operating Nominal 22 dB evrc nw opO fer 3 2 point 0 16 kHz sampling 396 FER 022 evrc nw opO fer 196 pls EVRC NW2K Nominal 22 dB evrc nw 196 p22 operating point 0 196 FER 196 pls fer 196 pls 196 022 16 kHz sampling evrc nw opoO pi2 EVRC NWZK operating High 12 dB evrc nw op0 o12 point 0 16 kHz sampling evrc nw opO dim 19o p EVRC NW2K operating Low 32 dB evrc_nw_op0 dim_1 32 point
72. pcm o verify pkt e EvrcFIt i verify pkt o verify dec d If the output files mstr ref pkt and mstr ref dec exactly match the verify pkt and the verify dec respectively then verification of the master codec s operation is complete Because of differences in the way that floating point arithmetic is done in different computing environments it will not always be true that the floating point master C simulation will produce identical output in response to the same input when compiled and run on different compiler hardware platforms even though the simulation is operating correctly In the event that the exact match described in the preceding paragraph is not obtained it is recommended that the user verify that the version of GCC used is version 2 7 2 or later 3 1 4 Fixed Point Bit Exact Codec for SO 3 This section describes the C simulation of the speech codec specified by 1 The speech codec C simulation is based on finite precision fixed point arithmetic operations and is required to be used as a reference codec to verify the performance of a bit exact EVRC implementation of the fixed point C simulation of a test codec The bit exact EVRC codec along with the appropriate test vectors to verify the bit exactness performance are included in the associated Software Distribution There are two options for compiling the fixed point EVRC simulation One option uses the 31 bit long multiply DSP math library and the other uses the 32 bit librar
73. s22 8k EVRC NW operating point Nominal 22 dB evrc_nw_op6 p22 6 8 kHz sampling src s12 8k EVRC NW operating point High 12 dB evrc_nw_op6 p12 6 8 kHz sampling src s32 8k EVRC NW operating point Low 32 dB evrc nw op6 p32 6 8 kHz sampling src s22 8k EVRC NW operating point Nominal 22 dB evrc_nw_op6 dim_1 p22 6 8 kHz sampling 196 d amp b src s22 8k EVRC NW operating point Nominal 22 dB evrc nw op7 p22 7 8 kHz sampling Src c15 8k EVRC NW operating point Nominal 22 dB evrc nw opl1 pc 1 8 kHz sampling 15 dB car noise src s15 8k EVRC NW operating point Nominal 22 dB evrc nw opi ps 1 8 kHz sampling 15 dB street noise src b20 8k EVRC NW operating point Nominal 22 dB evrc nw op1 pb 1 8 kHz sampling 20 dB babble noise Src c15 8k EVRC NW operating point Nominal 22 dB evrc nw op6 pc 6 8 kHz sampling 15 dB car noise src s15 8k EVRC NW operating point Nominal 22 dB evrc nw op6 ps 6 8 kHz samplin 15 dB street noise 8 41 3GPP2 50018 v2 0 Input File Operating Point Condition beu RS PIE src b20 8k EVRC NW operating point Nominal 22 dB evrc nw op6 pb 6 8 kHz sampling 20 dB babble noise Src c15 8k EVRC NW operating point Nominal 22 dB evrc nw op7 pc 7 8 kHz samplin 15 dB car noise Note 9 3 kbps mode is generated using anchor operating point 0 and 5 8 kbps mode is generated using anchor operating point 2 Table 3 4 4 5 3 SO 73 Suite A Decoder Bit exact
74. samples listeners rate the Background Quality on a five point rating scale with the points labeled Not noticeable Fairly noticeable Noticeable but not intrusive Fairly conspicuous somewhat intrusive Very conspicuous very intrusive A C For the third sub sample listeners rate the Overall quality on a five point rating scale with the points labeled Excellent Good Fair Poor Bad ou Data from 32 listeners shall be used for Experiments 2 and 4 four listeners for each listening panel where each listening panel uses a different randomization Before starting the test the listeners should be given instructions for performing the subjective test An example set of instructions for the P 835 test are presented below The instructions may be modified to allow for variations in laboratory data gathering apparatus Instructions for P 835 Speech Rating Experiment In this speech rating experiment each trial will involve three sentences and you will give a rating for each sentence For the first sentence in each trial you will be asked to attend only to the speech signal and rate how natural or conversely how degraded the speech signal sounds to you You will use the rating scale shown in the figure below to register your ratings of the speech signal Your task will be to choose the numbered phrase from the list below that best describes your opinion of the SPEECH SIGNAL ALONE and then enter the corresponding numbe
75. scale below 5 Excellent Good 4 3 Fair 2 Poor 1 Bad During the session you will hear samples varying in different aspects of quality Please take into account your total impression of each sample rather than concentrating on any particular aspect Figure 2 2 8 1 1 Instructions for Listeners 2 2 8 2 P 835 Listening Test Procedures Experiment II Experimental Il uses the P 835 test methodology described in ITU T Rec P 835 13 The P 835 methodology is specifically designed to evaluate the quality of speech in background noise It yields a measure of Signal Quality SIG a measure of Background Quality BAK and a measure of Overall Quality OVRL In general OVRL scores are highly correlated with MOS but the OVRL score provides greater sensitivity and precision in test conditions involving background noise While the OVRL score is of most interest here the SIG and BAK scores also provide valuable diagnostic information For each trial in a P 835 test listeners are presented with three sub samples where each sub sample is a single sentence approx 4 sec duration processed through the same test condition In one of the first two sub samples listeners rate the Signal Quality on a five point rating scale with the points labeled Very natural no distortion Fairly natural little distortion Somewhat natural some distortion Fairly unnatural fairly distorted Very unnatural very distorted C 2 29 20
76. standard concentrates specifically on the EVRC whether implemented at the mobile station or the base station or elsewhere in the cellular system This standard covers the operation of this component only to the extent that compatibility with the specific EVRC compatible variable rate codec is ensured 1 1 Scope This document specifies the procedures to test implementations of EVRC A EVRC B EVRC WB EVRC NW or EVRC NW2K compatible variable rate speech codecs either by meeting the bit exact implementation or meeting recommended minimum performance requirements The EVRC A is the Service Option 3 SO 3 speech codec the EVRC B is the Service Option 68 SO 68 speech codec the EVRC WB is the Service Option 70 SO 70 speech codec the EVRC NW is the Service Option 73 SO 73 speech codec and the EVRC NW2K is the Service Option 77 SO 77 speech codec all described in 1 The procedures specified in this document for the SO 3 speech codec are fully consistent with those contained in 3 The SO 3 speech codec is used to digitally encode the speech signal for transmission at a variable data rate of 8550 bps 4000 bps or 800 bps The SO 68 speech codec is used to digitally encode the speech signal for transmission at a variable data rate of 8550 bps 4000 bps 2000 bps or 800 bps The SO 70 speech codec is used to digitally encode the speech signal for transmission at a variable data rate of 8550 bps 4000 bps or 800 bps The SO 73 speech codec is us
77. the tables in Section 2 5 2 3 2 5 4 Processing of Speech Material for SO 77 Testing The source speech material shall be processed by the various combinations of encoders and decoders listed in the descriptions of the experiments given in Section 2 5 2 The master codec software described in Section 3 5 3 shall be used in the processing involving the master codec Generally the master codec encoder and decoder outputs have been provided in the respective directories so77 subjctv exp m pkt and so77 subjctv exp m m Execution of the master codec software is needed only for the test encoder master decoder combination for each experiment condition All codec processing shall be done digitally Noise suppression and post filter options shall be enabled for both the master and the test codecs The digital format of the speech files is described in Section 3 5 4 4 The naming convention of the processed speech is as follows For the packet files in the so77 subjctv exp 1 3 m_pkt directory the p12 files are the master packet files for the s12 source file Likewise the p22 and p32 files are the respective packet files for the s22 and s32 source files The pf8 files are the impaired packet files which will be described in Section 2 4 4 3 Similarly the directory so77 subjctv exp 2 4 m pkt contains the master packet files for the respective experiments Here the pc10 pb20 and ps files are the master packet files for the c
78. under a variety of conditions depending on a number of parameters These conditions include channel impairments audio background noise and different input levels Figure 2 5 2 1 illustrates a decision tree to arrive at the suite of tests that are needed to demonstrate Minimum Performance Spec compliance of a Test implementation of SO 77 for different profiles of equipment that support SO 77 2 90 3GPP2 C S0018 E v2 0 Is 16kHz Sampling Rate Supported Yes No Run Test Suite A Run Test Suite B Figure 2 5 2 1 SO 77 Subjective test suite decision flowchart An implementation may support SO 77 only for 8 kHz sample rate input output for example a Base station transcoder or a Media Gateway An implementation may support SO 77 for both 16 kHz and 8 kHz sample rates for example a mobile station that supports wideband electro acoustics Therefore the main parameter in the decision tree is 16 kHz support in the implementation Depending on the implementation profile of the device under test one of 2 possible Test Suites are to be used to demonstrate SO 77 compliance These 2 test suites named Test suites A and B and the individual tests comprising the Test suites are highlighted in Table 2 5 2 1 Table 2 5 2 1 Test Suites for SO 77 compliance Test Suites Set of Experiments A Experiment 1 2 3 4 5 6 7 and 8 B Experiment 3 4 6 7 and 8 Each of the individual experiments are further defined
79. validate the implementation of the speech codec being tested using the master codec defined in 3 4 3 as a reference Experiments 1 and 3 are based on the Absolute Category Rating ACR method which yields the Mean Opinion Score MOS as described in 10 Experiments 2 and 4 are based on the ITU T Recommendation P 835 described in 13 2 4 2 2 Method of Measurement The subjective tests involve a listening only assessment of the quality of the codec being tested using the master codec as a reference Subjects from the general population of telephone users will rate the various conditions of the test Material supplied with this standard for use with this test includes source speech impaired packet files from the master codec encoder and source speech processed by various Modulated Noise Reference Unit MNRU conditions and other references The basic Absolute Category Rating test procedure involves rating all conditions using a five point scale describing the opinion of the test condition This procedure is fully described in 10 The P 835 test method involves rating all conditions on scales of Signal Background and Overall quality and is fully described in 13 2 4 2 3 Test Conditions and Test Design for SO 73 Listening experiments 1 and 3 for SO 73 are performed as ACR listening tests Experiments 2 and 4 for SO 73 are performed as P 835 listening tests 2 4 2 3 1 Subjective Experiment 1 for SO 73 The Test Parameters
80. 0 21 22 23 24 25 26 27 28 29 30 31 32 33 34 3GPP2 C S0018 E v2 0 emee oes es es Listeners Votes e a 32 2 oe Listeners Voters per Listening Panel Lau Fat a l s Votes per Condition 256 192 256 102 2 4 3 Speech Material for SO 73 Testing The source speech files used for SO 73 compliance testing consist of Harvard sentences pairs which are preprocessed to include proper level adjustment and noise mixing for use in the subjective experiments The talkers used in these files consist of adult males and adult females and are native speakers of North American English For the following discussion it may be useful to refer to Table 4 3 for the composition of the Software Distribution database The source speech material for subjective Experiments is contained in directory so73 subjctv exp source Each file has been appropriately pre filtered level adjusted and noise processed These files are named src The speech database also includes samples processed through the various reference conditions in directory so73 subjctv exp ref The reference conditions are named ref for the respective conditions given in the tables in Section 2 4 2 3 2 4 4 Processing of Speech Material for SO 73 Testing The source speech material shall be processed by the various combinations of encoders and decoders listed in the descriptions of the experiments given
81. 0 Table 2 4 4 5 2 Cutting Points for the astrip Software Tool for the SO 73 Experiments 2 and 4 Start sample Length samples 1 57758 Sentence a 5 _ 2 02 288424 55912 2s02 3404062 59225 9 mso 3463288 51884 10 m3s02 515172 59593 8 8 9 1 16 18 15 P 835 Test Experiment II P 835 Start Length sample samples 60 3s10 3404618 58012 Start Length Sentence sample samples 66 2 12 3734538 55821 68 22512 3852015 60190 69 mssiip 3912115 55432 80 festa 4567054 55052 86 misis 4900324 51588 89 mzsi5 5080387 65752 90 mzsis 5146139 63251 95 3518 5508241 51792 Table 2 4 4 5 3 Composition of the Sentence Triad Samples for the Experiments 2 and 4 P 835 Test TA ince n TIG n ce n UE 2 4 5 Randomization For each of the first four subjective experiments each presentation sample consists of a speech sample processed under a condition of the test For the ACR Experiments 1 and 3 the sample consists of a pair of concatenated sentences of approximately 8 sec duration For the P 835 Experiments 2 and 4 the sample consists of three sub samples where each sub sample is a single 2 80 20 21 22 23 3GPP2 C S0018 E v2 0 sentence of approximately 4 sec duration The samples shall be presented to the listeners in a randomized presentation order The l
82. 0 16 kHz sampling 1 d amp B 032 evrc_nw_op0 pc1 EVRC NW2K operating Nominal 22 dB evrc nw opO oc1 point 0 16 kHz sampling 10 dB car noise evrc nw fer 396 p EVRC NW2K Nominal 22 dB evrc nw opO fer 3 c2 operating point 0 20 dB car noise oc2 16 kHz sampling fer 396 evrc nw opO ps EVRC NW2K operating Nominal 22 dB evrc nw op0 os point 0 16 kHz sampling 15 dB street noise evrc nw opoO pb EVRC NW2K operating Nominal 22 dB evrc_nw_op0 ob point 0 16 kHz sampling 20 dB babble noise evrc nw opO fer 3 EVRC NW2K operating Generic audio evrc nw opO fer 3 point 0 16 kHz sampling signal fer 396 om evrc_nw_op1 fer_3 p2 EVRC NW2K operating Nominal 22 dB evrc nw op1 fer 396 2 point 1 8 kHz sampling FER 396 022 8k evrc nw opi pi2 EVRC NW2K operating High 12 aB evrc nw opt o12 8k point 1 8 kHz sampling evrc nw op1 p32 EVRC NW2K operating Low 32 dB evrc nw op1 032 8k point 1 8 kHz sampling evrc nw op7 p22 EVRC NW2K operating Nominal 22 dB evrc nw op7 o22 8k point 7 8 kHz sampling 3 55 3GPP2 50018 v2 0 Reference output Input Packet File Operating Point Condition speech files for bit exact compliance evrc nw opi dim 196 pl EVRC NW2K operating Nominal 22 dB evrc nw opti dim 196 196 p22 point 1 8 kHz sampling 196 d amp b 196 pls pls 196 022 8k evrc nw op1 pc EVRC NW2K ope
83. 0 2698895 2815196 2920207 304491 3160678 3280125 339051 3502478 rc Length samples n 60 GD GW GI rn KO COF oy KOL S 2 79 NA Sentence Start mips 3615102 fips 3728548 2 1204461 43223 4427918 Oo e Jo N N p n n I Co oy W n Oy CO Co B m p5 4 5 mipo 4539257 fipo m2p 1 2 6 866256 rape rip 5570050 6142186 6243725 mips 6362601 fips mp8 20 1200 2 7 932 SELE 5 2 8 maps 7089854 Fape_ 7194312 1 5 4 1 p5 n4p5 f4p5 1 TA n4p6 5232827 f4p6 5342393 1 7 4 1 p7 n4p7 4 7 n1p8 4 4 p8 n4p8 f4p8 Length samples W ws n WO OY 11506 gt I 129258 7851 05606 339 723 10581 122664 1164 1274 10956 10880 118850 097 NO NO 6957 112149 01539 8876 8866 116617 130938 123975 115326 12153 10445 10290 sf po sf po W 3GPP2 C S0018 E v2
84. 0 8 EVRC NWZK operating Nominal 22 dB evrc nw op3 pb k point 3 8 kHz sampling 20 dB babble noise Table 3 5 4 5 5 SO 77 Suite B Decoder Bit exact Test Conditions Reference output Input Packet File Operating Point Condition speech files for bit exact compliance evrc nw opO fer 396 p EVRC NWZ2K Nominal 22 dB evrc nw opO fer 396 22 operating point 0 3 FER 022 8k 8 kHz sampling evrc nw opoO pi2 EVRC NWZ2K High 12 dB evrc nw op0 012 8k operating point 0 8 kHz sampling evrc_nw_op0 p32 EVRC NWZ2K Low 32 dB evrc nw 0op0 032 8k operating point 0 8 kHz sampling evrc nw opO dim 196 p EVRC NWZ2K Nominal 22 dB evrc nw opO dim 196 ls 196 p22 operating point 0 8 kHz 196 d amp b 1 pls pls 196 022 8k sampling evrc nw opO pc EVRC NWZ2K Nominal 22 dB evrc nw opO0 oc 8k operating point 0 8 kHz sampling 15 dB car noise evrc nw opO ps EVRC NW2k operating point 0 8 kHz sampling Nominal 22 dB 15 dB street noise evrc_nw_op0 os 8k evrc_nw_op0 fer_2 p b EVRC NW2k operating point 0 8 kHz sampling Nominal 22 dB 20 dB babble noise 2 FER evrc_nw_op0 fer_2 ob 8k evrc_nw_op0 fer_3 p m EVRC NW2k operating point 0 8 kHz sampling Generic audio signal fer_3 evrc_nw_op0 fer_3 om 8k evrc_nw_op1 fer_3 p 22 EVRC NW2K operating point 1 8 kHz sampling 3 58 Nominal 22 dB FER 396 evrc nw opi fer 396 022 8k
85. 098 20 dB 40 dB MNRU SNR Car Noise Figure 2 3 10 2 1 Typical P 835 Score Profiles for Reference Conditions 2 4 Performance Testing for SO 73 2 4 4 Objective Performance Testing for SO 73 The objective testing portion of this specification consists of an average data rate test and compliance to End to End Algorithmic Delay and Unity gain requirements 2 4 1 1 Average Data Rate Test An implementation may support SO 73 for 16 kHz sample rates for example a mobile station that supports wideband electro acoustics The average data rate for the test codec shall be measured using seven source speech files that are contained in the so73 subjctv exp 1 2 source directories Each file exhibits a different condition power levels 12 dB 22 dB and 32 dB and background noise conditions 20 dB SNR babble noise 10 dB SNR car noise 20 dB SNR car noise and 15 dB SNR street noise The input source files used in the average data rate test have an approximate voice activity factor of 0 6 and are the same input files used in the subjective portion of the experiment An implementation may support SO 73 only for 8 kHz sample rate input output for example a Base station transcoder or a Media Gateway or mobile station The average data rate for the test codec for this case shall be measured using six narrow band source speech files that are contained in the so73 subjctv exp 3 4 source directories Each file exhibits a differ
86. 1 4 3 mips 1 4 2 4 mips ESN 1 1 2 1 3 1 f4pl n1 pl 41 pl n2p1 2 1 n3p1 3p n4p1 4p 4 p2 n4p2 f4p2 n1p3 fes p3 n4p3 f4p3 nip4 41 p4 f2p4 f4p 10 16 18 19 21 e Ke Ko Of Of O17 OTF on OIF Of OTF Bol A GOT Fe CO Go 1817228 1920188 2038580 2160485 2281 2385937 2494010 2591520 2698895 2815196 2920207 304491 3160678 3280125 339051 3502478 rc Length samples n 60 GD GW GI rn KO COF oy KOL S 2 54 NA Sentence Start mips 3615102 fips 3728548 2 1204461 43223 4427918 Oo e Jo N N p n n I Co oy W n Oy CO Co B m p5 4 5 mipo 4539257 fipo m2p 1 2p6 1866256 rape rip 5570050 6142186 6243725 mips 6362601 fips mp8 20 1200 2 7 932 SELE 5 2 8 maps 7089854 Fape_ 71
87. 1 32 33 34 35 36 3GPP2 50018 v2 0 There exists two options for compiling the fixed point EVRC simulation One option uses the 31 bit long multiply DSP math library and the other uses the 32 bit library A parallel set of bit exact test vectors is provided so that a CODEC may qualify as bit exact using either library By default the DSP math library compiles the 32 bit long multiply routines In order to compile with the 31 bit long multiply routines the following lines in so3 simul fixed dspmath makefile must be commented uncommented Change from 32 bit library Uncomment the following line to use alternate double precision multiplies CCAUXFLAGS DUSE_ALT_DP31 amp comment the following line out CCAUXFLAGS to 31 bit library Uncomment the following line to use alternate double precision multiplies CCAUXFLAGS DUSE_ALT_DP31 amp comment the following line out CCAUXFLAGS 3 1 43 Running the Fixed Point Codec Simulation The EVRC executable files use command line arguments to receive all information regarding input and output files and various parameters used during execution Executing EvrcFix with no command line arguments will display a brief description of the required and optional command line arguments The options are described below i infn required o outf required d f max h max min Specifies the name of the input speech file or the name of the input pac
88. 15 b20 and s15 source files respectively For the master encode master decode directories so77 subjctv exp m m the naming convention of the speech files is such that the first two characters of the file name indicate the codec combination and the suffix indicates the condition numbers in Table 2 4 2 3 1 2 and Table 2 4 2 3 2 2 Naming conventions for the remaining two experiments follow accordingly Detailed descriptions of all processing operations are given in Section 6 2 5 41 Encoding by the Test Codec All of the source files will be encoded by the test codec to produce encoded packet files For ease of reference it is recommended that directories so77 subjctv exp t pkt be created to deposit the test encoder output packets and that the naming conventions be made consistent with the master codec 2 5 4 2 Decoding by the Master Test Codecs The encoded packet files generated from the various encoders conditions shall be processed through the master and test decoders 2 5 4 3 Introduction of Impairments For the frame error conditions the impaired master codec encoded packet files are provided in the so77 subjctv exp m_pkt directory Unlike other conditions this condition uses only the test decoder and not the test encoder 2 105 20 21 22 23 24 25 3GPP2 50018 v2 0 For the Dim and Burst processing and also the Packet Level Signaling conditions the processing requires inputs from a signaling fi
89. 2 dB 15 dB street 9 3 ps src s15 EVRC B 5 8 kbps Nominal 22 dB 15 dB street 5 8 ps Note 9 3 kbps mode is generated using anchor operating point 0 and 5 8 kbps mode is generated using anchor operating point 2 3 18 3GPP2 C S0018 E v2 0 Table 3 2 4 5 2 SO 68 Decoder Bit exact Test Conditions Input Reference output Packet Operating Point Condition speech files for bit File exact compliance 9 3 p22 EVRC B 9 3 kbps Nominal 22 dB 9 3 022 5 8 p22 EVRC B 5 8 kbps Nominal 22 dB 5 8 022 4 8 p22 EVRC B 4 8kbps Nominal 22 dB 4 8 022 9 3 p32 EVRC B 9 3 kbps Low 32 dB 1 d amp b 196 pls 9 3 032 5 8 p32 EVRC B 5 8 kbps Low 32 dB 1 d amp b 1 pls 5 8 032 9 3 p12 EVRC B 9 3 kbps High 12 dB 9 3 012 5 8 p12 EVRC B 5 8 kbps High 12 dB 5 8 012 9 EVRC B 9 3 kbps Nominal 22 dB 15 dB carnoise 9 3 0c 5 8 EVRC B 5 8 kbps Nominal 22 dB 15 dB carnoise 5 8 0C 9 EVRC B 9 3 kbps Nominal 22 dB 20 dB babble 9 3 ob 5 8 EVRC B 5 8 kbps Nominal 22 dB 20 dB babble 5 8 ob 9 3 ps EVRC B 9 3 kbps Nominal 22 dB 15 dB street 9 3 0s 5 8ps EVRC B 5 8 kbps Nominal 22 dB 15 dB street 5 8 05 3 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 3GPP2 C S0018 E v2 0 3 3 Specific Standard Test Conditions for SO 70 3 3 1 Audio Path and
90. 29 30 31 32 33 34 35 36 37 38 3GPP2 C S0018 E v2 0 Similarly the directory so3 subjctv exp2 m pkt contains the master packet files for Experiment Il Here the p22 files are the master packet files for the s22 source files and the pc pb and ps files are the master packet files for the car bab and str source files respectively For the master encode master decode directories soS3 subjctv exp m m the naming convention of the speech files is such that the first two characters of the suffix indicate the codec combination and third indicates the condition number 1 through 5 It is required that this convention be used for the other codec combinations mt tm and tt so that the supplied randomization lists see Section 2 1 5 are valid Two exceptions to this naming convention is the master encoder master decoded 3 reverse link FER files which shall be assigned the extension tm4 and the 396 forward link FER files shall be assigned the extension mm4 2 1 4 4 Encoding by the Test Codec All of the source files will be encoded by the test codec to produce encoded packet files For ease of reference it is recommended that directories so3 subjctv exp1 t and so3 subjctv exp2 t pkt be created to deposit the test encoder output packets and that the naming conventions be made consistent with the master codec 2 1 4 2 Decoding by the Master Test Codecs The encoded packet files generated
91. 3 6 1 2 47 3GPP2 50018 v2 0 1 Table 2 3 2 3 6 1 SO 70 Listening Experiment 6 Test Parameters P NSA P 835 Narrowband Number of talkers 3 males 3 females Test conditions Car Noise 15 dB SNR Mode 0 Street Noise 15 dB SNR Mode 0 Babble 20 dB SNR 2 FER Mode 0 Car Noise 15 dB SNR Mode 0 2 d amp b Car Noise 15 dB SNR Mode 0 1 pls Encoder Decoder Combinations 4 M M T T T M The Test Conditions for Listening Experiment 6 are presented in Table 2 3 2 3 6 2 4 Table 2 3 2 3 6 2 SO 70 Listening Experiment 6 Test Conditions Exp 6 Narrowband P 835 Reference Conditions File MNRU f01 MNRU 40dB SNR 40dB Reference 02 MNRU 40aB SNR 20dB Reference MNRU 40aB SNR 0dB Reference 04 MNRU ZOdB SNR 40dB Reference 05 MNRU 20aB SNR 40dB Reference 06 MNRU 10aB SNR 10dB Reference 07 MNRU 20aB SNR 20dB Reference 08 MNRU 40aB SNR 30dB Reference Test Conditions File Condition Enc Dec 09 Car 15dB SNR Mode 0 LB portion of Wideband mode decoder test only M M 10 Car 15dB SNR Mode 0 LB portion of Wideband mode decoder test only M T Hi Street 15dB SNR Mode 0 LB portion of Wideband mode decoder test only M M 12 Street 15dB SNR Mode 0 LB portion of Wideband mode decoder test only M T 19 Babble 20dB SNR 2 FER Mode 0 LB portion of Wideband mode decoder test only M M 2 48 3GPP2 50018 v2 0
92. 35 7 WB music decoder test ACR Mobile supporting 16 kHz Fs 8 NB music decoder test ACR BS supporting 8 kHz Fs 2 3 21 Definition The codec subjective test is intended to validate the implementation of the speech codec being tested using the master codec defined in 3 3 3 as a reference Experiments 1 3 and 5 are based on the Absolute Category Rating ACR method which yields the Mean Opinion Score MOS as described in 10 Experiments 2 4 and 6 are based on the ITU T Recommendation P 835 described in 13 2 38 3GPP2 C S0018 E v2 0 2 3 2 2 Method of Measurement The subjective tests involve a listening only assessment of the quality of the codec being tested using the master codec as a reference Subjects from the general population of telephone users will rate the various conditions of the test Material supplied with this standard for use with this test includes source speech impaired packet files from the master codec encoder and source speech processed by various Modulated Noise Reference Unit MNRU conditions and other references The basic Absolute Category Rating test procedure involves rating all conditions using a five point scale describing the opinion of the test condition This procedure is fully described in 10 The P 835 test method involves rating all conditions on scales of Signal Background and Overall quality and is fully described in 13 2 3 2 8 Test Conditions and Test Design for
93. 4 SO 3 or Section 3 2 4 SO 68 or Section 3 3 4 SO 70 or Section 3 4 4 SO 73 or Section 3 5 4 SO 77 respectively e Following the objective and subjective testing procedures set forth in Sections 2 1 1 and 2 1 2 or Sections 2 2 1 and 2 2 2 or Sections 2 3 1 and 2 3 2 or Section 2 4 1 and 2 4 2 SO 73 or Section 2 5 1 and 2 5 2 SO 77 of this standard With the exception of Sections 3 1 4 3 2 4 3 3 4 3 4 4 and 3 5 4 the remaining text applies only to implementations that do not satisfy the requirement for bit exactness 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 3GPP2 C S0018 E v2 0 Testing the codec is based on two classes of procedures objective tests and subjective tests In the event that the test codec fails any of the objective or subjective tests the test codec fails the compliance test Objective tests are based upon actual measurements from the speech codec function Subjective tests are based on listening tests to judge overall speech quality The minimum subjective requirement for the test codec is based upon the ability of the test codec to demonstrate performance equivalent to or better than that of the specific EVRC floating point bit exact codec within a fixed allowable statistical error The purpose of the testing is not only to ensure adequate performance between one manufacturer s encoder and decoder but also that this level of perf
94. 4 1 SO 73 ITU T P 311 P 341 Transmit Mask and Filter responses 3 4 3 Master Codec for SO 73 This section describes the C simulation of the speech codec specified by 1 The master codec C simulation used for verifying the performance of a non bit exact EVRC NW implementation shall be the floating point master C simulation included in the associated Software Distribution 1a 3 36 20 21 22 23 24 26 27 28 29 30 31 32 33 34 35 3GPP2 C S0018 E v2 0 3 4 3 1 Compiling the Master Codec Simulation The source code for floating point simulation can be compiled using the GNU G compiler and make utility A G compatible makefile has been included in the appropriate sub directory in 1a Typing make this directory will compile and link the code and create the executable file called Evrc nw Evrc nw exe on Win32 systems which will be placed in the same directory The included makefile may require some user modification for a particular hardware platform and or operating system 3 4 3 2 Running the Master Codec Simulation The EVRC NW floating point executable Evrc nw files use command line arguments to receive all information regarding input and output files and various parameters used during execution Executing Evrc nw with no command line arguments will display a brief description of the required and optional command line arguments The options are described below i infn required Spec
95. 4 2 3 5 1 SO 73 Listening Experiment 5 Test Parameters ACR P 800 Wideband ca of genres Background noise none ambient Audio Input Level 22 dB Filter characteristics P 341 refer Section 3 3 2 4 Reference conditions 4 Specified reference conditions Test conditions 0 FER and 3 FER Encoder Decoder Combinations 2 M M M T 2 74 2 5 6 9 3GPP2 C S0018 E v2 0 The Test Conditions for Listening Experiment 5 are presented in Table 2 4 2 3 5 2 Table 2 4 2 3 5 2 SO 73 Listening Experiment 5 Test Conditions Exp 5 Wideband Music File Reference Condition 01 MNRU 15dB Reference 02 MNRU 25dB Reference e03 MNRU 35dB Reference 04 Reference File Test Condition Enc Dec e05 0 FER M M e06 0 FER M T e07 3 FER M M e08 3 FER M T 2 4 2 3 6 Subjective Experiment 6 for SO 73 The Test Parameters for Listening Experiment 6 are presented in Table 2 4 2 3 6 1 Table 2 4 2 3 6 1 SO 73 Listening Experiment 6 Test Parameters of test of test 2 800 P 800 Narrowband Number of genres Background noise none ambient Audio Input Level 22 dB Filter characteristics MIRS Reference conditions 4 Specified reference conditions 0 FER and 3 FER Encoder Decoder Combinations 2 M M M T Test conditions The Test Conditions for Listening Experiment 6 are presented in Table 2 4 2 3 6 2 2 75
96. 4 a06 m4 3 mi f1 m2 m3 f1 m2 f2 m2 f4 mi f1 mi 20 21 22 23 24 25 26 27 28 29 30 31 3GPP2 C S0018 E v2 0 The randomization lists for each of the eight listening panels for each experiment are provided in so68 subjctv exp1 data play lst and so68 subjctv exp2 data play Ist respectively 2 2 6 Presentation Presentation of speech materials for the SO 68 codec listening tests shall be made with one side of high fidelity supra aural headphones with the other ear uncovered The speech material delivery system shall meet the requirements of Section 3 2 1 1 The listeners should be seated in a quiet room with an ambient noise level of 30 dBA or below 2 2 7 Listeners The listener sample is intended to represent the population of telephone users with normal hearing acuity The listeners should be naive with respect to telephony technology issues that is they should not be experts in telephone design digital voice encoding algorithms and so on They should not be trained listeners that is they should not have been trained in these or previous listening studies using feedback trials Age distribution and gender should be nominally balanced across listening panels Each listener shall provide data only once for a particular evaluation A listener may participate in different evaluations but test sessions performed with the same listener should be at least two months apart so as t
97. 40 EVRC B 5 8 kbps Nominal 22 dB 396 FER M T 2 2 2 8 Subjective Experiment Il for SO 68 The Test Parameters for Listening Experiment II are presented in Table 2 2 2 3 2 1 Table 2 2 2 3 2 1 SO 68 Listening Experiment Il Test Parameters P NSA P 835 Type of test Type oftest n Test conditions Encoder Decoder Combinations 4 M M M T T T T M a Car Noise 15 dB S N 9 3 5 8 4 8 kbps b Street Noise 15 dB S N 9 3 5 8 kbps c Office Babble 20 dB S N 9 3 5 8 kbps 2 20 3GPP2 50018 v2 0 The Test Conditions for Listening Experiment 11 are presented in Table 2 2 2 3 2 2 Table 2 2 2 3 2 2 SO 68 Listening Experiment Il Test Conditions Label Operating Point Impairment Condition Encoder Decoder Combinations b01 Reference Car Noise 40 dB SNR MNRU 40 dB b02 Reference Car Noise 20 dB SNR MNRU 40 dB b03 Reference Car Noise 0 dB SNR MNRU 40 dB b04 Reference Car Noise 40 dB SNR MNRU 0 dB b05 Reference Car Noise 40 dB SNR MNRU 20 dB b06 Reference Car Noise 10 dB SNR MNRU 10 dB b07 Reference Car Noise 20 dB SNR MNRU 20 dB b08 Reference Car Noise 30 dB SNR MNRU 30 dB b09 EVRC B 9 3 kbps Car Noise 15 dB M M b10 EVRC B 9 3 kbps Car Noise 15 dB M T b11 EVRC B 9 3 kbps Car Noise 15 dB T T b12 EVRC B 9 3 kbps Car Noise 15 dB T M b13 EVRC
98. 41 Tx Magnitude dB 40 i i i d iii i i i i i i i il 1 2 3 4 10 10 10 10 Frequency Hz Figure 3 3 2 4 1 SO 70 ITU T P 311 P 341 Transmit Mask and Filter responses 3 89 8 Master Codec for SO 70 This section describes the C simulation of the speech codec specified by 1 The master codec C simulation used for verifying the performance of a non bit exact EVRC WB implementation shall be the floating point master C simulation included in the associated Software Distribution 1a 3 3 3 1 Compiling the Master Codec Simulation The source code for floating point simulation can be compiled using the GNU G compiler and make utility A G compatible makefile has been included in the appropriate sub directory in 1a Typing make this directory will compile and link the code and create the executable file called Evrc_wb 3 22 20 21 22 23 24 25 26 27 28 29 30 3GPP2 C S0018 E v2 0 Evrc wb exe on Win32 systems which will be placed in the same directory The included makefile may require some user modification for a particular hardware platform and or operating system 3 3 3 2 Running the Master Codec Simulation The EVRC WB floating point executable Evrc wb files use command line arguments to receive all information regarding input and output files and various parameters used during execution Executing Evrc wb with no command line a
99. 5 10 1 1 Typical Plot of MOS versus MNRU 2 118 Figure 2 5 10 2 1 Typical P 835 Score Profiles for Reference Conditions 2 119 Figure 3 1 Basic Test nennen nnne nnne nnns 3 1 Figure 3 2 Subjective Testing Equipment Configuration sse 3 2 Figure 3 3 2 4 1 SO 70 ITU T P 311 P 341 Transmit Mask and Filter responses 3 22 Figure 3 3 4 5 1 SO 70 Fixed point bit exact test suite decision 3 26 Figure 3 4 2 4 1 SO 73 ITU T P 311 P 341 Transmit Mask and Filter responses 3 36 Figure 3 4 4 5 1 SO 73 Fixed point bit exact test suite decision 3 40 Figure 3 5 2 4 1 SO 77 ITU T P 311 P 341 Transmit Mask and Filter responses 3 48 Figure 3 5 4 5 1 SO 77 Fixed point bit exact test suite decision 3 52 xiii 20 21 22 23 24 25 26 27 28 29 30 31 32 3GPP2 C S0018 E v2 0 LIST OF TABLES Table 2 1 2 3 1 1 SO 3 Listening Experiment 2 3 Table 2 1 2 3 1 2 SO Listening Experiment Design 2 4 Table 2 1 2 3 2 1 SO Listening Experiment II Conditions
100. 50018 v2 0 b08 involved in the P 835 Experiments 2 and 4 and ten reference conditions 01 10 involved in the P 835 Experiment 8 respectively In general results are expected for these reference conditions such that the obtained score profiles are similar to those shown in Figure 2 5 10 2 1 P 835 Scores w SNR 40dB Car Noise 50 B BAK OVRL P 835 Scores 20dB MNRU 40 dB gt w n MNRU 40dB P 835 Scores 20dB 40 dB SNR Car Noise gt w MNRU SNR SG 8 OVAL 10 10 dB 20 20 dB 30 30 dB 40 40 dB SNR Car Noise Figure 2 5 10 2 1 Typical P 835 Score Profiles for Reference Conditions 2 119 3GPP2 C S0018 E v2 0 This page intentionally left blank 2 120 21 22 23 24 25 26 3GPP2 C S0018 E v2 0 CODEC STANDARD TEST CONDITIONS This section describes the conditions equipment and the software tools necessary for the performance of the tests of Section 2 The software tools and the speech database associated with Sections 3 1 2 through 3 1 4 SO 3 3 2 2 through 3 2 4 SO 68 3 3 2 through 3 3 4 SO 70 3 4 2 through 3 4 4 SO 73 or 3 5 2 through 3 5 4 SO 77 can be found in the Software Distribution associated with this document The objective and subjective testing requires that speech data files can be input to the spe
101. 68 support T M d31 Car 15dB SNR Mode 6 interoperable with Mode 6 of SO 68 support M M d32 Car 15dB SNR Mode 6 interoperable with Mode 6 of SO 68 support M T d33 Car 15dB SNR Mode 6 interoperable with Mode 6 of SO 68 support T T d34 Car 15dB SNR Mode 6 interoperable with Mode 6 of SO 68 support T M d35 Street 15dB SNR Mode 6 interoperable with Mode 6 of SO 68 support M M d36 Street 15dB SNR Mode 6 interoperable with Mode 6 of SO 68 support M T d37 Street 15dB SNR Mode 6 interoperable with Mode 6 of SO 68 support TT d38 Street 15dB SNR Mode 6 interoperable with Mode 6 of SO 68 support T M d39 Babble 20dB SNR 2 FER Mode 6 interoperable with Mode 6 of SO 68 support M M d40 Babble 20dB SNR 296 FER Mode 6 interoperable with Mode 6 of SO 68 support M T d41 Babble 20dB SNR 2 FER Mode 6 interoperable with Mode 6 of SO 68 support TT d42 Babble 20dB SNR 296 FER Mode 6 interoperable with Mode 6 of SO 68 support T M 2 5 2 3 5 Subjective Experiment 5 for SO 77 The Test Parameters for Listening Experiment 5 are presented in Table 2 5 2 3 5 1 Table 2 5 2 3 5 1 SO 77 Listening Experiment 5 Test Parameters ACR P 800 Wideband ca of genres Background noise none ambient Audio Input Level 22 dB Filter characteristics P 341 refer Section 3 3 2 4 Reference conditions 4 Specified reference conditions Test conditions 0 FER and 3 FER Encoder Decoder Combinations 2 M M M T 2 99 3
102. 7 38 39 40 41 3GPP2 C S0018 E v2 0 8550 bps rate Rate 1 2 frames use the 4000 bps rate Rate 1 4 frames use the 2000 bps rate and Rate 1 8 frames use the 800 bps rate Rates for SO 77 Same as SO 73 ROLR Receive Objective Loudness Rating a measure of receive audio sensitivity ROLR is a frequency weighted ratio of the line voltage input signal to a reference encoder to the acoustic output of the receiver 17 defines the measurement of sensitivity and 18 defines the calculation of objective loudness rating Supra aural Headphones Headphones that cover but do not surround the entire ear Tmax The maximum undistorted sinusoidal level that can be transmitted through the interfaces between the EVRC and the PCM based network This is taken to be a reference level of 3 17 1 5 Test Model for the Speech Codec For the purposes of this standard a speech encoder is a process that transforms a stream of binary data samples of speech into an intermediate low bit rate parameterized representation As mentioned elsewhere in this document the reference method for the performance of this process is given in 1 This process may be implemented in real time as a software program or otherwise at the discretion of the manufacturer Likewise a speech decoder is a process that transforms the intermediate low bit rate parameterized representation of speech given 1 back into a stream of binary data samples suitab
103. 94312 1 5 4 1 p5 n4p5 f4p5 1 TA n4p6 5232827 f4p6 5342393 1 7 4 1 p7 n4p7 4 7 n1p8 4 4 p8 n4p8 f4p8 Length samples W ws n WO OY 11506 gt I 129258 7851 05606 339 723 10581 122664 1164 1274 10956 10880 118850 097 NO NO 6957 112149 01539 8876 8866 116617 130938 123975 115326 12153 10445 10290 sf po sf po W 3GPP2 C S0018 E v2 0 Table 2 3 4 5 2 Cutting Points for the astrip Software Tool for the SO 70 Experiments 2 4 and Start sample Length samples 1 57758 Sentence a 5 _ 2 02 288424 55912 2s02 3404062 59225 9 mso 3463288 51884 10 m3s02 515172 59593 8 8 9 1 16 18 15 6 P 835 Test Experiment II P 835 Start Length sample samples 60 3s10 3404618 58012 Start Length Sentence sample samples 66 2 12 3734538 55821 68 22512 3852015 60190 69 mssiip 3912115 55432 80 festa 4567054 55052 86 misis 4900324 51588 89 mzsi5 5080387 65752 90 mzsis 5146139 63251 95 3518 5508241 51792 Table 2 3 4 5 3 Composition of the Sentence Triad Samples for the Experiments 2 4 and 6 P 835 Test TA ince n TIG n ce n UE
104. A 396 FER For amp Rev M T a18 EVRC A 396 FER For amp Rev T M a19 EVRC A 396 FER For amp Rev T T a20 15 96 3 FER For amp Rev R R a21 EVRC A HR Max Nominal 22 dB M M a22 EVRC A HR Max Nominal 22 dB M T a23 EVRC A HR Max Nominal 22 dB T M a24 EVRC A HR Max Nominal 22 dB T T 2 4 3GPP2 C S0018 E v2 0 Label Operating Point Condition Enc Dec Connection a25 1S 96 C HR Max Nominal 22 dB R R a26 Reference MNRU 5aB a27 Reference MNRU 15aB a28 Reference MNRU 20aB a29 Reference MNRU 25aB a30 Reference G 728 a31 Reference u Law Source 2 2 1 2 3 2 Subjective Experiment Il for SO 3 3 The Test Conditions for Listening Experiment II are presented in Table 2 1 2 3 2 1 4 Table 2 1 2 3 2 1 SO 3 Listening Experiment Il Conditions Test conditions 1 Clean 2 Car Noise IRS at 15 dB S N 3 Street Noise flat at 12 dB S N 4 Office Babble flat at 20 dB S N 5 Tandem Number of codecs 5 MM T M T T IS 96 C Encoding stages single and tandem e The Test Design for Listening Experiment Il are presented in Table 2 1 2 3 2 2 7 Table 2 1 2 3 2 2 SO 3Listening Experiment Il Design Label Operating Point Condition Enc Dec Connection b01 EVRC A Clean Nominal 22 dB M M b02 EVRC A Clean Nominal 22 dB M T b03 EVRC A Clean Nominal 22 dB T M b04 EVRC A Clean Nominal 22 dB T T 2 5 3GPP2 C S0018 E v2 0
105. ARIB ARIB STD T64 C S0018 E v2 0 Minimum Performance Specification for the Enhanced Variable Rate Codec Speech Service Options 3 68 70 73 and 77 for Wideband Spread Spectrum Digital Systems Refer to Industrial Property Rights IPR in the preface of ARIB STD T64 for Related Industrial Property Rights Refer to Notice in the preface of ARIB STD T64 for Copyrights 1 5 8 Original Specification This standard ARIB STD T64 C S0018 E v2 0 was prepared by 3GPP2 WG of Association of Radio Industries and Businesses ARIB based upon the 3GPP2 specification C S0018 E v2 0 Modification to the original specification None Notes None 3GPP2 C S0018 E v2 0 3mRD GENERATION PARTNERSHIP PROJECT Z 3GPP2 March 2013 Minimum Performance Specification for the Enhanced Variable Rate Codec Speech Service Options 3 68 70 73 and 77 for Wideband Spread Spectrum Digital Systems 2013 3GPP2 3GPP2 and its Organizational Partners claim copyright in this document and individual Organizational Partners may copyright and issue documents or standards publications in individual Organizational Partner s name based on this document Requests for reproduction of this document should be directed to the 3GPP2 Secretariat at secretariat 3qpp2 org Requests to reproduce individual Organizational Partner s documents should be directed to that Organizational Partner See www 3gpp2 org for more information 3GP
106. Assessment of Quality Annex D Modified IRS Send and Receive Characteristics MIRS February 1996 13 ITU T Recommendation P 835 Subjective test methodology for evaluating speech 1 3 2 14 15 16 17 18 communication systems that include noise suppression algorithm November 2003 Informative References 3GPP2 C S0011 C Recommended Minimum Performance Standards for cdma2000 Spread Spectrum Mobile Stations March 2006 3GPP2 C S0010 C Recommended Minimum Performance Standards for cdma2000 Spread Spectrum Base Stations March 2006 TIA EIA 95 B R2004 Mobile Station Base Station Compatibility Standard for Wideband Spread Spectrum Cellular Systems October 2004 IEEE 269 2002 R2002 Standard Method for Measuring Transmission Performance of Telephone Sets Handsets and Headsets April 2003 IEEE STD 661 1979 R2008 EEE Standard Method for Determining Objective Loudness Ratings of Telephone Connections September 2008 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 3GPP2 50018 v2 0 19 ISO IEC 9899 1999 C2001 C2004 Programming Languages C 20 Dunnett C W A multiple comparison procedure for comparing several treatments with a control Journal of the American Statistical Association vol 50 1955 pp 1096 1121 21 SGPP TS 26 131 3rd Generation Partnership Project Technical Specification Group Services and System Aspects Termi
107. B SNR car noise condition and 12 dB SNR street noise The background noise has been introduced by mixing the clean speech recording with the noise recording at the appropriate levels The benchmark recording employed in the average data rate test is a single sided recording similar to a telephone conversation It exhibits an approximate voice activity factor of 0 35 The processed files are not used in the subjective portion of the experiment The length of each of the benchmark files is approximately 480 seconds 2 1 1 1 1 Average Data Rate Computation The average data rate for the test codec shall be computed for each of the benchmark files as follows 9600 N4 4800 No 1200 Ng N where number of frames encoded at Rate 1 No number of frames encoded at Rate 1 2 Ng number of frames encoded at Rate 1 8 and N1 Ng The total average data rate for the test codec is then given by 2 This section does not apply whenever a codec has demonstrated bit exactness See 3 1 4 3 2 4 3 3 4 or 3 4 4 2 1 Oo a A CQ m 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 3GPP2 C S0018 E v2 0 Ravg 0833 R babble noise segment 12dB R car noise segment 12dB R street noise segment 12dB R ambient background segment 12dB R babble noise segment 22dB R car noise segment 22dB R street noise segment 22dB R ambie
108. Calibration for SO 70 3 3 1 1 Audio Path The audio path for wideband test conditions Experiments 1 and 2 must meet the following requirements for electro acoustic performance measured between the output of the D A converter and the output of the headphone 1 Frequency response shall be flat to within 2 dB between 50 Hz and 7000 Hz and below 50 Hz the response shall roll off at a minimum of 12 dB per octave Equalization may be used in the audio path to achieve this A suitable reconstruction filter shall be used for playback 2 Total harmonic distortion shall be less than 1 for signals between 50 Hz and 8000 Hz 3 Noise over the audio path shall be less than 35 dBA measured at the ear reference plane of the headphone 4 Signal shall be delivered to the headphone on the listener s preferred telephone listening ear and the other ear shall be uncovered No signal shall be delivered to the other headphone The audio path for narrowband test conditions Experiments 3 4 5 and 6 must meet the following requirements for electro acoustic performance measured between the output of the D A converter and the output of the headphone 1 Frequency response shall be flat to within 2 dB between 200 Hz and 3400 Hz and below 200 Hz the response shall roll off at a minimum of 12 dB per octave Equalization may be used in the audio path to achieve this A suitable reconstruction filter shall be used for playback 2 Total harmonic distortion
109. Directory containing input test vectors so70 testvec source suiteA so70 testvec source suiteB so70 testvec source s so70 testvec source suiteD Files in the so70 testvec directory are provided for the purpose of qualifying a test codec as bit exact and conform to the file naming convention described in Section 2 2 4 The so70 testvec directory is divided into 2 subdirectories so70 testvec source and so70 testvec fixed The so70 testvec source directory contains input source files as well as packet files injected with frame erasures The so70 testvec fixed directory contains files processed with the EVRC WB fixed point reference software The files in these directories are the reference files for bit exact compliance A test codec is bit exact if it can reproduce all of the reference files in the Iso70 testvec fixed directory exactly The outputs of the encoder and decoder of the test codec are to be obtained for the conditions given below in Table 3 3 4 5 2 Table 3 3 4 5 9 The processing steps for these conditions are illustrated in Section 6 Table 3 3 4 5 2 SO 70 Encoder Suite A Bit exact Test Conditions ae Operating Point Condition src s22 EVRC WB operating point 0 Nominal 22 dB evrc wb op0 p22 16 kHz sampling src s12 EVRC WB operating point 0 High 12 dB
110. ER 2 1 4 4 Tandem Conditions The clear channel tandem condition shall be performed by e encoding the appropriate source file e decoding the encoder s output file 2 9 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 3GPP2 C S0018 E v2 0 e normalizing signal power to 22dB e companding the modified decoded speech file to u law PCM format e encoding the law PCM companded version of the decoded speech file e decoding the resultant encoder s output file to generate the processed speech file e normalizing signal power to 22dB e companding the modified decoded speech file to u law PCM This process is performed for each combination of master encode test decode test encode master decode and test encode test decode The master test combinations for tandem processing represent master encode test decode master encode test decode and vice versa for the test master combination The master encode master decode files are provided The following four conditions are tested e MM e e T M T M e T T T T To expedite processing it may be possible to use the output files for Experiment 11 condition 1 tm1 and tt1 as the input for the three test combinations It is also worth noting that the front end algorithmic delay through the master codec is 13 ms or 104 samples which can be accounted for by the noise suppression overlap delay plus the LPC
111. Experiment 8 are presented in Table 2 3 2 3 8 1 4 Table 2 3 2 3 8 1 SO 70 Listening Experiment 8 Test Parameters Type of test ncn P 800 Narrowband Number of genres Background noise none ambient Audio Input Level 22 dB Filter characteristics MIRS Reference conditions 4 Specified reference conditions Test conditions 0 FER and 3 FER Encoder Decoder Combinations 2 M M M T e The Test Conditions for Listening Experiment 8 are presented in Table 2 3 2 3 8 2 7 Table 2 3 2 3 8 2 SO 70 Listening Experiment 8 Test Conditions Exp 8 Narrowband Music File Reference Condition h01 MNRU 10dB Reference h02 MNRU 20dB Reference h03 MNRU 30dB Reference h04 Source Reference File Test Condition Enc Dec h05 0 FER M M h06 0 FER M T h07 3 FER M M h08 3 FER M T 23 2 8 Numerical Parameters for the SO 70 Listening Experiments 10 Table 2 3 2 3 9 1 describes the resultant numerology that is used for the eight SO 70 listening 11 experiments The first column is the description of the parameter and columns 2 to 9 show the 12 numerical value for each of the parameters for the eight listening experiments For each listening experiment the different Encode Decode Test conditions include various interconnections between 14 the Master and Test Encoders and the Master and Test Decoders There are eight reference 2 50 3GPP2 C S0018 E v2 0 conditions in each of the experiments1 thr
112. F The software Evrc wb iwf cc can be compiled to yield a simulation utility Evrc wb iwf with usage defined as Evrc wb iwf s signaling mask fil i encoded packet fil o dimmed packet fil where Evrc wb converts full rate frames the input encoded packet file to half rate frames at packet level that is using a simple scaling down of the packet instead of a complicated transcoding method 3 3 2 4 P 341 Tx Filter The software utility p341_tx c can be compiled to yield a Tx filtering utility p341 tx with usage defined as 3 21 3GPP2 C S0018 E v2 0 p341 tx input file name output file name where p341_ tx is the 3GPP2 Tx filter compliant to ITU T P 341 Figure 3 3 2 4 1 shows the frequency response of p341 tx filter Also shown in this figure is the response of the ITU T P 341 STL 2000 filter implementation as well as the transmit masks for the ITU T P 341 P 311 and the wideband transmit response from Table 9 in the 3GPP electro acoustics specification 21 From this figure it can be seen that the STL 2000 filter response in red does not meet the frequency response of the 3GPP electro acoustics specification while the p341 tx filter response in green meets both the P 341 P 311 masks as well as the 3GPP electro acoustics specification mask ITU T 3GPP Transmit Masks vs Filter Responses 20 STL 2000 P 341 3GPP2 P 341 Tx 3GPP Handset Send 5dB ITU T P 311 3
113. GPP2 C S0018 E v2 0 2 The Test Conditions for Listening Experiment 5 are presented in Table 2 5 2 3 5 2 3 Table 2 5 2 3 5 2 SO 77 Listening Experiment 5 Test Conditions Exp 5 Wideband Music File Reference Condition 01 MNRU 15dB Reference 02 MNRU 25dB Reference e03 MNRU 35dB Reference e04 Source Reference File Test Condition Enc Dec e05 0 FER M M e06 0 FER M T e07 3 FER M M e08 3 FER M T 4 2 5 2 8 B0 Subjective Experiment 6 for SO 77 s The Test Parameters for Listening Experiment 6 are presented in Table 2 5 2 3 6 1 6 Table 2 5 2 3 6 1 SO 77 Listening Experiment 6 Test Parameters 04 Audio Input Level 22 dB Filter characteristics MIRS Reference conditions 4 Specified reference conditions Test conditions 096 FER and 396 FER Encoder Decoder Combinations 2 M M M T The Test Conditions for Listening Experiment 6 are presented in Table 2 5 2 3 6 2 2 100 2 3 3GPP2 C S0018 E v2 0 Table 2 5 2 3 6 2 SO 77 Listening Experiment 6 Test Conditions Exp 6 Narrowband Music File Reference Condition f01 MNRU 10dB Reference 02 MNRU 20dB Reference f03 MNRU 30dB Reference 04 Source Reference File Test Condition Enc Dec 05 0 FER 06 096 FER M T f07 396 FER M M 08 396 FER M T 2 5 2 3 7 Subjective Experiment 7 for SO 77 The Test Parameters for Listening Experiment 7 are presented
114. Here the pc pb and ps files are the master packet files for the c15 b20 and s15 source files respectively For the master encode master decode directories so68 subjctv exp m m the naming convention of the speech files is such that the first two characters of the file name indicate the codec combination and the suffix indicates the condition numbers in Table 2 2 2 3 1 2 and Table 2 2 2 3 2 2 Detailed descriptions of all processing operations are given in Section 6 2 23 20 21 22 23 24 25 26 27 28 29 30 31 32 33 3GPP2 C S0018 E v2 0 2 2 41 Encoding by the Test Codec All of the source files will be encoded by the test codec to produce encoded packet files For ease of reference it is recommended that directories so68 subjctv exp1 t_pkt and so68 subjctv exp2 t pkt be created to deposit the test encoder output packets and that the naming conventions be made consistent with the master codec 2 2 4 2 Decoding by the Master Test Codecs The encoded packet files generated from the various encoders conditions shall be processed through the master and test decoders 2 2 4 3 Introduction of Impairments For the 3 frame error condition Experiment condition d the impaired master codec encoded packet files are provided in the so68 subjctv exp1 m_pkt directory Unlike other conditions this condition uses only the test decoder and not the test encoder For the Dim and Burs
115. IER PT Total Stimuli Experiment 1536 1152 3072 2016 96 1596 1182 Listening Pans e e e a a a e oo SimuiperliteringPend 192 144 252 24 192 ET EROR DUI E Listeners voters perListeningPanei 4 a 4 4 e e 4 Votes per Gonditon 258 see ase 192 oo oo 256 192 2 5 8 Speech Material for SO 77 Testing The source speech files used for SO 77 compliance testing consist of Harvard sentences pairs which are preprocessed to include proper level adjustment and noise mixing for use in the subjective experiments The talkers used in these files consist of adult males and adult females and are native speakers of North American English For the following discussion it may be useful to refer to Table 4 5 for the composition of the Software Distribution database 2 104 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 3GPP2 50018 v2 0 The source speech material for subjective Experiments is contained directory so77 subjctv exp source Each file has been appropriately pre filtered level adjusted and noise processed These files are named src The speech database also includes samples processed through the various reference conditions in directory so77 subjctv exp ref The reference conditions are named ref for the respective conditions given in
116. Interworking Function IWF The software EvrcB_iwf cc can be compiled to yield a simulation utility EvrcB iwf with usage defined as EvrcB iwf s signaling mask fil i encoded packet fil o dimmed packet fil where EvrcB iwf converts full rate frames the input encoded packet file to half rate frames at packet level that is using a simple scaling down of the packet instead of a complicated transcoding method 3 2 8 Master Codec for SO 68 This section describes the C simulation of the speech codec specified by 1 The master codec C simulation used for verifying the performance of a non bit exact EVRC B implementation shall be the floating point master C simulation included in the associated Software Distribution 1a 3 2 3 1 Compiling the Master Codec Simulation The source code for floating point simulation can be compiled using the GNU G compiler and make utility A G compatible makefile has been included in the appropriate sub directory in 1a Typing make this directory will compile and link the code and create the executable file called EvrcB EvrcB exe on Win32 systems which will be placed in the same directory The included makefile may require some user modification for a particular hardware platform and or operating system 3 2 8 2 Running the Master Codec Simulation The EVRC B floating point executable EvrcB files use command line arguments to receive all information regarding input and ou
117. LB portion of Wideband mode decoder test M M only d10 Car 15dB SNR Mode 0 LB portion of Wideband mode decoder test M T only d11 Car 15dB SNR Mode 1 interoperable with Mode 0 of SO 68 support M M di2 Car 15dB SNR Mode 1 interoperable with Mode 0 of SO 68 support M T d13 Car 15dB SNR Mode 1 interoperable with Mode 0 of SO 68 support T T 914 Car 15dB SNR Mode 1 interoperable with Mode 0 of SO 68 support T M di5 Car 15dB SNR Mode 7 interoperable with Mode 0 of SO 68 support T T di6 Car 15dB SNR Mode 7 interoperable with Mode 0 of SO 68 support T M di7 Car 15dB SNR Mode 7 interoperable with Mode 0 of SO 68 support M M 918 Car 15dB SNR Mode 7 interoperable with Mode 0 of SO 68 support M T d19 Street 15dB SNR Mode 0 LB portion of Wideband mode decoder test M M only d20 Street 15dB SNR Mode 0 LB portion of Wideband mode decoder test M T only d21 Street 15dB SNR Mode 1 interoperable with Mode 0 of SO 68 support M M d22 Street 15dB SNR Mode 1 interoperable with Mode 0 of SO 68 support M T d23 Street 15dB SNR Mode 1 interoperable with Mode 0 of SO 68 support T T 2 73 1 2 3GPP2 50018 v2 0 Exp 4 Narrowband P 835 d24 Street 15dB SNR Mode 1 interoperable with Mode 0 of SO 68 support T M d25 Babble 20dB SNR 2 FER Mode 0 LB portion of Wideband mode decoder test M M only d26 Babble 20dB SNR 2 FER Mode 0 LB portion of
118. O 70 2 36 Xe AX 2 38 2 3 2 2 Method of Measurement nennen erre 2 39 2 3 2 3 Test Conditions and Test Design for SO 70 sss 2 39 2 3 2 3 1 Subjective Experiment 1 for SO 70 sse 2 39 2 3 2 8 Subjective Experiment 2 for SO 70 ssssssssssseeeeeene 2 41 2 3 2 3 3 Subjective Experiment for SO 70 2 43 2 3 2 3 4 Subjective Experiment 4 for SO 70 sssssssssssseeeeneen 2 44 2 3 2 3 5 Subjective Experiment 5 for SO 70 sse 2 46 2 3 2 3 6 Subjective Experiment 6 for SO 70 2 47 2 3 2 3 7 Subjective Experiment 7 for SO 70 1 2 49 2 3 2 3 8 Subjective Experiment 8 for SO 70 1 2 50 2 3 2 3 9 Numerical Parameters for the SO 70 Listening Experiments 2 50 23 3 Speech Material for SO 70 Testing nnne 2 51 20 21 22 23 24 25 26 27 28 29 30 31 32 3GPP2 C S0018 E v2 0 2 3 4 Processing of Speech Material for SO 70 Testing A 2 51 2 8 1 Encoding by the Test Codec 2 52 2 3 4 2 Decoding by the Master Test Codecs 2 52 2 3 4 3 Introduction of Impairments nnnnnnnnnnnnnnn nnne nennen tnn enenatis 2 52 2 3 4 4 Ensuring Proper Encoded Frame Packet Files mm 2 52 2 3 4 5 Post processing of test condition output
119. P2 C S0018 E v2 0 REVISION HISTORY Revision Description Date C S0018 0 v1 0 Minimum Performance Specification for the Enhanced Variable Rate Codec Speech Service Option 3 for Spread Spectrum Digital Systems December 1999 C S0018 B v1 0 Minimum Performance Specification for the August 2007 Enhanced Variable Rate Codec Speech Service Options 3 and 68 for Spread Spectrum Digital Systems C S0018 C v1 0 Minimum Performance Specification for the December 2007 Enhanced Variable Rate Codec Speech Service Options 3 68 and 70 for Spread Spectrum Digital Systems C S0018 D v1 0 Minimum Performance Specification for the Enhanced Variable Rate Codec Speech Service Options 3 68 70 and 73 for Spread Spectrum Digital Systems January 25 2010 C S0018 E v1 0 Minimum Performance Specification for the December 2011 Enhanced Variable Rate Codec Speech Service Options 3 68 70 73 and 77 for Spread Spectrum Digital Systems C S0018 E v2 0 Minimum Performance Specification for the March 2013 Enhanced Variable Rate Codec Speech Service Options 3 68 70 73 and 77 for Spread Spectrum Digital Systems 3GPP2 C S0018 E v2 0 FOREWORD This foreword is not part of this document This document specifies the procedures to test implementations of EVRC A EVRC B EVRC WB EVRC NW or EVRC NW2K compatible variable rate speech codecs either by meeting the bit exact implementation or meeting
120. Point Codec Simulation The source code for the fixed point codec simulation has been written in C and can be compiled using any general purpose compiler such as the GNU G compiler and make utility Two GCC compatible makefiles have been included in the build directory Typing make in the build directory will compile and link the code and create the executable file called Evrc_nw_fx Evrc nw fx exe on Win32 systems which will be placed in the build directory The included makefiles may require some user modification for a particular hardware platform and or operating system 3 5 43 Running the Fixed Point Codec Simulation The EVRC NW2K executable files use command line arguments to receive all information regarding input and output files and various parameters used during execution Executing Evrc nw fx with no command line arguments will display a brief description of the required and optional command line arguments The options are described below i infn required Specifies the name of the input speech file or the name of the input packet file if only decoding is being performed see the d option below o outf required Specifies the name of the output speech file or the name of the output packet file if only encoding is being performed see the e option below 3 50 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 3GPP2 C S0018 E v2 0 d Instructs the simulat
121. RATE REDUC 111 3 29 1 5 The above table provides the maximum allowable average channel rate including full half and eighth rate for the different operating points These maximum allowable average channel rates were obtained by processing the 7 wide band benchmark files for the 16 kHz case and 6 narrow band benchmark files for the 8kHz case through the master floating point software See Section 3 3 2 1 for details in using the provided software tool that can be used to aid in making this calculation 2 3 1 1 2 Average Data Rate Requirement for SO 70 The total average data rate Rayg for each operating point shall not exceed the target average data rate by more than the tolerance level in Table 2 3 1 1 1 1 otherwise the test codec fails the compliance test 2 3 1 2 Unity Gain Requirement The specific EVRC WB test codec shall output speech with unity gain when compared with the input speech The unity gain measurement output active speech level input active speech level will be performed over the entire input speech database for the clean nominal level source conditions for each mode The measurement should be made using the STL 2000 tool 6 6a actlev and must not 2 35 20 21 22 23 24 25 26 27 28 29 30 31 32 3GPP2 C S0018 E v2 0 show more than 0 5 dB deviation between input and output active speech levels This procedure is fully described in 9 2 3 1 3 End to end Algorithmic D
122. Rate Requirement for SO 73 2 65 2 4 1 2 Unity Gain Requirement 2 65 2 4 1 3 End to end Algorithmic Delay Recommendation e 2 65 2 4 2 Subjective Performance Testing for SO 73 nnne 2 65 24 2 1 BYETI NA ONA kad ki kap rr errr sere err errr ere rer ER 2 67 2 4 2 2 Method of ener enne 2 67 2 4 2 3 Test Conditions and Test Design for SO 73 sss 2 67 2 4 2 3 1 Subjective Experiment 1 for SO 7 2 67 2 4 2 8 Subjective Experiment 2 for SO 73 2 69 2 4 2 3 3 Subjective Experiment for SO 7 2 70 vi 20 21 22 23 24 25 26 27 28 29 30 31 32 33 3GPP2 C S0018 E v2 0 2 4 2 3 4 Subjective Experiment 4 for SO 73 2 72 2 4 2 3 5 Subjective Experiment 5 for SO 73 2 74 2 4 2 3 6 Subjective Experiment 6 for SO 73 2 75 2 4 2 8 7 Numerical Parameters for the SO 73 Listening Experiments 2 76 2 4 3 Speech Material for SO 73 Testing nnne 2 77 2 4 4 Processing of Speech Material for SO 73 Testing M 2 77 2 4 4 1 Encoding by the Test Codec tnnt 2 78 2 4 4 2 Dec
123. SO 70 Listening experiments 1 3 5 for SO 70 are performed as ACR listening tests Experiments 2 4 and 6 for SO 70 are performed as P 835 listening tests 2 3 2 3 1 Subjective Experiment 1 for SO 70 The Test Parameters for Listening Experiment 1 are presented in Table 2 3 2 3 1 1 Condition Description 4 males 4 females none ambient Reference conditions 8 Specified reference conditions Test conditions Low Audio Input Level 32 dB 1 d amp b Nominal Audio Input Level 22 dB High Audio Input Level 12 dB 3 FER and 1 FER 2 pls at Nominal Audio Input Level 22 Encoder Decoder Combinations 4 M M M T T T T M Table 2 3 2 3 1 1 SO 70 Listening Experiment 1 Test Parameters Condition Description Type of test MOS P 800 Wideband Number of talkers 4 males 4 females Background noise none ambient Audio Input Level 22 dB 32 dB 12 dB Filter characteristics P 341 refer Section 3 3 2 4 Reference conditions 8 Specified reference conditions Test conditions o Low Audio Input Level 32 dB 1 d amp b o Nominal Audio Input Level 22 dB 2 39 3GPP2 50018 v2 0 Condition Description o High Audio Input Level 12 dB o 3 FER and 1 FER 2 pls at Nominal Audio Input Level 22 Encoder Decoder Combinations 4 MM M T T T T M The Test Conditions for Listening Experiment 1 are presented in Table 2 3 2 3 1 2 Table 2 3 2 3 1 2 SO 70 Listening Exp
124. T a19 High level T T a20 High level T M a21 196 FER 196 PLS M M a22 196 FER 196 PLS M T a23 3 FER M M a24 3 FER M T 2 68 2 5 2 4 2 8 Subjective Experiment 2 for SO 73 3GPP2 C S0018 E v2 0 The Test Parameters for Listening Experiment 2 are presented in Table 2 3 2 3 2 1 Table 2 4 2 3 2 1 SO 74 Listening Experiment 2 Test Parameters 3 males 3 females Type of test Number of talkers Reference conditions 8 Specified reference conditions Test conditions o Car Noise 10 dB SNR o Car Noise 20 dB SNR 296 FER o Street Noise 15 dB SNR o Babble noise 20 dB S N Encoder Decoder Combinations 4 M M T T T M The Test Conditions for Listening Experiment 2 are presented in Table 2 4 2 3 2 2 Table 2 4 2 3 2 2 SO 73 Listening Experiment 2 Test Conditions Exp 2 Wideband P 835 Reference Conditions File MNRU SNR 01 MNRU 40aB SNR 40dB Reference b02 MNRU 40aB SNR 20dB Reference b03 MNRU 40aB SNR 0dB Reference 004 MNRU 0dB SNR 40dB Reference b05 MNRU 20aB SNR 40dB Reference b06 MNRU 10aB SNR 10dB Reference b07 MNRU 20aB SNR 20dB Reference b08 MNRU 40aB SNR 30dB Reference Test Conditions File Condition Enc Dec b09 Car 10dB SNR M M b10 Car 10dB SNR M T bi 1 Car 10dB SNR T T b12 Car 10dB SNR T M 2 69 2 3 6 3GPP2 C S0018 E v2 0 Exp 2 Wideband P 835 b13 Car 20
125. The IS 96 C codec specified in 2 is included for all conditions as an additional codec 2 1 2 3 1 Subjective Experiment for SO 3 The Test Conditions for Listening Experiment are presented in Table 2 1 2 3 1 1 Table 2 1 2 3 1 1 SO 3 Listening Experiment Conditions 4 males 4 females REESE Filter characteristics IRS Reference conditions law source 5 15 20 25 dBQ G 728 2 3 3GPP2 C S0018 E v2 0 Test conditions 1 Clean 2 High Audio Input Level 12 dB 4 396 FER forward and reverse 5 Rate 1 2 Maximum 1 2 3 Low Audio Input Level 32 dB 4 S Number of codecs 5 M M T M M T T T IS 96 C Encoding stages The Test Design for Listening Experiment are presented in Table 2 1 2 3 1 2 Table 2 1 2 3 1 2 SO 3 Listening Experiment Design Label Operating Point Condition Enc Dec Connection 01 EVRC A Clean Nominal 22 dB M M a02 EVRC A Clean Nominal 22 dB M T a03 EVRC A Clean Nominal 22 dB T M a04 EVRC A Clean Nominal 22 dB T T a05 IS 96 C Clean Nominal 22 dB R R a06 EVRC A High 12 dB M M a07 EVRC A High 12 dB M T a08 EVRC A High 12 dB T M a09 EVRC A High 12 dB T T a10 15 96 High 12 dB R R a11 EVRC A Low 32 dB M M a12 EVRC A Low 32 dB M T a13 EVRC A Low 32 dB T M 14 EVRC A Low 32 dB T T a15 15 96 Low 32 dB R R a16 EVRC A 396 FER For amp Rev M M 17 EVRC
126. The Test Conditions for Listening Experiment 2 are presented in Table 2 3 2 3 2 2 2 41 3GPP2 C S0018 E v2 0 Table 2 3 2 3 2 2 SO 70 Listening Experiment 2 Test Conditions Exp 2 Wideband P 835 Reference Conditions File MNRU SNR 01 MNRU 40aB SNR 40dB Reference b02 MNRU 40dB SNR 20dB Reference b03 MNRU 40dB SNR 0dB Reference b04 MNRU 0dB SNR 40dB Reference b05 MNRU 20dB SNR 40dB Reference b06 MNRU 10dB SNR 10dB Reference b07 MNRU 20dB SNR 20dB Reference b08 MNRU 40dB SNR 30dB Reference Test Conditions File Condition Enc Dec b09 Car 10dB SNR M M b10 Car 10dB SNR M T bit Car 10dB SNR T T b12 Car 10dB SNR T M b13 Car 20dB SNR 2 FER M M b14 Car 20dB SNR 2 FER M T b15 Car 20dB SNR 2 FER T T b16 Car 20dB SNR 2 FER T M b17 Street 15dB SNR M M bi8 Street 15dB SNR M T b19 Street 15dB SNR T T b20 Street 15dB SNR T M b21 Babble 20dB SNR M M b22 Babble 20dB SNR M T b23 Babble 20dB SNR T T b24 Babble 20dB SNR T M 2 42 1 2 5 3GPP2 C S0018 E v2 0 2 3 2 3 3 Subjective Experiment 3 for SO 70 The Test Parameters for Listening Experiment 3 are presented in Table 2 3 2 3 3 1 Table 2 3 2 3 3 1 SO 70 Listening Experiment 3 Test Parameters Type oftest ACR P80 Narrowband 4 males 4 females Type of test Reference conditions 8 Specified reference conditions Test condi
127. The input file must contain packets of compressed data e Instructs the simulation to perform only the encoding function The output file will contain packets of compressed data If neither the d or the e option is invoked the coder performs both the encoding and decoding functions by default 3 24 20 21 22 23 24 25 26 27 28 29 30 3GPP2 C S0018 E v2 0 M max Sets the maximum allowable data rate to max where max is element of 4 3 1 using the codes specified in the first column of Table 3 1 3 3 1 m min Sets the minimum allowable data rate to min where min is element of 4 3 1 using the codes specified in the first column of Table 3 1 3 3 1 If neither the M nor m option is invoked the coder allows the data rate to vary between Rate 1 and Rate 1 8 In addition if max min the data rate varies between max and min using the same rate decision algorithm where the data rate is set to max if the selected data rate is gt max and the data rate is set to min if the selected data rate is min 3 3 4 4 File Formats Files of speech contain 2 s complement 16 bit samples with the least significant byte first The packet file contains twelve 16 bit words with the low byte ordered first followed by the high byte The first word in the packet contains the data rate while the remaining 11 words contain the encoded speech data packed in accordance with the tables specified in 1 The packet f
128. The listeners shall listen to each sample and rate the quality of the test sample using a five point scale with the points labeled 5 Excellent 4 Good 3 Fair 2 Poor 1 Bad Data from 32 listeners shall be used for Experiments 1 and 3 four listeners for each listening panel where each listening panel uses a different randomization Before starting the test the listeners should be given instructions for performing the subjective test An example set of instructions for the ACR are presented in Figure 2 4 8 1 1 The instructions may be modified to allow for variations in laboratory data gathering apparatus 2 82 3GPP2 C S0018 E v2 0 This is an experiment to determine the perceived quality of speech over the telephone You will be listening to a number of recorded speech samples spoken by several different talkers and you will be rating how good you think they sound Use the single headphone on the ear you normally use for the telephone On each trial a two sentence sample will be played After you have listened to the sample determine the category from the list below which best describes the overall quality of the sample Press the numeric key on your keyboard corresponding to your rating for how good or bad that particular passage sounded The quality of the speech should be rated according to the scale below 5 Excellent Good 4 3 Fair 2 Poor 1 Bad During the session you will hear samples varying in different aspects
129. U 25aB b30 Reference G 728 b31 Reference u Law Source 2 6 3GPP2 C S0018 E v2 0 1 2 1 2 3 3 Numerical Parameters for SO Listening Experiments 2 Table 2 1 2 3 3 1 describes the resultant numerology that is used for each of the two SO listening s X experiments The first column is a variable name given to each of the parameters the second column 4 isthe description of the parameter the third column shows the required calculation for determining the s X value of the parameter if it is dependent upon other parameter values and the last column shows the e numerical value for each of the parameters For each listening experiment four codecs plus the IS 96 C codec are evaluated The number of reference conditions in each of the two listening experiments is six and the number of test conditions is five Table 2 1 2 3 3 1 Numerical Parameters for SO 3 Listening Experiments Parameter Calculation Experiment Experiment Il Value Value UN o2__ oodecTestGondiions 03_ Reterence Conditions e e cs rakes pe co BimupeTaer e Co eSesos C istenes voes C12 isteners Voters per File Session encas 8 JvotesperCond on _______ s2 sr 11 2 1 3 Source Speech Material for SO 3 Testing 12 All source material is derived from the Harvard Sentence Pair Database and matched in overall level There are a total of 64 original source f
130. a certain course of action is preferred but not necessarily required or that in the negative form a certain possibility or course of action is discouraged but not prohibited May and need not indicate a course of action permissible within the limits of the document Can and cannot are used for statements of possibility and capability whether material physical or causal 1 3 References The following standards contain provisions which through reference in this text constitute provisions of this Standard At the time of publication the editions indicated were valid All standards are subject to revision and parties to agreements based on this Standard are encouraged to investigate the possibility of applying the most recent editions of the standards indicated below ANSI 3GPP2 TIA and ITU T maintain registers of currently valid national and international standards published by them 1 3 1 Normative References 1 3GPP2 C S0014 E v1 0 Enhanced Variable Rate Codec Speech Service Options 3 68 70 73 and 77 for Wideband Spread Spectrum Digital Systems September 2011 1a 3GPP2 C S0014 D v2 0 Enhanced Variable Rate Codec Speech Service Options 3 68 70 and 73 for Wideband Spread Spectrum Digital Systems January 2010 1 3 20 21 22 23 24 25 26 28 29 30 31 32 33 34 35 36 37 3GPP2 C S0018 E v2 0 1b 6a 11 12 3GPP2 C R0014 C v1 0 Software Distribution
131. a rate is gt max and the data rate is set to min if the selected data rate is lt min W target active speech channel adr Specifies the target active speech channel average data rate in bps that the EVRC B encoder should target For example W 7500 for 7 5 kbps 3 17 22 23 24 25 3GPP2 50018 v2 0 3 2 4 4 File Formats Files of speech contain 2 s complement 16 bit samples with the least significant byte first The packet file contains twelve 16 bit words with the low byte ordered first followed by the high byte The first word in the packet contains the data rate while the remaining 11 words contain the encoded speech data packed in accordance with the tables specified in 1 The packet file value for each data rate is shown in Table 3 2 3 3 1 Unused bits are set to 0 For example in a Rate 1 8 frame the packet file will contain the word 0x0100 byte swapped 0x0001 followed by one 16 bit word containing the 16 data bits for the frame in byte swapped form followed by ten 16 bit words containing all zero bits 3 2 4 5 Verifying Bit Exact Performance of the Fixed Point Test Codec Files in the so68 testvec directory are provided for the purpose of qualifying a test codec as bit exact and conform to the file naming convention described in Section 2 2 4 The so68 testvec directory is divided into 2 subdirectories so68 testvec source and so68 testvec fixed The s068 testvec source directory c
132. adr gt Specifies the target active speech channel average data rate in kbps that the EVRC B encoder should target For example W 7 5 for 7 5 kbps 3 2 3 8 File Formats for SO 68 Files of speech contain 2 s complement 16 bit samples with the least significant byte first The packet file contains twelve 16 bit words with the low byte ordered first followed by the high byte The first word in the packet contains the data rate while the remaining 11 words contain the encoded speech data packed in accordance with the tables specified in 1 The packet file value for each data rate is shown in Table 3 2 3 3 1 Table 3 2 3 3 1 Packet File Structure From Master Codec Channel Error Model Value in Packet File Rate Data Bits per Frame 4 030004 171 s oxo0s u Unused bits are set to 0 For example in a Rate 1 8 frame the packet file will contain the word 0x0100 byte swapped 0x0001 followed by one 16 bit word containing the 16 data bits for the frame in byte swapped form followed by ten 16 bit words containing all zero bits 3 2 4 Fixed Point Bit Exact Codec for SO 68 This section describes the C simulation of the speech codec specified by 1 The speech codec C simulation is based on finite precision fixed point arithmetic operations and is recommended to be used as a reference codec to verify the performance of a bit exact EVRC B implementation of the fixed point C simulation of a test codec The bit exact EVRC B cod
133. ake all in that directory A GCC compatible makefile has been provided for this purpose in the so3 tools directory The makefile creates the executables avg rate exe mu exe and sv56 exe in the so3 tools bin directory This makefile may need to be modified to conform to the user s hardware platform Those non 3GPP2 supplied tools I mu exe and sv56 exe available in C code form from 6 and 6a and compiled using GCC are identified and are to be used supplementary to those available on the Software Distribution The program descriptions that follow all use the convention of enclosing optional command line arguments in angle brackets lt gt 3 1 2 1 Average Data Rate Determination Utility avg rate c This utility program is used to determine the average data rate at which a test codec encodes a set of benchmark speech files The source code avg rate c is a 3GPP2 supplied tool and is located the so3 tools avg rate directory of the associated Software Distribution The input to the program is a list of packet file names where each packet file referred to in the list conforms to the format described in Section 3 1 3 3 The output of the program is for each file referred to by the input file list The file name the number of packets contained in the file and the average data rate calculated as described in Section 2 1 1 1 The average data rate utility is intended to be used on the packet files created by
134. al 22 dB M M c14 Mode 6 Nominal 22 dB M T c15 Mode 6 Nominal 22 dB T T c16 Mode 6 Nominal 22 dB T M c17 Mode 7 Nominal 22 dB M M c18 Mode 7 Nominal 22 dB M T c19 Mode 7 Nominal 22 dB T T c20 Mode 7 Nominal 22 dB T M c21 Mode 1 Low 32 dB 1 d amp b 1 pls M M c22 Mode 1 Low 32 dB 1 d amp b 1 pls M T c23 Mode 1 Low 32 dB 1 d amp b 1 pls T T c24 Mode 1 Low 32 dB 1 d amp b 1 pls T M c25 Mode 6 Low 32 dB 1 d amp b 1 pls M M c26 Mode 6 Low 32 dB 1 d amp b 1 pls M T c27 Mode 6 Low 32 dB 1 d amp b 1 pls T T c28 Mode 6 Low 32 dB 1 d amp b 1 pls T M c29 Mode 1 High 12 dB M M c30 Mode 1 High 12 dB M T c31 Mode 1 High 12 dB T T c32 Mode 1 High 12 dB T M c33 Mode 6 High 12 dB M M c34 Mode 6 High 12 dB M T c35 Mode 6 High 12 dB T T c36 Mode 6 High 12 dB T M c37 Mode 1 Nominal 22 dB 396 FER M M 2 71 2 3 6 3GPP2 50018 v2 0 Label Opseatn Condition pog c38 Mode 1 Nominal 22 dB 396 FER M T c39 Mode 6 Nominal 22 dB 396 FER M M c40 Mode 6 Nominal 22 dB 396 FER M T c41 Mode 0 Nominal LB portion of Widemode mode decoder M M test only c42 Mode 0 Nominal LB portion of Widemode mode decoder M T test only c43 Mode 0 Low 1 D amp B 1 PLS LB portion of Widemode M M mode decoder test only c44 Mode 0 Low 1 D amp B 1 PLS LB portion of Widemode M T mode decoder test only c45 Mode 0 H
135. ample Randomization for the Experiments 1 3 and 7 ACR Test y o f4s8 0251 251 1255 f2s4 1151 157 1355 357 1152 f4s5 m4s8 f2 m4 n gt N N o rh NS nln oy E owe Ol BS Fe fan s6 o Fh Q m4s7 fls5 1251 157 1351 w n 2037251 a08m3s7 a22m2s6 a21 F4 58 2102231 a23m2s3 a06m2s4 a05 F4 58 al 94356 Im 352 a09f1s7 m3s4 a2imis2 a03m3s4 a21m2s6 al7m3s4 a04f4s1 vo oe Ron Ron Re a06 a04 al4 17 a05 a01 a07 al0 a22 a20 a23 a02 al6 al2 a08 al8 a21 all a24 al9 al3 a09 al5 F2s6 m4s4 252 1158 fls6 mis6 357 1251 254 1457 358 1255 457 1455 452 1252 fls8 m3s4 f4s6 m3s4 f1ls2 mis6 358 7 a 01 255 a05f3s8 a09f4s8 a06m3s5 alom287 a03f4s7 L2 252 al5f2s4 a23m4s2 01m3s3 a20m3s1 a09f2s4 al2m1s8 13 353 al4f1s4 m4s7 a21m4s3 al3m2s2 al0f3s2 a07m4s5 16 152 a20mis4 11 451 al5m4s5 al8f3s1 a04m1s3 24 152 al4m3s4 al7f2s2 a05m2s6 al9f4sl 2062184 2021184 2072256 The randomization lists for each of the eight listening panels for each experiment are provided so77 subjctv exp data play Ist 2 5 6 Presentation Presentation of speech materials
136. ar reference plane of the headphone 4 Signal shall be delivered to the headphone on the listener s preferred telephone listening ear and the other ear shall be uncovered No signal shall be delivered to the other headphone 3 5 1 2 Calibration The audio circuit shall deliver an average sound level of the stimuli to the listener at 18 dBPa 76 dB SPL at the ear reference plan This level was chosen because it is equivalent to the level delivered by a nominal ROLR handset driven by the average signal level on the PSTN network This level may be calibrated using a suitable artificial ear with circum aural headphone adapter and microphone A test file with a reference signal is included with the source speech database for the purpose of calibration The file cal 1004 16k is located in the directory so77 cal of the companion software The calibration file contains a 22 dB 1004 Hz reference signal The audio circuit shall be calibrated so that the test signal has a level of 15 dBPa at the ear reference plane while maintaining compliance with Section 3 4 1 1 3 46 21 22 23 24 25 26 27 28 29 30 31 32 33 34 3GPP2 C S0018 E v2 0 3 5 2 Software Test Tools for SO 77 This section describes a set of software tools useful for performing the MPS tests The code has been developed and compiled using the GNU g compiler and software maintenance utilities The tools have been verified under various representati
137. are the master packet files for the s12 source file Likewise the p22 and p32 files are the respective packet files for the s22 and s32 source files The pf3 files are the impaired packet files which will be described in Section 2 3 4 3 Similarly the directory so70 subjctv exp 2 4 6 m_pkt contains the master packet files for the respective experiments Here the pc10 pb20 and ps files are the master packet files for the 15 b20 and s15 source files respectively For the master encode master decode directories so70 subjctv exp m m the naming convention of the speech files is such that the first two characters of the file name indicate the codec combination and the suffix indicates the condition numbers in Table 2 3 2 3 1 2 and Table 2 3 2 3 2 2 Naming conventions for the remaining two experiments follow accordingly Detailed descriptions of all processing operations are given in Section 6 2 3 41 Encoding by the Test Codec All of the source files will be encoded by the test codec to produce encoded packet files For ease of reference it is recommended that directories so70 subjctv exp t pkt be created to deposit the test encoder output packets and that the naming conventions be made consistent with the master codec 2 3 4 2 Decoding by the Master Test Codecs The encoded packet files generated from the various encoders conditions shall be processed through the master and test decoders 2 3 4 3 Introd
138. ate for the experiment Table 2 2 4 5 1 shows the cutting points to be used with the astrip tool for producing the two sentence samples for the Experiment ACR test Table 2 2 4 5 2 shows the cutting points to be used with the astrip tool for producing the single sentence sub samples for the Experiment 11 P 835 test Table 2 2 4 5 3 shows the sub samples that make up the samples i e sentence triads for the P 835 test 2 24 3GPP2 C S0018 E v2 0 Table 2 2 4 5 1 Cutting Points for the astrip Software Tool for the Experiment ACR Test Sentence pair EH MEN m 3 xg 1 98817 152321 209921 257537 304897 51020 614657 714241 865281 91084 96102 101299 106700 111641 1170177 126924 1367297 142156 147481 152524 158156 949664 49665 49152 98817 53504 152321 57600 209921 47616 257537 47360 304897 52736 510209 53504 5637139 _ 50944 614657 51456 714241 51712 865281 45568 1170177 51908 1222145 47104 1269249 47104 1316353 50944 1367297 54272 Experiment I ACR eng SeNntsimce ENS eng samples pair Sample samples 1636353 19564 1686017 50432 49664 49152 53504 5760 4761 4736 5273 5068 5017 53504 50944 5145 4812 4992 4940 4556 5017 5196 5401 4940 5376 5196 47104 47104 50944 54272 5324 5043 5632 5478 4 4 HS GO EIN 25 50 60
139. ation Encoder A device for the coding of a signal into a digital representation For the purpose of this standard a device compatible with a specific EVRC implementation FER Frame Error Rate equals the number of full rate frames received in error divided by the total number of transmitted frames IRS Intermediate Reference System 12 MGW Media Gateway MIRS Modified Intermediate Reference System 12 MNRU Modulated Noise Reference Unit A procedure to add speech correlated noise to a speech signal in order to produce distortions that are subjectively similar to that produced by logarithmically companded PCM systems The amount of noise is expressed as a signal to noise ratio value in dB and is usually referred to as dBQ 11 Mobile Station A station in the Domestic Public Cellular Radio Telecommunications Service It is assumed that mobile stations include portable tranceivers for example hand held personal tranceivers and tranceivers installed in vehicles MOS Mean Opinion Score The result of a subjective test based on an absolute category rating ACR where listeners associate a quality adjective with the speech samples to which they are listening These subjective ratings are transferred to a numerical scale and the arithmetic mean is the resulting MOS number 10 Rates for SO 3 The allowable traffic frame rates for SO 3 Rate 1 frames use the 9600 bps rate Rate 2 frames use the 4800 bps rate Rate 4 fra
140. ation lists for each of the eight listening panels for each experiment are provided in so73 subjctv exp data play Ist 2 81 20 21 22 23 24 25 26 27 28 29 3GPP2 C S0018 E v2 0 2 4 6 Presentation Presentation of speech materials for the SO 73 codec listening tests shall be made with one side of high fidelity supra aural headphones with the other ear uncovered The speech material delivery system shall meet the requirements of Section 3 4 1 1 The listeners should be seated in a quiet room with an ambient noise level of 30 dBA or below 2 4 7 Listeners The listener sample is intended to represent the population of telephone users with normal hearing acuity The listeners should be naive with respect to telephony technology issues that is they should not be experts in telephone design digital voice encoding algorithms and so on They should not be trained listeners that is they should not have been trained in these or previous listening studies using feedback trials Age distribution and gender should be nominally balanced across listening panels Each listener shall provide data only once for a particular evaluation A listener may participate in different evaluations but test sessions performed with the same listener should be at least two months apart so as to reduce the cumulative effects of experience 2 4 8 Listening Test Procedures 2 4 8 1 ACR Listening Test Procedures Experiments 1 and 3
141. b_op4 fer_3 p22 EVRC WB operating point 4 8 kHz sampling Nominal 22 dB FER 3 evrc wb op4 fer 396 022 8k evrc wb op4 p12 EVRC WB operating point 4 8 kHz sampling High 12 dB evrc wb op4 012 8k evrc wb op4 p32 EVRC WB operating point 4 8 kHz sampling Low 32 dB evrc wb op4 032 8k evrc wb op7 p22 EVRC WB operating point 7 8 kHz sampling Nominal 22 dB evrc wb op7 022 8k evrc wb op4 dim 196 pls 196 p22 EVRC WB operating point 4 8 kHz sampling Nominal 22 dB 1 d amp b 1 pls evrc wb op4 dim 196 pls 196 022 8k evrc wb op4 pc EVRC WB operating point 4 8 kHz sampling Nominal 22 dB 15 dB car noise evrc wb op4 oc 8k evrc wb op7 pc EVRC WB operating point 7 8 kHz sampling Nominal 22 dB 15 dB car noise evrc wb op7 oc 8k evrc wb op4 ps EVRC WB operating point 4 8 kHz sampling Nominal 22 dB 15 dB street noise evrc wb op4 os 8k evrc wb op4 fer 296 pb EVRC WB operating point 4 8 kHz samplin 3 33 Nominal 22 dB 15 dB babble noise evrc wb op4 fer 296 pb 8k 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 3GPP2 C S0018 E v2 0 3 4 Specific Standard Test Conditions for SO 73 3 4 1 Audio Path and Calibration for SO 73 3 4 1 1 Audio Path The audio path for wideband test conditions Experiments 1 and 2 must meet the followi
142. been appropriately pre filtered level adjusted and noise processed These files are named src The speech database also includes samples processed through the various reference conditions in directory so70 subjctv exp ref The reference conditions are named ref for the respective conditions given in the tables in Section 2 3 2 3 2 3 4 Processing of Speech Material for SO 70 Testing The source speech material shall be processed by the various combinations of encoders and decoders listed in the descriptions of the experiments given in Section 2 3 2 The master codec software described in Section 3 3 3 shall be used in the processing involving the master codec Generally the master codec encoder and decoder outputs have been provided in the respective 2 51 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 3GPP2 C S0018 E v2 0 directories so70 subjctv exp m pkt and so70 subjctv exp m_m Execution of the master codec software is needed only for the test encoder master decoder combination for each experiment condition All codec processing shall be done digitally Noise suppression and post filter options shall be enabled for both the master and the test codecs The digital format of the speech files is described in Section 3 3 4 4 The naming convention of the processed speech is as follows For the packet files in the so70 subjctv exp 1 3 5 m_pkt directory the p12 files
143. binations 4 M M T T T M The Test Conditions for Listening Experiment 2 are presented in Table 2 5 2 3 2 2 Table 2 5 2 3 2 2 SO 77 Listening Experiment 2 Test Conditions Exp 2 Wideband P 835 Reference Conditions File MNRU SNR 01 MNRU 40aB SNR 40dB Reference b02 MNRU 40dB SNR 20dB Reference b03 MNRU 40dB SNR 0dB Reference b04 MNRU 0dB SNR 40dB Reference b05 MNRU 20dB SNR 40dB Reference b06 MNRU 10dB SNR 10dB Reference b07 MNRU 20dB SNR 20dB Reference b08 MNRU 40dB SNR 30dB Reference Test Conditions File Condition Enc Dec b09 Car 10dB SNR M M b10 Car 10dB SNR M T bit Car 10dB SNR T T b12 Car 10dB SNR T M bi3 Car 20dB SNR 296 FER M M 2 94 1 2 3GPP2 50018 v2 0 Exp 2 Wideband P 835 b14 Car 20dB SNR 2 FER M T b15 Car 20dB SNR 2 FER T T b16 Car 20dB SNR 2 FER T M b17 Street 15dB SNR M M b18 Street 15dB SNR M T b19 Street 15dB SNR T T b20 Street 15dB SNR T M b21 Babble 20dB SNR M M b22 Babble 20dB SNR M T b23 Babble 20dB SNR T T b24 Babble 20dB SNR T M 2 5 2 3 3 Subjective Experiment 3 for SO 77 The Test Parameters for Listening Experiment 3 are presented in Table 2 5 2 3 3 1 Table 2 5 2 3 3 1 SO 77 Listening Experiment 3 Test Parameters ACR P 800 Narrowband Type of test Typeoftest O 4 males 4 females Reference condi
144. ble 2 3 1 1 1 1 Target ADR vs Capacity Operating Point 2 35 Table 2 3 2 1 Test Suites for SO 70 compliance eee 2 38 Table 2 3 2 2 Experiments for SO 70 compliance sse 2 38 Table 2 3 2 3 1 1 SO 70 Listening Experiment 1 Test 2 39 Table 2 3 2 3 1 2 SO 70 Listening Experiment 1 Test 1 2 40 Table 2 3 2 3 2 1 SO 70 Listening Experiment 2 Test 2 41 Table 2 3 2 3 2 2 SO 70 Listening Experiment 2 Test 2 42 Table 2 3 2 3 3 1 SO 70 Listening Experiment 3 Test 2 43 Table 2 3 2 3 3 2 SO 70 Listening Experiment 3 Test 2 43 Table 2 3 2 3 4 1 SO 70 Listening Experiment 4 Test 2 45 Table 2 3 2 3 4 2 SO 70 Listening Experiment 4 Test 2 45 Table 2 3 2 3 5 1 SO 70 Listening Experiment 5 Test 2 46 Table 2 3 2 3 5 2 SO 70 Listening Experiment 5 Test 2 47 Table 2 3 2 3 6 1 SO 70 Liste
145. ble difference i j k 1 is given by Oli j k 1 0 12 c i j k VA SER SER 1j k 1 Oli j k 1 0 12 c i j k TA SER GJ E SER 1j k 1 2 1 10 5 where i j k and are as defined above and the multipliers are given in Table 2 1 10 1 The standard errors SEA i j k for each condition are defined as gt GU ksn MOSQ j KYY SRE hse eee 261632 Y km MOSG j KY SER i j k Pe oo van 2 1 10 6 Specifically stating the requirement Equations 2 1 10 2 and 2 1 10 3 shall be true for all cases otherwise the test codec fails the compliance test Table 2 1 10 1 Multipliers for Equations 2 1 10 4 and 2 1 10 5 Experiment Condition Description k 2 ven 4 Hf ten Lim pann Low Audio Input Level 2 94 396 FER N A Oo f s revom 265 2 on tf f 20 817 2 Canoe 200 200 200 Street Noise _ 200 2 00 200 4 _ 226 322 349 5 Tamem 258 465 2 14 3GPP2 C S0018 E v2 0 2 1 11 Expected Results for Reference Conditions The MNRU conditions have been included to provide a frame of reference for the MOS test Also they provide anchor conditions for comparing results between test laboratories In listening evaluations where test conditions span approximately the same range of quality the MOS results for similar conditions should be approximately the same Data from previous studie
146. car noise evrc wb op4 oc 8k evrc wb op7 pc EVRC WB operating point 7 8 kHz sampling Nominal 22 dB 15 dB car noise evrc wb op7 oc 8k evrc wb op4 ps EVRC WB operating point 4 8 kHz sampling Nominal 22 dB 15 dB street noise evrc wb op4 os 8k evrc wb op4 fer 296 pb EVRC WB operating point 4 8 kHz samplin 3 30 Nominal 22 dB 15 dB babble noise evrc wb op4 fer 296 0b 8k 3GPP2 C S0018 E v2 0 2 Table 3 3 4 5 6 SO 70 Encoder Suite C Bit exact Test Conditions Input File Operating Point Condition Reference packet files for bit exact compliance 4 Table 3 3 4 5 7 No need encoder tests if SO 70 Suite C Decoder Bit exact Test Conditions Reference output speech Input Packet File Operating Point Condition files for bit exact compliance evrc wb opO dim 196 fer 296 p22 EVRC WB Nominal 22 dB evrc wb opO dim 196 fer 296 operating point 0 196 d amp b 022 8k 8 kHz sampling 2 FER evrc wb opO pls 196 p22 EVRC WB Nominal 22 dB evrc wb opO pls 196 022 8k operating point 0 1 pls 8 kHz sampling evrc_wb_op0 p12 EVRC WB High 12 dB evrc wb op0 012 8k operating point 0 8 kHz sampling evrc_wb_op0 p32 EVRC WB Low 32 dB evrc wb op0 o32 8k operating point 0 8 kHz sampling evrc wb opoO dim 296 pc EVRC WB Nominal 22 dB evrc wb opO dim 296 0c 8k operating point 0 15 dB car noise 8 kHz sampling 2 d amp b
147. ce ANOVA for the effects of E DC x Subjects where the E DC factor includes the four E DC s three test E DC s plus the Reference E DC and the Subjects factor includes the 32 subjects involved in the subjective test If the F ratio for the E DC effect is significant i e lt 05 then there is significant variation among the scores for the E DC s and the Dunnett s test proceeds to the second stage of the process An F ratio that is not significant indicates that there is no significant variation among the Test and Reference E DC s A non significant F ratio indicates that the means for all four E DC s are statistically equivalent therefore all Test E DC s are not worse than the Reference E DC and all pass the MPS In the second stage of Dunnett s Test each of the Test E DC means is compared statistically to the Reference E DC mean and the mean difference is evaluated for significance The three statistical tests use a common estimate of the Standard Error of the Mean Difference derived from the Error Mean Square from the ANOVA 5 1 Stage 1 Analysis of Variance Table 5 1 1 shows the generalized Variance Source Table for the stage 1 ANOVA s involved in the Dunnett s Tests The Error Sum of Squares SoS in the ANOVA is the residual SoS after removal of the systematic effects due to the E DC and the Subjects factors 10 The scores for each subject are average values over talkers 5 1 3GPP2 C S0018 E v2 0 Table 5 1
148. ch data to a file 3 1 3GPP2 50018 v2 0 The choice of the host computer and the nature of the interfaces between the host computer and the speech codec are not subject to standardization It is expected that the host computer would be some type of personal computer or workstation with suitable interfaces and adequate disk storage The interfaces may be serial or parallel and will be determined by the interfaces available on the particular hardware realization of the speech codec Figure 3 2 shows a generic block diagram of the audio path for the subjective test using four listeners per session The audio path is shown as a solid line the data paths for experimental control are shown as broken lines This figure is for explanatory purposes and does not prescribe a specific implementation Digital Speech 2 Software Response Terminals Files Control Program D A Converter Reconstruction Filter Bandpass Filter Attenuator or Electronic Amplifier Switch Headphones Figure 3 2 Subjective Testing Equipment Configuration For the purposes of this standard speech codecs under test are not required to provide performance across ranges of temperature humidity or other typical physical environmental variables 3 2 20 21 22 23 24 25 26 27 28 29 30 3GPP2 C S0018 E v2 0 3 1 Specific Standard Test Conditions for SO 3 3 1 1 Audio Path and Calib
149. ch trial in a P 835 test listeners are presented with three sub samples where each sub sample is a single sentence approx 4 sec duration processed through the same test condition In one of the first two sub samples listeners rate the Signal Quality on a five point rating scale with the points labeled C Very natural no distortion Fairly natural little distortion Somewhat natural some distortion Fairly unnatural fairly distorted Very unnatural very distorted 2 58 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 3GPP2 C S0018 E v2 0 For the other of the first two sub samples listeners rate the Background Quality on a five point rating scale with the points labeled Not noticeable Fairly noticeable Noticeable but not intrusive Fairly conspicuous somewhat intrusive Very conspicuous very intrusive A C For the third sub sample listeners rate the Overall quality on a five point rating scale with the points labeled Excellent Good Fair Poor Bad ou Data from 32 listeners shall be used for Experiments 2 4 and 6 four listeners for each listening panel where each listening panel uses a different randomization Before starting the test the listeners should be given instructions for performing the subjective test An example set of instructions for the P 835 test are presented below The instructions may be modified to allow for variations
150. cked in accordance with the tables specified in 1 The packet file value for each data rate is shown in Table 3 5 3 3 1 Table 3 5 3 3 1 Packet File Structure from Master Codec Channel Error Model Value in Packet File Rate Data Bits per Frame 4 0 004 171 3 49 20 21 22 23 24 25 26 27 28 29 30 3GPP2 50018 v2 0 Value in Packet File Rate Data Bits per Frame 3 0x0003 1 2 EDEN DIM Unused bits are set to 0 For example in a Rate 1 8 frame the packet file will contain the word 0x0100 byte swapped 0x0001 followed by one 16 bit word containing the 16 data bits for the frame in byte swapped form followed by ten 16 bit words containing all zero bits 3 5 4 Fixed Point Bit Exact Codec for SO 77 This section describes the C simulation of the speech codec specified by 1 The speech codec C simulation is based on finite precision fixed point arithmetic operations and is recommended to be used as a reference codec to verify the performance of a bit exact EVRC NW2K implementation of the fixed point C simulation of a test codec The bit exact EVRC NW2K codec along with the appropriate test vectors to verify the bit exactness performance are included in the associated Software Distribution 3 5 4 1 Fixed Point Codec Program Files This section describes the C program files which are provided in the associated software distribution for this document 3 5 4 2 Compiling the Fixed
151. d in the associated software distribution for this document 3 3 4 2 Compiling the Fixed Point Codec Simulation The source code for the fixed point codec simulation has been written in C and can be compiled using any general purpose compiler such as the GNU G compiler and make utility Two GCC compatible makefiles have been included in the build directory Typing make in the build directory will compile and link the code and create the executable file called Evrc wb fx Evrc wb fx exe on Win32 systems which will be placed in the build directory The included makefiles may require some user modification for a particular hardware platform and or operating system 3 3 4 8 Running the Fixed Point Codec Simulation The EVRC WB executable files use command line arguments to receive all information regarding input and output files and various parameters used during execution Executing Evrc wb fx with no command line arguments will display a brief description of the required and optional command line arguments The options are described below i infn required Specifies the name of the input speech file or the name of the input packet file if only decoding is being performed see the d option below o outf required Specifies the name of the output speech file or the name of the output packet file if only encoding is being performed see the e option below d Instructs the simulation to perform only the decoding function
152. dB evrc_nw_op1 os 8k operating point 1 8 kHz 15 dB street noise sampling evrc nw opi fer 296 p EVRC NW2kK Nominal 22 dB evrc_nw_op1 fer_2 b operating point 1 8 kHz 15 dB babble noise ob 8k sampling evrc_nw_op6 ps EVRC NWZ2K Nominal 22 dB evrc nw op6 0s 8k operating point 6 8 kHz 15 dB street noise samplin 3 59 3GPP2 50018 v2 0 Reference output Input Packet File Operating Point Condition speech files for bit exact compliance evrc_nw_op6 fer_2 p EVRC NW2kK Nominal 22 dB evrc_nw_op6 fer_2 b operating point 6 8 kHz 15 dB babble noise ob 8k sampling evrc_nw_op3 fer_3 p EVRC NW2kK Nominal 22 dB evrc_nw_op3 fer_3 22 operating point 3 8 kHz FER 3 022 8k sampling evrc nw op3 p12 EVRC NW2kK High 12 dB evrc nw op3 012 8k operating point 3 8 kHz sampling evrc nw op3 p32 EVRC NWZ2K Low 32 dB evrc nw 0op3 032 8k operating point 3 8 kHz sampling 15 196 p22 operating point 3 8 kHz 9 A9 pls 196 022 8k sampling evrc nw op3 pc EVRC NW2kK Nominal 22 dB evrc nw op3 oc 8k operating point 3 8 kHz 15 dB car noise sampling evrc nw op3 ps EVRC NWZ2K Nominal 22 dB evrc nw op3 os 8k operating point 3 8 kHz 15 dB street noise sampling evrc nw op3 fer 296 p EVRC NWZ2K Nominal 22 dB evrc nw op3 fer 296 b operating point 3 8 kHz 15 dB babble noise ob 8k samplin 3 60 3GPP2 C S0018 E v2 0 4 CONTENTS OF SOFTWARE DISTRIBUTION T
153. dB SNR 2 FER M M b14 Car 20dB SNR 2 FER M T b15 Car 20dB SNR 2 FER T T b16 Car 20dB SNR 2 FER T M b17 Street 15dB SNR M M b18 Street 15dB SNR M T b19 Street 15dB SNR T T b20 Street 15dB SNR T M b21 Babble 20dB SNR M M b22 Babble 20dB SNR M T b23 Babble 20dB SNR T T b24 Babble 20dB SNR T M 2 4 2 3 3 Subjective Experiment 3 for SO 73 The Test Parameters for Listening Experiment 3 are presented in Table 2 4 2 3 3 1 Table 2 4 2 3 3 1 SO 73 Listening Experiment 3 Test Parameters 4 males 4 females MRS Filter characteristics MIRS Test conditions Nominal level Modes 0 4 7 Low level Modes 0 4 High Level Mode 0 4 1 d amp b 1 pls Modes 0 4 3 FER Modes 0 4 Encoder Decoder Combinations 4 M M M T T T T M The Test Conditions for Listening Experiment 3 are presented in Table 2 4 2 3 3 2 2 70 3GPP2 C S0018 E v2 0 Table 2 4 2 3 3 2 SO 73 Listening Experiment 3 Test Conditions Label Operating Condition greed c01 Reference MNRU 5dB c02 Reference MNRU 10dB c03 Reference MNRU 15dB c04 Reference MNRU 20dB c05 Reference MNRU 25dB c06 Reference MNRU 30dB c07 Reference MNRU 35dB c08 Reference Direct c09 Mode 1 Nominal 22 dB M M c10 Mode 1 Nominal 22 dB M T c11 Mode 1 Nominal 22 dB T T c12 Mode 1 Nominal 22 dB T M c13 Mode 6 Nomin
154. de test encode master decode master encode test decode and test encode test decode or more simply M M T M M T and T T respectively The decoded speech material is then appropriately gain adjusted inversely to input gain and formatted u Law PCM for SO 3 and 16 bit linear PCM for SO 68 SO 70 SO 73 and SO 77 to form the final outputs The representation of output speech is the same as that for input speech material 3GPP2 C S0018 E v2 0 Master Encoder External Rate Control Test Decoder Test Encoder Intermediate Packet Format Figure 1 5 1 Test Model Various implementations of the encoder and decoder especially those in hardware may not be designed to deliver or accept a continuous data stream as previously described It is the responsibility of the manufacturer to implement a test platform that is capable of delivering and accepting these formats in order to complete the performance tests described in the following sections This may involve a custom hardware interface or a fair implementation of the algorithm in software or some other mechanism A fair implementation in software shall yield bit exact output with reference to any hardware implementation that it is claimed to represent The input speech material has been precision limited by an 8 bit p law quantization algorithm in which the inverse quantized linear samples fill the entire 16 bit linear range As specified within Section 3 of 1 the master c
155. decision flowchart An implementation may support SO 70 only for 8 KHz sample rate input output for example a Base station transcoder or a Media Gateway An implementation may support SO 70 for both 16 kHz and 8 kHz sample rate for example a mobile station that supports wideband electro acoustics Further the implementation supporting SO 70 might already have demonstrated compliance to SO 68 Minimum Performance Spec This means that such an equipment has also demonstrated the Minimum Performance requirements for RATE REDUC operating points 4 and 7 of SO 70 which exactly correspond to the RATE REDUC operating points 0 and 7 of SO 68 Therefore the main parameters in the decision tree are a 16 kHz support in the implementation b SO 68 compliance of the test implementation Depending on the implementation profile of the Device under test one of 4 possible Test Suites are to be used to demonstrate SO 70 compliance These 4 test suites named Test suites A B C D and the individual input test vectors comprising the Test suites are highlighted in Table 3 3 4 5 1 3 26 3GPP2 C S0018 E v2 0 Table 3 3 4 5 1 Test Suites of input test vectors for SO 70 compliance Test Suites Notes Mobile application already supporting SO 68 compliance Mobile application NOT already supporting SO 68 compliance Infra MGW application already supporting SO 68 compliance Infra MGW application NOT already supporting SO 68 compliance
156. decoded speech files The files are to be processed as follows Decode EvrcFix d i file pkt o file dec Table 3 1 4 6 2 4 Decoder Output Test Vector Files Packet Source File Decoded Speech File vec 07 1 pkt vec 07 1 dec vec 07 2 pkt vec 07 2 dec vec 07 3 pkt vec 07 3 dec vec 08 1 dec vec 08 2 pkt vec 08 2 dec vec 08 3 pkt vec 08 3 dec vec 10 1 pkt vec 10 1 dec vec 10 2 pkt vec 10 2 dec vec 10 3 pkt vec 10 3 dec 3 2 Specific Standard Test Conditions for SO 68 3 2 4 Audio Path and Calibration for SO 68 3 2 1 1 Audio Path The audio path must meet the following requirements for electro acoustic performance measured between the output of the D A converter and the output of the headphone 1 Frequency response shall be flat to within 2 dB between 200 Hz and 3400 Hz and below 200 Hz the response shall roll off at a minimum of 12 dB per octave Equalization may be used in the audio path to achieve this A suitable reconstruction filter shall be used for playback 2 Total harmonic distortion shall be less than 1 for signals between 100 Hz and 4000 Hz 3 Noise over the audio path shall be less than 35 dBA measured at the ear reference plane of the headphone 4 Signal shall be delivered to the headphone on the listener s preferred telephone listening ear and the other ear shall be uncovered No signal shall be delivered to the other headphone 3 2 1 2 Calibration The audio circuit shall deliver an average sound l
157. e 2 21 3GPP2 50018 v2 0 1 value of the parameter if it is dependent upon other parameter values and the last two columns show 2 the numerical value for each of the parameters for the two listening experiments For each listening experiment four codecs are evaluated with a differing number of conditions three for the EVRC B 9 3 4 and 6 6 kbps codecs and one for the EVRC B 5 8 and 4 8 kbps codecs There are eight reference 5 conditions in both experiments Table 2 2 2 3 3 1 Numerical Parameters for the SO 68 Listening Experiments Parameter Calculation Experiment Experiment Il Value Pince Ca feoaeos tor Testcorators o ca ps pewecommtoeomdonis L a ps ontnatons Condion eem C4 C6 C7 Listening Cis listeners Voters 32 C16 Listeners Voters per Listening Panel 8 2 2 3 Speech Material for SO 68 Testing The source speech files used for SO 68 compliance testing consist of 128 Harvard sentences which 10 are preprocessed to include proper level adjustment and noise mixing for use in the two subjective 11 experiments The talkers used in these files consist of four adult males and four adult females and 1 are native speakers of North American English 18 For the following discussion it may be useful to refer
158. e numbered phrase from the list below that best describes your opinion of the SPEECH SIGNAL ALONE and then enter the corresponding number on your keyboard 2 30 3GPP2 50018 v2 0 Attending ONLY to the SPEECH SIGNAL select the category which best describes the sample you just heard the SPEECH SIGNAL in this sample was VERY NATURAL NO DEGRADATIO FAIRLY NATURAL LITTLE DEGRA SOMEWHAT NATURAL SOM FAIRLY UNNATURAL FAI VERY UNNATURAL VERY For the second sentence in each trial you will be asked to attend only to the background and rate how noticeable intrusive and or conspicuous the background sounas to you You will use the rating scale shown in the figure below to register your ratings of the background Your task will be to choose the numbered phrase from the list below that best describes your opinion of the BACKGROUND ALONE and then enter the corresponding number on your keyboard Attending ONLY to the BACKGROUND select the category which best describes the sample you just heard the BACKGROUND in this sample was 5 NOT NOTICEABLE SOMEWHAT NOTICEABLE NOTICEABLE BUT NOT INTRUSIVE FAIRLY CONSPICUOUS SOMEWHAT INTRUSIV VERY CONSPICUOUS VERY INTRUSIV For the third and final sentence in each trial you will be asked to attend to the entire sample b
159. e 0 3 FER LB portion of Wideband mode decoder test only M T c39 Mode 4 3 FER interoperable with Mode 0 of SO 68 support M M c40 Mode 4 3 FER interoperable with Mode 0 of SO 68 support M T 2 2 3 2 3 4 Subjective Experiment 4 for SO 70 The Test Parameters for Listening Experiment 4 are presented in Table 2 3 2 3 4 1 2 44 3GPP2 C S0018 E v2 0 Table 2 3 2 3 4 1 SO 70 Listening Experiment 4 Test Parameters Type of test P NSA P 835 Narrowband Number of talkers Background noise Specified test conditions Audio Input Level 22 dB Filter characteristics MIRS Reference conditions 8 Specified reference conditions Test conditions 3 males 3 females o Car Noise 15 dB SNR Modes 0 4 7 o Street Noise 15 dB SNR Modes 0 4 o Babble noise 20 dB SNR 2 FER Modes 0 4 Encoder Decoder Combinations 4 M M M T T T T M The Test Conditions for Listening Experiment 4 are presented in Table 2 3 2 3 4 2 Table 2 3 2 3 4 2 SO 70 Listening Experiment 4 Test Conditions Exp 4 Narrowband P 835 Reference Conditions File MNRU 901 MNRU 40aB SNR 40dB Reference 902 MNRU 40cB SNR 20dB Reference do3 MNRU 40aB SNR 0dB Reference 904 MNRU 0dB SNR 40dB Reference 905 MNRU 20cB SNR 40dB Reference do6 MNRU 10aB SNR 10dB Reference 907 MNRU 20cB SNR 20dB Reference kos MNRU 40aB SNR 30dB Reference Test Conditions File Cond
160. e 2 1 11 1 MOS versus MNRU essent 2 15 Figure 2 2 8 1 1 Instructions for 1 enne eene 2 29 Figure 2 2 10 1 1 MOS versus ennemi 2 33 Figure 2 2 10 2 1 P 835 Score Profiles for Reference 2 34 Figure 2 3 2 1 SO 70 Subjective test suite decision 2 37 Figure 2 3 8 1 1 Instructions for 2 58 Figure 2 3 10 1 1 Typical Plot of MOS versus MNRU 2 62 Figure 2 3 10 2 1 Typical P 835 Score Profiles for Reference Conditions 2 63 Figure 2 4 2 1 SO 73 Subjective test suite decision 2 66 Figure 2 4 8 1 1 Instructions for Listeners esses tnnt tnn 2 83 Figure 2 4 10 1 1 Typical Plot of MOS versus MNRU nennen 2 87 Figure 2 4 10 2 1 Typical P 835 Score Profiles for Reference Conditions 2 88 Figure 2 5 2 1 SO 77 Subjective test suite decision 2 91 Figure 2 5 8 1 1 Instructions for 1 2 113 Figure 2
161. e Condition Enc Dec c11 Street 15dB SNR M M c12 Car 15dB SNR M M c13 Babble 15dB SNR M M c14 Street 15dB SNR M T c15 Car 15dB SNR M T c16 Babble 15dB SNR M T c17 Street 15dB SNR T M c18 Car 15dB SNR T M c19 Babble 15dB SNR T M c20 Street 15dB SNR T T c21 Car 15dB SNR T T c22 Babble 15dB SNR T T 2 103 3GPP2 C S0018 E v2 0 Exp 4 Narrowband P 835 c23 Street 15dB SNR 296 FER M M c24 Street 15dB SNR 2 FER M T 2 5 2 3 9 Numerical Parameters for the SO 77 Listening Experiments Table 2 5 2 3 9 1 describes the resultant numerology that is used for the eight SO 77 listening experiments The first column is the description of the parameter and columns 2 to 9 show the numerical value for each of the parameters for the eight listening experiments For each listening experiment the different Encode Decode Test conditions include various interconnections between the Master and Test Encoders and the Master and Test Decoders There are eight reference conditions in each of the experiments 1 through 4 and 7 four reference conditions in experiments 5 and 6 and 10 reference conditions for experiment 8 Table 2 5 2 3 9 1 Numerical Parameters for the SO 77 Listening Experiments Parameter Expi Exp2 Exp 3 Exp 4 5 Exp 4 3 Exp Exp Encode Decode Test conditens i6 16 4o 34 4 4 16 ia e ta lm a 2 EE ESE EMT ENIM PES E E
162. e SO 70 compliance These 4 test suites named Test suites A B C D and the individual tests comprising the Test suites are highlighted in Table 2 3 2 1 2 37 3 3GPP2 50018 v2 0 Table 2 3 2 1 Test Suites for SO 70 compliance Test Set of Experiments Notes Suites A Experiment 1 2 and 7 Mobile MGW already supporting SO 68 compliance B Experiment 1 2 3 4 7 and 8 Mobile MGW NOT already supporting SO 68 compliance C Experiment 5 6 and 8 Infra MGW already supporting SO 68 compliance D Experiment 3 4 and 8 Infra MGW NOT already supporting SO 68 compliance Each of the individual experiments are further defined in detail by Table 2 3 2 2 Table 2 3 2 2 Experiments for SO 70 compliance Experiment Individual tests Notes 1 WB clean level FER signaling ACR Mobile supporting 16 kHz Fs 2 WB noise FER P 835 Mobile supporting 16 kHz Fs 3 NB clean level FER signaling including BS supporting 8 kHz and MS supporting SO 68 interoperable mode tests 8 16 kHz SO 68 compliance not PROVEN 4 NB noise FER including SO 68 BS supporting 8 kHz and MS supporting interoperable mode tests P 835 8 16 kHz SO 68 compliance not PROVEN 5 NB clean level FER signaling NOT BS supporting 8 kHz SO 68 compliance including SO 68 interoperable mode already PROVEN tests ACR 6 NB noise FER signaling NOT including BS supporting 8 kHz SO 68 compliance SO 68 interoperable mode tests already PROVEN P 8
163. e SO 77 Listening Experiments 2 104 2 5 3 Speech Material for SO 77 Testing nnne 2 104 2 5 4 Processing of Speech Material for SO 77 Testing m teen eee 2 105 2 54 1 Encoding by the Test rtrtrererereosososososososonononoasasososasosososososososososososososoonnonn 2 105 2 5 4 2 Decoding by the Master Test Codecs 2 105 2 5 4 3 Introduction of Impairments nnne tnnt trennen nennen 2 105 2 5 4 4 Ensuring Proper Encoded Frame Packet Files mm 2 106 2 5 4 5 Post processing of test condition output 1 2 106 2 5 5 ied dedere e 2 111 256 PresentatllOnsiessetnteeee da r k PUn eek 2 112 25 7 LISteAEN sete mere nd 2 112 2 5 8 Listening Test Procedures nnne nennen nnne tnn eate tatnen 2 113 2 5 8 1 ACR Listening Test Procedures Experiments 1 3 and 7 mms 2 113 2 5 8 2 P 835 Listening Test Procedures Experiments 2 4 and 8 2 114 2 5 9 Analysis of 2 117 2 5 9 1 Basic Results for the SO 77 Listening tests m 2 117 2 5 9 Minimum Subjective Requirement for SO 77 Listening Tests 2 117 2 5 10 Expected Results for Reference Conditions e 2 118 2 5 10 1 Reference Conditions
164. e file ratedec c for more information p flag If flag is set to 0 the post filter is disabled If the flag is set to 1 the post filter is enabled If the p option is not invoked the post filter is enabled during decoding n flag If flag is set to 0 noise suppression is disabled If the flag is set to 1 noise suppression is enabled If the n option is not invoked noise suppression is enabled during encoding 3 1 4 4 File Formats Files of speech contain 2 s complement 16 bit samples with the least significant byte first The packet file contains twelve 16 bit words with the low byte ordered first followed by the high byte The first word in the packet contains the data rate while the remaining 11 words contain the encoded speech data packed in accordance with the tables specified in 1 The packet file value for each data rate is shown in Table 3 1 3 3 1 Unused bits are set to 0 For example in a Rate 1 8 frame the packet file will contain the word 0x0100 byte swapped 0x0001 followed by one 16 bit word containing the 16 data bits for the frame in byte swapped form followed by ten 16 bit words containing all zero bits 3 1 4 5 Verifying Proper Operation of the Fixed Point Codec Files are provided for the purpose of verifying the fixed point codec executable The files soS simul fixed test source pcm contain the original unprocessed speech files The files in soS simul fixed test fixed32 contain the encoded packet file
165. e for 3 1 1 2 listening tests so3 tools Source code for the software 3 1 2 tools so3 testvec Test vectors for verifying bit 3 1 4 6 exact EVRC implementations 4 1 3GPP2 C S0018 E v2 0 Table 4 2 Description of EVRC B Software Distribution Contents So68 EVRCB FX source code for the bit exact 3 2 4 fixed point code so68 subjctv Speech and other material 2 2 1 1 2 2 3 2 2 4 2 2 5 necessary to perform Subjective Experiments and Il so68 cal Output level calibration file for 3 2 1 2 listening tests so68 tools Source code for the software 3 2 2 tools so68 testvec Test vectors for verifying bit 3 2 4 5 exact EVRC implementations Table 4 3 Description of EVRC WB Software Distribution Contents so70_73 EVRCWB_NW_FX source code for the bit exact 3 3 4 fixed point code so70 subjctv Speech and other material 2 3 1 1 2 3 3 2 3 4 2 3 5 necessary to perform subjective experiments so70 cal Output level calibration file for 3 3 1 2 listening tests so70 tools Source code for the software 3 3 2 tools so70 testvec Test vectors for verifying bit 3 3 4 5 exact EVRC implementations 4 2 3GPP2 C S0018 E v2 0 Table 4 4 Description of EVRC NW Software Distribution Contents so70 73 EVRCWB NW FX source code for the bit exact 3 3 4 fixed point code so73 subjctv Speech and other material 2 3 1 1 2 4 3 2 3 4 2 3 5 necessary to perform subjective experiments so73 cal Output level calibrati
166. e tables specified in 1 The packet file value for each data rate is shown in Table 3 4 3 3 1 Table 3 4 3 3 1 Packet File Structure from Master Codec Channel Error Model Value in Packet File Rae Data Bits per Frame 4 0x0004 171 3 37 20 21 22 23 24 25 26 27 29 3GPP2 50018 v2 0 3 0x0003 1 2 Value in Packet File Rate Data Bits per Frame EDEN DIM Unused bits are set to 0 For example in a Rate 1 8 frame the packet file will contain the word 0x0100 byte swapped 0x0001 followed by one 16 bit word containing the 16 data bits for the frame in byte swapped form followed by ten 16 bit words containing all zero bits 3 4 4 Fixed Point Bit Exact Codec for SO 73 This section describes the C simulation of the speech codec specified by 1 The speech codec C simulation is based on finite precision fixed point arithmetic operations and is recommended to be used as a reference codec to verify the performance of a bit exact EVRC NW implementation of the fixed point C simulation of a test codec The bit exact EVRC NW codec along with the appropriate test vectors to verify the bit exactness performance are included in the associated Software Distribution 3 4 4 1 Fixed Point Codec Program Files This section describes the C program files which are provided in the associated software distribution for this document 3 4 4 2 Compiling the Fixed Point Codec Simulation The source code f
167. e with Mode 0 of SO 68 support M M c18 Nominal Mode 7 interoperable with Mode 0 of SO 68 support M T c19 Low Mode 0 LB portion of Wideband mode decoder test only c20 Low Mode 0 LB portion of Wideband mode decoder test only M T c21 Low Mode 4 interoperable with Mode 0 of SO 68 support M M c22 Low Mode 4 interoperable with Mode 0 of SO 68 support M T c23 Low Mode 4 interoperable with Mode 0 of SO 68 support T T C24 Low Mode 4 interoperable with Mode 0 of SO 68 support T M c25 High Mode 0 LB portion of Wideband mode decoder test only M M c26 High Mode 0 LB portion of Wideband mode decoder test only M T c27 High Mode 4 interoperable with Mode 0 of SO 68 support M M c28 High Mode 4 interoperable with Mode 0 of SO 68 support M T c29 High Mode 4 interoperable with Mode 0 of SO 68 support T T c30 High Mode 4 interoperable with Mode 0 of SO 68 support T M c31 Mode 0 1 D amp B 1 PLS LB portion of Wideband mode decoder test only c32 Mode 0 1 D amp B 1 PLS LB portion of Wideband mode decoder test only M T c33 Mode 4 1 D amp B 1 PLS interoperable with Mode 0 of SO 68 support M M C34 Mode 4 1 D amp B 1 PLS interoperable with Mode 0 of SO 68 support M T c35 Mode 4 1 D amp B 1 PLS interoperable with Mode 0 of SO 68 support T T c36 Mode 4 1 D amp B 1 PLS interoperable with Mode 0 of SO 68 support T M c37 Mode 0 3 FER LB portion of Wideband mode decoder test only c38 Mod
168. ec along with the appropriate test vectors to verify the bit exactness performance are included in the associated Software Distribution 3 2 4 1 Fixed Point Codec Program Files This section describes the C program files which are provided in the associated software distribution for this document All of the files needed to compile run and verify the fixed point codec are located in the directory so68 EVRCB_FX 3 16 20 21 22 23 24 25 26 28 29 30 31 32 33 34 36 37 3GPP2 C S0018 E v2 0 3 2 4 2 Compiling the Fixed Point Codec Simulation The source code for the fixed point codec simulation has been written in C and can be compiled using any general purpose compiler such as the GNU G compiler and make utility Refer to Section 3 3 for information regarding obtaining GCC make and relevant documentation Two GCC compatible makefiles have been included in the so68 EVRCB FX build directory Typing make in the build directory will compile and link the code and create the executable file called EvrcB fx EvrcB fx exe on Win32 systems which will be placed in the build directory The included makefiles may require some user modification for a particular hardware platform and or operating system 3 243 Running the Fixed Point Codec Simulation The EVRC B executable files use command line arguments to receive all information regarding input and output files and various parameters used during
169. ec combination and the suffix indicates the condition numbers in Table 2 4 2 3 1 2 and Table 2 4 2 3 2 2 2 77 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 3GPP2 C S0018 E v2 0 Naming conventions for the remaining two experiments follow accordingly Detailed descriptions of all processing operations are given in Section 6 2 4 41 Encoding by the Test Codec All of the source files will be encoded by the test codec to produce encoded packet files For ease of reference it is recommended that directories so73 subjctv exp t be created to deposit the test encoder output packets and that the naming conventions be made consistent with the master codec 2 4 4 2 Decoding by the Master Test Codecs The encoded packet files generated from the various encoders conditions shall be processed through the master and test decoders 2443 Introduction of Impairments For the frame error conditions the impaired master codec encoded packet files are provided in the so73 subjctv exp m pkt directory Unlike other conditions this condition uses only the test decoder and not the test encoder For the Dim and Burst processing and also the Packet Level Signaling conditions the processing requires inputs from a signaling file to control maximum encoding rate An external software utility Evrc nw iwf in Section 3 4 2 3 is also needed to reduce the data rate of certain packets from full
170. ech encoder and that the output data stream can be saved to a set of files It is also necessary to input data stream files into the speech decoder and have the output speech data saved to a set of files This process suggests the use of a computer based data acquisition system to interface to the codec under test Since the hardware realizations of the speech codec may be quite varied it is not desirable to precisely define a set of hardware interfaces between such a data acquisition system and the codec Instead only a functional description of these interfaces will be defined A host computer system is necessary to handle the data files that must be input to the speech encoder and decoder and to save the resulting output data to files These data files will contain either sampled speech data or speech codec parameters hence all the interfaces are digital The generic Standard Equipment is shown in Figure 3 1 Host Computer Host Computer Digital Digital Data Data Speech Speech Encoder or Decoder Encoder or Decoder Figure 3 1 Basic Test Equipment The host computer has access to the data files needed for testing For encoder testing the host computer has the source speech data files which it outputs to the speech encoder The host computer simultaneously saves the speech parameter output data from the encoder Similarly for decoder testing the host computer outputs speech parameters from a disk file and saves the decoder output spee
171. econd experiment for SO 68 is performed as a P 835 listening test 2 2 2 3 1 Subjective Experiment for SO 68 The Test Parameters for Listening Experiment are presented in Table 2 2 2 3 1 1 Table 2 2 2 3 1 1 SO 68 Listening Experiment Test Parameters Type of test MOS P 800 Number of talkers 4 males 4 females Background noise Audio Input Level 22 dB 32 dB 12 dB Filter characteristics MIRS Reference conditions 8 Direct 3 9 15 21 27 33 39 dBQ 2 18 3GPP2 C S0018 E v2 0 Test conditions a Low Audio Input Level 32 dB 9 3 5 8 kbps 1 d amp b 1 pls b Nominal Audio Input Level 22 dB 9 3 5 8 4 8 kbps c High Audio Input Level 12 dB 9 3 5 8 kbps d Nominal Audio Input Level 22 dB 9 3 5 8 kbps 396 FER M M M T Only Encoder Decoder Combinations 4 MM M T T T T M Conditions a c 2 M M M T Condition d The Test Conditions for Listening Experiment are presented in Table 2 2 2 3 1 2 Table 2 2 2 3 1 2 SO 68 Listening Experiment I Test Conditions Label Operating Point Condition Encoder Decoder Combinations a01 Reference MNRU 3dB a02 Reference MNRU 9dB a03 Reference MNRU 15dB a04 Reference MNRU 21dB a05 Reference MNRU 27dB a06 Reference MNRU 33dB a07 Reference MNRU 39dB a08 Reference Direct a09 EVRC B 9 3 kbps Nominal 22 dB M M a10 EVRC B 9 3 kbps Nominal 22 dB M T
172. ed Point Bit Exact Codec for SO 70 3 24 3 34 1 Fixed Point Codec Program FilQS eeeeeesesesesososovonononsnonouoasasososososososososososnsosnsnsononnnn 3 24 3 3 4 2 Compiling the Fixed Point Codec Simulation e 3 24 3 3 4 8 Running the Fixed Point Codec 5 3 24 3 944 Eile Formats 5 io oe adio cat idet ere Tie 3 25 3 3 4 5 Verifying Bit Exact Performance of the Fixed Point Test 3 25 3 4 Specific Standard Test Conditions for SO 73 3 34 341 Audio Path and Calibration for SO 73 nemen 3 34 3411 Audio 3 34 FA 1D GallbratiOri ier tkt ee htt hr the ket ce it teh ED 3 34 342 Software Test Tools for SO 73 eene nen 3 35 3 4 2 1 Channel Model Utilities exsig29 exe em HH HH 3 35 3 4 2 2 Channel Error and Signaling Masks sssssseeeeennn 3 35 3 4 3 EVRC NW Interworking Function IW F e enn 3 35 25 26 27 28 29 30 31 32 33 3GPP2 50018 v2 0 3424 P 341 TX File 9e reb nere exe Ra APRAN a a bak beh DOUA den DRA 3 36 343 Master Codec for SO 73 3 36 3 4 3 1 Compiling the Master Codec Simulation IIR 3 37 3 4 8 2 Running the Master Codec Simulation
173. ed for sv56 differs by SdB from the value required for this specification For example in order to adjust speech files 22dB in accordance with this specification the calling sequence is sv56 25 File In File Out 8 4 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 3GPP2 C S0018 E v2 0 3 1 2 3 j Law Companding mu l c This program applies Law companding to the sample values in a linearly quantized speech file according to 7 The source code mu l c is available from 6 and The input to the program is the speech file to be companded The output is the companded speech file Both files are linearly quantized speech files in accordance with Section 3 1 3 3 of this document The program is invoked as follows mu linput filename output filename 3 4 8 Master Codec for SO 3 This section describes the C simulation of the speech codec specified by 1 The master codec C simulation used for verifying the performance of a non bit exact EVRC implementation shall be the floating point master C simulation included in the associated Software Distribution 1a 3 1 3 1 Compiling the Master Codec Simulation The source code for floating point C simulation has been written in ANSI C and compiled using the GNU GCC C compiler and make utility Refer to Section 3 1 2 for information regarding obtaining GCC make and relevant documentation A GCC compatible makefile ha
174. ed in the following ways for the two experiments 1 A test sample for each codec combination talker and level channel condition or background noise level Experiment or Il or MNRU value and talker shall be presented exactly once 2 Randomization has been done in blocks such that one sample of each codec level codec channel condition or codec background noise level again depending on Experiment I or or MNRU value will be presented once with a randomly selected talker in each block This ensures that listeners rate each codec condition being tested equally often in the initial middle and final parts of the session and will mitigate the effects of practice and fatigue A block contains 31 file samples A session will consist of eight blocks of 31 file samples plus one practice block of 31 at the beginning of each session for each experiment There are a total of eight sessions per experiment A particular randomization session shall not be presented to more than eight listeners 3 Talkers shall be chosen so that the same talker is never presented on two consecutive trials within the same block The randomization lists for each of the eight file sets of each experiment are given in so3 subjctv exp1 data play Ist and soS subjctv exp2 data play lst respectively 2 1 6 Presentation Presentation of speech material for the SO 3 codec listening tests shall be made with one side of high fidelity circum aural headphones The s
175. ed to digitally encode the speech signal for transmission at a variable data rate of 8550 bps 4000 bps 2000 bps or 800 bps The SO 77 is same as SO 73 except it includes a maximum rate 2 kbps mode speech codec Like some other speech coding standards this standard provides a bit exact method of verifying the test codec for minimum performance In this optional procedure a given set of test vectors are input to the test codec and the output vectors from the test codec must be bit exact with the output vectors given in the software distribution which is associated with this standard If they are bit exact the test codec passes the minimum performance requirement and no further testing is required The bit exact mode of testing however is only applicable to codecs whose design conforms in all respects to the algorithmic description of the specific EVRC service option including the noise suppression rate determination and post filter components Should the candidate EVRC differ in any of these components the test codec shall be tested using the objective and subjective tests prescribed by this standard That is EVRC compliance of a test codec can be achieved by either e Complying with Sections 2 1 1 and 2 1 2 SO 3 or Sections 2 2 1 and 2 2 2 SO 68 or Sections 2 3 1 and 2 3 2 SO 70 or Section 2 4 1 and 2 4 2 SO 73 or Section 2 5 1 and 2 5 2 SO 77 and demonstrating bit exactness according to the procedure described in Section 3 1
176. eference plane of the headphone 4 Signal shall be delivered to the headphone on the listener s preferred telephone listening ear and the other ear shall be uncovered No signal shall be delivered to the other headphone 3 4 1 2 Calibration The audio circuit shall deliver an average sound level of the stimuli to the listener at 18 dBPa 76 dB SPL at the ear reference plan This level was chosen because it is equivalent to the level delivered by a nominal ROLR handset driven by the average signal level on the PSTN network This level may be calibrated using a suitable artificial ear with circum aural headphone adapter and microphone A test file with a reference signal is included with the source speech database for the purpose of calibration The file cal_1004 16k is located in the directory so73 cal of the companion software The calibration file contains a 22 dB 1004 Hz reference signal The audio circuit shall be calibrated so that the test signal has a level of 15 dBPa at the ear reference plane while maintaining compliance with Section 3 4 1 1 3 34 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 3GPP2 C S0018 E v2 0 3 4 2 Software Test Tools for SO 73 This section describes a set of software tools useful for performing the MPS tests The code has been developed and compiled using the GNU g compiler and software maintenance utilities The tools have been verified under various representa
177. egory which best describes the sample you just heard for purposes of everyday speech communication the OVERALL SPEECH SAMPLE was EXCELLENT GOOD FAIR POOR BAD 2 85 20 21 22 23 24 25 26 27 28 3GPP2 C S0018 E v2 0 2 4 9 Analysis of Results The response data from the practice blocks shall be discarded Data sets with missing responses from listeners shall not be used i e a complete set of data is required for 24 listeners four for each of six listening panels Responses from the different listening panels for the corresponding test conditions shall be treated as equivalent in the analysis 2 4 9 1 Basic Results for the SO 73 Listening tests The votes for each of the test conditions for SO 73 Experiments and II shall be averaged to produce an associated mean score M as shown in Equation 2 4 9 1 1 and a Standard Deviation SD as shown in Equation 2 4 9 1 2 where L is the number of listeners and T is the number of talkers involved in the experiment pL LxT LxT 1 lt 2 4 9 1 1 SD 2 4 9 1 2 2 4 9 2 Minimum Subjective Requirement for SO 73 Listening Tests The Terms of Reference for the MPS tests state that the mean score for each of the Test Encoder Decoder Combinations E DC should be not worse than the mean score for the Reference E DC For most of the test conditions involved in the subjective experiments the
178. elay Recommendation The algorithmic delay for the specific EVRC WB test codec should be calculated analytically by the codec manufacturer In considering the algorithmic delay it can be assumed that all transmission channels have infinite bandwidth and that all processing elements have infinite throughput Algorithmic delay is defined as the sum of all sequential filter delays and buffering delays in the encode decode path The maximum end to end algorithmic delay should be no greater than that of the master codec For the master codecs defined in 1 the algorithmic delay is given as Delay Source Delay ms Signal Preprocessing Delay 0 0 Filterbank Analysis 0 8 LPC Analysis Look ahead 10 0 LPC Analysis Window 20 0 Highband excitation generation delay 1 5 Highband synthesis overlap and add delay 2 0 Filterbank Synthesis Delay 1 1 Total 35 4 Therefore the total algorithmic delay imposed by a SO 70 test codec should not exceed 35 4 milliseconds 2 3 2 Subjective Performance Testing for SO 70 This section outlines the subjective testing methodology of the subjective performance test The purpose of this testing is to evaluate the quality of the test codec under a variety of conditions which may occur in the CDMA system To accomplish this suites of listening experiments have been designed to test speech codec quality under a variety of conditions depending on a number of parameters These conditions include channel impairment
179. ementation may support SO 73 for both 16 kHz and 8 kHz sample rate for example a mobile station that supports wideband electro acoustics The main parameter in the decision tree is 16 kHz support in the implementation Depending on the implementation profile of the Device under test one of 2 possible Test Suites are to be used to demonstrate SO 73 compliance These 2 test suites named Test suites A and B and the individual input test vectors comprising the Test suites are highlighted in Table 3 4 4 5 1 Table 3 4 4 5 1 Test Suites of input test vectors for SO 73 compliance Test Suites Directory containing input test vectors so73 testvec source suiteA B so73 testvec source suiteB Files in the so73 testvec directory are provided for the purpose of qualifying a test codec as bit exact and conform to the file naming convention described in Section 2 2 4 The so73 testvec directory is divided into 2 subdirectories so73 testvec source and so73 testvec fixed The so73 testvec source directory contains input source files as well as packet files injected with frame erasures The so73 testvec fixed directory contains files processed with the EVRC NW fixed point reference software The files in these directories are the reference files for bit exact compliance A test codec is bit exact if it can reproduce all of the reference files in the s073 testvec fixed directory exactly The outputs of the encoder and decoder o
180. en as Delay Source Delay ms Signal Preprocessing Delay 0 0 Filterbank Analysis 0 8 LPC Analysis Look ahead 10 0 LPC Analysis Window 20 0 Highband excitation generation delay 1 5 Highband synthesis overlap and add delay 2 0 Filterbank Synthesis Delay 1 1 Total 35 4 Therefore the total algorithmic delay imposed by a SO 73 test codec should not exceed 35 4 milliseconds 2 4 2 Subjective Performance Testing for SO 73 This section outlines the subjective testing methodology of the subjective performance test The purpose of this testing is to evaluate the quality of the test codec under a variety of conditions which may occur in the CDMA system To accomplish this suites of listening experiments have been designed to test speech codec quality under a variety of conditions depending on a number of parameters These conditions include channel impairments audio background noise and different input levels Figure 2 4 2 1 illustrates a decision tree to arrive at the suite of tests that are needed to demonstrate Minimum Performance Spec compliance of a Test implementation of SO 73 for different profiles of equipment that support SO 73 2 65 3GPP2 50018 v2 0 Is 16kHz Sampling Rate Supported Yes No Run Test Suite A Run Test Suite B Figure 2 4 2 1 SO 73 Subjective test suite decision flowchart An implementation may support SO 73 only for 8 kHz sample rate input output for example a Base station tran
181. ener texte eaa 2 18 2 2 2 2 Method of Measurement enemies 2 18 2 2 2 3 Test Conditions and Test Design for SO 68 2 18 2 2 2 3 1 Subjective Experiment for SO 68 2 18 2 2 2 8 Subjective Experiment Il for SO 68 2 20 2 2 2 3 3 Numerical Parameters for the SO 68 Listening Experiments 2 21 2 2 3 Speech Material for SO 68 Testing nnne 2 22 2 2 8 1 Source Speech Material for SO 68 Experiment l M 2 23 2 2 3 2 Source Speech Material for SO 68 Experiment ll 2 23 2 2 4 Processing of Speech Material for SO 68 Testing RA 2 23 2 2 4 1 Encoding by the Test Codec entree tanen tenens 2 24 2 24 2 Decoding by the Master Test Codecs nnns 2 24 2 2 4 3 Introduction of Impairments nnne nennen nete 2 24 2 2 44 Ensuring Proper Encoded Frame Packet Files 1 2 24 2 2 4 5 Post processing of test condition output 2 24 20 21 22 23 24 25 26 27 28 29 30 31 32 33 3GPP2 C S0018 E v2 0 2 2 5 iet etui enu ebrio reu eren equo 2 26 2 26 2 28 OOF 2 28 2 2 8 Listening Test Procedures
182. ent condition power levels 12 dB 22 dB and 32 dB and background noise conditions 20 dB SNR babble noise 15 dB SNR car noise and 15 dB SNR street noise The input source files used in the average data rate test have an approximate voice activity factor of 0 6 and are the same input files used in the subjective portion of the experiment 2 4 1 1 1 Average Data Rate Computation for SO 73 The average channel data rate for the test codec shall be computed for each of the benchmark files as follows R 9600 1 4800 No 2400 Nq4 1200 Ng N where N4 number of frames encoded at Rate 1 2 63 20 21 22 23 24 25 3GPP2 C S0018 E v2 0 No number of frames encoded at Rate 1 2 N4 number of frames encoded at Rate 1 4 Ng number of frames encoded at Rate 1 8 and 1 N4 Ng For the 16 kHz input The total average channel data rate for the test codec is then given by Ravg 1 7 R ambient background segment 12dB R ambient background segment 32dB R ambient background segment 22dB R 20 dB SNR babble noise segment 22dB R 10 dB SNR car noise segment 22dB R 20 dB SNR car noise segment 22dB R 15 dB SNR street noise segment 22aB For the 8 kHz input The total average channel data rate for the test codec is then given by Ravg 1 6 R ambient background segment 12dB R ambient background segment 32dB R ambient background segment
183. eriment 1 Test Conditions Exp 1 Wideband ACR Reference Conditions File MNRU a01 7dB MNRU Reference a02 14dB MNRU Reference a03 21dB MNRU Reference a04 28dB MNRU Reference a05 35dB MNRU Reference a06 42dB MNRU Reference 07 49dB MNRU Reference a08 Direct Source Reference Test Conditions File Condition Enc Dec a09 Nominal level M M a10 Nominal level M T a11 Nominal level T T 12 Nominal level T M a13 Low level 196 d amp b M M 14 Low level 1 d amp b M T 15 Low level 1 d amp b T T a16 Low level 1 d amp b T M 17 High level M M a18 High level M T a19 High level T T a20 High level T M a21 1 FER 1 PLS M M a22 1 FER 1 PLS M T a23 3 FER M M a24 3 FER M T 2 40 3GPP2 C S0018 E v2 0 2 2 3 2 3 2 Subjective Experiment 2 for SO 70 3 The Test Parameters for Listening Experiment 2 are presented in Table 2 3 2 3 2 1 4 Table 2 3 2 3 2 1 SO 70 Listening Experiment 2 Test Parameters Number of talkers Test conditions o Car Noise 10 dB SNR o Car Noise 20 dB SNR 2 FER o Street Noise 15 dB SNR o Babble noise 20 dB S N Encoder Decoder Combinations 4 M M M T T T T M Number of talkers 3 males 3 females Test conditions o Car Noise 10 dB SNR o Noise 20 dB SNR 2 FER o Street Noise 15 dB SNR o Babble noise 20 dB S N Encoder Decoder Combinations 4 M M T T T M 7
184. es They are designed to exercise the decoder s frame error handling vec 07 1 pkt vec 07 2 pkt vec 07 3 pkt vec 08 1 pkt vec 08 2 pkt vec 08 3 pkt vec 10 1 pkt vec 10 2 pkt vec 10 3 pkt Encoded packet w 196 FER Encoded packet w 296 FER Encoded packet w 396 FER Encoded packet w 196 FER Encoded packet w 296 FER Encoded packet w 396 FER Encoded packet w 196 FER Encoded packet w 2 FER Encoded packet w 396 FER 3 1 4 6 2 Instructions for Processing Bit Exact Test Vectors The following table is a list of source files to be processed in DEFAULT MODE rates 1 3 4 allowed and the names of the corresponding reference files The files are to be processed as follows Encode EvrcFix e i file pcm o file pkt Decode EvrcFix d i file pkt o file dec Table 3 1 4 6 2 1 Source and Bit exact Default Mode Test Vector Files PCM Source File Encoded Packet File Decoded Speech File rda_high pcm rda_low pcm rda_mod pcm rda_test pcm vec_01 pcm rda_high pkt rda_high dec rda_low pkt rda_low dec rda_mod pkt rda_mod dec rda_test pkt rda_test dec vec_01 pkt vec_01 dec 3 11 3GPP2 C S0018 E v2 0 PCM Source File Encoded Packet File Decoded Speech File vec 02 pcm vec 03 pcm vec 04 pcm vec 05 pcm vec 06 pcm vec 07 pcm vec 08 pcm vec 09 pcm vec 10 pcm vec 11 pcm vec 12 pcm vec 02 pkt vec 03 pkt vec 04 pkt vec 05 pkt vec 06 pkt vec 07 pkt vec 08 pkt vec 09 pkt vec 10 pkt vec 11 pkt vec 12 p
185. evel of the stimuli to the listener at 15 dBPa 79 dB SPL at the ear reference plan This level was chosen because it is equivalent to the level delivered by a nominal ROLR handset driven by the average signal level on the PSTN network This 3 13 20 21 22 23 24 25 26 27 28 29 3GPP2 C S0018 E v2 0 level may be calibrated using a suitable artificial ear with circum aural headphone adapter and microphone A test file with a reference signal is included with the source speech database for the purpose of calibration The file cos1004 290 is located in the directory so68 cal of the companion software The calibration file contains a 22 dB 1004 Hz reference signal The audio circuit shall be calibrated so that the test signal has a level of 15 dBPa at the ear reference plane while maintaining compliance with Section 3 2 1 1 3 2 2 Standard Software Test Tools for SO 68 This section describes a set of software tools useful for performing the MPS tests The code has been developed and compiled using the GNU g 9 compiler and software maintenance utilities The tools have been verified under various representative operating systems on a number of different hardware platforms The 3GPP2 supplied tools are all located in the so68 tools directory in the associated Software Distribution and can be built using the GNU g compiler Other software tools such as scaldemo actlev filter andastrip are available in 6 3 2 8
186. f s signaling mask fil i encoded packet fil o dimmed packet fil where Evrc nw converts full rate frames the input encoded packet file to half rate frames at packet level that is using a simple scaling down of the packet instead of a complicated transcoding method 3 47 3GPP2 50018 v2 0 3 5 24 P 341 Tx Filter The software utility p341_tx c can be compiled to yield a Tx filtering utility p341 tx with usage defined as p341 tx input file name output file name where p341 tx is the 3GPP2 Tx filter compliant to ITU T P 341 Figure 3 5 2 4 1shows the frequency response of p341_ tx filter Also shown in this figure is the response of the ITU T P 341 STL 2000 filter implementation as well as the transmit masks for the ITU T P 341 P 311 and the wideband transmit response from Table 9 in the 3GPP electro acoustics specification 21 From this figure it can be seen that the STL 2000 filter response in red does not meet the frequency response of the 3GPP electro acoustics specification while the p341 tx filter response in green meets both the P 341 P 311 masks as well as the 3GPP electro acoustics specification mask ITU T 3GPP Transmit Masks vs Filter Responses 20 STL 2000 P 341 3GPP2 P 341 Tx 3GPP Handset Send 5dB ITU T P 311 341 Tx Magnitude dB LL ii i i L jot i 4 10 10 Frequency
187. f 0 78 and are the same input files used in the subjective portion of the experiment 2 2 1 1 1 Average Data Rate Computation for SO 68 The average channel data rate for the test codec shall be computed for each of the benchmark files as follows R 9600 N4 4800 No 2400 N4 1200 Ng N where N4 number of frames encoded at Rate 1 No number of frames encoded at Rate 1 2 N4 number of frames encoded at Rate 1 4 Ng number of frames encoded at Rate 1 8 and 1 N4 Ng The total average channel data rate for the test codec is then given by Ravg 1 6 R ambient background segment 12dB R ambient background segment 32dB R ambient background segment 22dB R 20 dB SNR babble noise segment 22dB R 15 dB SNR car noise segment 22dB R 15 dB SNR street noise segment 22dB The above files are to be processed with EVRC B encoder at various capacity operating points defined by the active speech average channel rate shown in Table 2 2 1 1 1 1 Table 2 2 1 1 1 1 Target ADR vs Capacity Operating Point Capacity Operating Point Target Average Channel Data Rate kbps active speech average channel data rate EVRC B 9 3k bits sec 6 93 41 596 EVRC B 8 5 bits sec 6 42 41 596 EVRO B 7 5k bits sec 5 52 41 596 2 16 20 21 22 23 24 25 26 27 28 29 30 3GPP2 C S0018 E v2 0 Capacity Operating Po
188. f the test codec are to be obtained for the conditions given below in Table 3 4 4 5 2 Table 3 4 4 5 5 The processing steps for these conditions are illustrated in Section 6 3 40 3GPP2 C S0018 E v2 0 Table 3 4 4 5 2 SO 73 Encoder Suite A Bit exact Test Conditions Reference packet files for bit Input File Operating Point Condition exact compliance src s22 EVRC NW operating point Nominal 22 dB evrc nw _ 0 22 0 16 kHz sampling src s12 EVRC NW operating point High 12 dB evrc_nw_op0 p12 0 16 kHz sampling Src s32 EVRC NW operating point Low 32 dB evrc nw op0 dim 196 p32 0 16 kHz sampling 196 d amp b src c10 EVRC NW operating point Nominal 22 dB evrc nw opO pc1 0 16 kHz sampling 10 dB car noise src c20 EVRC NW operating point Nominal 22 dB evrc_nw_op0 pc2 0 16 kHz sampling 20 dB car noise src s15 EVRC NW operating point Nominal 22 dB evrc nw opO ps 0 16 kHz sampling 15 dB street noise src b20 EVRC NW operating point Nominal 22 dB evrc_nw_op0 pb 0 16 kHz sampling 20 dB babble noise src s22 8k EVRC NW operating point Nominal 22 dB evrc_nw_op1 p22 1 8 kHz sampling src s12 8k EVRC NW operating point High 12 dB evrc_nw_op1 p12 1 8 kHz sampling src s32 8k EVRC NW operating point Low 32 dB evrc_nw_op1 p32 1 8 kHz sampling src s22 8k EVRC NW operating point Nominal 22 dB evrc nw opti dim 196 p22 1 8 kHz sampling 196 d amp b src
189. for Enhanced Variable Hate Codec Speech Service Options 3 68 and 70 for Wideband Spread Spectrum Digital Systems September 2007 3GPP2 C S0009 0 v1 0 Speech Service Option Standard for Wideband Spread Spectrum Systems December 1999 3GPP2 C S0018 0 v1 0 Minimum Performance Specification for the Enhanced Variable Rate Codec Speech Service Option 3 for Spread Spectrum Digital Systems December 1999 ANSI S1 4 1983 R2006 Sound Level Meters Specification for March 2006 ANSI S1 4A 1985 R2006 Sound Level Meters Specifications for Supplement to ANSI S1 4 1983 March 2006 ITU T Recommendation G 191 Software Tools for Speech and Audio Coding Standardization September 2005 User s Group on Software Tools ITU T Software Tool Library 2005 User s Manual Distributed with the software for STL2000 September 2005 ITU T Recommendation G 711 Pulse code modulation PCM of voice frequencies November 1988 ITU T Recommendation G 728 Coding of speech at 16 kbit s using low delay code excited linear prediction September 1992 ITU T Recommendation P 56 Objective Measurement of Active Speech Level March 1993 ITU T Recommendation P 800 Methods for Subjective Determination of Transmission Quality Annex B Listening Tests Absolute Category Rating ACR August 1996 ITU T Recommendation P 810 Modulated Noise Reference Unit MNRU February 1996 ITU T Recommendation P 830 Methods for Objective and Subjective
190. for SO 68 emen 3 16 20 21 22 23 24 25 26 27 28 29 30 31 32 3GPP2 50018 v2 0 3 2 4 1 Fixed Point Codec Program Files tenentes 3 16 3 2 4 2 Compiling the Fixed Point Codec Simulation IH 3 17 3 2 4 3 Running the Fixed Point Codec 5 3 17 3 2 4 4 File Formats ease P Haee E dea BUD eae 3 18 3 2 4 5 Verifying Bit Exact Performance of the Fixed Point Test 3 18 3 3 Specific Standard Test Conditions for SO 70 eene 3 20 3 81 Audio Path and Calibration for SO 70 3 20 3311 eaa aiiai iaa aaaea pitaa 3 20 9 450 A tc te 3 20 3 82 Software Test Tools for SO 70 eene en ener 3 21 3 3 2 1 Channel Model Utilities exsig28 exe 3 21 3 3 2 2 Channel Error and Signaling Masks cccscceeeceeeeeeeeeeaeeeeaeeseeeeesaeeeeaeeseneeteaees 3 21 3 3 2 3 EVRC WB Interworking Function IWF rnnt 3 21 3 3 OAs 3234 TX Ell roti easter cay 3 21 3 3 3 Master Codec for SO 70 3 22 3 3 3 1 Compiling the Master Codec Simulation nnn 3 22 3 3 3 2 Running the Master Codec Simulation sse 3 23 3 3 3 3 File Formats for SO 70 nnne enne 3 23 3 3 4 Fix
191. from the various encoders conditions shall be processed through the master and test decoders For all conditions the signal power shall be normalized to 22 dB The signal shall then be law companded into PCM files See Sections 3 1 2 2 and 3 1 2 3 for details in using the provided software tools that can be used for this post processing 2 1 4 3 Introduction of Impairments For the 396 frame error condition Experiment condition 4 the impaired master codec encoded packet files are provided in the so3 subjctv exp1 m_pkt directory Unlike other conditions this condition uses only the test decoder and not the test encoder The performance of the test decoder is compared to that of master decoder using master encoder generated packets from two different frame error models 3 forward FER and 3 reverse FER The 3 forward FER packets pf3 are then used by the test decoder to generate the master encoder test decoder combination mt4 and the 396 reverse FER packets pr3 are used by the test decoder to generate the master encoder test decoder combination tt4 The respective master decoder outputs are the mm4 and tm4 To clarify the naming convention the following four conditions are tested e mm4 master encoder master decoder 3 forward link FER e tm4 master encoder master decoder 3 reverse link FER e mt4 master encoder test decoder 3 forward link FER e 14 master encoder test decoder 396 reverse link F
192. g how good you think they sound Use the single headphone on the ear you normally use for the telephone On each trial a two sentence sample will be played After you have listened to the sample determine the category from the list below which best describes the overall quality of the sample Press the numeric key on your keyboard corresponding to your rating for how good or bad that particular passage sounded The quality of the speech should be rated according to the scale below 5 4 3 2 1 Excellent Good Fair Poor Bad During the session you will hear samples varying in different aspects of quality Please take into account your total impression of each sample rather than concentrating on any particular aspect Figure 2 3 8 1 1 Instructions for Listeners 2 3 8 2 835 Listening Test Procedures Experiments 2 4 and 6 Experiments 2 4 and 6 use the P 835 test methodology described in ITU T Rec P 835 13 The P 835 methodology is specifically designed to evaluate the quality of speech in background noise lt yields a measure of Signal Quality SIG a measure of Background Quality BAK and a measure of Overall Quality OVRL In general OVRL scores are highly correlated with MOS but the OVRL score provides greater sensitivity and precision in test conditions involving background noise While the OVRL score is of most interest here the SIG and BAK scores also provide valuable diagnostic information For ea
193. g mask fil i encoded packet fil o dimmed packet fil where Evrc nw converts full rate frames the input encoded packet file to half rate frames at packet level that is using a simple scaling down of the packet instead of a complicated transcoding method 3 35 3GPP2 C S0018 E v2 0 3 4 2 4 P 341 Tx Filter The software utility p341_tx c can be compiled to yield a Tx filtering utility p341 tx with usage defined as p341 tx input file name output file name where p341 tx is the 3GPP2 Tx filter compliant to ITU T P 341 Figure 3 4 2 4 1 shows the frequency response of p341 tx filter Also shown in this figure is the response of the ITU T P 341 STL 2000 filter implementation as well as the transmit masks for the ITU T P 341 P 311 and the wideband transmit response from Table 9 in the 3GPP electro acoustics specification 21 From this figure it can be seen that the STL 2000 filter response in red does not meet the frequency response of the 3GPP electro acoustics specification while the p341 tx filter response in green meets both the P 341 P 311 masks as well as the SGPP electro acoustics specification mask ITU T 3GPP Transmit Masks vs Filter Responses 20 STL 2000 P 341 3GPP2 P 341 Tx 3GPP Handset Send 5dB ITU T P 311 341 Tx Magnitude dB iiiiii i i i Gi 4 10 10 Frequency Hz Figure 3 4 2
194. he source code for the master codec fixed point bit exact codec and software tools as well as the material needed to perform the objective and subjective tests described in this document are provided within an associated Software Distribution The directory structure of the Software Distribution is represented in Table 4 1 Table 4 2 Table 4 3 Table 4 4 and Table 4 5 Table 4 1 contains a brief description of the Software Distribution for the EVRC A MPS Table 4 2 contains a brief description of the Software Distribution for the EVRC B MPS Table 4 3 contains a brief description of the Software Distribution for the EVRC WB MPS Table 4 4 contains a brief description of the Software Distribution for the EVRC NW MPS and Table 4 5 contains a brief description of the Software Distribution for the EVRC NW2K MPS The prime sub directories of these distributions are so3 so68 so70 so73 or so77 respectively These tables contain brief descriptions of the contents of these directories as well as cross references to the sections of this document in which they are described in detail Table 4 1 Description of EVRC A Software Distribution Contents References so3 simul fixed source code for the bit exact 3 1 4 fixed point code so3 subjctv Speech and other material 2 1 3 2 1 4 2 1 5 necessary to perform Subjective Experiments and Il so3 objctv Speech material necessary to perform the Average Data Rate so3 cal Output level calibration fil
195. he test A test session consists of the same number of blocks as there are talkers involved in the test Each session is presented to a listening panel of four listeners Randomizations are constructed such that talker gender is alternated on successive trials resulting in the same talker never being presented on consecutive trials Table 2 3 5 1 shows an example randomization for a single listening panel Each entry in the table is the file name for a sample with the following file naming convention xxyy zzz where xx is the talker yy is the sample and zzz is the test condition Table 2 3 5 1 Example Randomization for the Experiments 1 3 and 5 ACR Test 1 20 21 al5f4s7 07 157 27 adim2s6 aa CES ae l 0294 20 DL ucc 24 allm4s5 a20f a04f1s4 al3 a03m4s3 a22f3s3 al8 a04 a23 2091252 a05n3s5 al mdsi 2092337 2182136 a17m2s5 alinis2 2011453 219 454 a06m4s8 2151253 256 242234 a13 4s6 24 a02m3s1 a2 f4s8 1251 a03f2s1 a08m3s7 156 a05 al8m3s1 a07m3s1 f4s8 fls7 a02f1s4 al7m4s1 18 156 a01m4s3 352 al5m2s8 al3f4s6 F3sl1 F352 a22m2s6 10 251 a06m2s4 al9f3s6 a20 a al f4s5 1152 f2s8 m4s6 s4 s2 s7 m2s6 s6 s7 fls5 m3s5 f4s1 a23m3s8 a03f3s4 a05m1s5 a09f1s7 olejnej gt HS of HS ofw Fh Phy S h a n
196. hms and so on They should not be trained listeners that is they should not have been trained in these or previous listening studies using feedback trials Age distribution and gender should be nominally balanced across listening panels Each listener shall provide data only once for a particular evaluation A listener may participate in different evaluations but test sessions performed with the same listener should be at least two months apart so as to reduce the cumulative effects of experience 2 3 8 Listening Test Procedures 2 3 8 1 ACR Listening Test Procedures Experiments 1 and 5 The listeners shall listen to each sample and rate the quality of the test sample using a five point scale with the points labeled 5 Excellent 4 Good 3 Fair 2 Poor 1 Bad Data from 32 listeners shall be used for Experiments 1 3 and 5 four listeners for each listening panel where each listening panel uses a different randomization Before starting the test the listeners should be given instructions for performing the subjective test An example set of instructions for the ACR are presented in Figure 2 3 8 1 1 The instructions may be modified to allow for variations in laboratory data gathering apparatus 2 57 3GPP2 C S0018 E v2 0 This is an experiment to determine the perceived quality of speech over the telephone You will be listening to a number of recorded speech samples spoken by several different talkers and you will be ratin
197. ifies the name of the input speech file or the name of the input packet file if only decoding is being performed see the d option below o outf required Specifies the name of the output speech file or the name of the output packet file if only encoding is being performed see the e option below d Instructs the simulation to perform only the decoding function The input file must contain packets of compressed data e Instructs the simulation to perform only the encoding function The output file will contain packets of compressed data If neither the d or the e option is invoked the coder performs both the encoding and decoding functions by default M max Sets the maximum allowable data rate to max where max is element of 4 3 2 1 using the codes specified in the first column of Table 3 4 3 3 1 m min Sets the minimum allowable data rate to min where min is element of 4 3 2 1 using the codes specified in the first column of Table 3 4 3 3 1 If neither the M nor m option is invoked the coder allows the data rate to vary between Rate 1 and Rate 1 8 3 4 8 3 File Formats for SO 73 Files of speech contain 2 s complement 16 bit samples with the least significant byte first The packet file contains twelve 16 bit words with the low byte ordered first followed by the high byte The first word in the packet contains the data rate while the remaining 11 words contain the encoded speech data packed in accordance with th
198. igh LB portion of Widemode mode decoder test M M only c46 Mode 0 High LB portion of Widemode mode decoder test M T only c47 Mode 0 Nominal 3 FER LB portion of Widemode mode M M decoder test only c48 Mode 0 Nominal 3 FER LB portion of Widemode mode M T decoder test only 2 4 2 3 4 Subjective Experiment 4 for SO 73 The Test Parameters for Listening Experiment 4 are presented in Table 2 4 2 3 4 1 Table 2 4 2 3 4 1 SO 74 Listening Experiment 4 Test Parameters P NSA P 835 Narrowband Number of talkers 3 males 3 females Test conditions o Car Noise 15 dB SNR Modes 0 4 7 o Street Noise 15 dB SNR Modes 0 4 o Babble noise 20 dB SNR 2 FER Modes 0 4 Encoder Decoder Combinations 4 M M M T T T T M The Test Conditions for Listening Experiment 4 are presented in Table 2 4 2 3 4 2 2 72 3GPP2 C S0018 E v2 0 Table 2 4 2 3 4 2 SO 73 Listening Experiment 4 Test Conditions Exp 4 Narrowband P 835 Reference Conditions File MNRU 901 40 SNR 40dB Reference 902 MNRU 40cB SNR 20dB Reference do3 MNRU 40aB SNR 0dB Reference 904 0 SNR 40dB Reference d05 MNRU 20cB SNR 40dB Reference do6 MNRU 10aB SNR 10dB Reference d07 MNRU 20cB SNR 20dB Reference 908 MNRU 40aB SNR 30dB Reference Test Conditions File Condition Enc Dec d09 Car 15dB SNR Mode 0
199. il 4 where iis the experiment number jis the condition number k is the codec combination number 1 M M with 3 forward link FER 2 M T with 3 forward link FER 3 T M with 3 reverse link FER 4 T T with 3 reverse link FER and v is the associated listener vote Then the per combination condition requirement can be defined as i 1 2 j 5 2 4 1 4 iefi 2 jel 5 k ep 4 2 1 10 2 tj 1 4 MOS 11 MOS i j k x i j k MONS i j 1 MOS i j k i except for the 396 FER condition i 1 j 4 where the following requirement is defined 3 Refer to Section 2 1 4 3 In this case M M and M T are respectively the outputs of the master and test decoders in response to packets generated by the master encoder that have been corrupted using a 396 forward link FER model Similarly T M and T T are the outputs of the master and test decoders in response to packets generated by the master encoder that have been corrupted using a 396 reverse link error model 2 13 3GPP2 50018 v2 0 MOS 4 k MOS 4 k 1 lt 6 4 amp 1 k 1 3 MOS 1 4 k MOS 1 4 k 1 lt 60 4 k 1 ki 13 2 1 10 3 In Equation 2 1 10 2 the maximum allowable difference i j k is given by SRS max 0 12 ci j kN SER i j 1 SER 1 jk oy pe max 0 12 Cli j k SERA j 1 SER i jk 2 1 10 4 Similarly in Equation 2 1 10 3 the maximum allowa
200. ile value for each data rate is shown in Table 3 1 3 3 1 Unused bits are set to 0 For example in a Rate 1 8 frame the packet file will contain the word 0x0100 byte swapped 0x0001 followed by one 16 bit word containing the 16 data bits for the frame in byte swapped form followed by ten 16 bit words containing all zero bits 3 3 4 5 Verifying Bit Exact Performance of the Fixed Point Test Codec This section outlines the methodology of verifying whether a Fixed point Test codec is bit exact to the Fixed point reference software The purpose of this testing is to evaluate the bit exactness of the test codec under a variety of conditions which may occur To accomplish this suites of test vectors have been designed to test for bit exactness of the Test Codec under a variety of conditions depending on a number of parameters These conditions include channel impairments audio background noise and different input levels Figure 3 3 4 5 1 illustrates a decision tree to arrive at the suite of test vectors that are needed to demonstrate Minimum Performance Spec compliance through bit exactness of a Test implementation of SO 70 for different profiles of equipments that support SO 70 3 25 3GPP2 C S0018 E v2 0 Is 16kHz Sampling Rate Supported No SO 68 Compliant SO 68 Compliant Run Test Suite D Yes Run Test Suite C Run Test Suite A Figure 3 3 4 5 1 SO 70 Fixed point bit exact test suite
201. iles from 8 different talkers While individual sentences are 14 repeated every sample uses a distinct sentence pairing Talkers were chosen to have distinct voice qualities and are native speakers of North American English 16 For the following discussion it may be useful to refer to Table 4 1 for the configuration of the associated Software Distribution 18 21 31 Source Speech Material for Experiment 19 source speech material for subjective Experiment is contained in directory 20 so3 subjctv expl source Each sentence is IRS filtered gain adjusted and u Law companded in 2 7 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 3GPP2 C S0018 E v2 0 accordance with 7 The talkers in subjective Experiment consist of four adult males and four adult females The source material for Experiment consists of 8 sentence pairs from 8 different speakers for a total of 64 speech files for both of the nominal input conditions conditions 1 and 5 These files are named 522 This directory also contains the source material for each of the high and low level input conditions which are named s12 and s32 respectively for a total of 3 x 64 192 files The speech database also includes samples processed through the various reference conditions in directory so3 subjctv exp1 ref The reference conditions are named q05 through q25 for the res
202. in Section 2 4 2 The master codec software described in Section 3 4 3 shall be used in the processing involving the master codec Generally the master codec encoder and decoder outputs have been provided in the respective directories so73 subjctv exp m pkt and so73 subjctv exp m m Execution of the master codec software is needed only for the test encoder master decoder combination for each experiment condition All codec processing shall be done digitally Noise suppression and post filter options shall be enabled for both the master and the test codecs The digital format of the speech files is described in Section 3 4 4 4 The naming convention of the processed speech is as follows For the packet files in the so73 subjctv exp 1 3 m_pkt directory the p12 files are the master packet files for the s12 source file Likewise the p22 and p32 files are the respective packet files for the s22 and s32 source files The pf3 files are the impaired packet files which will be described in Section 2 4 4 3 Similarly the directory so73 subjctv exp 2 4 m pkt contains the master packet files for the respective experiments Here the pc10 pb20 and ps files are the master packet files for the c15 b20 and s15 source files respectively For the master encode master decode directories so73 subjctv exp m m the naming convention of the speech files is such that the first two characters of the file name indicate the cod
203. in Table 2 5 2 3 7 1 Table 2 5 2 3 7 1 SO 77 Listening Experiment 7 Test Parameters Type of test ACR P 800 Narrowband Number of talkers 4 males 4 females Background noise none ambient Audio Input Level 22 dB 32 dB 12 dB Filter characteristics MIRS Reference conditions 8 Specified reference conditions Test conditions Nominal level Low level High Level 1 FER 3 FER Encoder Decoder Combinations 4 M M T T T M The Test Conditions for Listening Experiment 7 are presented in Table 2 5 2 3 7 2 Table 2 5 2 3 7 2 SO 77 Listening Experiment 7 Test Conditions Label Condition Qo reg a01 Reference MNRU 3aB a02 Reference MNRU 9dB a03 Reference MNRU 15dB 2 101 3GPP2 50018 v2 0 Label Operating condition ponen a04 Reference MNRU 21dB a05 Reference MNRU 27dB a06 Reference MNRU 33dB 07 Reference MNRU 39dB a08 Reference Direct a09 2kbps Nominal 22 dB M M a10 2kbps High 12 dB M M 11 2kbps Low 32 dB M M a12 2kbps Nominal 22 dB M T a13 2kbps High 12 dB M T al4 2kbps Low 32 dB M T a15 2kbps Nominal 22 dB T M 16 2kbps High 12 dB T M a17 2kbps Low 32 dB T M 18 2kbps Nominal 22 dB T T a19 2kbps High 12 dB T T a20 2kbps Low 32 dB T T a21 2kbps Nominal 22 dB 196 FER M M a22 2kbps Nominal 22 dB 3 FER M M a23 2kbps Nominal 22 dB
204. in laboratory data gathering apparatus Instructions for P 835 Speech Rating Experiment In this speech rating experiment each trial will involve three sentences and you will give a rating for each sentence For the first sentence in each trial you will be asked to attend only to the speech signal and rate how natural or conversely how degraded the speech signal sounds to you You will use the rating scale shown in the figure below to register your ratings of the speech signal Your task will be to choose the numbered phrase from the list below that best describes your opinion of the SPEECH SIGNAL ALONE and then enter the corresponding number on your keyboard 2 59 3GPP2 50018 v2 0 Attending ONLY to the SPEECH SIGNAL select the category which best describes the sample you just heard the SPEECH SIGNAL in this sample was VERY NATURAL NO DEGRADATIO FAIRLY NATURAL LITTLE DEGRA SOMEWHAT NATURAL SOM FAIRLY UNNATURAL FAI VERY UNNATURAL VERY For the second sentence in each trial you will be asked to attend only to the background and rate how noticeable intrusive and or conspicuous the background sounas to you You will use the rating scale shown in the figure below to register your ratings of the background Your task will be to choose the numbered phrase from the list below that best describes your opinion of the BACKGROUND ALONE
205. ing point 0 1 d amp B 3GPP2 50018 v2 0 Reference output speech Input Packet File Operating Point Condition files for bit exact compliance evrc_wb_op0 pc1 EVRC WB Nominal 22 dB evrc_wb_op0 oc1 operating point 0 16 kHz sampling 10 dB car noise evrc wb opo fer 3Yo pc2 EVRC WB Nominal 22 dB evrc wb opO fer 396 0c2 operating point 0 20 dB car noise 16 kHz sampling fer 396 evrc wb op0 ps EVRC WB Nominal 22 dB evrc wb opO0 os operating point 0 16 kHz sampling 15 dB street noise evrc wb opO0 po EVRC WB operating point 0 16 kHz sampling Nominal 22 dB 20 dB babble noise evrc_wb_op0 ob evrc_wb_op0 fer_3 pm EVRC WB operating point 0 16 kHz sampling Generic audio signal fer 396 evrc wb opO fer 3 evrc wb op4 fer 3 22 EVRC WB Nominal 22 dB evrc wb op4 fer 396 022 8k operating point 4 FER 396 8 kHz sampling evrc wb op4 p12 EVRC WB High 12 dB evrc wb op4 012 8k operating point 4 8 kHz sampling evrc wb op4 p32 EVRC WB Low 32 dB evrc wb op4 032 8k operating point 4 8 kHz sampling evrc wb op7 p22 EVRC WB Nominal 22 dB evrc wb op7 022 8k operating point 7 8 kHz sampling evrc wb op4 dim 196 pls 196 p22 EVRC WB Nominal 22 dB evrc wb op4 dim 196 pls 196 operating point 4 8 kHz sampling 196 d amp b 196 pls 022 8k evrc wb op4 pc EVRC WB operating point 4 8 kHz sampling Nominal 22 dB 15 dB
206. int Target Average Channel Data Rate kbps active speech average channel data rate EVRC B 7 0k bits sec 5 24 1 5 EVRC B 6 6k bits sec 4 82 1 5 EVRC B 6 2k bits sec 4 62 1 5 EVRC B 5 8k bits sec 4 45 41 596 EVRC B Half Rate Max 4 8k bits sec 3 75 1 5 The above table provides the maximum allowable average channel rate including full half quarter and eighth rate for the different capacity operating points These maximum allowable average channel rates were obtained by processing the 6 bench mark files through the master floating point software See Section 3 2 2 1 for details in using the provided software tool that can be used to aid in making this calculation 2 2 1 1 2 Average Data Rate Requirement for SO 68 The total average data rate Ravg for each capacity operating point shall not exceed the target average data rate by more than the tolerance level in Table 2 2 1 1 1 1 otherwise the test codec fails the compliance test 2 2 1 2 Unity Gain Requirement The specific EVRC B test codec shall output speech with unity gain when compared with the input speech The unity gain measurement output active speech level input active speech level will be performed over the entire input speech database for the clean nominal level source conditions for each mode The measurement should be made using the STL 2000 tool 6 6a actlev and must not show more than 0 5 dB deviation between input and output active s
207. ion of Wideband mode decoder test only M T d27 Babble 20dB SNR 2 FER Mode 4 interoperable with Mode 0 of SO 68 support M M d28 Babble 20dB SNR 2 FER Mode 4 interoperable with Mode 0 of SO 68 support M T i d29 Babble 20dB SNR 2 FER Mode 4 interoperable with Mode 0 of SO 68 support T T d30 Babble 20dB SNR 2 FER Mode 4 interoperable with Mode 0 of SO 68 support T M 1 2 2 3 2 3 5 Subjective Experiment 5 for SO 70 The Test Parameters for Listening Experiment 5 are presented in Table 2 3 2 3 5 1 4 Table 2 3 2 3 5 1 SO 70 Listening Experiment 5 Test Parameters 4 males 4 females Test conditions Nominal level Mode 0 Low level Mode 0 High level Mode 0 Nominal level Mode 0 1 d amp b Nominal level Mode 0 10 d amp b 2 FER Mode 0 1 d amp b 6 FER Mode 0 10 d amp b Nominal Mode 0 1 pls Encoder Decoder Combinations 4 M M T T T M e Test Conditions for Listening Experiment 5 are presented in Table 2 3 2 3 5 2 2 46 3 4 Table 2 3 2 3 5 2 SO 70 Listening Experiment 5 Test Conditions 3GPP2 C S0018 E v2 0 Exp 5 Narrowband ACR Reference Conditions File MNRU e01 5dB MNRU Reference e02 10dB MNRU Reference e03 15dB MNRU Reference e04 20dB MNRU Reference e05 25dB MNRU Reference e06 B0dB MNRU
208. ion to perform only the decoding function The input file must contain packets of compressed data e Instructs the simulation to perform only the encoding function The output file will contain packets of compressed data If neither the d or the e option is invoked the coder performs both the encoding and decoding functions by default M max Sets the maximum allowable data rate to max where max is element of 4 3 2 1 using the codes specified in the first column of Table 3 5 3 3 1 m min Sets the minimum allowable data rate to min where min is element of 4 3 2 1 using the codes specified in the first column of Table 3 5 3 3 1 If neither the M nor m option is invoked the coder allows the data rate to vary between Rate 1 and Rate 1 8 In addition if max min the data rate varies between max and min using the same rate decision algorithm where the data rate is set to max if the selected data rate is gt max and the data rate is set to min if the selected data rate is min 3 5 4 4 File Formats Files of speech contain 2 s complement 16 bit samples with the least significant byte first The packet file contains twelve 16 bit words with the low byte ordered first followed by the high byte The first word in the packet contains the data rate while the remaining 11 words contain the encoded speech data packed in accordance with the tables specified in 1 The packet file value for each data rate is shown in Table 3
209. isteners for each file set shall be presented with practice trials for subjective Experiments 1 and 3 and Experiments 2 and 4 The randomization of the test samples has been accomplished with the following constraints for each of the two experiments 1 Atrial i e a test sample for the combination of each test condition and each talker shall be presented exactly once to each listening panel i e trials panel conditions x talkers Randomization is in blocks such that one sample of each test condition is presented once with a randomly selected talker in each block This ensures that listeners rate each test condition equally often in the initial middle and final parts of the block and controls for the effects of time and order of presentation A block contains the same number of samples as there are test conditions involved in the test A test session consists of the same number of blocks as there are talkers involved in the test Each session is presented to a listening panel of four listeners Randomizations are constructed such that talker gender is alternated on successive trials resulting in the same talker never being presented on consecutive trials Table 2 4 5 1 shows an example randomization for a single listening panel Each entry in the table is the file name for a sample with the following file naming convention xxyy zzz where xx is the talker yy is the sample and zzz is the test condition 1 20
210. istortion and background noise intrusiveness Table 2 4 2 3 2 2 shows the eight reference conditions 601 608 involved in the 2 87 20 21 22 23 24 25 26 27 28 29 30 31 32 3GPP2 50018 v2 0 P 835 Experiments 2 and 4 In general results are expected for these reference conditions such that the obtained score profiles are similar to those shown in Figure 2 3 10 1 1 SNR 40dB Car Noise MNRU 40dB MNRU SNR E a a SG SG B BAK SG _ BAK P 835 Scores w P 835 Scores w P 835 Scores w A OVRL OVAL oaB 20aB 40aB 098 20 dB 40 dB 10 10 dB 20 20 dB 30 30 dB 40 40 dB MNRU SNR Car Noise SNR Car Noise Figure 2 4 10 2 1 Typical P 835 Score Profiles for Reference Conditions 2 5 Performance Testing for SO 77 2 5 1 Objective Performance Testing for SO 77 The objective testing portion of this specification consists of an average data rate test and compliance to End to End Algorithmic Delay and Unity gain requirements 2 5 1 1 Average Data Rate Test An implementation may support SO 77 for 16 kHz sample rates for example a mobile station that supports wideband electro acoustics The average data rate for the test codec shall be measured using seven source speech files that are contained in the s
211. ite A Decoder Bit exact Test Conditions 3 42 Table 3 4 4 5 4 SO 73 Encoder Suite B Bit exact Test Conditions 3 43 Table 3 4 4 5 5 SO 73 Suite B Decoder Bit exact Test Conditions 3 44 Table 3 5 3 3 1 Packet File Structure from Master Codec Channel Error Model 3 49 Table 3 5 4 5 1 Test Suites of input test vectors for SO 77 compliance 3 52 Table 3 5 4 5 2 SO 77 Encoder Suite A Bit exact Test Conditions 3 53 Table 3 5 4 5 3 SO 77 Suite A Decoder Bit exact Test Conditions 3 55 Table 3 5 4 5 4 SO 77 Encoder Suite B Bit exact Test Conditions 3 57 Table 3 5 4 5 5 SO 77 Suite B Decoder Bit exact Test Conditions 3 58 Table 4 1 Description of EVRC A Software Distribution Contents 4 1 Table 4 2 Description of EVRC B Software Distribution Contents 4 2 Table 4 3 Description of EVRC WB Software Distribution 4 2 Table 4 4 Description of EVRC NW Software Distribution Contents
212. ith practice trials for subjective Experiments 1 3 and 7 and Experiments 2 4 and 8 The randomization of the test samples has been accomplished with the following constraints for each of the two experiments 1 Atrial i e a test sample for the combination of each test condition and each talker shall be presented exactly once to each listening panel i e trials panel conditions x talkers 2 Randomization is in blocks such that one sample of each test condition is presented once with a randomly selected talker in each block This ensures that listeners rate each test condition equally often in the initial middle and final parts of the block and controls for the effects of time and order of presentation A block contains the same number of samples as there are test conditions involved in the test A test session consists of the same number of blocks as there are talkers involved in the test Each session is presented to a listening panel of four listeners 3 Randomizations are constructed such that talker gender is alternated on successive trials resulting in the same talker never being presented on consecutive trials Table 2 4 5 1 shows an example randomization for a single listening panel Each entry in the table is the file name for a sample with the following file naming convention xxyy zzz where xx is the talker yy is the sample and zzz is the test condition 2 111 3GPP2 C S0018 E v2 0 Table 2 5 5 1 Ex
213. ition Enc Dec do9 Car 15dB SNR Mode 0 LB portion of Wideband mode decoder test only d10 Car 15dB SNR Mode 0 LB portion of Wideband mode decoder test only M T 911 Car 15dB SNR Mode 4 interoperable with Mode 0 of SO 68 support M M 012 Car 15dB SNR Mode 4 interoperable with Mode 0 of SO 68 support M T d13 Car 15dB SNR Mode 4 interoperable with Mode 0 of SO 68 support T T di4 Car 15dB SNR Mode 4 interoperable with Mode 0 of SO 68 support T M d15 Car 15dB SNR Mode 7 interoperable with Mode 0 of SO 68 support T T d16 Car 15dB SNR Mode 7 interoperable with Mode 0 of SO 68 support 2 45 3GPP2 C S0018 E v2 0 Exp 4 Narrowband P 835 di7 Car 15dB SNR Mode 7 interoperable with Mode 0 of SO 68 support M M d18 Car 15dB SNR Mode 7 interoperable with Mode 0 of SO 68 support M T di9 Street 15dB SNR Mode 0 LB portion of Wideband mode decoder test only M M d20 Street 15dB SNR Mode 0 LB portion of Wideband mode decoder test only M T d21 Street 15dB SNR Mode 4 interoperable with Mode 0 of SO 68 support M M d22 Street 15dB SNR Mode 4 interoperable with Mode 0 of SO 68 support M T d23 Street 15dB SNR Mode 4 interoperable with Mode 0 of SO 68 support T T d24 Street 15dB SNR Mode 4 interoperable with Mode 0 of SO 68 support T M 925 Babble 20dB SNR 2 FER Mode 0 LB portion of Wideband mode decoder test only d26 Babble 20dB SNR 2 FER Mode 0 LB port
214. itri nee el npe nenne na ce intorno ka ve uan 2 9 20 21 22 23 24 25 26 27 28 29 30 31 32 3GPP2 C S0018 E v2 0 2 1 4 5 Rate 1 2 Maximum Processing tne nnne tnt tnn 2 10 2 1 4 6 Ensuring Proper Encoded Frame Packet Files 2 10 2 1 5 Randomization aee eere ene erede doe eet hee epe 2 10 2 1 6 Presentation idet Elie acte die eet Mine co 2 11 23 7 curet eie te eben rc t eta verte vr e evt 2 11 2 1 8 Listening Test Procedures ertt eae Uer tetas 2 11 249 Analysis Of Besults eee 2 12 2 1 10 Minimum Subjective 2 13 2 1 11 Expected Results for Reference 2 15 2 2 Performance Testing for SO 68 ener nnne nennen 2 16 2 2 4 Objective Performance Testing for SO 68 eme 2 16 2 2 1 1 Average Data Rate Test eeterererrrereeresoooososososososooosoooonononononononousnsusososososososononononsoon 2 16 2 2 1 1 1 Average Data Rate Computation for SO 68 2 16 2 2 1 1 2 Average Data Rate Requirement for SO 68 2 17 2 24 2 Unity Gain Requirement nnne nennt ene n tnnt teta thin 2 17 2 2 1 3 End to end Algorithmic Delay Recommendation 2 17 2 2 2 Subjective Performance Testing for SO 68 te Im 2 18 2 0 t
215. kHz sampling 196 d amp b src s22 8k EVRC NW operating point 7 Nominal 22 dB evrc nw op7 p22 8 kHz sampling Src c15 8k EVRC NW operating point 1 Nominal 22 dB evrc nw opl1 pc 8 kHz sampling 15 dB car noise src s15 8k EVRC NW operating point 1 Nominal 22 dB evrc nw opi ps 8 kHz sampling 15 dB street noise src b20 8k EVRC NW operating point 1 Nominal 22 dB evrc nw op1 pb 8 kHz sampling 20 dB babble noise src c15 8k EVRC NW operating point 6 Nominal 22 dB evrc nw op6 pc 8 kHz sampling 15 dB car noise src s15 8k EVRC NW operating point 6 Nominal 22 dB evrc nw op6 ps 8 kHz sampling 15 dB street noise src b20 8k EVRC NW operating point 6 Nominal 22 dB evrc nw op6 pb 8 kHz sampling 20 dB babble noise Src c15 8k EVRC NW operating point 7 Nominal 22 dB evrc nw op7 pc 8 kHz samplin 15 dB car noise Table 3 4 4 5 5 SO 73 Suite B Decoder Bit exact Test Conditions Reference output Input Packet File Operating Point Condition speech files for bit exact compliance evrc nw opO fer 396 p EVRC NW Nominal 22 dB evrc nw opO fer 396 22 operating point 0 3 FER 022 8k 8 kHz sampling evrc_nw_op0 p12 EVRC NW High 12 dB evrc_nw_op0 012 8k operating point 0 8 kHz sampling evrc_nw_op0 p32 EVRC NW Low 32 dB evrc_nw_op0 032 8k operating point 0 8 kHz sampling evrc_nw_op0 dim_1 p EVRC NW operating Nominal 22 dB evrc nw opO dim 196 ls 196 p22 point 0 8 kHz sa
216. kHz sampling 20 dB babble noise Src c15 8k EVRC WB operating point 7 Nominal 22 dB evrc wb op7 pc 8 kHz samplin 15 dB car noise Table 3 3 4 5 9 SO 70 Suite D Decoder Bit exact Test Conditions Reference output speech Input Packet File Operating Point Condition files for bit exact compliance evrc wb opoO fer 3 22 EVRC WB Nominal 22 dB evrc wb opO0 fer 396 022 8k operating point 0 3 FER 8 kHz sampling evrc wb opO0 p12 EVRC WB High 12 dB evrc wb op0 012 8k operating point 0 8 kHz sampling evrc_wb_op0 p32 EVRC WB Low 32 dB evrc_wb_0p0 032 8k operating point 0 8 kHz sampling evrc_wb_op0 dim_1 pls_1 p22 EVRC WB Nominal 22 dB evrc_wb_op0 dim_1 pls_1 operating point 0 1 d amp b 1 pls 022 8k 8 kHz sampling evrc_wb_op0 pc EVRC WB Nominal 22 dB evrc_wb_op0 oc 8k operating point 0 15 dB car noise 8 kHz samplin 3 32 3GPP2 50018 v2 0 Reference output speech Input Packet File Operating Point Condition files for bit exact compliance evrc wb opO0 ps EVRC WB Nominal 22 dB evrc wb opO0 os 8k operating point 0 8 kHz sampling 15 dB street noise evrc_wb_op0 fer_2 pb EVRC WB operating point 0 8 kHz sampling Nominal 22 dB 20 dB babble noise 2 FER evrc wb opoO fer 296 0b 8k evrc_wb_op0 fer_3 pm EVRC WB operating point 0 8 kHz sampling Generic audio signal fer_3 evrc_wb_op0 fer_3 om 8k evrc_w
217. ket file if only decoding is being performed see the d option below Specifies the name of the output speech file or the name of the output packet file if only encoding is being performed see the e option below Instructs the simulation to perform only the decoding function The input file must contain packets of compressed data Instructs the simulation to perform only the encoding function The output file will contain packets of compressed data If neither the d or the e option is invoked the coder performs both the encoding and decoding functions by default Sets the maximum number of frames to be processed Sets the maximum allowable data rate to max where max is element of 4 3 1 using the codes specified in the first column of Table 3 1 3 3 1 Sets the minimum allowable data rate to min where min is element of 4 3 1 using the codes specified in the first column of Table 3 1 3 3 1 3 8 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 3GPP2 C S0018 E v2 0 If neither the h nor I option is invoked the coder allows the data rate to vary between Rate 1 and Rate 1 8 In addition if max min the data rate varies between max and min using the same rate decision algorithm where the data rate is set to max if the selected data rate is gt max and the data rate is set to min if the selected data rate is lt min See the select rate routine in th
218. kt vec 02 dec vec 03 dec vec 04 dec vec 05 dec vec 06 dec vec 07 dec vec 08 dec vec 09 dec vec 10 dec vec 11 dec vec 12 dec vec 13 dec The following table is a list of source files to be processed in Rate 1 2 Maximum rates 1 3 allowed and the names of the corresponding reference files The files are to be processed as follows Encode EvrcFix e h 3 i file pcm o file h pkt Decode EvrcFix d i file h pkt o file h dec Table 3 1 4 6 2 2 Source and Bit exact Rate 1 2 Max Test Vector Files PCM Source File Encoded Packet File Decoded Speech File vec 05 pcm vec 05 h pkt vec 05 h dec vec 06 pcm vec 06 h pkt vec 06 h dec vec 08 pcm vec 08 h pkt vec 08 h dec The following table is a list of source files to be processed in FULL RATE ONLY MODE only rate 4 allowed and the names of the corresponding reference files The files are to be processed as follows Encode EvrcFix e l 4 i file pcm o file pkt Decode EvrcFix d i file pkt o file dec Table 3 1 4 6 2 3 Source and Bit exact Full Rate Only Test Vector Files PCM Source File Encoded Packet File Decoded Speech File shiftl pcm shiftl pkt shiftl dec shiftr pcm shiftr pkt shiftr dec 3 12 1 4 20 21 22 23 24 3GPP2 C S0018 E v2 0 The following table is a list of source packet files to be decoded and the names of the corresponding reference files Note that it is not necessary to reproduce the source packet files only the
219. le 2 5 2 3 4 1 SO 77 Listening Experiment 4 Test Parameters Type oftest P835 Narrowband Number of talkers Type of test Reference conditions 8 Specified reference conditions Test conditions o Car Noise 15 dB SNR Modes 0 4 7 o Street Noise 15 dB SNR Modes 0 4 O Babble noise 20 dB SNR 2 FER Modes 0 4 Encoder Decoder Combinations 4 M M T T T M The Test Conditions for Listening Experiment 4 are presented in Table 2 5 2 3 4 2 2 97 3GPP2 50018 v2 0 Table 2 5 2 3 4 2 SO 77 Listening Experiment 4 Test Conditions Exp 4 Narrowband P 835 Reference Conditions File MNRU 901 40 SNR 40dB Reference 902 MNRU 40cB SNR 20dB Reference do3 MNRU 40aB SNR 0dB Reference 904 0 SNR 40dB Reference d05 MNRU 20cB SNR 40dB Reference do6 MNRU 10aB SNR 10dB Reference d07 MNRU 20cB SNR 20dB Reference 908 MNRU 40aB SNR 30dB Reference Test Conditions File Condition Enc Dec d09 Car 15dB SNR Mode 0 LB portion of Wideband mode decoder test M M only d10 Car 15dB SNR Mode 0 LB portion of Wideband mode decoder test M T only d11 Car 15dB SNR Mode 1 interoperable with Mode 0 of SO 68 support M M di2 Car 15dB SNR Mode 1 interoperable with Mode 0 of SO 68 support M T d13 Car 15dB SNR Mode 1 interoperable with Mode 0 of SO 68 support T T 914 Car 15dB SNR
220. le Randomization for the Experiments 1 3 and 5 ACR Test 2 56 Table 2 4 1 1 1 1 Target ADR vs Capacity Operating Point 2 64 Table 2 4 2 1 Test Suites for SO 73 compliance eene 2 66 Table 2 4 2 2 Experiments for SO 73 compliance sse 2 66 Table 2 4 2 3 1 1 SO 73 Listening Experiment 1 Test 2 67 Table 2 4 2 3 1 2 SO 73 Listening Experiment 1 Test Conditions 2 68 Table 2 4 2 3 2 1 SO 74 Listening Experiment 2 Test 2 69 Table 2 4 2 3 2 2 SO 73 Listening Experiment 2 Test Conditions 2 69 Table 2 4 2 3 3 1 SO 73 Listening Experiment 3 Test 2 70 Table 2 4 2 3 3 2 SO 73 Listening Experiment 3 Test 2 71 Table 2 4 2 3 4 1 SO 74 Listening Experiment 4 Test 2 72 Table 2 4 2 3 4 2 SO 73 Listening Experiment 4 Test 2 73 Table 2 4 2 3 5 1 SO 73 Listening Experiment 5 Test 2 74 Table 2 4 2 3 5 2 SO 73 Listening Experiment 5 Test Conditions 2 75 Table 2 4 2 3 6 1 SO 73 Listening Experiment 6 Test
221. le for input to a digital to analog converter followed by an electro acoustic transducer The test model compares the output streams of the test encoder and or decoder to those of a master encoder or decoder when driven by the same input stream Figure 1 5 1 shows how the various combinations of outputs are generated Various test conditions will dictate the specific source material and the functions of the gain blocks the frame error model block and the external rate control The input stream for an encoder is a sequence of 16 bit linear binary 2 s complement samples of speech source material The speech can be clean no background noise or can have background noise added depending on the condition being tested The source is passed through the gain block which can amplify or attenuate the signal depending on the condition being tested This signal is then processed by both the master and test encoders with the ability to control the maximum packet rate externally The output of the test encoder for a given rate must conform to the packet files formats specified in 1 The master encoded speech packets can be presented to a frame error model which simulates packet loss over a CDMA air interface The potentially corrupted encoded speech packets from the master and test encoders are then used as inputs to each of the master and test decoders forming four combinations of decoded outputs The four output combinations are master encode master deco
222. le to control maximum encoding rate An external software utility Evrc_nw_iwf in Section 3 4 2 3 is also needed to reduce the data rate of certain packets from full rate to half rate Details of these operations are given in Section 6 The signaling file and other utilities are provided in so77 tools directory 2 5 44 Ensuring Proper Encoded Frame Packet Files All encoded frame packet files shall be examined to ensure that the files only contain data in those file locations where data should exist for a given data rate The examination of the encoded frame packet files should indicate the occurrence of any improper data in the files but the examination must not alter the encoded frame packet files in any way 2 5 45 Post processing of test condition output files In order to build the play sets to be presented to the listening panels the output files for the various test conditions must be processed to provide the appropriate listening conditions In addition the concatenated output files must be partitioned into the samples representing the combination of test condition and talker The listening conditions for Narrowband experiments are provided by filtering the output files using the STL software tool filter with the MIRS receive filter mask The listening conditions for Wideband experiments are provided by mixing STL tool oper the output files with Psophometrically filtered noise STL tool filter PSO filter mask at 74dBov STL tool ast
223. les as there are test conditions involved in the test A test session consists of the same number of blocks as there are talkers involved in the test Each session is presented to a listening panel of four listeners resulting in the same talker never being presented on consecutive trials Randomizations are constructed such that talker gender is alternated on successive trials Table 2 2 5 1 shows an example randomization for a single listening panel Each entry in the table is the file name for a sample with the following file naming convention xxyy zzz where xx is the talker yy is the sample and zzz is the test condition Table 2 2 5 1 Example Randomization for the Experiment ACR Test 1 3 5 7 10 11 12 13 14 15 16 17 18 19 21 23 25 27 30 31 32 33 34 35 36 37 38 39 3 mip6 a09 m1p3 a23 3 m3p4 a27 m4p2 a34 m2p6 al4 f2p6 a30 f1p2 a27 f1p5 a01 m2p2 a26 m4p6 a18 fip7 a29 f1p3 a03 m3p5 a01 f1 m2 f4 m4 f2 p3 al p8 al6 pl a38 p7 al p5 al2 m2p8 a08 flp4 al m4p4 a02 3p4 a37 mip4 a39 f3pl al19 mip6 a05 3p8 a03 m2p6 a22 f3p2 all m4pl a40 4 m4p8 f2p7 m2p2 ml1p6 a18 m2p3 all m2p3 a29 m4p4 a27 m3p4 a02 m3p1 a13 m2p8 a20 flpl al4 3p4 a34 f1p2 a20 f1p4 a39 m3p8 a04 m4pl a31 m3p7 a18 m2p2 a36 2p6 a07 f1p6 a08 f3p2 a14 3p7 a09 m3p7 a12 m2p8 a03 m2p7 a35 m2p3 al9 pl a27 p6 a30 pl a29 pl a32
224. level of 3 17 dBm0 Nominal input speech level is defined to be approximately 22 dB below this reference tone level and is equivalent to 25 dBov or 19 dBm0 For 16 bit signed integers a sine wave with a peak amplitude of 32768 corresponds to 0 dB according to this definition Because a sine wave with amplitude A has a RMS value of A140 the level in dB of a voice active segment of speech x n X n N 1 quantized with 16 bit two s complement linear data spanning 32768 32767 is given by 2 n N 1 10log Q 29 o dBA A weighted sound pressure level expressed in decibels obtained by the use of a metering characteristic and the weighting A specified in 4 and 5 dBmoO Power relative to 0 transmission level point 7 specifies a theoretical load capacity with a full scale sine wave to be 3 17 dBmO for law PCM coding and 3 14 dBmO for A Law PCM coding dBPa Sound level with respect to one Pascal 20 10910 Pressure 1 Pa 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 3GPP2 50018 v2 0 dB SPL Sound Pressure Level in decibels with respect to 0 002 dynes cm2 20 log10 Pressure 0 002 dynes cm2 dBPa is preferred Decoder A device for the translation of a signal from a digital representation into an analog format For the purposes of this standard a device compatible with a specific EVRC implement
225. look ahead This 13 ms delay will ensure the proper tandem processing It may be beneficial for the test codec to incur the same delay as the master codec to avoid potential quality differences due to framing skew This kind of delay ensures asynchronous tandem processing 2 1 4 5 Rate 1 2 Maximum Processing The appropriate speech files will be processed through the codecs for the Rate 1 2 Maximum processing test conditions The test speech codec shall be constrained to operate such that Rate 1 coding is not used 2 1 4 6 Ensuring Proper Encoded Frame Packet Files All encoded frame packet files shall be examined to ensure that the files only contain data in those file locations where data should exist for a given data rate The examination of the encoded frame packet files should indicate the occurrence of any improper data in the files but the examination must not alter the encoded frame packet files in any way 2 1 5 Randomization For each of the two subjective experiments each presentation sample consists of one sentence pair processed under a condition of the test The samples shall be presented to the listeners in a random order The listeners for each file set shall be presented with practice trials for subjective Experiments 2 10 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 3GPP2 C S0018 E v2 0 land Il The randomization of the test samples has been constrain
226. mes use the 2400 bps rate and Rate 1 8 frames use the 1200 bps rate The allowable speech encoding frame rates for SO 3 Rate 1 frames use the 8550 bps rate Rate 2 frames use the 4000 bps rate Rate frames are not used in Service Option 3 and Rate 1 8 frames use the 800 bps rate Rates for SO 68 The allowable traffic frame rates for SO 68 Rate 1 frames use the 9600 bps rate Rate 1 2 frames use the 4800 bps rate Rate 4 frames use the 2400 bps rate and Rate 1 8 frames use the 1200 bps rate The allowable speech encoding frame rates for SO 68 Rate 1 frames use the 8550 bps rate Rate 1 2 frames use the 4000 bps rate Rate 1 4 frames use the 2000 bps rate and Rate 1 8 frames use the 800 bps rate Rates for SO 70 The allowable traffic frame rates for SO 70 Rate 1 frames use the 9600 bps rate Rate 12 frames use the 4800 bps rate and Rate 1 8 frames use the 1200 bps rate The allowable speech encoding frame rates for SO 70 Rate 1 frames use the 8550 bps rate Rate 2 frames use the 4000 bps rate and Rate 1 8 frames use the 800 bps rate Rates for SO 73 The allowable traffic frame rates for SO 73 Rate 1 frames use the 9600 bps rate Rate 1 2 frames use the 4800 bps rate Rate 1 4 frames use the 2400 bps rate and Rate 1 8 frames use the 1200 bps rate The allowable speech encoding frame rates for SO 73 Rate 1 frames use the 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 3
227. minal 22 dB evrc_wb_op4 p22 4 8 kHz sampling src s12 8k EVRC WB operating point High 12 dB evrc_wb_op4 p12 4 8 kHz sampling src s32 8k EVRC WB operating point Low 32 dB evrc_wb_op4 p32 4 8 kHz sampling src s22 8k EVRC WB operating point Nominal 22 dB evrc wb op4 dim 196 p22 4 8 kHz sampling 196 d amp b src s22 8k EVRC WB operating point Nominal 22 dB evrc wb op7 p22 7 8 kHz sampling Src c15 8k EVRC WB operating point Nominal 22 dB evrc wb op4 pc 4 8 kHz sampling 15 dB car noise src s15 8k EVRC WB operating point Nominal 22 dB evrc wb op4 ps 4 8 kHz sampling 15 dB street noise src b20 8k EVRC WB operating point Nominal 22 dB evrc wb op4 pb 4 8 kHz sampling 20 dB babble noise Src c15 8k EVRC WB operating point Nominal 22 dB evrc wb op7 pc 7 8 kHz samplin 15 dB car noise Table 3 3 4 5 5 SO 70 Suite B Decoder Bit exact Test Conditions Reference output speech 16 kHz sampling 3 29 Input Packet File Operating Point Condition files for bit exact compliance evrc wb opO fer 396 p22 EVRC WB Nominal 22 dB evrc wb opO0 fer 396 022 operating point 0 3 FER 16 kHz sampling evrc_wb_op0 fer_1 pls_1 p22 EVRC WB Nominal 22 dB evrc_wb_op0 operating point 0 3 FER fer 196 pls 196 022 16 kHz sampling evrc wb opO0 p12 EVRC WB High 12 dB evrc wb op0 012 operating point 0 16 kHz sampling evrc_wb_op0 dim_1 p32 EVRC WB Low 32 dB evrc_wb_op0 dim_1 032 operat
228. mpling 196 d amp b 196 pls pls 196 022 8k evrc nw opO pc EVRC NW Nominal 22 dB evrc nw opO0 oc 8k operating point 0 15 dB car noise 8 kHz sampling 3 44 3GPP2 C S0018 E v2 0 Reference output Input Packet File Operating Point Condition speech files for bit exact compliance evrc nw opO ps EVRC NW Nominal 22 dB evrc nw opO0 os 8k operating point 0 8 kHz sampling 15 dB street noise evrc_nw_op0 fer_2 p b EVRC NW operating point 0 8 kHz sampling Nominal 22 dB 20 dB babble noise 2 FER evrc_nw_op0 fer_2 ob 8k evrc_nw_op0 fer_3 p m EVRC NW operating point 0 8 kHz sampling Generic audio signal fer_3 evrc_nw_op0 fer_3 om 8k evrc_nw_op1 fer_3 p EVRC NW operating Nominal 22 dB evrc nw opi fer 396 22 point 1 8 kHz sampling FER 396 022 8k evrc nw opi pi2 EVRC NW operating High 12 aB evrc nw opt o12 8k point 1 8 kHz sampling evrc nw op1 p32 EVRC NW operating Low 32 dB evrc nw 1 032 8 point 1 8 kHz sampling evrc nw op6 fer 396 p EVRC NW operating Nominal 22 dB evrc nw op6 fer 396 22 point 6 8 kHz sampling FER 3 022 8k evrc nw op6 p12 EVRC NW operating High 12 dB evrc nw 0p6 012 8k point 6 8 kHz sampling evrc nw op6 p32 EVRC NW operating Low 32 dB evrc nw op6 032 8k point 6 8 kHz sampling evrc nw op7 p22 EVRC NW operating Nominal 22 dB evrc nw op7 o22 8k point 7 8 kHz sampling
229. mpling 196 d amp b src s22 8k EVRC NW2K operating Nominal 22 dB evrc nw op7 p22 point 7 8 kHz sampling Src c15 8k EVRC NW2K operating Nominal 22 dB evrc nw opl1 pc point 1 8 kHz sampling 15 dB car noise src s15 8k EVRC NW2K operating Nominal 22 dB evrc_nw_op1 ps point 1 8 kHz sampling 15 dB street noise src b20 8k EVRC NW2K operating Nominal 22 dB evrc nw op1 pb point 1 8 kHz sampling 20 dB babble noise Src c15 8k EVRC NW2K operating Nominal 22 dB evrc nw op6 pc point 6 8 kHz sampling 15 dB car noise src s15 8k EVRC NW2K operating Nominal 22 dB evrc_nw_op6 ps point 6 8 KHz sampling 15 dB street noise src b20 8k EVRC NW2K operating Nominal 22 aB evrc nw op6 pb point 6 8 kHz sampling 20 dB babble noise src c15 8k EVRC NW2K operating Nominal 22 dB evrc nw op7 pc point 7 8 kHz sampling 15 dB car noise src 2kbps s22 8k EVRC NW2K operating Nominal 22 dB evrc_nw_op3 p22 point 3 8 kHz sampling src 2kbps s12 8k EVRC NW2K operating High 12 dB evrc nw op3 p12 point 3 8 kHz sampling src 2kbps s32 8k EVRC NW2K operating Low 32 dB evrc_nw_op3 p32 point 3 8 kHz sampling 3 57 3GPP2 50018 v2 0 Input File Operating Point Condition src_2kbps c15 8k EVRC NW2K operating Nominal 22 dB evrc nw op3 pc point 3 8 kHz sampling 15 dB car noise src 2kbps s15 8k EVRC NW2K operating Nominal 22 dB evrc nw op3 ps point 3 8 kHz sampling 15 dB street noise src_2kbps b2
230. n 32 difference scores one for each listener df 231 2 32 20 21 3GPP2 C S0018 E v2 0 the criterion value for the appropriate test 2 09 for Dunnett s Test 1 70 for the t test then the E DC passes the MPS test Miu Mus L Test Vaiss 2 2 9 2 1 2 2 10 Expected Results for Reference Conditions 2 2 10 1 Experiment Reference Conditions The MNRU conditions have been included to provide a frame of reference for the Experiment MOS test In listening evaluations where test conditions span approximately the same range of quality the MOS results for similar conditions should be approximately the same Data from previous studies allows a generalization to be made concerning the expected MOS results for the MNRU reference conditions see Figure 2 2 10 1 1 MOS scores obtained for the MNRU conditions in any SO 68 validation test should be compared to those shown in the graph below Inconsistencies beyond a small shift in the means in either direction or a slight stretching or compression of the scale near the extremes may imply a problem in the execution of the evaluation test In particular MOS should be monotonic with MNRU within the limits of statistical resolution and the contour of the relation should show a similar slope MOS 6 10 20 30 40 50 Figure 2 2 10 1 1 MOS versus MNRU 2 2 10 2 Experiment Il Reference Conditions Reference conditions for P 835 tests are constructed as a combina
231. nal acoustic characteristics for telephony Requirements V9 0 0 March 2009 1 4 Definitions Base Station A station in the Domestic Public Cellular Radio Telecommunications Service other than a mobile station used for radio communications with mobile stations Bit Exact A test procedure for codecs by which a set of prescribed vectors are input to the test codecs and output vectors from the codecs correspond exactly bit for bit with output vectors prescribed by this standard CELP Code Excited Linear Predictive Coding This technique uses codebooks to vector quantize the excitation residual signal of a Linear Predictive Codec LPC Circum aural Headphones Headphones that surround and cover the entire ear Codec The combination of an encoder and decoder in series encoder decoder Compand The process of compressing and expanding a signal In this text the process is described in terms of Law 7 dB Normally taken to be defined as X dB 2010910 x In the context of digitized speech the unit dB is used to represent the average power level of a speech signal with respect to full scale For the purposes of this document full scale is defined as the maximum sinusoidal input level which does not result in clipping where 0 dB corresponds to the output level measured according to 9 for a full scale 1 kHz sinusoidal input This corresponds to a digitally referenced input level of 3 dBov and 7 defined tone
232. nd decoders listed in the descriptions of the two experiments given in Section 2 2 2 The master codec software described in Section 3 2 3 shall be used in the processing involving the master codec Generally the master codec encoder and decoder outputs have been provided in the respective directories so68 subjctv exp m pkt and so68 subjctv exp m m Execution of the master codec software is needed only for the test encoder master decoder combination for each experiment condition All codec processing shall be done digitally Noise suppression and post filter options shall be enabled for both the master and the test codecs The digital format of the speech files is described in Section 3 2 4 4 The naming convention of the processed speech is as follows For the packet files in the so68 subjctv exp1 m pkt directory Experiment l the p12 files are the master packet files for the 12 source file Likewise the 22 and p32 files are the respective packet files for the s22 and 532 source files For the packet files the file name 9 3 indicates an output from the master encoder at 9 3 kbps active speech channel rate Likewise the file names 5 8 and 4 8 indicate an output from the master encoder at the respective active speech channel rates The pf3 files are the impaired packet files which will be described in Section 2 2 4 3 Similarly the directory so68 subjctv exp2 m pkt contains the master packet files for Experiment Il
233. nd 4 shall be averaged to produce an associated mean score M as shown in Equation 2 3 9 1 1 and a Standard Deviation SD as shown in Equation 2 3 9 1 2 where L is the number of listeners and T is the number of talkers involved in the experiment 2 3 9 1 1 LxT spake Lx 1 2 3 9 1 2 2 3 9 2 Minimum Subjective Requirement for SO 70 Listening Tests The Terms of Reference for the MPS tests state that the mean score for each of the Test Encoder Decoder Combinations E DC should be not worse than the mean score for the Reference E DC For most of the test conditions involved in the subjective experiments there are three Test E DC s M T T M and T T which means there are three statistical tests against the Reference E DC M M The three statistical tests are not independent however Since they all involve the same ratings for the Reference E DC t tests are not appropriate The appropriate statistical test for multiple Test conditions against a common Reference condition is Dunnett s Test A complete description of Dunnett s Test is contained in Appendix B The critical value for the Dunnett s testis 2 09 one sided test p lt 05 4 E DC s df 93 For those test conditions where a single Test E DC T T is compared against the Reference E DC the appropriate statistical test is Student s t test The critical value for the Student s t test is 1 70 one sided test p lt 05 df 31 In b
234. ne entree enne nnns nnns entente 2 88 2 5 1 Objective Performance Testing for SO 77 nnne 2 88 2 5 1 1 Average Data Rate 2 88 2 5 1 1 1 Average Data Rate Computation for SO 77 sss 2 88 2 5 1 1 2 Average Data Rate Requirement for SO 77 sss 2 89 2 5 1 2 Unity Gain Requirement rennen nnne nnn enne atate teens 2 90 2 5 1 3 End to end Algorithmic Delay Recommendation e 2 90 2 5 2 Subjective Performance Testing for SO 77 2 90 2 5521 i tetuer nsbo daki nennoa n bonbade ke etunup tein va sagin ed a RR E RR RR RR NR Ris 2 92 2 5 2 2 Method of Measurement recreanten anaa a nennen 2 92 vii 20 21 22 23 24 25 26 27 3GPP2 C S0018 E v2 0 2 5 2 3 Test Conditions and Test Design for SO 77 sss 2 92 2 5 2 3 1 Subjective Experiment 1 for SO 77 2 92 2 5 2 8 Subjective Experiment 2 for SO 77 2 94 2 5 2 3 3 Subjective Experiment for SO 77 2 95 2 5 2 3 4 Subjective Experiment 4 for SO 77 2 97 2 5 2 3 5 Subjective Experiment 5 for SO 77 2 99 2 5 2 3 Subjective Experiment 6 for SO 77 2 100 2 5 2 3 7 Subjective Experiment 7 for SO 77 2 101 2 5 2 3 8 Subjective Experiment 8 for SO 77 2 102 2 5 2 3 Numerical Parameters for th
235. ng requirements for electro acoustic performance measured between the output of the D A converter and the output of the headphone 1 Frequency response shall be flat to within 2 dB between 50 Hz and 7000 Hz and below 50 Hz the response shall roll off at a minimum of 12 dB per octave Equalization may be used in the audio path to achieve this A suitable reconstruction filter shall be used for playback 2 Total harmonic distortion shall be less than 1 for signals between 50 Hz and 8000 Hz 3 Noise over the audio path shall be less than 35 dBA measured at the ear reference plane of the headphone 4 Signal shall be delivered to the headphone on the listener s preferred telephone listening ear and the other ear shall be uncovered No signal shall be delivered to the other headphone The audio path for narrowband test conditions Experiments 3 and 4 must meet the following requirements for electro acoustic performance measured between the output of the D A converter and the output of the headphone 1 Frequency response shall be flat to within 2 dB between 200 Hz and 3400 Hz and below 200 Hz the response shall roll off at a minimum of 12 dB per octave Equalization may be used in the audio path to achieve this A suitable reconstruction filter shall be used for playback 2 Total harmonic distortion shall be less than 1 for signals between 100 Hz and 4000 Hz 3 Noise over the audio path shall be less than 35 dBA measured at the ear r
236. ning Experiment 6 Test 2 48 Table 2 3 2 3 6 2 SO 70 Listening Experiment 6 Test 2 48 xiv 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 3GPP2 C S0018 E v2 0 Table 2 3 2 3 7 1 SO 70 Listening Experiment 7 Test 2 49 Table 2 3 2 3 7 2 SO 70 Listening Experiment 7 Test 2 49 Table 2 3 2 3 8 1 SO 70 Listening Experiment 8 Test 2 50 Table 2 3 2 3 8 2 SO 70 Listening Experiment 8 Test 2 50 Table 2 3 2 3 9 1 Numerical Parameters for the SO 70 Listening Experiments 2 51 Table 2 3 4 5 1 Cutting Points for the astrip Software Tool for the SO 70 Experiments 1 3 and 5 hano 2 54 Table 2 3 4 5 2 Cutting Points for the astrip Software Tool for the SO 70 Experiments 2 4 and 6 P2835 Test ee e pd d age e ata lal f 2 55 Table 2 3 4 5 3 Composition of the Sentence Triad Samples for the Experiments 2 4 and 6 P 835 TOS iui editi pinea tidie t eO ree ERE noon EY RES Dou Mee xa te 2 55 Table 2 3 5 1 Examp
237. nt background segment 22dB R babble noise segment 32dB R car noise segment 32dB R street noise segment 32dB R ambient background segment 32dB See Section 3 1 2 1 for details in using the provided software tool that can be used to aid in making this calculation 2 1 1 1 2 Average Data Rate Requirement The total average data rate Rayg shall not exceed 4400 bps otherwise the test codec fails the compliance test 2 1 1 2 Unity Gain Requirement The specific EVRC test codec shall output speech with unity gain when compared with the input speech The unity gain measurement output active speech level input active speech level will be performed over the entire input speech database for the clean nominal level source conditions for each mode The measurement should be made using the STL 2000 tool 6 6a actlev and must not show more than 0 5 dB deviation between input and output active speech levels This procedure is fully described in 9 2 1 1 8 End to end Algorithmic Delay Recommendation The algorithmic delay for the specific EVRC test codec should be calculated analytically by the codec manufacturer In considering the algorithmic delay it can be assumed that all transmission channels have infinite bandwidth and that all processing elements have infinite throughput Algorithmic delay is defined as the sum of all sequential filter delays and buffering delays in the encode decode path The maximum end
238. nting the combination of test condition and talker The listening conditions for Narrowband experiments are provided by filtering the output files using the STL software tool filter with the MIRS receive filter mask The listening conditions for Wideband experiments are provided by mixing STL tool oper the output files with Psophometrically filtered noise STL tool filter PSO filter mask at 74dBov STL tool astrip is also used to split the concatenated files into the individual samples appropriate for the experiment Table 2 3 4 5 1 shows the cutting points to be used with the astrip tool for producing the two sentence samples for the Experiments 1 3 and 5 ACR test Table 2 3 4 5 2 shows the cutting points to be used with the astrip tool for producing the single sentence sub samples for the Experiments 2 4 and 6 P 835 test Table 2 3 4 5 3 shows the sub samples that make up the samples i e sentence triads for the P 835 test 2 53 3GPP2 50018 v2 0 Table 2 3 4 5 1 Cutting Points for the astrip Software Tool for the SO 70 Experiments 1 3 and 5 ACR Test Experiment I ACR Sentence pair N Start sample 113707 232293 344193 461679 572672 696242 80299 13867 68071 N N 1136889 261550 5 N Co 5 N 372671 14832774 N sf po A ra rn 1609353 708427 apa 4p2 ipi 5193 2
239. o reduce the cumulative effects of experience 2 2 8 Listening Test Procedures 2 2 8 1 ACR Listening Test Procedures Experiment l The listeners shall listen to each sample and rate the quality of the test sample using a five point scale with the points labeled 5 Excellent 4 Good 3 Fair 2 Poor 1 Bad Data from 32 listeners shall be used for Experiment four listeners for each listening panel where each listening panel uses a different randomization Before starting the test the listeners should be given instructions for performing the subjective test An example set of instructions for the ACR are presented in Figure 2 1 8 1 The instructions may be modified to allow for variations in laboratory data gathering apparatus 2 28 3GPP2 50018 v2 0 This is an experiment to determine the perceived quality of speech over the telephone You will be listening to a number of recorded speech samples spoken by several different talkers and you will be rating how good you think they sound Use the single headphone on the ear you normally use for the telephone On each trial a two sentence sample will be played After you have listened to the sample determine the category from the list below which best describes the overall quality of the sample Press the numeric key on your keyboard corresponding to your rating for how good or bad that particular passage sounded The quality of the speech should be rated according to the
240. o77 subjctv exp 1 2 source directories Each file exhibits a different condition power levels 12 dB 22 dB and 32 dB and background noise conditions 20 dB SNR babble noise 10 dB SNR car noise 20 dB SNR car noise and 15 dB SNR street noise The input source files used in the average data rate test have an approximate voice activity factor of 0 6 and are the same input files used in the subjective portion of the experiment An implementation may support SO 77 only for 8 kHz sample rate input output for example a Base station transcoder or a Media Gateway or mobile station The average data rate for the test codec for this case shall be measured using six narrow band source speech files that are contained in the so77 subjctv exp 3 4 source directories Each file exhibits a different condition power levels 12 dB 22 dB and 32 dB and background noise conditions 20 dB SNR babble noise 15 dB SNR car noise and 15 dB SNR street noise The input source files used in the average data rate test have an approximate voice activity factor of 0 6 and are the same input files used in the subjective portion of the experiment 2 5 1 1 1 Average Data Rate Computation for SO 77 The average channel data rate for the test codec shall be computed for each of the benchmark files as follows R 9600 1 4800 No 2400 N4 1200 Ng N where N4 number of frames encoded at Rate 1 No number of frames encoded at Rate 1 2 2 88
241. odec assumes a 16 bit integer input output normalization 20 21 22 23 24 25 26 27 28 29 30 31 3GPP2 C S0018 E v2 0 2 CODEC MINIMUM STANDARDS This section describes the validation procedures that shall be used to verify the quality and interoperability of an EVRC implementation The procedures are both comprehensive and backward compatible in that they are provided for the SO 3 SO 68 SO70 and SO 73 implementations of EVRC The validation procedures comprise a set of objective and subjective tests as well as a maximum algorithmic delay Recommendation These are described in the following sections 2 1 Performance Testing for SO 3 2 1 1 Objective Performance Testing for SO The objective testing portion of this specification consists of an average data rate test and compliance to End to End Algorithmic Delay and Unity gain requirements 2 1 1 1 Average Data Rate Test The average data rate for the test codec shall be measured using benchmark files that are contained on the accompanying Software Distribution in the so3 objctv subdirectory The average data rate for the test codec shall be measured using twelve benchmark files that are contained in the associated Software Distribution in the soS objctv subdirectory Each file exhibits a different combination of input level 12 dB 22 dB and 32 dB and background noise conditions ambient background noise 20 dB SNR babble noise condition 15 d
242. oding by the Master Test annt tna 2 78 2 4 4 3 Introduction of Impairments eeerereresesesesoosososososoooooooooononononononononsoousososososososononononoann 2 78 2 4 4 4 Ensuring Proper Encoded Frame Packet Files 2 78 2 4 4 5 Post processing of test condition output 2 78 2 4 5 Rarndomization ctt et 2 80 2 4 6 Presentation eire semina de ke pole ya ia vec ea sav inu nea davies emer erasa se e pen baka ravage beens 2 82 VIS EE ITO ULT LOIR NUI 2 82 2 4 8 Listening Test Procedures nnne nenne nnne ninth 2 82 2 4 8 1 ACR Listening Test Procedures Experiments 1 3 mms 2 82 2 4 8 2 P 835 Listening Test Procedures Experiments 2 and 4 2 83 2 4 9 Analysis of ReSUuIts RE A 2 86 2 4 9 1 Basic Results for the SO 73 Listening tests eee renee 2 86 2 4 9 2 Minimum Subjective Requirement for SO 73 Listening Tests 2 86 2 4 10 Expected Results for Reference Conditions ntet 2 87 2 4 10 1 Reference Conditions for Experiments 1 3 and 5 2 87 2 4 10 2 Reference Conditions for Experiments 2 and 4 2 87 2 5 Performance Testing for SO 77 sse ee
243. ominal 22 dB 1 d amp b 1 pls evrc nw op6 dim 196 pls 196 022 8k evrc nw op6 pc EVRC NW operating point 6 8 kHz sampling Nominal 22 dB 15 dB car noise evrc nw op6 oc 8k evrc nw op6 ps EVRC NW operating point 6 8 kHz sampling Nominal 22 dB 15 dB street noise evrc nw op6 os 8k evrc nw op6 fer 296 pb EVRC NW operating point 6 8 kHz sampling Nominal 22 dB 15 dB babble noise 296 FER evrc nw op6 fer 2 ob 8k Table 3 4 4 5 4 SO 73 Encoder Suite B Bit exact Test Conditions Input File Operating Point Condition po n src s22 8k EVRC NW operating point 1 Nominal 22 dB evrc nw opi p22 8 kHz sampling src s12 8k EVRC NW operating point 1 High 12 dB evrc nw opi pi2 8 kHz sampling src s32 8k EVRC NW operating point 1 Low 32 dB evrc nw op1 p32 8 kHz sampling src s22 8k EVRC NW operating point 1 Nominal 22 dB evrc nw opi dim 196 p22 8 kHz sampling 196 d amp b src s22 8k EVRC NW operating point 6 Nominal 22 dB evrc nw op6 p22 8 kHz sampling src s12 8k EVRC NW operating point 6 High 12 dB evrc nw op6 p12 8 kHz sampling 3 43 3GPP2 C S0018 E v2 0 Input File Operating Point Condition E or src s32 8k EVRC NW operating point 6 Low 32 dB evrc nw op6 p32 8 kHz sampling src s22 8k EVRC NW operating point 6 Nominal 22 dB evrc nw op6 dim 196 p22 8
244. on file for 3 3 1 2 listening tests so73 tools Source code for the software 3 tools so73 testvec Test vectors for verifying bit 3 3 4 5 exact EVRC implementations Table 4 5 Description of EVRC NW2K Software Distribution Contents so70 73 77 EVRCWB NW F source code for the bit exact 3 3 4 X fixed point code so77 subjctv Speech and other material 2 3 1 1 2 5 3 2 3 4 2 3 5 necessary to perform subjective experiments so77 cal Output level calibration file for 3 3 1 2 listening tests so77 tools Source code for the software 3 3 2 tools so77 testvec Test vectors for verifying bit 3 3 4 5 exact EVRC implementations 4 3 3GPP2 C S0018 E v2 0 This page intentionally left blank 4 4 21 22 23 24 3GPP2 C S0018 E v2 0 5 DUNNETT S TEST Most of the MPS statistical tests for SO 68 SO 70 and SO 73 compliance involve multiple Test Encoder Decoder Combinations E DC and a single Reference E DC The appropriate analysis for the statistical tests involved in the EVRC B MPS and EVRC WB MPS test is Dunnett s Test 20 Dunnett s Test is a special case of the more general Post Hoc Multiple Means Test where multiple treatment means are statistically compared to a common control mean In the case of the MPS tests the treatments are the three Test E DC s M T T M T T and the control is the Reference E DC M Dunnett s Test is conducted two stages The first stage involves an Analysis of Varian
245. ontains input source files as well as packet files injected with frame erasures The so68 testvec fixed directory contains files processed with the EVRC B fixed point reference software The files in these directories are the reference files for bit exact compliance A test codec is bit exact if it can reproduce all of the reference files in the Iso68 testvec fixed directory exactly The outputs of the encoder and decoder of the test codec are to be obtained for the conditions given below in Table 3 2 4 5 1 and Table 3 2 4 5 2 The processing steps for these conditions are illustrated in Section 6 Table 3 2 4 5 1 SO 68 Encoder Bit exact Test Conditions Input e u Reference packet File Operating Point Condition files for bit exact compliance src s22 EVRC B 9 3 kbps Nominal 22 dB 9 3 p22 src s22 EVRC B 5 8 kbps Nominal 22 dB 5 8 p22 src s22 EVRC B 4 8 kbps Nominal 22 dB 4 8 p22 src s32 EVRC B 9 3 kbps Low 32 dB 196 d amp b 9 3 p32 src s32 EVRC B 5 8 kbps Low 32 dB 196 d amp b 5 8 p32 src si2 EVRC B 9 3 kbps High 12 dB 9 3 pi2 src s12 EVRC B 5 8 kbps High 12 dB 5 8 p12 src c15 EVRC B 9 3 kbps Nominal 22 dB 15 dB carnoise 9 3 pc src c15 EVRC B 5 8 kbps Nominal 22 dB 15 dB carnoise 5 8 pc src b20 EVRC B 9 3 kbps Nominal 22 dB 20 dB babble 9 3 po src b20 EVRC B 5 8 kbps Nominal 22 dB 20 dB babble 5 8 po src s1i5 EVRC B 9 3 kbps Nominal 2
246. or the fixed point codec simulation has been written in C and can be compiled using any general purpose compiler such as the GNU G compiler and make utility Two GCC compatible makefiles have been included in the build directory Typing make in the build directory will compile and link the code and create the executable file called Evrc_nw_fx Evrc nw fx exe on Win32 systems which will be placed in the build directory The included makefiles may require some user modification for a particular hardware platform and or operating system 3 4 4 8 Running the Fixed Point Codec Simulation The EVRC NW executable files use command line arguments to receive all information regarding input and output files and various parameters used during execution Executing Evrc nw fx with no command line arguments will display a brief description of the required and optional command line arguments The options are described below i infn required Specifies the name of the input speech file or the name of the input packet file if only decoding is being performed see the d option below 3 38 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 3GPP2 C S0018 E v2 0 o outf required Specifies the name of the output speech file or the name of the output packet file if only encoding is being performed see the e option below d Instructs the simulation to perform only the decoding
247. or the test codec shall be measured using seven source speech files that are contained in the so70 subjctv exp 1 2 source directories Each file exhibits a different condition power levels 12 dB 22 dB and 32 dB and background noise conditions 20 dB SNR babble noise 10 dB SNR car noise 20 dB SNR car noise and 15 dB SNR street noise The input source files used in the average data rate test have an approximate voice activity factor of 0 6 and are the same input files used in the subjective portion of the experiment An implementation may support SO 70 only for 8 kHz sample rate input output for example a Base station transcoder or a Media Gateway or mobile station The average data rate for the test codec for this case shall be measured using six narrow band source speech files that are contained in the so70 subjctv exp 3 4 source directories Each file exhibits a different condition power levels 12 dB 22 dB and 32 dB and background noise conditions 20 dB SNR babble noise 15 dB SNR car noise and 15 dB SNR street noise The input source files used in the average data rate test have an approximate voice activity factor of 0 6 and are the same input files used in the subjective portion of the experiment 2 3 1 1 1 Average Data Rate Computation for SO 70 The average channel data rate for the test codec shall be computed for each of the benchmark files as follows R 9600 1 4800 2 1200 8 where N1 numbe
248. ormance is maintained with operation between any pairing of manufacturers encoders and decoders This interoperability issue is a serious one Any variation in implementing the exact standard must be avoided if it cannot be ensured that minimum performance levels are met when interoperating with all other manufacturers equipment meeting the standard This standard provides a means for measuring performance levels while trying to ensure proper interoperation with other manufacturers equipment The issue of interoperation can only be definitively answered by testing all combinations of encoder decoder pairings With the number of equipment manufacturers expected to supply equipment this becomes a prohibitive task therefore the objective and subjective tests rely upon the use of a master codec The master codec is defined as the floating point implementation of specific EVRC written in the C programming language The master codec software which is described in Section 3 1 3 SO 3 Section 3 2 3 SO 68 Section 3 3 3 SO 70 Section 3 4 3 SO 73 or Section 3 5 3 SO 77 is used as part of the interoperability testing 1 2 Requirements Language Shall and shall not identify requirements to be followed strictly to conform to this document and from which no deviation is permitted Should and should not indicate that one of several possibilities is recommended as particularly suitable without mentioning or excluding others that
249. ot independent however Since they all involve the same ratings for the Reference E DC t tests are not appropriate The appropriate statistical test for multiple Test conditions against a common Reference condition is Dunnett s Test A complete description of Dunnett s Test is contained in Appendix B The critical value for the Dunnett s test is 2 09 one sided test p 05 4 E DC s df 93 For those test conditions where a single Test E DC T T is compared against the Reference E DC M M the appropriate statistical test is Student s t test The critical value for the Student s t test is 1 70 one sided test p lt 05 df 31 In both the Dunnett s Test and the t test the MPS test is evaluated by dividing the difference between the mean score for the Test E DC and the mean score for the Reference ED C by the Standard Error of the Mean Difference SEyp as shown in Equation 2 5 9 2 1 If the resultant Test value is less than 7 The appropriate t test is a matched groups t test and the SEmp is based on the differences between individual listener s average ratings where the average is over talkers Therefore the SEwp is based on 32 difference scores one for each listener df 231 2 117 20 21 22 3GPP2 C S0018 E v2 0 the criterion value for the appropriate test 2 09 for Dunnett s Test 1 70 for the t test then the E DC passes the MPS test Miu Mus Test Vaiss 2 5 9 2 1 2 5 10 Expected Results for Refe
250. oth the speech signal and the background and rate your opinion of the sample for purposes of everyday speech communication Select the category which best describes the sample you just heard for purposes of everyday speech communication the OVERALL SPEECH SAMPLE was EXCELLENT GOOD FAIR POOR BAD 2 31 20 21 22 23 24 25 26 27 28 3GPP2 C S0018 E v2 0 2 2 9 Analysis of Results The response data from the practice blocks shall be discarded Data sets with missing responses from listeners shall not be used i e a complete set of data is required for 32 listeners four for each of eight listening panels Responses from the different listening panels for the corresponding test conditions shall be treated as equivalent in the analysis 2 2 9 4 Basic Results for the SO 68 Listening tests The votes for each of the test conditions for SO 68 Experiments and II shall be averaged to produce an associated mean score M as shown in Equation 2 2 9 1 1 and a Standard Deviation SD as shown in Equation 2 2 9 1 2 where L is the number of listeners and T is the number of talkers involved in the experiment zzv LxT LxT 1 lt 2 2 9 1 1 PECON SD 2 2 9 1 2 2 2 9 2 Minimum Subjective Requirement for SO 68 Listening Tests The Terms of Reference for the MPS tests state that the mean score for each of the Test Encoder Decoder Combinations
251. oth the Dunnett s Test and the t test the MPS test is evaluated by dividing the difference between the mean score for the Test E DC and the mean score for the Reference ED C by the Standard Error of the Mean Difference SEyp as shown in Equation 2 3 9 2 1 If the resultant Test value is less than 5 The appropriate t test is a matched groups t test and the SEmp is based on the differences between individual listener s average ratings where the average is over talkers Therefore the SEwp is based on 32 difference scores one for each listener df 231 2 61 20 21 22 3GPP2 C S0018 E v2 0 the criterion value for the appropriate test 2 09 for Dunnett s Test 1 70 for the t test then the E DC passes the MPS test Miu Mus L Test Vaiss 2 3 9 2 1 2 3 10 Expected Results for Reference Conditions 2 3 10 1 Reference Conditions for Experiments 1 3 and 5 The MNRU conditions have been included to provide a frame of reference for the Experiments 1 3 and 5 In listening evaluations where test conditions span approximately the same range of quality the MOS results for similar conditions should be approximately the same Data from previous studies allows a generalization to be made concerning the expected MOS results for the MNRU reference conditions see Figure 2 3 10 1 1 MOS scores obtained for the MNRU conditions in any SO 70 validation test should be compared to those shown in the graph below Inconsistencies beyond
252. ough 6 and four reference conditions in experiments 7 and Table 2 3 2 3 9 1 Numerical Parameters for the SO 70 Listening Experiments ena nui pwa bor ers eae tm EncodeDecode 16 is se z s e a Reference Condtions 8 e a e 4 TotalCondtions ao 2o se e s 5 8 6 8 6 e co Simul per Taker gere e a e a a a a Total Stimuli per Experiment 1536 1152 2560 1440 1596 768 o6 listoningPanes e s e e s a a Stimul per Listening Parei 192 144 sco 10 192 oo 24 24 Listeners votes se ae se so se se a Listeners Voters per Listening Panel ES ES FIF FIE EN Votes per 2 192 266 192 256 192 oo o6 2 3 8 Speech Material for SO 70 Testing The source speech files used for SO 70 compliance testing consist of Harvard sentences pairs which are preprocessed to include proper level adjustment and noise mixing for use in the subjective experiments The talkers used in these files consist of adult males and adult females and are native speakers of North American English For the following discussion it may be useful to refer to Table 4 3 for the composition of the Software Distribution database The source speech material for subjective Experiments is contained directory so70 subjctv exp source Each file has
253. ow 32 dB 1 d amp b 1 pls M M c26 Mode 6 Low 32 dB 1 d amp b 1 pls M T c27 Mode 6 Low 32 dB 1 d amp b 1 pls T T c28 Mode 6 Low 32 dB 1 d amp b 1 pls T M c29 Mode 1 High 12 dB M M c30 Mode 1 High 12 dB M T c31 Mode 1 High 12 dB T T c32 Mode 1 High 12 dB T M c33 Mode 6 High 12 dB M M c34 Mode 6 High 12 dB M T c35 Mode 6 High 12 dB T T c36 Mode 6 High 12 dB T M c37 Mode 1 Nominal 22 dB 396 FER M M c38 Mode 1 Nominal 22 dB 396 FER M T c39 Mode 6 Nominal 22 dB 396 FER M M c40 Mode 6 Nominal 22 dB 3 FER M T 2 96 2 3 6 3GPP2 50018 v2 0 Label Condition og c41 Mode 0 Nominal LB portion of Widemode mode decoder M M test only c42 Mode 0 Nominal LB portion of Widemode mode decoder M T test only c43 Mode 0 Low 1 D amp B 1 PLS LB portion of Widemode M M mode decoder test only c44 Mode 0 Low 1 D amp B 1 PLS LB portion of Widemode M T mode decoder test only c45 Mode 0 High LB portion of Widemode mode decoder test M M only c46 Mode 0 High LB portion of Widemode mode decoder test M T only c47 Mode 0 Nominal 3 FER LB portion of Widemode mode M M decoder test only c48 Mode 0 Nominal 3 FER LB portion of Widemode mode M T decoder test only 2 5 2 3 4 Subjective Experiment 4 for SO 77 The Test Parameters for Listening Experiment 4 are presented in Table 2 5 2 3 4 1 Tab
254. p6 a37 p3 a39 p5 a05 p5 a38 p7 a02 p6 a06 p8 a21 p3 a40 p2 a13 p3 a25 p4 a36 m2 fl m4 3 m4 f2 m4 fl ml fl m4 3 m4 f4 m3 N H N w mlp8 a40 m4p8 al9 4 f1p3 a05 1 7 35 f2p6 a18 2 8 31 f1p7 a32 f3p2 a22 f2p8 al6 4 m2p1l a33 m3p2 a26 m2pl1 a09 m2p8 a04 m3p5 a29 m3p3 a05 mip6 al9 m4p4 a22 4p3 a03 4 5 31 3p7 al7 m4p5 a03 m4p5 a06 f2p7 a09 3p6 a06 f2p7 a24 m4p8 al5 f2p2 a25 m2p1 a37 3 2 18 m4p8 a39 mipl al2 m2p6 a13 m3 3 mi mi f4 2 m4p6 a36 3p7 a05 mip3 al0 mlpl a26 m3p1 a20 mlp7 a34 2 27 m2p4 a05 m4p7 a23 f4pl all mip7 a04 mip8 a36 5 17 p4 a04 p8 a01 p6 a21 p3 a34 pl a05 p8 al m2 f1 m2 2 m3 f2 m3 f4 mi f1 mi 3 m4 f4 m4 f4 mi f4 m2 f2 mi 5 08 5 28 3 16 7 30 p6 al p6 a07 p4 a35 p4 a p6 a32 p4 a39 7 25 pl al p8 a24 n N W m4p2 al4 m m 3 a 1 1 A 2 E f3 m3 3 m4 f4 mi 2 m4 f4 m4 F2 m4 f2 mi f1 m3 f4 m3 T m2 f4 m2 f3 m3 p5 a20 7 16 p3 al2 p4 a09 p8 a05 p4 al4 p4 a03 p8 a33 p3 a37 pl a25 p3 a35 p5 al7 7 19 p3 a30 8 18 p8 a40 p3 al3 p6 a24 p2 a02 p5 a23 p7 a29 p2 a39 p3 a36 p3 a32 p3 all p3 a01 p5 a28 p6 a38 p2 a26 p5 a07 p3 a04 p3 a08 p7 a34 p7 a31 pl a27 p2 al5 p8 a21 p3 a22 p6 al0 p
255. pective MNRU conditions and 728 for the G 728 reference The samples processed by the IS 96 C codec for each of the five conditions are named qc1 through qc5 respectively and qc4 is replaced with qf3 and qr3 corresponding to the IS 96 C codec 3 forward and reverse FER respectively also reside here 2 1 3 2 Source Speech Material for Experiment II The source speech material for subjective Experiment is contained in directory so3 subjctv exp2 source Each sentence is flat filtered and law companded in accordance with 7 The talkers in subjective Experiment Il consist of four adult males and four adult females The clean source material for Experiment Il conditions 1 and 5 consists of 8 sentence pairs from 8 different speakers for a total of 64 speech files These files are named s22 This directory also contains the source material for the car street and babble noise conditions which are named car str and bab respectively for a total of 4 x 64 256 files The speech database also includes samples processed through the various reference conditions in directory so3 subjctv exp2 ref The reference conditions are named q05 through q25 for the respective MNRU conditions and 728 for the G 728 reference The samples processed by the IS 96 C codec for each of the five conditions named qc1 through qc5 respectively also reside here 2 1 4 Processing of Speech Material for SO Testing The source speech ma
256. peech levels This procedure is fully described in 9 2 2 1 3 End to end Algorithmic Delay Recommendation The algorithmic delay for the specific EVRC B test codec should be calculated analytically by the codec manufacturer In considering the algorithmic delay it can be assumed that all transmission channels have infinite bandwidth and that all processing elements have infinite throughput Algorithmic delay is defined as the sum of all sequential filter delays and buffering delays in the encode decode path The maximum end to end algorithmic delay should be no greater than that of the master codec For the master codecs defined in 1 the algorithmic delay is given as Delay Element SO 68 Signal Preprocessing Delay 3 milliseconds LPC Analysis Look ahead 10 milliseconds LPC Analysis Window 20 milliseconds Total 33 milliseconds 2 17 20 21 22 23 24 25 26 27 28 29 3GPP2 C S0018 E v2 0 Therefore the total algorithmic delay imposed by a SO 68 test codec should not exceed 33 milliseconds 2 2 2 Subjective Performance Testing for SO 68 This section outlines the subjective testing methodology of the subjective performance test The purpose of this testing is to evaluate the quality of the test codec under a variety of conditions which may occur in the CDMA system To accomplish this two listening experiments have been designed to test speech codec quality under a variety of conditions These condi
257. peech material delivery system shall meet the requirements of Section 3 1 1 1 The delivery system shall be calibrated to deliver an average listening level of 16 dBPa 78 dB SPL The equivalent acoustic noise level of the delivery system should not exceed 35 dBA as measured on a standard A weighted meter The listeners should be seated in a quiet room with an ambient noise of 40 dBA or below 2 1 7 Listeners The listener sample is intended to represent the population of telephone users with normal hearing acuity The listeners should be naive with respect to telephony technology issues that is they should not be experts in telephone design digital voice encoding algorithms and so on They should not be trained listeners that is they should not have been trained in these or previous listening studies using feedback trials The listeners should be adults of mixed sex and age Each listener shall provide data only once for a particular evaluation A listener may participate in different evaluations but test sessions performed with the same listener should be at least one month apart so as to reduce the effect of cumulative experience 2 1 8 Listening Test Procedures The listeners shall listen to each sample and rate the quality of the test sample using a five point scale with the points labeled 1 Bad 2 Poor 3 Fair 3GPP2 C S0018 E v2 0 4 Good 5 Excellent Data from 64 listeners shall be used for each of the two experimen
258. r of frames encoded at Rate 1 2 34 20 21 22 23 24 25 26 27 28 29 30 31 32 3GPP2 C S0018 E v2 0 N2 number of frames encoded at Rate 1 2 Ng number of frames encoded at Rate 1 8 and N N1 2 Ng For the 16 kHz input The total average channel data rate for the test codec is then given by Ravg 1 7 R ambient background segment 12dB R ambient background segment 32dB R ambient background segment 22dB R 20 dB SNR babble noise segment 22dB R 10 dB SNR car noise segment 9 22dB R 20 dB SNR car noise segment 22dB R 15 dB SNR street noise segment 22aB For the 8 kHz input The total average channel data rate for the test codec is then given by Ravg 1 6 R ambient background segment 12dB R ambient background segment 32dB R ambient background segment 22dB R 20 dB SNR babble noise segment 22dB R 15 dB SNR car noise segment 22dB R 15 dB SNR street noise segment 22dB The above files are to be processed with EVRC WB encoder at various capacity operating points defined by the active speech average channel rate shown in Table 2 3 1 1 1 1 Table 2 3 1 1 1 1 Target ADR vs Capacity Operating Point Capacity Operating Point Target Average Channel Data Rate kbps active speech average channel data rate EVRC WB RATE REDUC 000 5 641 596 EVRC WB RATE REDUC 100 5 921 596 EVRC WB
259. r on your keyboard 2 84 3GPP2 50018 v2 0 Attending ONLY to the SPEECH SIGNAL select the category which best describes the sample you just heard the SPEECH SIGNAL in this sample was VERY NATURAL NO DEGRADATIO FAIRLY NATURAL LITTLE DEGRA SOMEWHAT NATURAL SOM FAIRLY UNNATURAL FAI VERY UNNATURAL VERY For the second sentence in each trial you will be asked to attend only to the background and rate how noticeable intrusive and or conspicuous the background sounas to you You will use the rating scale shown in the figure below to register your ratings of the background Your task will be to choose the numbered phrase from the list below that best describes your opinion of the BACKGROUND ALONE and then enter the corresponding number on your keyboard Attending ONLY to the BACKGROUND select the category which best describes the sample you just heard the BACKGROUND in this sample was 5 NOT NOTICEABLE SOMEWHAT NOTICEABLE NOTICEABLE BUT NOT INTRUSIVE FAIRLY CONSPICUOUS SOMEWHAT INTRUSIV VERY CONSPICUOUS VERY INTRUSIV For the third and final sentence in each trial you will be asked to attend to the entire sample both the speech signal and the background and rate your opinion of the sample for purposes of everyday speech communication Select the cat
260. r or a Media Gateway An implementation may support SO 77 for both 16 kHz and 8 kHz sample rate for example a mobile station that supports wideband electro acoustics The main parameter in the decision tree is 16 kHz support in the implementation Depending on the implementation profile of the Device under test one of 2 possible Test Suites are to be used to demonstrate SO 77 compliance These 2 test suites named Test suites A and B and the individual input test vectors comprising the Test suites are highlighted in Table 3 5 4 5 1 Table 3 5 4 5 1 Test Suites of input test vectors for SO 77 compliance Test Suites Directory containing input test vectors so77 testvec source suiteA B so77 testvec source suiteB Files in the so77 testvec directory are provided for the purpose of qualifying a test codec as bit exact and conform to the file naming convention described in Section 2 2 4 The so77 testvec directory is divided into 2 subdirectories so77 testvec source and so77 testvec fixed The so77 testvec source directory contains input source files as well as packet files injected with frame erasures The so77 testvec fixed directory contains files processed with the EVRC NW2K fixed point reference software The files in these directories are the reference files for bit exact compliance A test codec is bit exact if it can reproduce all of the reference files in the so77 testvec fixed directory exactly The outpu
261. rating Nominal 22 dB evrc nw op1 oc 8k point 1 8 kHz sampling 15 dB car noise evrc nw op7 pc EVRC NW2K operating Nominal 22 dB evrc nw op7 oc 8k point 7 8 kHz sampling 15 dB car noise evrc nw op1 ps EVRC NWZK operating Nominal 22 dB evrc nw op1 os 8k point 1 8 kHz sampling 15 dB street noise evrc_nw_op1 fer_2 pb EVRC NW2K operating Nominal 22 dB evrc nw op1 fer 296 point 1 8 kHz sampling 15 dB babble ob 8k noise 2 FER evrc_nw_op6 fer_3 p2 EVRC NW2kK operating Nominal 22 dB evrc_nw_op6 fer_3 2 point 6 8 kHz sampling FER 3 022 8k evrc nw op6 p12 EVRC NW2K operating High 12 dB evrc nw op6 0o12 8k point 6 8 kHz sampling evrc nw op6 p32 EVRC NWZK operating Low 32 dB evrc nw op6 032 8k point 6 8 kHz sampling evrc nw op6 dim 196 pl EVRC NW2K operating Nominal 22 dB evrc nw op6 dim 196 S 196 p22 point 6 8 kHz sampling 1 d amp b 1 pls pls 196 022 8k evrc nw op6 pc EVRC NW2K operating Nominal 22 dB evrc nw op6 oc 8k point 6 8 kHz sampling 15 dB car noise evrc nw op6 ps EVRC NW2K operating Nominal 22 dB evrc nw op6 os 8k point 6 8 kHz sampling 15 dB street noise evrc_nw_op6 fer_2 pb EVRC NW2K operating Nominal 22 dB evrc nw op6 fer 296 point 6 8 kHz sampling 15 dB babble ob 8k noise 2 FER evrc_nw_op3 fer_3 p2 EVRC NW2K operating Nominal 22 dB evrc_nw_op3 fer_3 2 point 3 8 kHz sampling FER 3 022 8k evrc nw op3 p12 EVRC NW2K operating High 12 dB evrc nw op3 012 8k point 3
262. ration for SO 3 3 1 1 1 Audio Path The audio path must meet the following requirements for electro acoustic performance measured between the output of the D A converter and the output of the headphone 1 Frequency response shall be flat to within 2 dB between 200 Hz and 3400 Hz and below 200 Hz the response shall roll off at a minimum of 12 dB per octave Equalization may be used in the audio path to achieve this A suitable reconstruction filter shall be used for playback 2 Total harmonic distortion shall be less than 1 for signals between 100 Hz and 4000 Hz 3 Noise over the audio path shall be less than 35 dBA measured at the ear reference plane of the headphone 4 Signal shall be delivered to the headphone on the listener s preferred telephone ear No signal shall be delivered to the other headphone 3 1 1 2 Calibration The audio circuit shall deliver an average sound level of the stimuli to the listener at 16 dBPa 78 dB SPL at the ear reference plan This level was chosen because it is equivalent to the level delivered by a nominal ROLR handset driven by the average signal level on the PSTN network This level may be calibrated using a suitable artificial ear with circum aural headphone adapter and microphone A test file with a reference signal is included with the source speech database for the purpose of calibration The file cos1004_ 290 is located in the directory so3 cal of the companion software The calibration file
263. rdance with the tables specified in 1 The packet file value for each data rate is shown in Table 3 1 3 3 1 Table 3 1 3 3 1 Packet File Structure From Master Codec Channel Error Model Value in Packet File Rate Data Bits per Frame 4 0x0004 171 v o 1 0x0001 15 Ox000f Full Rate Probable 14 0x000e sue o o 00 O Unused bits are set to 0 For example in a Rate 1 8 frame the packet file will contain the word 0x0100 byte swapped 0x0001 followed by one 16 bit word containing the 16 data bits for the frame in byte swapped form followed by ten 16 bit words containing all zero bits 3 1 3 4 Verifying Proper Operation of the Master Codec Files are provided for the purpose of verifying the fixed point codec executable Three files mstr ref pcm mstr ref pkt and mstr_ref dec are included in the directory master test to provide a means for verifying proper operation of the master codec software The file mstr ref pcm is an unprocessed speech file The file mstr ref pkt is a packet file that was obtained by running EvrcFlt i mstr ref pcm o mstr_ref pkt e 3 6 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 3GPP2 50018 v2 0 The file mstr ref dec is a decoded speech file that was obtained by running EvrcFlt i mstr ref pkt o mstr ref dec d Once EvrcFlt is compiled verification files should be processed as follows EvrcFlt i mstr ref
264. re are three Test E DC s M T T M and T T which means there are three statistical tests against the Reference E DC M M The three statistical tests are not independent however Since they all involve the same ratings for the Reference E DC t tests are not appropriate The appropriate statistical test for multiple Test conditions against a common Reference condition is Dunnett s Test A complete description of Dunnett s Test is contained in Appendix B The critical value for the Dunnett s test is 2 09 one sided test p 05 4 E DC s df 93 For those test conditions where a single Test E DC T T is compared against the Reference E DC the appropriate statistical test is Student s t test The critical value for the Student s t test is 1 70 one sided test p lt 05 df 31 In both the Dunnett s Test and the t test the MPS test is evaluated by dividing the difference between the mean score for the Test E DC and the mean score for the Reference ED C by the Standard Error of the Mean Difference SEyp as shown in Equation 2 4 9 2 1 If the resultant Test value is less than 6 The appropriate t test is a matched groups t test and the SEmp is based on the differences between individual listener s average ratings where the average is over talkers Therefore the SEwp is based on 32 difference scores one for each listener df 231 2 86 20 21 22 3GPP2 C S0018 E v2 0 the criterion value for the appropriate te
265. recommended minimum performance requirements The EVRC A is the Service Option 3 SO 3 speech codec the EVRC B is the Service Option 68 SO 68 speech codec the EVRC WB is the Service Option 70 SO 70 speech codec EVRC NW is the Service Option 73 SO 73 speech codec and the EVRC NW2K is the Service Option 77 SO 77 speech codec 3GPP2 C S0018 E v2 0 This page intentionally left blank 22 23 24 29 30 31 32 3GPP2 C S0018 E v2 0 CONTENTS T als fe a ek oz tet Pee ter e a sd So n tet P epe Pave yn dtan 1 1 1711 2S60D6 ik ated det teta ero ted sivi 1 2 1 2 Requirements Language r iruretan raiant 1 3 1 3 References eiu pu eene 1 3 1 3 1 Normative References 1 3 1 3 2 Informative 1 4 1 4 Definitions P 1 5 1 5 Test Model for the Speech esee nennen enne 1 7 2 Codec Minimum Standards ee ete dete eod E eite 2 1 2 1 Performance Testing for SO 9 erue eite Pet t e ER redegi 2 1 2 1 14 Objective Performance Testing for SO 8 2 1 2 1 1 1 Average Data Rate ennt nnns 2 1 2 1 1 1 1 Average Data Rate
266. rectories are the reference files for bit exact compliance A test codec is bit exact if it can reproduce all of the reference files in either the so3 testvec fixed32 directory or the so3 testvec fixed31 directory 3 1 4 6 1 Description of Bit Exact Source Files The following source files are designed to exercise the majority of the bitstream slots vec 01 pcm 15dB babble 7 females 7 males vec 02 pcm 10aB car 7 females 7 males vec 03 pcm flat clean 7 females 7 males vec 04 pcm 15dB street 7 females 7 males vec_05 pcm high level 4 females 4 males vec_06 pcm low level 4 females 4 males vec_07 pcm irs clean 4 females 4 males vec_08 pcm flat clean 4 females 4 males vec_09 pcm 10aB car 4 females 4 males vec 10 pcm 15aB babble 4 females 4 males vec 11 pcm 12dB street 4 females 4 males vec 12 pcm mixed noise one sided conversation vec 13 pcm mixed noise one sided conversation The following source files are designed to exercise the RCELP algorithm NOTE These files must be processed in full rate only mode only rate 4 allowed shiftr pcm Frequency sweep shiftl pcm Frequency sweep 3 10 3GPP2 C S0018 E v2 0 The following source files are recordings of one sided conversations at different input levels and are designed to test the rate determination algorithm rda test pcm rda mod pcm rda high pcm rda low pcm The following source files are encoded packets which have been corrupted with frame erasure at different rat
267. reference files e Input and output vectors for bit exact testing An overview of the contents and formats of the software distribution is given in Section 4 of this document The EVRC A EVRC B EVRC WB EVRC NW and EVRC NW2K enhanced variable rate speech codecs collectively referred to as EVRC are intended to be used at mobile stations at compatible base stations in the cellular service This statement is not intended to preclude implementations in which codecs are placed at a Mobile Switching Center or elsewhere within the cellular system Indeed some mobile to mobile calls however routed may not require the implementation of a codec on the fixed side of the cellular system at all This standard is meant to define both verifications of bit exact implementations and the recommended minimum performance requirements of EVRC compatible variable rate codecs no matter where or how they are implemented in the cellular service Although the basic purpose of cellular telecommunications has been voice communication evolving usages for example data may allow the omission of some of the features specified herein provided that system compatibility is not compromised 1Numbers in brackets N refer to the reference document numbers For example 1 refers reference 1 in the reference list 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 3GPP2 50018 v2 0 This
268. rence Conditions 2 5 10 1 Reference Conditions for Experiments 1 3 5 and 7 The MNRU conditions have been included to provide a frame of reference for the Experiments 1 3 5 and 7 In listening evaluations where test conditions span approximately the same range of quality the MOS results for similar conditions should be approximately the same Data from previous studies allows a generalization to be made concerning the expected MOS results for the MNRU reference conditions see Figure 2 5 10 1 1 MOS scores obtained for the MNRU conditions in any SO 77 validation test should be compared to those shown in the graph below Inconsistencies beyond a small shift in the means in either direction or a slight stretching or compression of the scale near the extremes may imply a problem in the execution of the evaluation test In particular MOS should be monotonic with MNRU within the limits of statistical resolution and the contour of the relation should show a similar slope MOS 6 10 20 30 40 50 Figure 2 5 10 1 1 Typical Plot of MOS versus MNRU 2 5 10 2 Reference Conditions for Experiments 2 4 and 8 Reference conditions for P 835 tests are constructed as a combination of SNR and MNRU processing to provide degradation in overall speech quality in two dimensions signal distortion and background noise intrusiveness Table 2 5 2 3 2 2 and Table 2 5 2 3 7 2 shows the eight reference conditions bO1 2 118 3GPP2
269. rguments will display a brief description of the required and optional command line arguments The options are described below i infn required Specifies the name of the input speech file or the name of the input packet file if only decoding is being performed see the d option below o outf required Specifies the name of the output speech file or the name of the output packet file if only encoding is being performed see the e option below d Instructs the simulation to perform only the decoding function The input file must contain packets of compressed data e Instructs the simulation to perform only the encoding function The output file will contain packets of compressed data If neither the d or the e option is invoked the coder performs both the encoding and decoding functions by default M max Sets the maximum allowable data rate to max where max is element of 4 3 1 using the codes specified in the first column of Table 3 3 3 3 1 m min Sets the minimum allowable data rate to min where min is element of 4 3 1 using the codes specified in the first column of Table 3 1 3 3 1 If neither the M nor m option is invoked the coder allows the data rate to vary between Rate 1 and Rate 1 8 3 3 8 8 File Formats for SO 70 Files of speech contain 2 s complement 16 bit samples with the least significant byte first The packet file contains twelve 16 bit words with the low byte ordered first followed by the high by
270. rip is also used to split the concatenated files into the individual samples appropriate for the experiment Table 2 5 4 5 1 and Table 2 5 4 5 3 show the cutting points to be used with the astrip tool for producing the two sentence samples for the Experiments 1 and 3 ACR test and Experiment 7 ACR test respectively Table 2 5 4 5 2 and Table 2 5 4 5 4 show the cutting points to be used with the astrip tool for producing the single sentence sub samples for the Experiments 2 and 4 P 835 test and Experiment 8 P 835 test respectively Table 2 5 4 5 5 and Table 2 5 4 5 6 show the sub samples that make up the samples i e sentence triads for the P 835 test 2 106 3GPP2 C S0018 E v2 0 1 Table 2 5 4 5 1 Cutting Points for the astrip Software Tool for the SO 77 Experiments 1 and 3 2 Test Experiment I ACR 10 16 18 19 21 Sentence pair N Start sample 113707 232293 344193 461679 572672 696242 80299 13867 68071 N N 1136889 261550 5 N Co 5 N 372671 14832774 N sf po A ra rn 1609353 708427 apa 4p2 ipi 5193 21 4 3 mips 1 4 2 4 mips ESN 1 1 2 1 3 1 f4pl n1 pl 41 pl
271. s audio background noise and different input levels Figure 2 3 2 1 illustrates a decision tree to arrive at the suite of tests that are needed to demonstrate Minimum Performance Spec compliance of a Test implementation of SO 70 for different profiles of equipment that support SO 70 2 36 3GPP2 C S0018 E v2 0 Is 16kHz Sampling Rate Supported No SO 68 Compliant SO 68 Compliant Run Test Suite D Yes Run Test Suite C Run Test Suite A Figure 2 3 2 1 SO 70 Subjective test suite decision flowchart An implementation may support SO 70 only for 8 kHz sample rate input output for example a Base station transcoder or a Media Gateway An implementation may support SO 70 for both 16 kHz and 8 kHz sample rates for example a mobile station that supports wideband electro acoustics Further the implementation supporting SO 70 might already have demonstrated compliance to SO 68 Minimum Performance Spec This means that such equipment has also demonstrated the Minimum Performance requirements for RATE REDUC operating points 4 and 7 of SO 70 which exactly correspond to the RATE REDUC operating points 0 and 7 of SO 68 Therefore the main parameters in the decision tree are a 16 kHz support in the implementation and b SO 68 compliance of the test implementation Depending on the implementation profile of the device under test one of 4 possible Test Suites are to be used to demonstrat
272. s been developed and compiled using the GNU g compiler and software maintenance utilities The tools have been verified under various representative operating systems on a number of different hardware platforms The 3GPP2 supplied tools are all located in the so70 tools directory in the associated Software Distribution and can be built using the GNU g compiler Other software tools such as scaldemo actlev filter andastrip are available in 6 3 3 2 1 Channel Model Utilities fersig28 This utility program provides a the ability to introduce Frame Erasure channel impairment b the ability to verify use of half rate or lesser frame rate during dim and burst and packet level signaling the ability to measure the Average Data Rate from an encoded packet file A log output of ersig28 provides detail on the ADR performance of the preceding encoder In these applications the utility is invoked as in following examples for 3 FER and 1 signaling fersig28 c EVRC WB e fer 3 bin infile outfile fersig28 c EVRC WB s dim 15 e fer 3 bin infile outfile 3 3 2 2 Channel Error and Signaling Masks These binary Frame Error Rate and Signaling masks source level and packet level 1 byte of either 0 or 1 per frame are used with the fersig28 channel impairment and inter working simulation functions for the various conditions fer 3 bin dim 1 bin dim 1 pls bin 3 3 2 3 EVRC WB Interworking Function IW
273. s allows a generalization to be made concerning the expected MOS results for the MNRU reference conditions see Figure 2 1 11 1 MOS scores obtained for the MNRU conditions in any SO 3 validation test should be compared to those shown in the graph below Inconsistencies beyond a small shift in the means in either direction or a slight stretching or compression of the scale near the extremes may imply a problem in the execution of the evaluation test In particular MOS should be monatomic with MNRU within the limits of statistical resolution and the contour of the relation should show a similar slope Figure 2 1 11 1 MOS versus MNRU 2 15 20 21 22 23 24 25 26 27 29 3GPP2 C S0018 E v2 0 2 2 Performance Testing for SO 68 2 2 1 Objective Performance Testing for SO 68 The objective testing portion of this specification consists of an average data rate test and compliance to End to End Algorithmic Delay and Unity gain requirements 2 2 1 1 Average Data Rate Test The average data rate for the test codec shall be measured using six source speech files that are contained in the so68 subjctv exp source directory Each file exhibits a different condition power levels 12 dB 22 dB and 32 dB and background noise conditions 20 dB SNR babble noise condition 15 dB SNR car noise condition and 15 dB SNR street noise The input source files used in the average data rate test have an approximate voice activity factor o
274. s and the decoded speech files generated by the 32 bit long multiply DSP library Likewise files in so3 simul fixed test fixed31 were processed with the 31 bit DSP library The processed files have the following naming convention The encoded packet have the extension pkt and are generated by running EvrcFix i pcm o pkt e the decoded speech files dec are generated by running EvrcFix i pkt o dec d If the output files pkt and dec exactly match verify pkt and verify dec respectively then verification of the operation of the fixed point codec s operation is complete 3 1 4 6 Verifying Bit Exact Performance of the Fixed Point Test Codec Files in the so3 testvec directory are provided for the purpose of qualifying a test codec as bit exact The files in the so3 testvec directories are 16 bit PCM binary files in PC format LSB MSB and obey the following file extension naming convention 3 9 13 14 3GPP2 C S0018 E v2 0 Source speech pcm encoder output pkt decoder output dec The so3 testvec directory is divided into subdirectories so3 testvec source so3 testvec fixed31 and so3 testvec fixed32 The so3 testvec source directory contains input source files and includes original speech files as well as packet files injected with frame erasures The so3 testvec fixed31 so3 testvec fixed32 directory contains files processed with the 31 bit 32 bit DSP library The files in these di
275. s been included in 1a Typing make in the appropriate directory will compile and link the code and create the executable file called EvrcFlt evrcfit exe on Win32 systems The included makefile may require some user modification for a particular hardware platform and or operating system 3 1 3 2 Running the Master Codec Simulation The EVRC executable files use command line arguments to receive all information regarding input and output files and various parameters used during execution Executing EvrcFlt with no command line arguments will display a brief description of the required and optional command line arguments The options are described below i infn required Specifies the name of the input speech file or the name of the input packet file if only decoding is being performed see the d option below o outf required Specifies the name of the output speech file or the name of the output packet file if only encoding is being performed see the e option below d Instructs the simulation to perform only the decoding function The input file must contain packets of compressed data e Instructs the simulation to perform only the encoding function The output file will contain packets of compressed data If neither the d or the e option is invoked the coder performs both the encoding and decoding functions by default h max Sets the maximum allowable data rate to max where max is element of 4 3 1 using the codes
276. sample was VERY NATURAL NO DEGRADATIO FAIRLY NATURAL LITTLE DEGRA SOMEWHAT NATURAL SOM FAIRLY UNNATURAL FAI VERY UNNATURAL VERY For the second sentence in each trial you will be asked to attend only to the background and rate how noticeable intrusive and or conspicuous the background sounas to you You will use the rating scale shown in the figure below to register your ratings of the background Your task will be to choose the numbered phrase from the list below that best describes your opinion of the BACKGROUND ALONE and then enter the corresponding number on your keyboard Attending ONLY to the BACKGROUND select the category which best describes the sample you just heard the BACKGROUND in this sample was 5 NOT NOTICEABLE SOMEWHAT NOTICEABLE NOTICEABLE BUT NOT INTRUSIVE FAIRLY CONSPICUOUS SOMEWHAT INTRUSIV VERY CONSPICUOUS VERY INTRUSIV For the third and final sentence in each trial you will be asked to attend to the entire sample both the speech signal and the background and rate your opinion of the sample for purposes of everyday speech communication 2 115 3GPP2 C S0018 E v2 0 Select the category which best describes the sample you just heard for purposes of everyday speech communication the OVERALL SPEECH SAMPLE was EXCELLENT
277. sampling src s22 8k EVRC NW2K Nominal 22 dB evrc nw op6 p22 operating point 6 8 kHz sampling src s12 8k EVRC NW2K High 12 dB evrc nw op6 p12 operating point 6 8 kHz sampling src s32 8k EVRC NW2K Low 32 dB evrc nw op6 p32 operating point 6 8 kHz samplin 3 53 3GPP2 C S0018 E v2 0 Reference packet files for bit Input File Operating Point Condition exact compliance src s22 8k EVRC NW2K Nominal 22 dB evrc nw op6 dim 196 p22 operating point 6 8 kHz 196 d amp b sampling src s22 8k EVRC NW2K Nominal 22 dB evrc nw op7 p22 operating point 7 8 kHz sampling src c15 8k EVRC NW2K Nominal 22 dB evrc nw opi pc operating point 1 8kHz 15 dB car noise sampling src s15 8k EVRC NWZ2K Nominal 22 dB evrc nw opl1 ps operating point 1 8 kHz 15 aB street sampling noise src b20 8k EVRC NW2K Nominal 22 dB evrc nw op1 pb operating point 1 8 kHz 20 dB babble sampling noise Src c15 8k EVRC NW2K Nominal 22 dB evrc nw op6 pc operating point 6 8 kHz 15 dB car noise sampling src s15 8k EVRC NW2K Nominal 22 dB evrc nw op6 ps operating point 6 8 kHz 15 dB street sampling noise src b20 8k EVRC NWZ2K Nominal 22 dB evrc nw op6 pb operating point 6 8 kHz 20 dB babble sampling noise Src c15 8k EVRC NW2K Nominal 22 dB evrc nw _ 7 operating point 7 8 kHz 15 dB car noise sampling Sresrc 2kbps s22 8 EVRC NW2K Nominal 22 dB evrc nw op3 p22 k
278. scoder or a Media Gateway An implementation may support SO 73 for both 16 kHz and 8 kHz sample rates for example a mobile station that supports wideband electro acoustics Therefore the main parameter in the decision tree is 16 kHz support in the implementation Depending on the implementation profile of the device under test one of 2 possible Test Suites are to be used to demonstrate SO 73 compliance These 2 test suites named Test suites A and B and the individual tests comprising the Test suites are highlighted in Table 2 4 2 1 Table 2 4 2 1 Test Suites for SO 73 compliance Test Suites Set of Experiments A Experiment 1 2 3 4 5 and 6 B Experiment 3 4 and 6 Each of the individual experiments are further defined in detail by Table 2 4 2 2 Table 2 4 2 2 Experiments for SO 73 compliance Experiment Individual tests Notes 1 WB clean level FER signaling ACR Mobile supporting 16 kHz Fs 2 WB noise FER P 835 Mobile supporting 16 kHz Fs 3 NB clean level FER signaling including BS supporting 8 kHz and MS supporting SO 68 interoperable mode tests ACR 8 16 kHz 4 NB noise FER including SO 68 BS supporting 8 kHz and MS supporting interoperable mode tests P 835 8 16 kHz 5 WB music decoder test ACR Mobile supporting 16 kHz Fs 6 NB music decoder test ACR BS supporting 8 kHz Fs 2 66 22 23 24 3GPP2 C S0018 E v2 0 2 4 8 1 Definition The codec subjective test is intended to
279. shall be less than 196 for signals between 100 Hz and 4000 Hz 3 Noise over the audio path shall be less than 35 dBA measured at the ear reference plane of the headphone 4 Signal shall be delivered to the headphone on the listener s preferred telephone listening ear and the other ear shall be uncovered No signal shall be delivered to the other headphone 3 3 1 2 Calibration The audio circuit shall deliver an average sound level of the stimuli to the listener at 18 dBPa 76 dB SPL at the ear reference plan This level was chosen because it is equivalent to the level delivered by a nominal ROLR handset driven by the average signal level on the PSTN network This level may be calibrated using a suitable artificial ear with circum aural headphone adapter and microphone A test file with a reference signal is included with the source speech database for the purpose of calibration The file cal 1004 16k is located in the directory so70 cal of the companion software The calibration file contains a 22 dB 1004 Hz reference signal The audio circuit shall be calibrated so that the test signal has a level of 15 dBPa at the ear reference plane while maintaining compliance with Section 3 3 1 1 3 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 3GPP2 C S0018 E v2 0 3 3 2 Software Test Tools for SO 70 This section describes a set of software tools useful for performing the MPS tests The code ha
280. specified in the first column of Table 3 1 3 3 1 min Sets the minimum allowable data rate to min where min is element of 4 3 1 using the codes specified in the first column of Table 3 1 3 3 1 3 5 20 21 22 23 24 25 26 27 28 29 30 3GPP2 C S0018 E v2 0 If neither the h nor I option is invoked the coder allows the data rate to vary between Rate 1 and Rate 1 8 In addition if max min the data rate varies between max and min using the same rate decision algorithm where the data rate is set to max if the selected data rate is gt max and the data rate is set to min if the selected data rate is lt min See the select rate routine in the file ratedec c for more information p flag If flag is set to 0 the post filter is disabled If the flag is set to 1 the post filter is enabled If the p option is not invoked the post filter is enabled during decoding n flag If flag is set to 0 noise suppression is disabled If the flag is set to 1 noise suppression is enabled f the n option is not invoked noise suppression is enabled during encoding 3 1 8 3 File Formats Files of speech contain 2 s complement 16 bit samples with the least significant byte first The packet file contains twelve 16 bit words with the low byte ordered first followed by the high byte The first word in the packet contains the data rate while the remaining 11 words contain the encoded speech data packed in acco
281. st 2 09 for Dunnett s Test 1 70 for the t test then the E DC passes the MPS test Miu Mus Test 2 4 9 2 1 2 4 10 Expected Results for Reference Conditions 2 4 10 1 Reference Conditions for Experiments 1 3 and 5 The MNRU conditions have been included to provide a frame of reference for the Experiments 1 3 and 5 In listening evaluations where test conditions span approximately the same range of quality the MOS results for similar conditions should be approximately the same Data from previous studies allows a generalization to be made concerning the expected MOS results for the MNRU reference conditions see Figure 2 4 10 1 1 MOS scores obtained for the MNRU conditions in any SO 73 validation test should be compared to those shown in the graph below Inconsistencies beyond a small shift in the means in either direction or a slight stretching or compression of the scale near the extremes may imply a problem in the execution of the evaluation test In particular MOS should be monotonic with MNRU within the limits of statistical resolution and the contour of the relation should show a similar slope MOS W 10 20 30 40 50 dBQ Figure 2 4 10 1 1 Typical Plot of MOS versus MNRU 2 4 10 2 Reference Conditions for Experiments 2 and 4 Reference conditions for P 835 tests are constructed as a combination of SNR and MNRU processing to provide degradation in overall speech quality in two dimensions signal d
282. structions for Listeners 2 1 9 Analysis of Results The response data from the practice blocks shall be discarded Data sets with missing responses from listeners shall not be used Responses from the different sets of encoder decoder processed files shall be treated as equivalent in the analysis The votes for each of the 31 conditions and references for each of SO 3 Experiment and II shall be averaged in accordance with 10 to produce an associated mean opinion score MOS Additionally the standard error SER for each condition shall be calculated as described in the next section 2 12 3GPP2 C S0018 E v2 0 2 1 10 Minimum Subjective Requirement For each of the test combinations T M M T T T the MOS results are compared to those of the respective master codec M M The exception to this being the 3 FER case in which M T is compared to M M and T T is compared to T M3 If the MOS for the test combination condition is within an allowable difference as defined below of the MOS for the master combination condition then the subjective test is passed for that combination condition If any of the test combinations conditions exceeds the maximum allowable difference the test codec fails the compliance test These requirements can be clarified by first defining the MOS for a given combination condition as pg i 1 2 MOS jk 55 Zika jell 5 t keil 4 pg i 1 2 MOS i j k ap 2 0 jel 3 2 1 10 1 ke
283. t processing and also the Packet Level Signaling conditions in Experiment I the processing requires inputs from a signaling file to control maximum encoding rate An external software utility EvrcB_iwf in Section 3 2 2 3 is also needed to reduce the data rate of certain packets from full rate to half rate Details of these operations are given in Section 6 The signaling file and other utilities are provided in so68 tools directory 2 2 44 Ensuring Proper Encoded Frame Packet Files All encoded frame packet files shall be examined to ensure that the files only contain data in those file locations where data should exist for a given data rate The examination of the encoded frame packet files should indicate the occurrence of any improper data in the files but the examination must not alter the encoded frame packet files in any way 2 2 4 5 Post processing of test condition output files In order to build the play sets to be presented to the listening panels the output files for the various test conditions must be processed to provide the appropriate listening conditions In addition the concatenated output files must be partitioned into the samples representing the combination of test condition and talker The listening conditions are provided by filtering the output files using the STL software tool filter with the MIRS receive filter mask An STL tool astrip is also used to split the concatenated files into the individual samples appropri
284. te The first word in the packet contains the data rate while the remaining 11 words contain the encoded speech data packed in accordance with the tables specified in 1 The packet file value for each data rate is shown in Table 3 1 3 3 1 Table 3 3 3 3 1 Packet File Structure From Master Codec Channel Error Model Value in Packet File Rate Data Bits per Frame 4 0 004 171 w o 3 23 20 21 22 23 24 25 26 27 29 30 31 32 33 34 35 36 37 3GPP2 C S0018 E v2 0 Unused bits are set to 0 For example in a Rate 1 8 frame the packet file will contain the word 0x0100 byte swapped 0x0001 followed by one 16 bit word containing the 16 data bits for the frame in byte swapped form followed by ten 16 bit words containing all zero bits 3 3 4 Fixed Point Bit Exact Codec for SO 70 This section describes the C simulation of the speech codec specified by 1 The speech codec C simulation is based on finite precision fixed point arithmetic operations and is recommended to be used as a reference codec to verify the performance of a bit exact EVRC WB implementation of the fixed point C simulation of a test codec The bit exact EVRC WB codec along with the appropriate test vectors to verify the bit exactness performance are included in the associated Software Distribution 3 3 4 1 Fixed Point Codec Program Files This section describes the C program files which are provide
285. terial shall be processed by the various combinations of encoders and decoders listed in the descriptions of the two experiments given in Section 2 1 2 The master codec software described in Section 3 1 3 shall be used in the processing involving the master codec Generally the master codec encoder and decoder outputs have been provided in the respective so3 subjctv exp m pkt and so3 subjctv exp m m directories Execution of the master codec software is generally needed only for the test encoder master decoder combination for each experiment condition The exception to this is the tandem condition in Experiment 11 where double codec processing is required see Section 2 1 4 4 All codec processing shall be done digitally Noise suppression and post filter options shall be enabled for both the master and the test codecs The digital format of the speech files is described in Section 3 1 4 4 The naming convention of the processed speech is as follows For the packet files in the so3 subjctv exp1 m_pkt directory Experiment l the p12 files are the master packet files for the 12 source files Likewise the p22 and p32 files are the respective packet files for the s22 and 532 source files The pf3 and pr3 are the impaired packet files which will be described in Section 2 1 4 8 Condition five Rate 1 2 maximum it uses phr as the extension for the half rate max packets 2 8 21 22 23 24 25 26 27 28
286. th the astrip tool for producing the two sentence samples for the Experiments 1 and 3 ACR test Table 2 4 4 5 2 shows the cutting points to be used with the astrip tool for producing the single sentence sub samples for the Experiments 2 and 4 P 835 test Table 2 4 4 5 3 shows the sub samples that make up the samples i e sentence triads for the P 835 test 2 78 3GPP2 C S0018 E v2 0 1 Table 2 4 4 5 1 Cutting Points for the astrip Software Tool for the SO 73 Experiments 1 and 3 2 Test Experiment I ACR 10 16 18 19 21 Sentence pair N Start sample 113707 232293 344193 461679 572672 696242 80299 13867 68071 N N 1136889 261550 5 N Co 5 N 372671 14832774 N sf po A ra rn 1609353 708427 apa 4p2 ipi 5193 21 4 3 mips 1 4 2 4 mips ESN 1 1 2 1 3 1 f4pl n1 pl 41 pl n2p1 2 1 n3p1 3p n4p1 4p 4 p2 n4p2 f4p2 n1p3 fes p3 n4p3 f4p3 nip4 41 p4 f2p4 f4p OY Ke Ko Of Of O17 OTF on OIF Of OTF Bol A GOT Fe CO Go 1817228 1920188 2038580 2160485 2281 2385937 2494010 259152
287. tion of SNR and MNRU processing to provide degradation in overall speech quality in two dimensions signal distortion and background noise intrusiveness Table 2 2 2 3 2 2 shows the eight reference conditions 601 608 involved in the 2 33 20 21 22 23 24 25 26 27 28 29 30 31 32 3GPP2 50018 v2 0 P 835 Experiment Il In general results are expected for these reference conditions such that the obtained score profiles are similar to those shown in Figure 2 2 10 2 1 SNR 40dB Car Noise MNRU 4098 MNRU SNR E a a SG SG B BAK SG _ BAK P 835 Scores w P 835 Scores w P 835 Scores w A OVRL OVAL oaB 20aB 40aB 098 20 40 dB 10 10 dB 20 20 dB 30 30 dB 40 40 dB MNRU SNR Car Noise SNR Car Noise Figure 2 2 10 2 1 P 835 Score Profiles for Reference Conditions 2 3 Performance Testing for SO 70 2 3 4 Objective Performance Testing for SO 70 The objective testing portion of this specification consists of an average data rate test and compliance to End to End Algorithmic Delay and Unity gain requirements 2 3 1 1 Average Data Rate Test An implementation may support SO 70 for 16 kHz sample rates for example a mobile station that supports wideband electro acoustics The average data rate f
288. tions Nominal level Modes 0 4 7 Low level Modes 0 4 High Level Mode 0 4 196 d amp b 1 pls Modes 0 4 396 FER Modes 0 4 Encoder Decoder Combinations 4 M M M T T T T M The Test Conditions for Listening Experiment 3 are presented in Table 2 3 2 3 3 2 Table 2 3 2 3 3 2 SO 70 Listening Experiment 3 Test Conditions Exp 3 Narrowband ACR Reference Conditions File MNRU c01 5dB MNRU Reference c02 10dB MNRU Reference 15dB MNRU Reference c04 20dB MNRU Reference c05 25dB MNRU Reference 06 30dB MNRU Reference c07 35dB MNRU Reference c08 Direct Source Reference Test Conditions File Condition Enc Dec 09 Nominal Mode 0 LB portion of Wideband mode decoder test only M M c10 Nominal Mode 0 LB portion of Wideband mode decoder test only M T c11 Nominal Mode 4 interoperable with Mode 0 of SO 68 support M M c12 Nominal Mode 4 interoperable with Mode 0 of SO 68 support M T 2 43 3GPP2 C S0018 E v2 0 Exp 3 Narrowband ACR c13 Nominal Mode 4 interoperable with Mode 0 of SO 68 support T T c14 Nominal Mode 4 interoperable with Mode 0 of SO 68 support T M c15 Nominal Mode 7 interoperable with Mode 0 of SO 68 support T T c16 Nominal Mode 7 interoperable with Mode 0 of SO 68 support T M c17 Nominal Mode 7 interoperabl
289. tions 8 Specified reference conditions Test conditions Encoder Decoder Combinations Nominal level Modes 0 4 7 Low level Modes 0 4 High Level Mode 0 4 1 d amp b 1 pls Modes 0 4 3 FER Modes 0 4 4 M M M T T T T M The Test Conditions for Listening Experiment 3 are presented in Table 2 5 2 3 3 2 Table 2 5 2 3 3 2 SO 77 Listening Experiment 3 Test Conditions Operating a Encoder Decoder Label Point Condition Combinations c01 Reference MNRU 5dB c02 Reference MNRU 10aB 2 95 3GPP2 50018 v2 0 Label Operating Condition ponen c03 Reference MNRU 15dB c04 Reference MNRU 20dB c05 Reference MNRU 25dB c06 Reference MNRU 30dB c07 Reference MNRU 35dB c08 Reference Direct c09 Mode 1 Nominal 22 dB M M c10 Mode 1 Nominal 22 dB M T c11 Mode 1 Nominal 22 dB T T c12 Mode 1 Nominal 22 dB T M c13 Mode 6 Nominal 22 dB M M c14 Mode 6 Nominal 22 dB M T c15 Mode 6 Nominal 22 dB T T c16 Mode 6 Nominal 22 dB T M c17 Mode 7 Nominal 22 dB M M c18 Mode 7 Nominal 22 dB M T c19 Mode 7 Nominal 22 dB T T c20 Mode 7 Nominal 22 dB T M c21 Mode 1 Low 32 dB 1 d amp b 1 pls M M c22 Mode 1 Low 32 dB 1 d amp b 1 pls M T c23 Mode 1 Low 32 dB 1 d amp b 1 pls T T c24 Mode 1 Low 32 dB 1 d amp b 1 pls T M c25 Mode 6 L
290. tions include channel impairments audio background noise and different input levels 2 2 2 1 Definition The codec subjective test is intended to validate the implementation of the speech codec being tested using the master codec defined in 3 2 3 as a reference Experiment is based on the Absolute Category Rating ACR method which yields the Mean Opinion Score MOS as described in 10 Experiment Il is based on the ITU T Recommendation P 835 described in 13 2 2 2 2 Method of Measurement The subjective test involves a listening only assessment of the quality of the codec being tested using the master codec as a reference Subjects from the general population of telephone users will rate the various conditions of the test Material supplied with this standard for use with this test includes source speech impaired packet files from the master codec encoder and source speech processed by various Modulated Noise Reference Unit MNRU conditions and other references The basic Absolute Category Rating test procedure involves rating all conditions using a five point scale describing the opinion of the test condition This procedure is fully described in 10 The P 835 test method involves rating all conditions on scales of Signal Background and Overall quality and is fully described in 13 2 2 2 3 Test Conditions and Test Design for SO 68 The first listening experiment for SO 68 is performed as an ACR listening test The s
291. tive operating systems on a number of different hardware platforms The 3GPP2 supplied tools are all located in the so73 tools directory in the associated Software Distribution and can be built using the GNU g compiler Other software tools such as scaldemo actlev filter andastrip are available in 6 3 4 8 1 Channel Model Utilities exsig29 exe This utility program provides d the ability to introduce Frame Erasure channel impairment e the ability to verify use of half rate or lesser frame rate during dim and burst and packet level signaling f the ability to measure the Average Data Rate from an encoded packet file A log output of ersig29 provides detail on the ADR performance of the preceding encoder In these applications the utility is invoked as in following examples for 3 FER and 1 signaling fersig29 c EVRC NW e fer 3 bin infile outfile fersig29 c EVRC NW s dim 15 bin e fer 3 bin infile outfile 3 4 8 2 Channel Error and Signaling Masks These binary Frame Error Rate and Signaling masks source level and packet level 1 byte of either 0 or 1 per frame are used with the fersig29 channel impairment and inter working simulation functions for the various conditions fer 35 bin dim 1 bin dim 1 pls bin 3 4 2 3 EVRC NW Interworking Function IWF The software Evrc nw iwf cc can be compiled to yield a simulation utility Evrc nw iwf with usage defined as Evrc nw iwf s signalin
292. to 0 For example in a Rate 1 8 frame the packet file will contain the word 0x0100 byte swapped 0x0001 followed by one 16 bit word containing the 16 data bits for the frame in byte swapped form followed by ten 16 bit words containing all zero bits 3 4 4 5 Verifying Bit Exact Performance of the Fixed Point Test Codec This section outlines the methodology of verifying whether a Fixed point Test codec is bit exact to the Fixed point reference software The purpose of this testing is to evaluate the bit exactness of the test codec under a variety of conditions which may occur To accomplish this suites of test vectors have been designed to test for bit exactness of the Test Codec under a variety of conditions depending on a number of parameters These conditions include channel impairments audio background noise and different input levels Figure 3 4 4 5 1 illustrates a decision tree to arrive at the suite of test vectors that are needed to demonstrate Minimum Performance Spec compliance through bit exactness of a Test implementation of SO 73 for different profiles of equipments that support SO 73 3 39 3GPP2 C S0018 E v2 0 Is 16kHz Sampling Rate Supported Yes No Run Test Suite A Run Test Suite B Figure 3 4 4 5 1 SO 73 Fixed point bit exact test suite decision flowchart An implementation may support SO 73 only for 8 kHz sample rate input output for example a Base station transcoder or a Media Gateway An impl
293. to Table 4 2 for the composition of the Software 14 Distribution database 2 22 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 3GPP2 C S0018 E v2 0 2 2 3 1 Source Speech Material for SO 68 Experiment The source speech material for subjective Experiment is contained in directory so68 subjctv exp1 source Each file is MIRS filtered and level adjusted to 22 12 or 32 dB These files are named src s22 src s12 and src s32 respectively The speech database also includes samples processed through the various reference conditions in directory so68 subjctv exp1 ref The reference conditions are named a01 through a08 for the respective conditions given in Table 2 2 2 3 1 2 2 2 3 2 Source Speech Material for SO 68 Experiment II The source speech material for subjective Experiment is contained in directory so68 subjctv exp2 source This directory contains the source material for the car street and babble noise conditions which are named src c15 src s15 and src b20 respectively The speech database also includes samples processed through the various reference conditions directory so68 subjctv exp2 ref The reference conditions are named ref b01 through ref b08 for the respective conditions given in Table 2 2 2 3 2 2 2 2 4 Processing of Speech Material for SO 68 Testing The source speech material shall be processed by the various combinations of encoders a
294. to end algorithmic delay should be no greater than that of the master codec For the master codecs defined in 1 the algorithmic delay is given as Delay Element 0 3 Signal Preprocessing Delay 3 milliseconds LPC Analysis Look ahead 10 milliseconds LPC Analysis Window 20 milliseconds Total 33 milliseconds Therefore the total algorithmic delay imposed by a SO 3 test codec should not exceed 33 milliseconds 2 1 2 Subjective Performance Testing for SO 3 This section outlines the subjective testing methodology of the subjective performance test The purpose of this testing is to evaluate the quality of the test codec under a variety of conditions which may occur in the CDMA system To accomplish this two listening experiments have been designed to test speech codec quality under a variety of conditions These conditions include channel impairments codec tandem audio background noise and different input levels In addition half rate maximum operation of the codec will be examined 2 2 20 21 22 23 24 25 26 3GPP2 C S0018 E v2 0 2 1 2 1 Definition The codec subjective test is intended to validate the implementation of the speech codec being tested using the master codec defined in Section 3 1 3 as a reference The subjective tests for SO 3 are based on the Absolute Category Rating Mean Opinion Score MOS test as described in 10 2 1 2 2 Method of Measurement The subjective test involves a listening
295. tput files and various parameters used during execution Executing EvrcB with no command line arguments will display a brief description of the required and optional command line arguments The options are described below i infn required Specifies the name of the input speech file or the name of the input packet file if only decoding is being performed see the d option below o outf required Specifies the name of the output speech file or the name of the output packet file if only encoding is being performed see the e option below d Instructs the simulation to perform only the decoding function The input file must contain packets of compressed data e Instructs the simulation to perform only the encoding function The output file will contain packets of compressed data If neither the d or the e option is invoked the coder performs both the encoding and decoding functions by default M max Sets the maximum allowable data rate to max where max is element of 4 3 2 1 using the codes specified in the first column of Table 3 2 3 3 1 3 15 20 21 22 23 24 25 26 27 28 3GPP2 C S0018 E v2 0 m min Sets the minimum allowable data rate to min where min is element of 4 3 2 1 using the codes specified in the first column of Table 3 2 3 3 1 If neither the M nor m option is invoked the coder allows the data rate to vary between Rate 1 and Rate 1 8 W lt target_active_speech_channel_
296. ts The experiment may be run with up to eight listeners in parallel that is hearing the same random order of test conditions at the same time Before starting the test the listeners should be given the instructions in Figure 2 1 8 1 The instructions may be modified to allow for variations in laboratory data gathering apparatus This is an experiment to determine the perceived quality of speech over the telephone You will be listening to a number of recorded speech samples spoken by several different talkers and you will be rating how good you think they sound The sound will appear on one side of the headphones Use the live side on the ear you normally use for the telephone On each trial a sample will be played After you have listened to each passage the five buttons on your response box will light up Press the button corresponding to your rating for how good or bad that particular passage sounded During the session you will hear samples varying in different aspects of quality Please take into account your total impression of each sample rather than concentrating on any particular aspect The quality of the speech should be rated according to the scale below Bad Excellent Rate each passage by choosing the word from the scale which best describes the quality of speech you heard There will be 279 trials including 31 practice trials at the beginning Thank you for participating in this research Figure 2 1 8 1 In
297. ts of the encoder and decoder of the test codec are to be obtained for the conditions given below in Table 3 5 4 5 2 Table 3 5 4 5 5 The processing steps for these conditions are illustrated in Section 6 3 52 3GPP2 C S0018 E v2 0 Table 3 5 4 5 2 SO 77 Encoder Suite A Bit exact Test Conditions Input File Operating Point Condition VE aa a src s22 EVRC NW2K Nominal 22 dB evrc nw 22 operating point 0 16 kHz sampling Src s12 EVRC NW2K High 12 dB evrc nw 12 operating point 0 16 kHz sampling src s32 EVRC NW2K Low 32 dB evrc nw opO dim 196 p32 operating point 0 16 196 d amp b kHz sampling src c10 EVRC NW2K Nominal 22 dB evrc nw 1 operating point 0 16 10 dB car noise kHz sampling src c20 EVRC NW2K Nominal 22 dB evrc nw opO0 pc2 operating point 0 16 20 dB car noise kHz sampling src s15 EVRC NW2K Nominal 22 dB evrc nw operating point 0 16 15 dB street kHz sampling noise src b20 EVRC NW2K Nominal 22 dB evrc nw opoO pb operating point 0 16 20 dB babble kHz sampling noise src s22 8k EVRC NW2K Nominal 22 dB evrc nw opi p22 operating point 1 8 kHz sampling src s12 8k EVRC NW2K High 12 dB evrc nw opi pi2 operating point 1 8 kHz sampling src s32 8k EVRC NW2K Low 32 dB evrc nw opi p32 operating point 1 8 kHz sampling src s22 8k EVRC NW2K Nominal 22 dB evrc nw opi dim 196 p22 operating point 1 8 kHz 1 d amp b
298. uction of Impairments For the frame error conditions the impaired master codec encoded packet files are provided in the so70 subjctv exp m_pkt directory Unlike other conditions this condition uses only the test decoder and not the test encoder For the Dim and Burst processing and also the Packet Level Signaling conditions the processing requires inputs from a signaling file to control maximum encoding rate An external software utility Evrc_wb_iwf in Section 3 3 2 3 is also needed to reduce the data rate of certain packets from full rate to half rate Details of these operations are given in Section 6 The signaling file and other utilities are provided in so70 tools directory 2 3 44 Ensuring Proper Encoded Frame Packet Files All encoded frame packet files shall be examined to ensure that the files only contain data in those file locations where data should exist for a given data rate 2 52 3GPP2 C S0018 E v2 0 The examination of the encoded frame packet files should indicate the occurrence of any improper data in the files but the examination must not alter the encoded frame packet files in any way 2 3 45 Post processing of test condition output files In order to build the play sets to be presented to the listening panels the output files for the various test conditions must be processed to provide the appropriate listening conditions In addition the concatenated output files must be partitioned into the samples represe
299. users will rate the various conditions of the test Material supplied with this standard for use with this test includes source speech impaired packet files from the master codec encoder and source speech processed by various Modulated Noise Reference Unit MNRU conditions and other references The basic Absolute Category Rating test procedure involves rating all conditions using a five point scale describing the opinion of the test condition This procedure is fully described in 10 The P 835 test method involves rating all conditions on scales of Signal Background and Overall quality and is fully described in 13 2 5 2 8 Test Conditions and Test Design for SO 77 Listening experiments 1 3 and 7 for SO 77 are performed as ACR listening tests Experiments 2 4 and 8 for SO 77 are performed as P 835 listening tests 2 5 2 3 1 Subjective Experiment 1 for SO 77 The Test Parameters for Listening Experiment 1 are presented in Table 2 5 2 3 1 1 Table 2 5 2 3 1 1 SO 77 Listening Experiment 1 Test Parameters Type of test MOS P 800 Wideband Number of talkers 4 males 4 females Background noise none ambient Audio Input Level 22 dB 32 dB 12 dB Filter characteristics P 341 refer Section 3 3 2 4 Reference conditions 8 Specified reference conditions Test conditions o Low Audio Input Level 32 dB 196 d amp b o Nominal Audio Input Level 22 dB o High Audio Input Level 12 dB o 3 FER and 1 FER 2 pls at Nominal
300. ve operating systems on a number of different hardware platforms The 3GPP2 supplied tools are all located in the so77 tools directory in the associated Software Distribution and can be built using the GNU g compiler Other software tools such as scaldemo actlev filter andastrip are available in 6 3 5 2 4 Channel Model Utilities exsig29 exe This utility program provides g the ability to introduce Frame Erasure channel impairment h the ability to verify use of half rate or lesser frame rate during dim and burst and packet level signaling i the ability to measure the Average Data Rate from an encoded packet file A log output of ersig29 provides detail on the ADR performance of the preceding encoder In these applications the utility is invoked as in following examples for 3 FER and 1 signaling fersig29 c EVRC NW e fer 3 bin infile outfile fersig29 c EVRC NW s dim 15 e fer 3 bin infile outfile 3 5 2 2 Channel Error and Signaling Masks These binary Frame Error Rate and Signaling masks source level and packet level 1 byte of either 0 or 1 per frame are used with the fersig29 channel impairment and inter working simulation functions for the various conditions fer 1 bin fer 35 bin dim 1 bin dim 1 pls bin 3 5 2 8 EVRC NW Interworking Function IWF The software Evrc nw iwf cc can be compiled to yield a simulation utility Evrc nw iwf with usage defined as Evrc nw iw
301. vrc nw op1 p32 EVRC NW operating Low 32 dB evrc nw 0p1 032 8k point 1 8 kHz sampling evrc nw op7 p22 EVRC NW operating Nominal 22 dB evrc nw op7 o22 8k point 7 8 kHz sampling evrc nw opi dim 196 pl EVRC NW operating Nominal 22 dB evrc nw opti dim 196 1 22 point 1 8 kHz sampling 3 42 1 d amp b 1 pls pls 196 022 8k 3GPP2 C S0018 E v2 0 Input Packet File Operating Point Condition Reference output speech files for bit exact compliance evrc nw opi pc EVRC NW operating point 1 8 kHz sampling Nominal 22 dB 15 dB car noise evrc nw op1 oc 8k evrc nw op7 pc EVRC NW operating point 7 8 kHz sampling Nominal 22 dB 15 dB car noise evrc nw op7 oc 8k evrc nw opi ps EVRC NW operating point 1 8 kHz sampling Nominal 22 dB 15 dB street noise evrc nw op1 os 8k evrc nw 296 pb EVRC NW operating point 1 8 kHz sampling Nominal 22 dB 15 dB babble noise 296 FER evrc nw op1 fer 296 ob 8k evrc_nw_op6 fer_3 p2 2 EVRC NW operating point 6 8 kHz sampling Nominal 22 dB FER 396 evrc nw op6 fer 396 022 8k evrc nw op6 p12 EVRC NW operating point 6 8 kHz sampling High 12 dB evrc nw op6 012 8k evrc nw op6 p32 EVRC NW operating point 6 8 kHz sampling Low 32 dB evrc nw op6 032 8k evrc nw op6 dim 196 pl S 196 p22 EVRC NW operating point 6 8 kHz sampling N
302. what intrusive Very conspicuous very intrusive AC For the third sub sample listeners rate the Overall quality on a five point rating scale with the points labeled Excellent Good Fair Poor Bad Data from 32 listeners shall be used for Experiments 2 4 and 8 four listeners for each listening panel where each listening panel uses a different randomization Before starting the test the listeners should be given instructions for performing the subjective test An example set of instructions for the P 835 test are presented below The instructions may be modified to allow for variations in laboratory data gathering apparatus 2 114 3GPP2 C S0018 E v2 0 Instructions for P 835 Speech Rating Experiment In this speech rating experiment each trial will involve three sentences and you will give a rating for each sentence For the first sentence in each trial you will be asked to attend only to the speech signal and rate how natural or conversely how degraded the speech signal sounas to you You will use the rating scale shown in the figure below to register your ratings of the speech signal Your task will be to choose the numbered phrase from the list below that best describes your opinion of the SPEECH SIGNAL ALONE and then enter the corresponding number on your keyboard Attending ONLY to the SPEECH SIGNAL select the category which best describes the sample you just heard the SPEECH SIGNAL in this
303. y A parallel set of bit exact test vectors is provided so that a CODEC may qualify as bit exact using either library 3 1 4 1 Fixed Point Codec Program Files This section describes the C program files which are provided in the directory so3 simul fixed in the companion software All of the files needed to compile run and verify the fixed point codec are located in the directory so3 simul fixed 3 1 4 2 Compiling the Fixed Point Codec Simulation The source code for the fixed point codec simulation has been written in ANSI C and can be compiled using any general purpose compiler such as the GNU GCC C compiler and make utility Refer to Section 3 3 for information regarding obtaining GCC make and relevant documentation Two GCC compatible makefiles have been included in the so3 simul fixed code so3 simul fixed dspmath directory All of the files contained on the associated Software Distribution under the directory fixed should be copied onto a writable disk making sure to preserve the directory structure Typing make in the dspmath directory first followed by typing make in the directory code will compile and link the code and create the executable file called EvrcFix evrcfix exe on Win32 systems which will be placed in the bin directory The included makefiles may require some user modification for a particular hardware platform and or operating system 3 7 23 24 25 26 27 28 29 30 3

ARIB STD-T64-C.S0018-E v2.0 Minimum Performance Specification

Contents

Download Pdf Manuals

Related Search

Related Contents