Conformance Testing

By Sunil

Topics :

1. What is Conformance testing

2. Lab Setup
3. Tools used 4. Execution and test report 5. Test case

What is Conformance Testing

Conformance Testing is a type of testing which is used to determine whether a product or a system meets some specified standard Service providers, equipment manufacturers, and equipment suppliers rely on this data to ensure Quality of Service (QoS) through this conformance process UE Conformance GSM/GPRS - 3GPP Spec - 51.010 UMTS - 3GPP Spec 34.123

Lap set up

Lap Setup continued

TRX-1

Combiner

TRX-2

Anite Tester PC & Automation Test Server

Automation PC & NewATA Automation Test Client

RF LAN BNC

TRX-3 TRX-4 Target Sample

Anite SAT(4) AT Commands

UART USB

USB Interrupter

Tools SAT Controller PC - Contains Anite test toolsets 1. Development Toolsets Provides the environment to develop the test case in C/C++ or TTCN 2. Conformance Testing tools. Provides the interface to run the pre-written cases Validates the pass and fail criteria 3. Inter-operability Testing Transceiver Simulates the radio air-interface and along with radio frequency cable replaces the air interface.

Combiner This combines the RF signals from multiple transceivers into a single RF output for a terminal under test.
ABP Anite Baseband processor ABP sends and receives digital I and Q data samples over the LVDS interface and provides additional processing capabilities that are required for WCDMA , HSDPA and HSUPA operation For 3G testing , 3G capable Agilent 8960 and ABP is required.

Different Vendor equipments

Anite Agilent 8960: SAT(n) GCF/PTCRB GSM, GPRS/EDGE & 3G test cases are verified N refers to number of cells. RACAL Anite 6103E GCF/PTCRB GSM and GPRS and 3G test cases are verified Aeroflex RACAL AIME 6103G GCF / PTCRB GSM, GPRS/EDGE and UMTS test cases are verified R and S Rohde and Schwarz COMPRION IT3 (for SIM testing) This equipment is used for testing standalone SIM tests and for performing SIM tests involving Anite Network simulator

Tools SAT Controller PC - Contains Anite test toolsets 1. Development Toolsets Provides the environment to develop the test case in C/C++ or TTCN 2. Conformance Testing tools. Provides the interface to run the pre-written cases Validates the pass and fail criteria 3. Inter-operability Testing Transceiver Simulates the radio air-interface and along with radio frequency cable replaces the air interface.

Combiner This combines the RF signals from multiple transceivers into a single RF output for a terminal under test.
ABP Anite Baseband processor ABP sends and receives digital I and Q data samples over the LVDS interface and provides additional processing capabilities that are required for WCDMA , HSDPA and HSUPA operation For 3G testing , 3G capable Agilent 8960 and ABP is required.

Tools Test manager This GUI which is used add, remove and execute the test cases from the test suite and also provides the result. Logs All the below log files will be under the Anite PC 1. <Test_case_ID_randomnumber>.lfa This contains the message which are exchanged bewteen the UE and UTRAN over the air interface. 2. <Test_case_ID_randomnumber>.prm This contains the parameters the air interface 3 .<Test_case_ID_randomnumber>.vdc- This contains the verdict of the test case Mobile Logs - These are UE specify and Proprietary Ex Mobile analyzer , PCO ( point of control and abservation)

Execution , Bug Reporting and Test Report Execution Manual 1. Select the test in the Test manager 2. Configure the parameter in pics file which are related to particular SIM ex IMSI. 3. Execute the test case 4. Wait for the result Automation Set up the environment 1. Prepare the batch file which contains the test cases which needs to executed under automation 2. Once the test suite execution is completed , collect the results and prepare the test report

Execution , Bug Reporting and Test Report

Release Summary Sheet
System Test:
Release Name Type of Testing Platform Tested Sample Coverage Planned Release Start Date End Date x.yz Protocol Conformance Testing TP 2 Anite SAT(R) xxx GCF 3.29.0/PTCRB 3.14.0 rd 3 Sept 2007 th 10 Sept 2007 th 25 Sept 2007

Test Results:
PTCRB GSM AMR RLP SS SIM GPRS EDGE Total Percentage GCF GSM AMR RLP SS SIM GPRS EDGE Total Percentage Planned 248 23 5 41 95 309 130 851 Planned 240 23 5 86 102 328 130 914 Executed 240 23 5 86 102 328 130 914 Pass 226 23 3 84 93 311 130 870 95.18% Pass Executed 248 23 5 41 95 309 130 851 Pass 240 23 3 41 86 295 130 818 Fail 14 0 2 2 9 17 0 44 4.8 % FAIL Fail 8 0 2 0 9 14 0 33 Blocked 0 0 0 0 0 0 0 0 TC ISSUE 0 0 0 0 0 0 0 0 Blocked 0 0 0 0 0 0 0 0 TC ISSUE 0 0 0 0 0 0 0 0

96.12% Pass
Test Case
34.2.9.2

3.87% FAIL
Description CP-ACK is not sent in 5 seconds after CPDATA message is sent. MS trying for PDP initiation during T3380. MS is trying for routing area update on a wrong cell. Reset indication during acknowledged mode: MS not releasing TBF during data transfer.

SPR:
Sr. No 1 2 3 4 SPR ID PR-00146003 PR-00146006 PR-00146008 PR-00146957 Band 900 1900 1900 900

45.2.4.2.1 44.2.3.2.3.2 46.2.2.1.5

Observation: if any
The automation was not very helpful in this release. Most test cases were executed manually.

Typical Analysis Report

Hardware : xxxxxx ver2.0 Software : 2007 y.z Test case : 42.1.1.4.2 Band : 1900 & 850 Reproducible : 100% Traces attached : Yes Test Platform : Anite SAT(A) 4 Tester S/W version : GSM V25, GPRS V37 Test Purpose: To verify that for each attempt, the mobile station shall draw a random value R with uniform probability distribution. Expected sequence 1 PACKET SYSTEM INFORMATION TYPE 1 is updated with the P(I) values under test 2 SS waits 35 seconds to make sure MS has acquired the new P(I) values 3 SS -> MS PACKET PAGING REQUEST Sent on PPCH. 4 MS -> SS PACKET CHANNEL REQUEST 5 MS -> SS PACKET CHANNEL REQUEST M+3 SS SS waits for expiry of T3170 M+4 SS SS waits to allow Cell Reselection Behavior: Fails at step 4 MS is GPRS attached. The SS sends PACKET PAGING REQUEST message. The MS shall send between 0 and M+1 PACKET CHANNEL REQUEST message indicating page response. The SS verifies that the MS draw a random value R for each attach. Between two consecutive executions (for k and k+1), the SS must wait for amount of time which is enough to guarantee that the MS is in service (listening to its paging sub channel) i.e., SS waits for expiry of T3170,and waits to allow Cell Reselection but MS is sending packet channel request with in this time making test case to fail.

Case Study Test case 8.3.1.4 Cell Update: periodical cell update in CELL_PC 8.3.1.4.3 Test purpose 1. To confirm that the UE, in CELL_PCH state, executes a cell update procedure after the expiry of timer T305. 8.3.1.4.4 Method of test Initial Condition System Simulator: 2 cells Cell 1 and 2 are active. UE: CELL_PCH (state 6-12) in cell 1 as specified in clause 7.4 of TS 34.108. Test Procedure Table 8.3.1.4 Parameter Unit Cell 1 Cell 2 T0 T1 T0 T1 UTRA RF Channel Number Ch. 1 Ch. 1 CPICH Ec (FDD) dBm/3.84MHz -60 -69 -69 -60 P-CCPCH RSCP (TDD) dBm -60 -69 -69 -60

Case Study Test case 8.3.1.4 Cell Update: periodical cell update in CELL_PC 8.3.1.4.3 Test purpose 1. To confirm that the UE, in CELL_PCH state, executes a cell update procedure after the expiry of timer T305. 8.3.1.4.4 Method of test Initial Condition System Simulator: 2 cells Cell 1 and 2 are active. UE: CELL_PCH (state 6-12) in cell 1 as specified in clause 7.4 of TS 34.108. Test Procedure Table 8.3.1.4 Parameter Unit Cell 1 Cell 2 T0 T1 T0 T1 UTRA RF Channel Number Ch. 1 Ch. 1 CPICH Ec (FDD) dBm/3.84MHz -60 -69 -69 -60 P-CCPCH RSCP (TDD) dBm -60 -69 -69 -60

Test case 8.3.1.4 Run1.tc_8_3_1_4_090226_023239 Test fails at step 11 as per spec 34-123.
UE does the cell update with cause as cell reselection on cell2 having the SC=150 at SFN 959 ( line no : 103011 from MA logs). Ss sends the cell update confirm at SFN 978 ( at line no: 103067 from MA logs). Then UE enters into cell_pch state at SFN 215( at line no: 103091 from MA logs) with message URRCDC_STATE_IND. As expected UE does not send any cell update for 60 minutes. Then SS modifies the downlink transmission power of cell1 PS=100 to -60db and cell2 PS=150 to -69db. At SFN 615( at line no: 298047 from MA logs), RRC evaluate URRCM_RESEL_TRACE_EVAL and find the better cell with PC=100 with RSCP -58dbm. RRC sends reselection request URRCBP_CELL_RESEL_REQ on Sc=100 at SFN 615( 298050 from MA logs).L1U sends request to read MIB on SC=100 CPHY_READ_MIB_REQ to L1 at SFN 615( line no : 298052 from MA logs) At SFN 623(at line no:298057 from MA logs), RRC receives the message as CPHY_NO_BCH_IND . Hence cell reselection fails with cause as SI_READ_FAIL_Rej(3) at SFN 623(at line no:298060 from MA logs). L1U never forward this request to L1. But immediately responds with NO BCH Ind. Again L1U sends request to read MIB on SC=150 at SFN 624( 298059 from MA logs). For this request, RRC gets BCCH_DATA_IND at SFN 626 ( at line no 298063 from MA logs) . Hence remains on SC=150.

Case Study Test case 8.3.1.4

Run 2. tc_8_3_1_4_090226_043339 Initially UE does the cell update with cause as periodic cell update at SFN 537 at line no:103156. SS responds with Cell update confirm message. In the cell update confirm message , SS includes New C_RNTI value. Hence UE responds with UTRAN mobility information confirm with message URLC_RRC_AM_DATA_REQ At SFN 601( At line no : 103223) . SS responds with UTRAN mobility information with message URLC_RRC_AM_DATA_IND At SFN 622( at line no: 103271) and sets timer values. At SFN 623, UE sends UTRAN mobility information confirm message ( at 103285). After this SS modifies the downlink transmission power, such that cell 1=-69 db and cell2 =-60 db.

From L1 logs. At SFN 734 ( line no : 7710) L1 sends intra measurement report to L1U at SFN 734 ( line no : 7710) at SFN 739 ( at line no: 103376). L1 finds as the cell size as 2 with Cell Sc=100 of RSCP=60 and cell SC=150 with RSCP =54. But RRC layer never detect the another cell with SC=150. L1 keeps on the sending the intra measurement report at SFN 750( at 7732), 758(at line 7745), 766(at line 7752) and so on. But RRC layer never detect the another cell with SC=150. Hence RRC does not trigger the cell reselection to SC=150.

Hence test case fails.

Case Study Test case 8.3.1.4 Cell Update: periodical cell update in CELL_PC

Case Study Test case 8.3.1.4 Cell Update: periodical cell update in CELL_PC 8.3.1.4.3 Test purpose 1. To confirm that the UE, in CELL_PCH state, executes a cell update procedure after the expiry of timer T305. 8.3.1.4.4 Method of test Initial Condition System Simulator: 2 cells Cell 1 and 2 are active. UE: CELL_PCH (state 6-12) in cell 1 as specified in clause 7.4 of TS 34.108. Test Procedure Table 8.3.1.4 Parameter Unit Cell 1 Cell 2 T0 T1 T0 T1 UTRA RF Channel Number Ch. 1 Ch. 1 CPICH Ec (FDD) dBm/3.84MHz -60 -69 -69 -60 P-CCPCH RSCP (TDD) dBm -60 -69 -69 -60

Q and A

Thanks