ETSI Workshop on QoS / QoE / User experience focusing on speech / multimedia conference tools 21-22 September, Sophia Antipolis,

France

ETSI Workshop on QoS / QoE / User experience focusing on speech / multimedia conference tools
21-22 September 2010 Sophia Antipolis, France

Workshop Report

ETSI Workshop on QoS / QoE / User experience focusing on speech / multimedia conference tools 21-22 September, Sophia Antipolis, France

Overview
The ETSI Workshop on QoS / QoE / User experience focusing on speech / multimedia conference tools, organised and hosted by ETSI, took place on 21-22 September 2010 in Sophia Country Club in Sophia Antipolis, France. It counted around 55 participants with a vested interest in Multimedia QoS / QoE. The agenda included nine sessions and a feedback session, with presentations given by experts representing several organisations from around the world. The workshop provided interesting information on all topics covered, an excellent opportunity for professional networking, co-operation opportunities, and directions, recommendations and potential inputs for future standardisation work. It also revealed that there were many different definitions of QoE
o

Note : QoS means Quality of Service and QoE means Quality of Experience.

ETSI Workshop on QoS / QoE / User experience focusing on speech / multimedia conference tools 21-22 September, Sophia Antipolis, France

Session 1: Opening General Views .............................................................................. 4 Session 2: QoS / QoE Standardization, Guidelines and Definitions .......................... 6 Session 3: QOE STANDARDIZATION, GUIDELINES, DEFINITIONS................ 7 Session 4: USE OF CONFERENCING ...................................................................... 9 Session 5: AUDIO CONFERENCING TOOLS (INCLUDING QUALITY ASSESSMENT) .......................................................................................................... 12 Session 6: AUDIO CONFERENCING TOOLS (INCLUDING QUALITY ASSESSMENT) .......................................................................................................... 13 Session 7: VIDEO CONFERENCING TOOLS ........................................................ 14 Session 7: VIDEO CONFERENCING TOOLS ........................................................ 15 Session 8: BILLING VERIFICATION (STF 375) ................................................... 17 Session 9: CUSTOMER RELATIONSHIP STAGES QOS (STF 374) .................... 18 FEEDBACK FROM SESSION CHAIRS AND CONCLUSIONS ........................... 20

ETSI Workshop on QoS / QoE / User experience focusing on speech / multimedia conference tools 21-22 September, Sophia Antipolis, France

Session 1: Opening General Views

Moderator: Jean-Yves Monfort, ETSI STQ Chairman Welcome Jean-Yves Monfort, ETSI STQ Chairman Jean-Yves welcomed the participants to this workshop in Sophia Antipolis and he introduced the purpose, the expected outcome of QoE/QoE workshop and the wish that the different actors exchange on their QoS/QoE approaches to enhance further collaborations. He emphasized the importance of on-going work related to develop the electrical working tools under ETSI green agenda and introduced the Special Report(Roadmap for electronic working tools) under development. He mentioned that it is right time to have this workshop to realize this goal. Practical arrangements Nathalie Guinet Nathalie welcomed the participants to this workshop, in Sophia Country Club. She provided practical arrangements for the work during the workshop. ETSI introduction Dong Hi Sim, ETSI Technical Officer Dong Hi Sim provided a brief overview with regards to who ETSI is and what ETSI is doing. ETSI is a European Standards Organization setting globally-applicable standards for Information and Communication Technologies (ICT). ETSI is an independent, not-for-profit, organisation recognized by one of European Standards Organizations. At present ETSI counts over 700 Members in 60 countries across 5 continents. He introduced ETSI services such as standardization enabling services, interoperability testing, forum hosting etc. Dong Hi briefly outlined the cluster concept has been introduced this year. ETSI has introduced the cluster concept to provide a simplified, yet comprehensive, introduction to its activities in ICT standardization. This new approach is aimed at enabling the audience to easily grasp ETSIs endeavours in a variety of domains, all contributing to the overall vision of a connected world. He also introduced Better living with ICT cluster which is directly related to the workshop. STQ introduction - Jean-Yves Monfort, ETSI STQ Chairman Jean-Yves Monfort provided a brief introduction about the ETSI TC STQ. He presented the structure and the officials. He explained that TC STQ is standardizing relating to terminals and networks for speech and media quality, end-to-end single media and multimedia transmission performance, Quality of Service (QoS) parameters for networks and services. Jean-Yves Monfort highlighted the main ETSI TC STQ activities as follows: - Speech Terminal Quality - User related QoS - Multimedia Audiovisual QoS - Metrics and Measurement Methods

ETSI Workshop on QoS / QoE / User experience focusing on speech / multimedia conference tools 21-22 September, Sophia Antipolis, France NGN QoS and Interoperability QoS Indicator

He also introduced the speech terminals standards published by TC STQ and results obtained until now, related to VoIP (including wideband speech terminals). Ongoing work covers superwideband / fullband speech quality. These standards also apply to group terminals and conference systems. Jean-Yves Monfort pointed out that the standards are regularly reviewed and updated, and new ones are created according to the directions of the ETSI Membership. Finally he presented the web links for published documents and for accessing to TC STQ. STQ Mobile Working Group introduction - Klemens P.F. Adler, ETSI STQ Mobile WG Chair Klemens P.F. Adler presented a brief introduction of TC STQ Mobile WG. The STQ Mobile WG bridges the gap between the reality and its simulation by providing a Technical Specification (TS) on QoS aspects for mobile networks. It covers more than voice communications, looking also at multi-media services such as web browsing and video. Much work has been done on laying down specifications for measuring robustness of services and possible errors to be encountered on all service types. The benchmarking parameters and the guidance given by the series are addressing both the users point of view and the supplier side, so that the operators can plan improvements to their services and network.

ETSI Workshop on QoS / QoE / User experience focusing on speech / multimedia conference tools 21-22 September, Sophia Antipolis, France

Session 2: QoS / QoE Standardization, Guidelines and Definitions

Moderator: Philippe Coude, SFR Bridging QoE and QoS for Mobile Broadband Networks - David Soldani, VP Huawei European Research Centre David Soldani introduced the QoE definitions and industry trend. He emphasized the challenges which now operators are facing on QoE as follows: - Very little work available on end user perception of the services, especially from Internet, that are being offered - Gaps between network measured quality and user perceived experience - Mechanisms, procedures and tools to continuously monitor, operate and report QoE indicators, for strategic company decisions, network optimisation, supervision and operation. He stated that following items could be standardized - QoE as a combination of QoS parameters (e.g. ETSI TS 102 250-2) and truth functions (fuzzy logic) - Call Detail Records (CDR) for service data flows and relevant interfaces / elements, which shall support those - Requirements for embedded probes and terminal agents for multi-vendor support It was commented that only broadband aspects are considered. And the relation between QoS and QoE is an interesting topic but needs to be developed further with interactions among other ETSI works.

QoE Aspects of NGN Services - Himzo Bajri, BH Telecom Himzo Bajri started his presentation to describe a conceptual overview of QoE aspects of NGN services. He also address the drivers and benefits of QoE. He presented two definitions of QoE which are available in ITU-T P.10/G. 100 / ITU-T E.100 and ETSI TR 102 643. Further, he explained 5 QoE dimensions: technology performance, usability, subjective evaluation, expectation and context. As a conclusion he reminded that the conceptual overview of QoE aspects of NGN services are provided within his presentation. Referring to the fact that QoE is subjective matter, not only technology-centric, but also user-centric aspects that may impact the quality of users experience when using any NGN service are covered. Approaching these QoE aspects from a wide interdisciplinary perspective, his presentation provides better understanding of user requirements and expectations needed for QoE management.

ETSI Workshop on QoS / QoE / User experience focusing on speech / multimedia conference tools 21-22 September, Sophia Antipolis, France

Session 3: QOE STANDARDIZATION, GUIDELINES, DEFINITIONS

Moderator: Mike Pluke, TC HF Acting Chair, Castle Consulting Ltd Increasing the strength of QoE data with objective and quantified subjective measures - Peter Brooks, Teolys, Psychological Research eProjects Peter Brooks started his presentation to define who users are. He outlined that he paid attention to the users with limited attention (in detail, restricted sensory and working memory, automatically selective, perceptual blindness and easily disturbed) and the users who are unconscious about social responses. He outlined the existing approaches to measure quality from a user perspective, i.e., user-perceived QoS, subjective QoE and modelling. He explained the objective measures of QoE which are not dependent on the test participants but are dependent on the actual usage and types of users, tasks and communication situations. He emphasized that the opinion scales should be based on quantitative scales not qualitative scales. In conclusion, he emphasized following points: QoE measures with higher validity and better communicability Expand from user perception to user experience Direct measures of users Objective Quantitative subjective Widespread misunderstandings for the nature of user measures? He challenged the current definition of QoE which was defined in ITU-T P.10/G.100 The overall acceptability of an application or service, as perceived subjectively by the end user is misleading. And he would like to define a parameter combining subjective QoE with objective data he defined as objective QoE. QoE as a bridge between technologists and marketing people - Bjorn Hestnes, Senior Research Scientist, Telenor Corporate Development Bjorn Hestnes started his presentation to explain the context of QoE and the definition of it. And he outlined the QoE data structure would be defined as follows:

He pointed out that QoE has a larger audience than QoS and QoE concept can be used even for management side of companies such as CEO, CTO etc.

ETSI Workshop on QoS / QoE / User experience focusing on speech / multimedia conference tools 21-22 September, Sophia Antipolis, France He demonstrated the tutorial on user experience guidelines for real-time services developed by ETSI STF 354. He concluded his presentation to emphasize following facts: - The marketing people need measures to detect services that will be rejected or not used. - The salesmen need arguments why their services create good user experience. - Technologists need a relation between their QoS and how good this is for the users. - The mindset of QoE should be reflected in all standards, including ETSI.

E2E QoS Converged Service Delivery - Philippe Coude, SFR Philippe Coude outlined the End-to-End QoS in NGN for converged service delivery considering mobility, heterogeneous network environments and user. He mentioned that he took top-down approach with service composition (SoA) and user preferences. He showed one specific example of session continuity in terms of QoS E2E dynamic management as follows: What : End to End QoS in NGN Context (mobility, heterogeneous, User Centric) Why : Because each level contributes at this QoS Equipment : Computing delivery (CPU and memory) Network : Media delivery Service (components and platform) : Service delivery How : With top down approach, service composition (SoA) and user preferences After this presentation, a more general discussion started; Jean-Yves pointed out that he feels that it is good idea to combine the subjective data and objective data to measure QoE perceived by the users, but that it was needed to clarify the what was defined by the objective data. Peter was on the floor to present some slides he already presented in detail how to model or measure QoE with objective parameters. However a comment was raised from the floor to outline two points to challenge the measurement of subjective data. The first one is the user experience keeps changing so it would be difficult to measure the subjective data properly. Second point it that we might do field study based on real user experience, however, to quantify the results from field study still, we still might need to come back to lab study or measurements. In turn, looking at the lab test results to compare those to the results from field tests, we might be able to identify what is the issue or problem in the real field. A question also was raised where this approach is also applicable for non-real time communication. Peter mentioned that 3 work items in TC HF were approved to work on the extension of the mapping of QoS to QoE in the real time communication to non-real time applications.

ETSI Workshop on QoS / QoE / User experience focusing on speech / multimedia conference tools 21-22 September, Sophia Antipolis, France Lighting up the quality of experience initiative - Antonio Cuadra Sanchez, TMF, Telefonica Antonio Cuadra Sanchez introduced tmforum which was organized to settle a methodology across processes and value chainin by modeling the service and underlying attributes According to standard, recommendations and forums. He outlined that in order to lead in todays communications services market, operators and SPs must offer customers the best quality and do so profitably. Managing Customer Experience (MCE) is defining best practices for improving customer preference and loyalty for business with a Service Provider across its brand, products, quality (QoE), and touch point interactions. MCE solutions ultimately improve business effectiveness by growing market share, reducing churn, and operational costs where appropriate.Areas of Interest of MCE are metrics, probes, service quality & performance management and customer care & support. They are about to complete final drafts of MCE Phase 2 deliverables including: - Customer/CE (Customer Experience) lifecycle model & Touch Points details (interactions) - Consolidated CE Metrics - MCE Applications Architecture, including Harmony V Catalyst outputs - Probe Technologies with TM Forum Solution Frameworks He concluded his presentation to identify the goals for next steps. There was a comment that the approach for probes might be beneficial, but, it is more important how to analyze the probes which were gathered from the real network.

Session 4: USE OF CONFERENCING

Moderator: Bjorn Hestnes, Senior Research Scientist, Telenor Corporate Development Conferencing tools in ETSI - Pierre-Alain Cerdan, ETSI, ITC Vice President Pierre-Alain outlined his presentation, which provided an overview of conferencing tools in ETSI which were implemented from 2007: web conference, audio conference and video conference: - A web conference is where a presenter can deliver a presentation over the web to a group of geographically dispersed participants. - An audio conference is a telephone call in which identified participants are usually able to call into the conference by dialling into a special telephone number that connects to a bridge. - A video conference is where cameras are used to enable participants in a virtual conference to see the presenter and, sometimes, the other participants He pointed out that there are some feedbacks for ETSI electrical meeting tools as follows: (1) If there is a need to download software upgrade, it is advised to communicate this well in advance (2) Originally it was advised that no firewall settings to change, however, as there are different security policies among companies, it is very difficult to meet all needs if there are very strict security policies in certain companies.

ETSI Workshop on QoS / QoE / User experience focusing on speech / multimedia conference tools 21-22 September, Sophia Antipolis, France (3) There are some comments on audio quality (which may be poor and affects significantly the global quality) (4) It was advised to have toll free or local numbers for every country, however, it is very difficult to have local numbers for Asian countries as it is dependent on the contract the service provider of electrical meeting tools. He anticipated user expectations from 2011 as follows: - Windows/Mac/Linux platform compatibility As there are requests from certain ETSI members for Linux platform compatibility, ETSI IT department is working on this - Increased firewall independency - More flexible virtual meeting scheduling Pricing model was quite competitive when ETSI has adopted a certain tool for electrical meeting. There are limited numbers of licences for electrical meeting. Until now, there is no way for chairman to open electrical meeting, however, it will be flexible to allow that in the near future. - Call-back feature - Improved VoIP quality - Independency from Web Conferencing - Extended worldwide coverage (local numbers) - VIDEO CONFERENCE tool with advanced features He addressed that ETSI has been requested to select the tool which should not be based on proprietary solution which means it should be based on standard and based on open source. However, he emphasized that there is NO way to select such kind of tool in real IT world. He concluded his presentation to point out that ETSI Board strategic team is working on the recommendations how to set-up the electrical working tool. The STEEL Project: a Technological Integrated Platform for E-learning Michele Cinotti, Research Engineer, CNIT Nicolas Ramin started his presentation by explaining who CNIT is. The CNIT is a national consortium for telecommunications composed by 37 universities over the Italian territory specialized in advanced education and research activities. The CNIT was constituted on January 10th 1995 and it was legally recognized by the Italian Ministry of University and Research on March 4th 1997. And he introduced the STEEL project which is a research project that deals with the study of an innovative integrated technology platform satellite-terrestrial on which build and test a complete distance learning system, considering both synchronous and asynchronous session mode. Multimedia Conferencing for an Ambulatory Health Monitoring, Diagnosis and Treatment: Systematic Review of QoS requirements and users QoE Katarzina Wac, University of Geneva Katarzina started her presentation by showing current trend in healthcare domain. She outlined the need of use of multimedia conference scenarios for healthcare as follows: - timely diagnosis from remote specialist - under-staffed providers to timely diagnosis from remote specialist

10

ETSI Workshop on QoS / QoE / User experience focusing on speech / multimedia conference tools 21-22 September, Sophia Antipolis, France real-time treatment e.g. decision to deliver the life-saving, clot-busting drug known as tPA within 3 hours of stroke onset [Knox, 2009] - pandemic situations assist more patients, avoid them to use public areas She pointed that there is lack of standards, particularly for service provision for healthcare to combine all available specific specifications for certain areas for healthcare to realize the service. Also she introduced technical requirements in terms of QoS to realize healthcare service during her presentation. Mike Pluke commented that TC HF published one ES related to Personalization of eHealth systems by using eHealth user profiles which can answer the questions on security, privacy, role-based authentication and authorization. -

11

ETSI Workshop on QoS / QoE / User experience focusing on speech / multimedia conference tools 21-22 September, Sophia Antipolis, France

Session 5: AUDIO CONFERENCING TOOLS (INCLUDING QUALITY ASSESSMENT)

Moderator: Catherine Quinquis, France Telecom New ITU-T SG12 question on conferencing and telemeeting quality assessment Gunilla Berndtsson, Ericsson Gunilla started her presentation on the motivation for the new question. There is an increasing demand for effective telemeeting (teleconference) systems. If the perceived quality is good enough, telemeetings can be a complement to and to some extent replace face-to-face meetings. Yet there exist no standardized methods for evaluating the subjective quality of telemeeting systems. She introduced ITU-T SG12 which is lead study group on Quality of Service (QoS) and Quality of Experience (QoE) of Speech, Audio, Video and Multimedia. She summarized the list of open questions on asymmetric connections and multiparty interaction. She pointed out that there are some standardized subjective test methods for several components in a telemeeting, such as: - Speech and audio quality (codec, bandwidth, bitrate, background noise) - Video quality (codec, screens size, resolution, frame rate) - Synchronization between audio and video - Transmission impairments However, she questioned that their relative impact on the overall subjective quality of a telemeeting needs to be evaluated. She summarized numerous activities from various standardization organisations. She concluded her presentation by welcoming contributions to ITU-T SG12/Q18 for the next ITU-T SG12 meeting in Geneva, 18-27 January 2011. There was a question on whether ITU-T is considering the acoustics problems such as echos from all different terminals etc. She explained that these acoustics issues have been considered and will be included into the recommendation in future. Conference phones for every situation John-Erik Eriksson, Konftel John-Eriksson started his presentation who Konftel is and outlined the wireless trends within the SIP and VoIP domain. He outlined voice conferencing challenges. New user situations drive need for products with higher degrees of customizability. In addition to this technology trends introduces variation in speech quality, variations in speech bandwidth, up to full band, lower degrees of service stability and a high degree of interoperability challenges for terminal providers. Managing voice quality in audio conferencing will become increasingly important because of more unpredictable voice conferencing situations.He concluded his presentation by comparing three wideband voice codecs, i.e., wideband, super wideband and full band. He emphasized that a major parameter is the intelligibility and that it is needed to minimize the room influence and to avoid loudness variations.

A Novel Impairment Framework for Speech Quality Assessment under Frequency Bandwidth Changeover - Sofiene Jelassi, INRIA 12

ETSI Workshop on QoS / QoE / User experience focusing on speech / multimedia conference tools 21-22 September, Sophia Antipolis, France Sofiene started his presentation to compare the characteristics of traditional QoE of VoIP to those of VoIP transport systems with heterogeneity, user mobility and advanced special signal processing. He introduced the harmonization quality-affecting factors with CODEC and frequency bandwidth changeover and considerable alteration of temporal structure of wave signals. He explained the assessment methodology and framework to analyze QoE of VoIP services. He concluded his presentation with lessons learned and some future work items to be done. There are some questions how the test results are generated and some doubts on the test results (as they were obtained by PESQ which is not intended to be used for such applications) on the effect of a 1-iteration of Up and Down sampling technology on MOS-LQOw.

Session 6: AUDIO CONFERENCING TOOLS (INCLUDING QUALITY ASSESSMENT)

Moderator: Hans W. Gierlich, HEAD acoustics, Head of Telecom Division Pre-study for subjective evaluation of spatial audio conferencing systems Catherine Quinquis, Orange Labs, TECH/OPERA Catherine started her presentation to introduce the spatial audio conference system focusing on sound spatialization based on Ambisonics, hand-free and multi-listener conferencing. She outlined some questions investigated. She explained three cases for testing such as external circular array, frontal linear array and internal circular array. Later on, she showed the results obtained through proposed methodology. She concluded her presentation by stating that from a methodological point of view people are able to judge spatial audio technologies through proposed attributes such as naturalness, comfort, intelligibility, immersion, pleasure and finally global quality. But she has some open questions. She stated that when people talk about spatialization, the same attributes spontaneously came out for example not only comfort, pleasure, intelligibility but also attributes related to timbre, localisation, level, proximity. She assumed global quality seems more related to hedonistic notions, i.e., comfort, pleasure. She also addressed that, from spatial audio technologies point of view, for teleconferencing and videoconferencing, the linear /external arrays emerge as the recommended configurations and the internal array requires further optimization to enhance the sound rendering. There was a question whether these results can be re-used for car audio system. Catherine stated that it could be but she was not sure as she tested listening only situation. Also there was a question whether objective measures were considered for her research. Catherine replied that the objective measures were not considered but she feels that it is needed to define objective measure data set first to do so.

13

ETSI Workshop on QoS / QoE / User experience focusing on speech / multimedia conference tools 21-22 September, Sophia Antipolis, France Optimizing a 3D Audio Teleconference Application - Mansoor Hyder, University of Tubingen Mansoor started his presentation to address the drawbacks of classic teleconferencing. He demonstrated that mixed gender talkers were found easier to localize. He also pointed out that increase of table size brought increased localization and increased number of talkers leads to increased localization. In addition, he mentioned that decrease in room size results in increase of localization performance. He concluded his presentation with some future work items.

Interconnected systems - Hans W. Gierlich, HEAD acoustics, Head of Telecom Division Hans started his presentation to introduce the definition of QoE, the relation between QoS and QoE and the parameters influencing QoS. He outlined the communication systems and interconnection. And he summarized the impacts of delay, coding, noise/noise cancellation, insufficient echo loss/improper echo cancellation, cascading of signal processing and missing signalling information between endpoints and network elements. He demonstrated some audio samples in various situations for example the car audio with hand-free, the echo cancellation effect etc. He stated that delay aggregation is the biggest problem in modern network configurations. Potential problems in a connection may occur due to tandemed signal processing in terminals and networks. Network planning and quality evaluation needs to involve all components of a connection and their interaction. Currently no signalling is possible between terminals and networks for information exchange about signal processing active in the different devices. However he gave two references related to on-going work on this.

14

ETSI Workshop on QoS / QoE / User experience focusing on speech / multimedia conference tools 21-22 September, Sophia Antipolis, France

Session 7: VIDEO CONFERENCING TOOLS

Moderator: Jean-Yves Monfort, ETSI TC STQ Chairman, France Telecom Immersive Telepresence and how user experience becomes a key part of the overall picture - Patrick Luthi, Tandberg Patrick started his presentation with configuration of telepresence systems. He introduced Tanberg solution called Immersive for 3 large screens. He outlined telepresence standardization status as: Lack of telepresence interoperability Different and incompatible media codecs Different signaling mechanisms How to deal with multiple video streams What about multiple multichannel audio streams He also outlined the work on-going in IETF, ITU-T and IMTC (International Multimedia Telecommunication Consortium). IETF has a new work item in Dispatch WG for multiple streams control mechanism to specify the information about media streams from one endpoint to another endpoint. ITU-T has a new question 5/16 Telepresence Systems for full interworking between telepresence systems. He stated that there are some future works to be done, for example, multiple media streams, collaboration and interaction, user interfaces, video quality beyond HD and personal and mobile telepresence. He also introduced the future video codecs such as H.265. He concluded his presentation to state that telepresence is changing the way people communicate, but this is still not a very big market. He emphasized that without interoperability, telepresence will fail. He stated that we need even more efficient video codecs as telepresence studios of the future will be even more bandwidth hungry than today. The design of the room and furniture, the eye-to-eye connection, the clarity of video and the crisp audio are integral parts of the user experience are important for future telepresence system.

Enabling seamless videoconferencing the PERIMETER approach - Markus Fiedler, BTH Markus started his presentation to introduce PERIMETER project. PERIMETERs main objective is to establish a new paradigm for user-centricity in advanced networking architectures. In contrast with network-centric approaches, user-centric strategies could achieve seamless mobility driven by actual user needs rather than simply business considerations. Putting the users at the centre rather than the operator enables them to finely control the way their identity, preferences and credentials are sued. Furthermore seamless mobility is streamlined according to user preferences, enabling mobile users to be Always Best Connected (ABC) in multiple-access multiple-operator networks of the Future Internet. He introduced the situation the user is always not best connected due to out of coverage, overloaded network and suboptimal network. He stated that bad QoE increases churn risk. He also addressed that the network selection usually is done by automatic selection, however, it is advised that it should be based on the real evidence such as network measurements.

15

ETSI Workshop on QoS / QoE / User experience focusing on speech / multimedia conference tools 21-22 September, Sophia Antipolis, France He introduced the concept of QoDelivery and QoPresentation also. He summarized the potential standardization items such as QoS-QoE relationship. He concluded his presentation to state that seamless videoconferencing is an opportunity but objective QoE parameters need to be defined and generated. He proposed a specific workshop on QoE in collaboration with ETSI TC HF.

IMTC and interoperability : key to user experience, QoE and QoS, especially in high end and telepresence environments - Louise Olfson, Polycom Louise started her presentation to introduce the IMTC (International Multimedia Telecommunication Corsortium). The IMTC is an industry-leading, non-profit organization whose mission is to promote and facilitate the development and use of interoperable, real-time, multimedia telecommunication products and services based on open international standards. The IMTC hosts interoperability testing events and demonstrations throughout the world. She explained annual interoperability testing event for unified communications, multimedia products and services. SuperOp is a significant event for telecommunications, video conferencing and telepresence sector. It brings together engineers from the leading companies developing unified communications, video communications products and services worldwide. The event includes equipment and service interoperability on a combination of networks, and covers a broad range of technologies.

16

ETSI Workshop on QoS / QoE / User experience focusing on speech / multimedia conference tools 21-22 September, Sophia Antipolis, France

Session 8: BILLING VERIFICATION (STF 375)

Moderator: Pierre-Yves Hebert, AFUTT A new standard for billing verification. An overview of the new specifications + Implementation and results of a trial based on the new - Xavier Lesage, Araxxe SAS Xavier started his presentation with the introduction of ETSI STF 375. He introduced the need of a standard in billing verification of service providers. He addressed the pre-requisites before implementing the checking-up on metering and billing process and billing integrity principles. He also explained the process taken for billing verification. The present standard adopts a process approach for providing evidence measured by an Independent Observer according to a standardized process that a Service Provider complies with the billing integrity principles. He concluded his presentation by explaining some next steps to be done. Conformity Assessment of Billing verification - Armelle Trotin, LSTI Armelle started her presentation to introduce ETSI TS 102 845 which is to provide a high level of trust in billing verification. She explained the main features of TS 102 845, certification scheme, two types of certification and how to perform conformity audit based on TS. She concluded her presentation to state that the next steps.

17

ETSI Workshop on QoS / QoE / User experience focusing on speech / multimedia conference tools 21-22 September, Sophia Antipolis, France

Session 9: CUSTOMER RELATIONSHIP STAGES QOS (STF 374)

Moderator: Pierre-Yves Hebert, AFUTT Assessment of Quality of Service (QoS) of the Customer Relationship Stages other than utilization of the service - Jean-Yves Monfort, STF374 member, JYMC.I.S Jean-Yves started his presentation with introduction of STF374. He addressed the objective of STF374. Getting the maximum benefit in terms of choice, price, and quality, users should have reliable information on the Quality of Service (QoS), including Customer Relationship Stages other than utilization of the service. Until now the most of existing standards for QoS apply to in-use services (speech communication, video transmission, data communication) and define parameters expected to ensure the technical QoS of the service provided. From a customer point of view the overall quality of a Service Provider is based on the quality offered by these "in-use services" but also on all the different stages of the relationship between the service provided and the customer. STF 374 has produced one guide and two specifications to assess all these customer relationship stages. Customer relationship stages QoS - Georgios Karachos, Qualigon GmbH Georgios started his presentation by explaining steps for executing a real QoS assessment campaign as part of the validation related to STF374. Providing the conditions for executing a real assessment campaign, they drafted a questionnaire with selected parameters and provided a web questionnaire. To collect and to interprete results they invited contacts from User associations to participate to the web questionnaire and used a web application to collect responses and provided Excel summaries. He demonstrated the web questionnaire used. He concluded his presentation to address the future steps.

Analysis of the results of the trial and future perspectives - Francois Fischer, FSCOM

18

ETSI Workshop on QoS / QoE / User experience focusing on speech / multimedia conference tools 21-22 September, Sophia Antipolis, France Francois started his presentation with introduction of validation the questionnaire required to proceed with a real campaign executed with a user panel. The main goal of the validation was to check the phrasing of the questions and applicability of the QoS parameters. The campaign was prepared by STF374, based on a subset of the parameters in the ETSI guide EG 202 843 covering preliminary information, contract establishment, complaint Management / billing and parameters whose assessment has been identified to be done by a customer survey. The campaign results showed that the parameters defined in EG 202 843 are well suited to measure the QoE of users concerning the customer relationship stages. The way to prepare and execute an assessment campaign, provided in TS 102 852 was able to complete a successful trial. Results are consistent with User Association views according to the received complaints. Critical issues are still pointed out major differences exist between operators.

19

ETSI Workshop on QoS / QoE / User experience focusing on speech / multimedia conference tools 21-22 September, Sophia Antipolis, France

FEEDBACK FROM CONCLUSIONS

SESSION

CHAIRS

AND

Each moderator for individual session was invited to review the outcome of his/her session. These recommendations and conclusions are provided in two parts, the first for QoS/QoE/User Experience and the second for multimedia conference tools. QoS/QoE/User experience

- Definitions and approaches Quality of Experience is not a universally well understood concept QoE definitions, test & measurement are always items who needs complementary definition and clarification - Relationship between QoE/QoS The relation between QoS & QoE is an interesting point of view to go further and the connection between ETSI works. Need for objective and universal QoE based criteria for audiovisual systems and need for strong correlation between QoE-Qodelivery-QoS - How to better take into account the User view? It has been noted that a user's opinion is not always correlated with observed behaviour Subjective opinion scores will not always mean that users achieve their intended goals (measured by objective QoE measures) There is a claim that objective measures have "higher validity and better communicability" During the review of the conclusion of session 3, there was a comment that we need to careful to use terminology Objective QoE measurement as it might be misleading for non-participants of this workshop. It was clarified that the objective QoE measurement is still QoE measurement but by measurements of pre-set QoE parameters objectively rather than relying on the direct opinions from the users on QoE. The objective measurement of QoE was introduced and accepted by majority of the audience The managing Customer Experience (MCE) programme is intended to define best practices for improving customer preference and loyalty for business with the service provider. It was quoted that the MCE approach to probes is good but it is more important to analyze probes from real networks". - Potential actions Approaching the QoE aspects from a wide interdisciplinary perspective provides better understanding of user requirements and expectations needed for QoE Management. More work is needed to indentify the merits and applicability of the different measures of QoE. Collaborations should be initiated to identify how "MCE" could be linked with the results of the ETSI specifications and guides on billing verification and on QoS of the user relationship stages has defined respectively by STF 375 and STF374.

20

ETSI Workshop on QoS / QoE / User experience focusing on speech / multimedia conference tools 21-22 September, Sophia Antipolis, France Those proposing the benefits of "objective" QoE measures and those using "subjective" QoE need to continue a productive dialogue. Begin collaborative efforts to produce a map that shows how "objective" QoE and "subjective" QoE can be most appropriately used. ETSI Cluster "Better living with ICT" (which includes HF, STQ and User Group) should be the good place to provide further consolidated views on Qos/QoE and user experience. - Multimedia conference tools - Audiovisual systems New dimensions of quality are to be considered such as Variation of bandwidth during a call Synchronisation video/audio There is a need to develop new quality assessment tools Methodologies for subjective assessment of quality to be developed for multimedia conferencing systems For telepresence it was noted the Importance of eye-to-eye contact, room design Need for telepresence interoperability, compatible/efficient media codecs, relevant signalling mechanisms, - Audio systems Need for better coding scheme and loudness with different bandwidths It is also needed to develop new quality assessment tools, in particular for multiple users (in particular to take into account the distances to microphones) - Spatial audio and 3D systems Naturalness, comfort, intelligibility, immersion, pleasure and global quality can be distinguished subjectively, but it is not easy to assign the attributes used by subjects correctly. In 3D audio teleconference systems speaker localisation can be improved and some factors influence the localisation (eg mixed gender talkers, size of the table, number of talkers, room size,..) - Needed actions and improvements To understand the impact of all systems on conversational tasks To define subjective test scenarios for conference systems To investigate the impact of background noise on teleconference systems For interconnected systems several issues should be better solved: Delay aggregation is the biggest problem in modern networks Potential problems in a connection may occur due to tandemed signal processing in terminals and networks Network planning and quality evaluation need to involve all components of a connection and their interaction Signalling needs to be established possible between terminals and networks for information exchange about signal processing active (ongoing actions but it is needed that all the interested bodies implement the mechanisms under development). 21

ETSI Workshop on QoS / QoE / User experience focusing on speech / multimedia conference tools 21-22 September, Sophia Antipolis, France

- Collaborations to be reinforced Several standardization bodies (eg ITU-T, IETF, ETSI) have initiated work on high quality tools and close collaborations are needed: in particular for high quality conference systems: ETSI HF ETSI STQ ITU-T Q. 18/12 ITU-T Q5/16- IMTC IETF. Additionnal information Also there was a remark that in IEEE the objective QoE measurement means somehow QoS. However, in the recent special edition of the IEEE Network on QoE (March/April 2010) results from ETSI STF 354 are described and a definition of QoE provided that combines subjective and objective user measures that is based on ETSI TR 102 643. In addition, how to incorporate this QoE concept to eGovernment and eHealth is another future topic to be discussed.

22