State-of-the-Practice in Evaluation of Quality of Experience in Real-Life Environments

  • Marko Matulin FPZ
  • Štefica Mrvelj FPZ
Keywords: Quality of user Experience, QoE, Quality of Service, QoS, user perspective, quality assessment


Continuous monitoring of achieved level of service quality in packet-switched networks represents an activity of major importance for network and service providers. This is paramount for network resources provisioning which have to satisfy expectations of fickle customers. However, providing enough resources to specific user does not automatically increase their Quality of Experience (QoE), hence understanding of the relationship between these two is crucial in the network management process. Essentially, this requires subjective testing of service quality which is usually done in controlled environments such as laboratories. Nevertheless, the most accurate subjective evaluation of QoE includes real-life experiments in the environments where the services are actually used. Therefore, the aim of this paper is to provide a review of the current state-of-the-practice in evaluating QoE in real-life environments.


FNC: Internet monthly reports, Federal Networking Council, 1995

Mrvelj, Š.: Dynamic capacity allocation in the Internet node according to service requirements, Ph.D. thesis, Faculty of Transport and Traffic Sciences, Zagreb, 2008

Guo, X., Pattinson, C.: Quality of service requirements for multimedia communications, Time and the Web, 1997

Braden, R., Clark, D., Shenker, S.: Integrated Ser-vices in the Internet architecture: an overview, RFC 1633, Internet Engineering Task Force, 1994

Braden, R., Zhang, L., Berson, S., Herzog, S., Jamin, S.: Resource ReSerVation Protocol (RSVP), RFC 2205, Internet Engineering Task Force, 1997

Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., Weiss, W.: An architecture for differentiated service, RFC 2475, Internet Engineering Task Force, 1998

Xipeng, X., Ni, L.: Internet QoS: a big picture, Network, Vol. 13, No. 2, 1999, pp. 8-18

Braun, T., Diaz, M., Enríquez-Gabeiras, J., Staub, T.: End-to-end quality of service over heterogeneous networks, Springer, Berlin, 2008

ITU: End-user multimedia QoS categories, ITU-T Rec. G1010, International Telecommunication Union, 2001

ITU: QoS support for VPN services – Framework and characteristics, ITU-T Rec. Y.1315, International Telecommunication Union, 2006

Riedi, R., V´ehel, J.: Multifractal properties of TCP traffic: a numerical study, Technical report 3129, INRIA Rocquencourt, 1997

Matta, I., Guo, L.: Differentiated predictive fair service for TCP flows, Proceedings of the 8th IEEE International Conference on Network Protocols (ICNP), Osaka, 2000, pp. 49-58

Jacobs, P., Davie, B.: Technical challenges in the delivery of interprovider QoS, Communications Magazine, Vol. 43, No. 6, 2005, pp. 112-118

Cotroneo, D., Iannello, G., Russo, S., Ventre, G.: A real time-based architecture for QoS multimedia provisioning, Microprocessors and Microsystems, Vol. 27, 2003, pp. 55-63

ITU: Network performance objectives for IP-based services, ITU-T Rec. Y.1541, International Telecommunication Union, 2006

Claypool, M.: The effect of latency on user performance in Real-Time Strategy games, Computer Networks, Vol. 49, 2005, pp. 52-70

ITU: Internet protocol data communication service – IP packet transfer and availability performance parameters, ITU-T Rec. Y.1540, International Telecommunication Union, 1999

McKnight, L., Boroumand, J.: Pricing Internet services: after flat rate, Telecommunications Policy, Vol. 24, 2000, pp. 565-590

ISO: Quality management and quality assurance, ISO 8402, International Organization for Standardization, 1994

ISO: Quality management, ISO 9000, International Organization for Standardization, 2000

ITU: Terms and definitions related to quality of service and network performance including dependability, ITU-T Rec. E.800, International Telecommunication Union, 1994

ITU: Information technology - Open distributed processing - Reference model: Foundations, ITU-T Rec. X.902, International Telecommunication Union, 1995

Hardy, W. C.: Measurement and evaluation of telecommunications quality of service, Wiley, Chichester, 2001

ITU: Communications quality of service: A framework and definitions, ITU-T Rec. G.1000, International Telecommunication Union, 2001

ITU: Definitions of terms related to quality of service, ITU-T Rec. E.800, International Telecommunication Union, 2008

Bai, Y., Chu, Y., Ito, M. R.: Dynamic end-to-end QoS support for video over the Internet, International Journal of Electronics and Communications (AEÜ), Vol. 65, 2011, pp. 385-391

DSL Forum: Triple-play services quality of experience (QoE) requirements, DSL Forum WT-126 v.0.5, Digital Subscriber Line Forum, 2006

Kemp, A.: Getting what you paid for: quality of service and wireless connection to the Internet, International Journal of Information Management, Vol. 25, 2005, pp. 107-115

Oodan, A., Ward, K., Savolaine, C., Daneshmand, M., Hoath, P.: Telecommunications quality of service management, The Institution of Engineering and Technology, London, 2009

ITU: General aspects of quality of service and network performance in digital networks, including ISDN, ITU-T Rec. I.350, International Telecommunication Union, 1993

Ghinea, G., Thomas, J. P.: QoS impact on user perception and understanding of multimedia video clips, Proceedings of the ACM Multimedia ‘98, Bristol, 1998, pp. 49-54

Dick, M., Wellnitz, O., Wolf, L.: Analysis of factors affecting players’ performance and perception in multiplayer games, Proceedings of the NetGames 2005, Hawthorne, New York, 2005

Henderson, T., Bhatti, S.: Networked games - a QoS sensitive application for QoS insensitive users?, Proceedings of the ACM SIGCOMM 2003 Workshops, Karlsruhe, Germany, 2003

Ascom: Video streaming quality measurement with VSQI, Techincal paper, Ascom, 2009

ITU: Subjective video quality assessment methods for multimedia applications, ITU-T Rec. P.910, International Telecommunication Union, 2008

Ghinea, G., Angelides, M.: A user perspective of quality of service in m-Commerce, Multimedia Tools and Applications, Vol. 22, 2004, pp. 187-206

Udo, G. J., Bagchi, K. K., Kirs, P. J.: An assessment of customers’ e-service quality perception, satisfaction and intention, International Journal of Information Management, Vol. 30, 2010, pp. 481-492

Quax, P., Monsieurs, P., Lamotte, W., Vleeschauwer, D., Degrande, N.: Objective and subjective evaluation of the influence of small amounts of delay and jitter on a recent first person shooter game, Proceedings of the SIGCOMM 2004 Workshops, Portland, Oregon, 2004

Pantel, L., Wolf, L.: On the impact of delay on real-time multiplayer games, Proceedings of the NOSSDAV 2002, Miami, Florida, 2002

Lopez, D., Gonzalez, F., Bellido, L., Alonso, A.: Adaptive multimedia streaming over IP based on customer-oriented metrics, Proceedings of the 7th International Symposium on Computer Networks, Istanbul, 2006

Soldani, D., Li, M., Cuny, R.: QoS and QoE management in UMTS cellular systems, Wiley, Chichester, 2006

ITU: Vocabularity of terms on telephone transmission quality and telephone sets, ITU-T Rec. P.10, International Telecommunication Union, 1998

ITU: New definitions for inclusion in Recommendation ITU-T P.10/G.10, ITU-T Rec. P.10/G.100, International Telecommunication Union, 2008

ETSI: Review of available material on QoS requirements of Multimedia Services, ETSI TR 102 479 V1.1.1, European Telecommunications Standards Institute, 2006

Joskowicz, J., Carlos López Ardao, J.: A general parametric model for perceptual video quality estimation, Proceedings of the 2010 IEEE International Workshop Technical Committee on Communications Quality and Reliability (CQR), Vancouver, 2010

Theta Networks available at: (accessed February 2011)

Hamam, A., Eid, M., El Saddik, A., Georganas, N.: A fuzzy logic system for evaluating quality of experience of haptic-based applications, Proceedings of the EuroHaptics 2008, Madrid, 2008, pp.129-138

ITU: Reference guide to quality of experience assessment methodologies, ITU-T Rec. G.1011, International Telecommunication Union, 2010

ITU: In-service non-intrusive measurement device – Voice service measurements, ITU-T Rec. P.561, International Telecommunication Union, 2002

ITU: Methods for subjective determination of transmission quality, ITU-T Rec. P.800, International Telecommunication Union, 1996

ITU: Mean Opinion Score (MOS) terminology, ITU-T Rec. P.800.1, International Telecommunication Union, 2006

Staelens, N., Moens, S., Van den Broeck, W., Mariën, I., Vermeulen, B., Lambert, P., Van de Walle, R., Demeester, P.: Assessing Quality of Experience of IPTV and Video on Demand Services in Real-Life Environments, IEEE Transactions On Broadcasting, Vol. 56, No. 4, 2010, pp. 458-466

Jelassi, S., Rubino, G., Melvin, H., Youssef, H., Pujolle, G.: Quality of Experience of VoIP Service: A Survey of Assessment Approaches and Open Issues, IEEE Communications Surveys & Tutorials, Vol. 14, No. 2, 2012, pp. 491-513

ITU: User requirements for objective perceptual video quality measurements in digital cable television, ITU-T Rec. J.143, International Telecommunication Union, 2000

ITU: Objective perceptual video quality measurement techniques for digital cable television in the presence of a full reference, ITU-T Rec. J.144, International Telecommunication Union, 2004

ITU: Objective picture quality measurement method by use of in-service test signals, ITU-T Rec. J.147, International Telecommunication Union, 2002

ITU: Full reference and reduced reference calibration methods for video transmission systems with constant misalignment of spatial and temporal domains with constant gain and offset, ITU-T Rec. J.244, International Telecommunication Union, 2008

Joskowicz, J., Carlos López Ardao, J.: A parametric model for perceptual video quality estimation, Telecommunication Systems, 2010

ITU: Opinion model for video-telephony applications, ITU-T Rec. G.1070, International Telecommunication Union, 2007

ITU: The E-model: a computational model for use in transmission planning, ITU-T Rec. G.107, International Telecommunication Union, 2009

GPP: Technical Specification Group Services and System Aspects; End-to-end multimedia services performance metrics, 3GPP TR 26.944 (Release 9), 3rd Generation Partnership Project, 2009

GPP: Transparent end-to-end Packet-switched Streaming Service (PSS); Protocols and codecs, 3GPP TS 26.234 (Release 10), 3rd Generation Partnership Project, 2011

GPP: Multimedia Broadcast/Multicast Service (MBMS); Protocols and codecs, 3GPP TS 26.346 (Release 10), 3rd Generation Partnership Project, 2011

Jekosch, U.: Voice and speech quality perception - assessment and evaluation, Springer, Berlin, 2005

Piamrat, K., Viho, C., Ksentini, A., Bonnin, J. M.: Quality of Experience Measurements for Video Streaming over Wireless Networks, Proceedings of the Sixth International Conference on Information Technology: New Generations, 2009, pp. 1184-1189

How to Cite
Matulin M, Mrvelj Štefica. State-of-the-Practice in Evaluation of Quality of Experience in Real-Life Environments. PROMET [Internet]. 2013Jun.19 [cited 2019Nov.22];25(3):255-63. Available from: