State-of-the-Practice in Evaluation of Quality of Experience in Real-Life Environments
Abstract
Continuous monitoring of achieved level of service quality in packet-switched networks represents an activity of major importance for network and service providers. This is paramount for network resources provisioning which have to satisfy expectations of fickle customers. However, providing enough resources to specific user does not automatically increase their Quality of Experience (QoE), hence understanding of the relationship between these two is crucial in the network management process. Essentially, this requires subjective testing of service quality which is usually done in controlled environments such as laboratories. Nevertheless, the most accurate subjective evaluation of QoE includes real-life experiments in the environments where the services are actually used. Therefore, the aim of this paper is to provide a review of the current state-of-the-practice in evaluating QoE in real-life environments.References
FNC: Internet monthly reports, Federal Networking Council, 1995
Mrvelj, Š.: Dynamic capacity allocation in the Internet node according to service requirements, Ph.D. thesis, Faculty of Transport and Traffic Sciences, Zagreb, 2008
Guo, X., Pattinson, C.: Quality of service requirements for multimedia communications, Time and the Web, 1997
Braden, R., Clark, D., Shenker, S.: Integrated Ser-vices in the Internet architecture: an overview, RFC 1633, Internet Engineering Task Force, 1994
Braden, R., Zhang, L., Berson, S., Herzog, S., Jamin, S.: Resource ReSerVation Protocol (RSVP), RFC 2205, Internet Engineering Task Force, 1997
Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., Weiss, W.: An architecture for differentiated service, RFC 2475, Internet Engineering Task Force, 1998
Xipeng, X., Ni, L.: Internet QoS: a big picture, Network, Vol. 13, No. 2, 1999, pp. 8-18
Braun, T., Diaz, M., Enríquez-Gabeiras, J., Staub, T.: End-to-end quality of service over heterogeneous networks, Springer, Berlin, 2008
ITU: End-user multimedia QoS categories, ITU-T Rec. G1010, International Telecommunication Union, 2001
ITU: QoS support for VPN services – Framework and characteristics, ITU-T Rec. Y.1315, International Telecommunication Union, 2006
Riedi, R., V´ehel, J.: Multifractal properties of TCP traffic: a numerical study, Technical report 3129, INRIA Rocquencourt, 1997
Matta, I., Guo, L.: Differentiated predictive fair service for TCP flows, Proceedings of the 8th IEEE International Conference on Network Protocols (ICNP), Osaka, 2000, pp. 49-58
Jacobs, P., Davie, B.: Technical challenges in the delivery of interprovider QoS, Communications Magazine, Vol. 43, No. 6, 2005, pp. 112-118
Cotroneo, D., Iannello, G., Russo, S., Ventre, G.: A real time-based architecture for QoS multimedia provisioning, Microprocessors and Microsystems, Vol. 27, 2003, pp. 55-63
ITU: Network performance objectives for IP-based services, ITU-T Rec. Y.1541, International Telecommunication Union, 2006
Claypool, M.: The effect of latency on user performance in Real-Time Strategy games, Computer Networks, Vol. 49, 2005, pp. 52-70
ITU: Internet protocol data communication service – IP packet transfer and availability performance parameters, ITU-T Rec. Y.1540, International Telecommunication Union, 1999
McKnight, L., Boroumand, J.: Pricing Internet services: after flat rate, Telecommunications Policy, Vol. 24, 2000, pp. 565-590
ISO: Quality management and quality assurance, ISO 8402, International Organization for Standardization, 1994
ISO: Quality management, ISO 9000, International Organization for Standardization, 2000
ITU: Terms and definitions related to quality of service and network performance including dependability, ITU-T Rec. E.800, International Telecommunication Union, 1994
ITU: Information technology - Open distributed processing - Reference model: Foundations, ITU-T Rec. X.902, International Telecommunication Union, 1995
Hardy, W. C.: Measurement and evaluation of telecommunications quality of service, Wiley, Chichester, 2001
ITU: Communications quality of service: A framework and definitions, ITU-T Rec. G.1000, International Telecommunication Union, 2001
ITU: Definitions of terms related to quality of service, ITU-T Rec. E.800, International Telecommunication Union, 2008
Bai, Y., Chu, Y., Ito, M. R.: Dynamic end-to-end QoS support for video over the Internet, International Journal of Electronics and Communications (AEÜ), Vol. 65, 2011, pp. 385-391
DSL Forum: Triple-play services quality of experience (QoE) requirements, DSL Forum WT-126 v.0.5, Digital Subscriber Line Forum, 2006
Kemp, A.: Getting what you paid for: quality of service and wireless connection to the Internet, International Journal of Information Management, Vol. 25, 2005, pp. 107-115
Oodan, A., Ward, K., Savolaine, C., Daneshmand, M., Hoath, P.: Telecommunications quality of service management, The Institution of Engineering and Technology, London, 2009
ITU: General aspects of quality of service and network performance in digital networks, including ISDN, ITU-T Rec. I.350, International Telecommunication Union, 1993
Ghinea, G., Thomas, J. P.: QoS impact on user perception and understanding of multimedia video clips, Proceedings of the ACM Multimedia ‘98, Bristol, 1998, pp. 49-54
Dick, M., Wellnitz, O., Wolf, L.: Analysis of factors affecting players’ performance and perception in multiplayer games, Proceedings of the NetGames 2005, Hawthorne, New York, 2005
Henderson, T., Bhatti, S.: Networked games - a QoS sensitive application for QoS insensitive users?, Proceedings of the ACM SIGCOMM 2003 Workshops, Karlsruhe, Germany, 2003
Ascom: Video streaming quality measurement with VSQI, Techincal paper, Ascom, 2009
ITU: Subjective video quality assessment methods for multimedia applications, ITU-T Rec. P.910, International Telecommunication Union, 2008
Ghinea, G., Angelides, M.: A user perspective of quality of service in m-Commerce, Multimedia Tools and Applications, Vol. 22, 2004, pp. 187-206
Udo, G. J., Bagchi, K. K., Kirs, P. J.: An assessment of customers’ e-service quality perception, satisfaction and intention, International Journal of Information Management, Vol. 30, 2010, pp. 481-492
Quax, P., Monsieurs, P., Lamotte, W., Vleeschauwer, D., Degrande, N.: Objective and subjective evaluation of the influence of small amounts of delay and jitter on a recent first person shooter game, Proceedings of the SIGCOMM 2004 Workshops, Portland, Oregon, 2004
Pantel, L., Wolf, L.: On the impact of delay on real-time multiplayer games, Proceedings of the NOSSDAV 2002, Miami, Florida, 2002
Lopez, D., Gonzalez, F., Bellido, L., Alonso, A.: Adaptive multimedia streaming over IP based on customer-oriented metrics, Proceedings of the 7th International Symposium on Computer Networks, Istanbul, 2006
Soldani, D., Li, M., Cuny, R.: QoS and QoE management in UMTS cellular systems, Wiley, Chichester, 2006
ITU: Vocabularity of terms on telephone transmission quality and telephone sets, ITU-T Rec. P.10, International Telecommunication Union, 1998
ITU: New definitions for inclusion in Recommendation ITU-T P.10/G.10, ITU-T Rec. P.10/G.100, International Telecommunication Union, 2008
ETSI: Review of available material on QoS requirements of Multimedia Services, ETSI TR 102 479 V1.1.1, European Telecommunications Standards Institute, 2006
Joskowicz, J., Carlos López Ardao, J.: A general parametric model for perceptual video quality estimation, Proceedings of the 2010 IEEE International Workshop Technical Committee on Communications Quality and Reliability (CQR), Vancouver, 2010
Theta Networks available at: http://www.thetanetworks.com/resources/qoe.html (accessed February 2011)
Hamam, A., Eid, M., El Saddik, A., Georganas, N.: A fuzzy logic system for evaluating quality of experience of haptic-based applications, Proceedings of the EuroHaptics 2008, Madrid, 2008, pp.129-138
ITU: Reference guide to quality of experience assessment methodologies, ITU-T Rec. G.1011, International Telecommunication Union, 2010
ITU: In-service non-intrusive measurement device – Voice service measurements, ITU-T Rec. P.561, International Telecommunication Union, 2002
ITU: Methods for subjective determination of transmission quality, ITU-T Rec. P.800, International Telecommunication Union, 1996
ITU: Mean Opinion Score (MOS) terminology, ITU-T Rec. P.800.1, International Telecommunication Union, 2006
Staelens, N., Moens, S., Van den Broeck, W., Mariën, I., Vermeulen, B., Lambert, P., Van de Walle, R., Demeester, P.: Assessing Quality of Experience of IPTV and Video on Demand Services in Real-Life Environments, IEEE Transactions On Broadcasting, Vol. 56, No. 4, 2010, pp. 458-466
Jelassi, S., Rubino, G., Melvin, H., Youssef, H., Pujolle, G.: Quality of Experience of VoIP Service: A Survey of Assessment Approaches and Open Issues, IEEE Communications Surveys & Tutorials, Vol. 14, No. 2, 2012, pp. 491-513
ITU: User requirements for objective perceptual video quality measurements in digital cable television, ITU-T Rec. J.143, International Telecommunication Union, 2000
ITU: Objective perceptual video quality measurement techniques for digital cable television in the presence of a full reference, ITU-T Rec. J.144, International Telecommunication Union, 2004
ITU: Objective picture quality measurement method by use of in-service test signals, ITU-T Rec. J.147, International Telecommunication Union, 2002
ITU: Full reference and reduced reference calibration methods for video transmission systems with constant misalignment of spatial and temporal domains with constant gain and offset, ITU-T Rec. J.244, International Telecommunication Union, 2008
Joskowicz, J., Carlos López Ardao, J.: A parametric model for perceptual video quality estimation, Telecommunication Systems, 2010
ITU: Opinion model for video-telephony applications, ITU-T Rec. G.1070, International Telecommunication Union, 2007
ITU: The E-model: a computational model for use in transmission planning, ITU-T Rec. G.107, International Telecommunication Union, 2009
GPP: Technical Specification Group Services and System Aspects; End-to-end multimedia services performance metrics, 3GPP TR 26.944 (Release 9), 3rd Generation Partnership Project, 2009
GPP: Transparent end-to-end Packet-switched Streaming Service (PSS); Protocols and codecs, 3GPP TS 26.234 (Release 10), 3rd Generation Partnership Project, 2011
GPP: Multimedia Broadcast/Multicast Service (MBMS); Protocols and codecs, 3GPP TS 26.346 (Release 10), 3rd Generation Partnership Project, 2011
Jekosch, U.: Voice and speech quality perception - assessment and evaluation, Springer, Berlin, 2005
Piamrat, K., Viho, C., Ksentini, A., Bonnin, J. M.: Quality of Experience Measurements for Video Streaming over Wireless Networks, Proceedings of the Sixth International Conference on Information Technology: New Generations, 2009, pp. 1184-1189
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
