Subjective Evaluation of User Quality of Experience for Omnidirectional Video Streaming

  • Štefica Mrvelj University of Zagreb, Faculty of Transport and Traffic Sciences
  • Marko Matulin University of Zagreb, Faculty of Transport and Traffic Sciences https://orcid.org/0000-0003-1676-354X
  • Sergo Martirosov University of West Bohemia
Keywords: 360-degree video, streaming, video resolution, frame drop, quality of experience

Abstract

This paper reports on the results of subjective testing of user Quality of Experience (QoE) for omnidirectional video (ODV) streaming quality. The test was conducted among 20 test subjects who watched three ODVs using a Head Mounted Display (HMD) system. The length of the videos was between two and three minutes. The first video was used for training purposes and contained no quality degradations. The quality of the other two ODVs was degraded by manipulating the resolution or by introducing different frame drop patterns. While watching the pre-prepared videos the subjects indicated if they noticed the changes in the quality and then rated it. After watching each video, the subjects completed a separate questionnaire, which evaluated their level of enjoyment and discomfort with the video. The results showed that the degradation of both objective parameters (video resolution and frame rate) impacted the subjects’ perception of quality; however, the impact was somewhat alleviated in ODV which contained dynamic scenes and fast camera movements.

Author Biographies

Štefica Mrvelj, University of Zagreb, Faculty of Transport and Traffic Sciences

Department of Information and Communication Traffic, full time professor

Marko Matulin, University of Zagreb, Faculty of Transport and Traffic Sciences

Department of Information and Communication Traffic, Assistant Professor

Sergo Martirosov, University of West Bohemia

Department of Industrial Engineering and Innovation Management, PhD student

References

Ericsson Consumerlab. Merged Reality: Understanding How Virtual and Augmented Realities Could Transform Everyday Reality. Ericsson; 2017. Available from: https://www.ericsson.com/en/trends-and-nsights/consumerlab/

consumer-insights/reports/merged-reality [Accessed June 2019].

Mrvelj Š, Matulin M. Impact of Packet Loss on the Perceived Quality of UDP-based Multimedia Streaming: A Study of User Quality of Experience in Real-life Environments. Multimedia Systems. 2018;24(1): 33-53. Available from: doi:10.1007/s00530-016-0531-8

Matulin M, Mrvelj Š. Modelling User Quality of Experience from Objective and Subjective Data Sets using Fuzzy Logic. Multimedia Systems. 2018;24(6): 645-667 Available from: doi:10.1007/s00530-018-0590-0

https://www.fpz.unizg.hr/qoe4vr/ [Accessed September 2019].

Cisco. Cisco Visual Networking Index: Global Mobile Data Traffic Forecast Update, 2017-2022. White Paper. Cisco; 2019. Available from: https://www.cisco.com/c/en/us/solutions/collateral/service-provider/visual-networking-index-vni/white-paper-c11-738429.html [Accessed June 2019].

Yu M, Lakshman H, Girod B. A Framework to Evaluate Omnidirectional Video Coding Schemes, Mixed and Augmented Reality. Proceedings of the IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 29 September – 3 October 2015, Fukuoka, Japan; 2015. p. 31-36. Available from: doi:10.1109/ISMAR.2015.12

Sun Y, Lu A, Yu L. WS-PSNR for 360 Video Objective Quality Evaluation. MPEG Joint Video Exploration Team. Vol. 116. Chengdu; 2016.

Sun Y, Lu A, Yu L. AHG8: WS-PSNR for 360 Video Objective Quality Evaluation. Joint Video Exploration Team of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11, JVET-D0040, 4th Meeting. Chengdu; 2016.

He Y, Vishwanath B, Xiu X, Ye Y. AHG8: InterDigital’s Projection Format Conversion Tool. Joint Video Exploration Team of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11, JVET-D0021, 4th Meeting. Chengdu; 2016.

Zakharchenko V, Alshina E, Singh A, Dsouza A. AHG8: Suggested Testing Procedure for 360-degree Video. Joint Video Exploration Team of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11, JVET-D0027, 4th Meeting. Chengdu; 2016.

Wu S, Chen X, Fu J, Chen Z. Efficient VR Video Representation and Quality Assessment. Journal of Visual Communication and Image Representation. 2018;57: 107-117. Available from: doi:10.1016/j.jvcir.2018.10.018

Chen S, Zhang Y, Li Y, Chen Z, Wang Z. Spherical Structural Similarity Index for Objective Omnidirectional Video Quality Assessment. Proceedings of the IEEE International Conference on Multimedia and Expo (ICME), 23-27 July 2018, San Diego, United States; 2018. p. 1-6. Available from: doi:10.1109/ICME.2018.8486584

Wang Z, Simoncelli EP, Bovik AC. Multiscale Structural Similarity for Image Quality Assessment. Proceedings of the 37th Asilomar Conference on Signals, Systems & Computers, 9-12 November 2003, Pacific Grove, United States; 2003. p. 1398-1402. Available from: doi:10.1109/ACSSC.2003.1292216

Ozcinar C, Cabrera J, Smolic A. Omnidirectional Video Streaming Using Visual Attention-Driven Dynamic Tiling for VR. Proceedings of the IEEE International Conference on Visual Communications and Image Processing (VCIP) 2018, 9-12 December 2018, Taichung, Taiwan; 2018. p. 1-4. Available from: doi:10.1109/VCIP.2018.8698638

Ozcinar C, Cabrera J, Smolic A. Visual Attention-Aware Omnidirectional Video Streaming Using Optimal Tiles for Virtual Reality. IEEE Journal on Emerging and Selected Topics in Circuits and Systems. 2019;9(1): 217-230. Available from: doi:10.1109/JETCAS.2019.2895096

Xu M, Li C, Chen Z, Wang Z, Guan Z. Assessing Visual Quality of Omnidirectional Videos. IEEE Transactions on Circuits and Systems for Video Technology. 2019. Available from: doi:10.1109/TCSVT.2018.2886277

Birkbeck N, Brown C, Suderman R. Quantitative Evaluation of Omnidirectional Video Quality. Proceedings of the 9th International Conference on Quality of Multimedia Experience (QoMEX), 31 May - 2 June 2017, Erfurt, Germany; 2017. Available from: doi:10.1109/QoMEX.2017.7965684

Zakharchenko V, Pyo Choi K, Alshina E, Hoon Park J. Omnidirectional Video Quality Metrics and Evaluation Process. Proceedings of the Data Compression Conference (DCC), 4-7 April 2017, Snowbird, United States; 2017. p. 472-472. Available from: doi:10.1109/DCC.2017.90

Chen Z, Li Y, Zhang Y. Recent Advances in Omnidirectional Video Coding for virtual Reality: Projection and Evaluation. Signal Processing. 2018;146: 66-78. Available from: doi:10.1016/j.sigpro.2018.01.004

Upenik E, Řeřábek M, Ebrahimi T. Testbed for Subjective Evaluation of Omnidirectional Visual Content. Picture Coding Symposium (PCS), 4-7 December 2016, Nuremberg, Germany; 2016. p. 1-5. Available from: doi:10.1109/PCS.2016.7906378

Pérez P, Escobar J. MIRO360: A Tool for Subjective Assessment of 360 Degree Video for ITU-T P.360-VR. Proceedings of the 11th International Conference on Quality of Multimedia Experience (QoMEX), 5-7 June 2019, Berlin, Germany; 2019. Available from: doi:10.1109/QoMEX.2019.8743216

Sassatelli L, Winckler M, Fisichella T, Dezarnaud A, Lemaire J, Aparicio-Pardo R, Trevisan D. New Interactive Strategies for Virtual Reality Streaming in Degraded Context of Use. Computers & Graphics. 2019. Available from: doi:10.1016/j.cag.2019.10.005

Schatz R, Sackl A, Timmerer C, Gardlo B. Towards Subjective Quality of Experience Assessment for Omnidirectional Video Streaming. Proceedings of the 9th International Conference on Quality of Multimedia Experience (QoMEX), 31 May - 2 June 2017, Erfurt, Germany; 2017. Available from: doi:10.1109/QoMEX.2017.7965657

Yao S-H, Fan C-L, Hsu C-H. Towards Quality-of-Experience Models for Watching 360° Videos in Head-Mounted Virtual Reality. Proceedings of the 11th International Conference on Quality of Multimedia Experience (QoMEX), 5-7 June 2019, Berlin, Germany; 2019. Available from: doi:10.1109/QoMEX.2019.8743198

Liotou E, Tsolkas D, Passas N. A roadmap on QoE metrics and models. Proceedings of the 23rd International Conference on Telecommunications (ICT), 16-18 May 2016, Thessaloniki, Greece; 2016. Available from: doi:10.1109/ICT.2016.7500363

Datta P, Izdebski L, Kumar N, Suh K. “It came to me in a stream…” The upward arc of online video, driven by consumers. Cisco white paper. Available from: https://www.cisco.com/c/dam/en_us/about/ac79/docs/sp/Online-

Video-Consumption_Consumers.pdf [Accessed 22nd Feb 2020].

Farrokhi F, Mahmoudi-Hamidabad A. Rethinking convenience sampling: defining quality criteria. Theory and Practice in Language Studies. 2012;2(4): 784-792. Available from: doi:10.4304/tpls.2.4.784-792

Fiedler M, Hossfeld T, Tran-Gia P. A Generic Quantitative Relationship between Quality of Experience and Quality of Service. IEEE Network. 2010;24(2): 36-41. Available from: doi:10.1109/MNET.2010.5430142

Published
2020-05-10
How to Cite
1.
Mrvelj Štefica, Matulin M, Martirosov S. Subjective Evaluation of User Quality of Experience for Omnidirectional Video Streaming. Promet - Traffic&Transportation. 2020;32(3):409-21. DOI: 10.7307/ptt.v32i3.3444
Section
Articles