Search by item HOME > Access full text > Search by item

JBE, vol. 27, no. 5, pp.773-793, September, 2022

DOI: https://doi.org/10.5909/JBE.2022.27.5.773

Performance Analysis of 6DoF Video Streaming Based on MPEG Immersive Video

Jong-Beom Jeong, Soonbin Lee, Inae Kim, and Eun-Seok Ryu

C.A E-mail: Virtual reality, MIV, 6DoF, HEVC, VVC, Immersive video, Metaverse

Abstract:

The moving picture experts group (MPEG) immersive video (MIV) coding standard has been established to support six degrees of freedom (6DoF) in virtual reality (VR) by transmitting high-quality multiple immersive videos. The MIV exploits two approaches considering tradeoff between bandwidth and computational complexity: 1) eliminating correlation between multi-view videos or 2) selecting representative videos. This paper presents performance analysis on intermediate synthesized views on source view positions and synthesized pose traces using high-efficiency video coding (HEVC) and versatile video coding (VVC) for above-mentioned two approaches. 



Keyword: Virtual reality, MIV, 6DoF, HEVC, VVC, Immersive video, Metaverse

Reference:
[1] M.-L. Champel, T. Stockhammer, T. Fautier, E. Thomas, R. Koenen. 2016. Quality Requirements for VR. 116th MPEG meeting of ISO/IEC JTC1/SC29/ WG11, MPEG 116/m39532.
[2] T. T. Le, J. B. Jeong, E. -S. Ryu. 2019. Efficient transcoding and encryption for live 360 CCTV system. Applied Sciences, Vol. 9, No. 4. doi: https://doi.org/10.3390/app9040760
[3] J. -B. Jeong, S. Lee, I. Kim, S. Lee, E. -S. Ryu. 2020. Implementing VVC Tile Extractor for 360-degree Video Streaming Using Motion-Constrained Tile Set. Journal of Broadcast Engineering (JBE), Vol. 25, No. 7, pp. 1073-1080. doi: https://doi.org/10.5909/JBE.2020.25.7.1073
[4] T. T. Le, J. -B. Jeong, S. S. Lee, J. Kim, E. -S. Ryu. 2021. An Efficient Viewport-Dependent 360 VR System Based on Adaptive Tiled Streaming. CMC-Computers, Materials & Continua, Vol. 66, No. 3, pp. 2627-2643. doi: https://doi.org/10.32604/cmc.2021.013399
[5] J. -B. Jeong, S. Lee, I. Kim, E. -S. Ryu. 2021. Implementing Viewport Tile Extractor for Viewport-Adaptive 360-Degree Video Tiled Streaming. International Conference on Information Networking 2022 (ICOIN2022), pp. 8-12. doi: https://doi.org/10.1109/ICOIN50884.2021.9333964
[6] S. Lee, D. Jang, J. B. Jeong, E. -S. Ryu. 2019. Motion-constrained tile set based 360-degree video streaming using saliency map prediction. ACM Network and Operating System Support for Digital Audio and Video 2019 (NOSSDAV2019), pp. 20-24. doi: https://doi.org/10.1145/3304112.3325614
[7] A. Vetro, T. Wiegand, G. J. Sullivan. 2011. Overview of the stereo and multiview video coding extensions of the H. 264/MPEG-4 AVC standard. Proceedings of the IEEE, Vol. 99, No. 4, pp. 626-642. doi: https://doi.org/10.1109/JPROC.2010.2098830
[8] M. M. Hannuksela, Y. Yan, X. Huang, H. Li. 2015. Overview of the multiview high efficiency video coding (MV-HEVC) standard. IEEE International Conference on Image Processing 2015 (ICIP2015), pp. 2154-2158. doi: https://doi.org/10.1109/ICIP.2015.7351182
[9] R. Doré, F. Thudor. 2018. Outperforming 3DoF+ Anchors: first evidence. 123th MPEG meeting of ISO/IEC JTC1/SC29/ WG11, MPEG2018/m43504.
[10] J. -B. Jeong, S. Lee, D. Jang, E. -S. Ryu. 2019. Towards 3DoF+ 360 Video Streaming System for Immersive Media. IEEE Access, Vol. 7, pp. 136399-136408. doi: https://doi.org/10.1109/ACCESS.2019.2942771
[11] ISO/IEC JTC1/SC29/WG11. 2019. Call for Proposals on 3DoF+ Visual. 125th MPEG meeting of ISO/IEC JTC1/SC29/ WG11, MPEG/n18145.
[12] J. Fleureau, F. Thudor, R. Doré, M. Dmitrichenko, B. Salahieh, J. Boyce. 2019. Description of Technicolor Intel response to MPEG-I 3DoF+ Call for Proposal. 126th MPEG meeting of ISO/IEC JTC1/SC29/ WG11, MPEG2019/m47445.
[13] M. Domański, A. Dziembowski, D. Mieloch, O. Stankiewicz, J. Stankowski, A. Grzelka, G. Lee, J. Y. Jeong, J. Seo. 2019. Technical description of proposal for Call for Proposals on 3DoF+ Visual prepared by Poznań University of Technology (PUT) and Electronics and Telecommunications Research Institute (ETRI). 126th MPEG meeting of ISO/IEC JTC1/SC29/ WG11, MPEG2019/m47407.
[14] B. Kroon, B. Sonneveldt. 2019. Philips response to CfP on 3DoF+. 126th MPEG meeting of ISO/IEC JTC1/SC29/ WG11, MPEG2019/m47179.
[15] V. K. M. Vadakital, K. Roimela, L. Ilola, J. Keränen, M. Pesonen, S. Schwarz, J. lainema, M. Hannuksela. 2019. Description of Nokia’s response to CFP for 3DOF+ visual. 126th MPEG meeting of ISO/IEC JTC1/SC29/ WG11, MPEG2019/m47372.
[16] B. Wang, Y. Sun, L. Yu. 2019. Description of Zhejiang University’s response to 3DoF+ Visual CfP. 126th MPEG meeting of ISO/IEC JTC1/SC29/ WG11, MPEG2019/m47684.
[17] ISO/IEC JTC 1/SC 29/WG 4. 2021. Text of ISO/IEC FDIS 23090-12 MPEG Immersive Video. 135th MPEG meeting of ISO/IEC JTC 1/SC 29/WG 4, MPEG/n00111.
[18] B. Bross, Y. -K. Wang, Y. Ye, S. Liu, J. Chen, G. J. Sullivan, J. -R. Ohm. 2021. Overview of the versatile video coding (VVC) standard and its applications. IEEE Transactions on Circuits and Systems for Video Technology, Vol. 31, No. 10, pp. 3736-3764. doi: https://doi.org/10.1109/TCSVT.2021.3101953
[19] Jongseok Lee, Juntaek Park, Hansol Choi, Joohyung Byeon, Donggyu Sim, “Overview of VVC“, Broadcast and Media Magazine, pp.10-25. Vol. 24, No. 3, Oct. 2019.
[20] E. S. Jang, M. Preda, K. Mammou, A. M. Tourapis, J. Kim, D. B. Graziosi, S. Rhyu, M. Budagavi. 2019. Video-based point-cloud-compression standard in MPEG: From evidence collection to committee draft [standards in a nutshell]. IEEE Signal Processing Magazine, Vol. 36, No. 3, pp. 118-123. doi: https://doi.org/10.1109/MSP.2019.2900721
[21] D. B. Graziosi, O. Nakagami, S. Kuma, A. Zaghetto, T. Suzuki, A. Tabatabai. 2020. An overview of ongoing point cloud compression standardization activities: Video-based (V-PCC) and geometry-based (G-PCC). APSIPA Transactions on Signal and Information Processing, Vol. 9, pp. 1-17. doi: https://doi.org/10.1017/ATSIP.2020.12
[22] DaeHyeok Gwon, Heeji Han, Haechul Choi , "Video based Point Cloud Compression with Versatile Video Coding" 2020 The Korean Institute of Broadcast and Media Engineers Summer Conference, pp. 375-377, July, 2020.
[23] ISO/IEC JTC 1/SC 29/WG 7. 2021. Text of ISO/IEC DIS 23090-5 Visual Volumetric Video-based Coding and Video-based Point Cloud Compression 2nd Edition. 135th MPEG meeting of ISO/IEC JTC 1/SC 29/WG 7, MPEG/n00188.
[24] J. -B. Jeong, S. Lee, I. -W. Ryu, T. T. Le, E. -S. Ryu. 2020. Towards Viewport-dependent 6dof 360 Video Tiled Streaming for Virtual Reality Systems. ACM International Conference on Multimedia 2020 (MM 2020), pp. 3687–3695. doi: https://doi.org/10.1145/3394171.3413712
[25] S. Lee, J. -B. Jeong, I. Kim, S. Lee, E. -S. Ryu. 2020. Screen Content Coding Analysis to Improve Coding Efficiency for Immersive Video. Journal of Broadcast Engineering (JBE), Vol. 25, No. 6, pp. 911-921. doi: https://doi.org/10.5909/JBE.2020.25.6.911
[26] L. Kondrad, V. K. M. Vadakital, L. Ilola. 2020. .CE-1.3: frame packed video sub-bitstream type. 131th MPEG meeting of ISO/IEC JTC 1/SC 29/WG 11, MPEG/m54274.
[27] B. Salahieh, G. Naf, J. Boyce. 2021. Frame Packing Implementation in TMIV. 134th MPEG meeting of ISO/IEC JTC 1/SC 29/WG 4, MPEG/m56827.
[28] J. -B. Jeong, S. Lee, E. -S. Ryu. 2021. [MIV] Extraction and Merging on Frame Packed Video. 134th MPEG meeting of ISO/IEC JTC 1/SC 29/WG 4, MPEG/m56591.
[29] J. -B. Jeong, S. Lee, E. -S. Ryu. 2021. Sub-bitstream packing based lightweight tiled streaming for 6 degree of freedom immersive video. Electronics Letters, Vol. 57, No. 25, pp. 973-976. doi: https://doi.org/10.1049/ell2.12329
[30] J. -B. Jeong, S. Lee, E. -S. Ryu. 2021. DWS-BEAM: Decoder-Wise Subpicture Bitstream Extracting and Merging for MPEG Immersive Video. International Conference on Visual Communications and Image Processing 2021 (VCIP2021), pp. 1-5.doi: https://doi.org/10.1109/VCIP53242.2021.9675419 [31] ISO/IEC JTC 1/SC 29/WG 11. 2018. Reference View Synthesizer (RVS) manual. 124th MPEG meeting of ISO/IEC JTC 1/SC 29/WG 11, MPEG/n18068.
[32] S. Facheada, D. Bonatto, A. Schenkel, G. Lafruit. 2018. Depth image based view synthesis with multiple reference views for virtual reality. The True Vision-Capture, Transmission and Display of 3D Video 2018 (3DTV-CON 2018), pp. 1-4. doi: https://doi.org/10.1109/3DTV.2018.8478484

[33] S. Kwak, J. Yun, J. Jeong, W. -S. Cheong, J. Seo. 2020. [MPEG-I Visual] Ray-based blending weight for 6DoF view synthesis. 131th MPEG meeting of ISO/IEC JTC 1/SC 29/WG 11, MPEG/m54409.
[34] J. -B. Jeong, S. Lee, E. -S. Ryu. 2021. On OpenGL-based View Synthesizer Optimization for Real-time 6DoF Immersive Video Streaming. IEEE P3079.1 Contribution (DCN: 3079-21-0057-00- 0001).
[35] D. Bonatto, S. Fachada, G. Lafruit. 2020. RaViS: Real-time accelerated View Synthesizer for immersive video 6DoF VR. Electronic Imaging, pp. 382-1-382-9. doi: https://doi.org/10.2352/ISSN.2470-1173.2020.13.ERVR-381
[36] H. Ahn, M. -j. Lee. 2020. MPEG-I RVS Software Speed-up for Real-time Application. Journal of Broadcast Engineering (JBE), Vol. 25, No. 5, pp. 655-664. doi: https://doi.org/10.5909/JBE.2020.25.5.655
[37] S. Kwak, J. Yun, J. -Y. Jeong, Y. Kim, I. Ihm, W. -S. Cheong, J. Seo. 2021. View synthesis with sparse light field for 6DoF immersive video. ETRI Journal, Vol. 44, No. 1, pp. 24-37. doi: https://doi.org/10.4218/etrij.2021-0205
[38] J. Jung, B. Kroon. 2021. Common Test Conditions for MPEG Immersive Video. 133th MPEG meeting of ISO/IEC JTC 1/SC 29/WG 4, MPEG/n00051.
[39] B. Salahieh, J. Jung, A. Dziembowski, C. Bachhuber. 2021. Test Model 8 for MPEG Immersive Video. 133th MPEG meeting of ISO/IEC JTC 1/SC 29/WG 4, MPEG/n00050.
[40] HM reference software, [Online]. Available at: https://hevc.hhi. fraunhofer.de/svn/svn_HEVCSoftware/.
[41] VTM reference software, [Online]. Available at: https://vcgit.hhi. fraunhofer.de/jvet/VVCSoftware VTM/.
[42] Y. Sun, A. Lu, L. Yu. 2017. Weighted-to-spherically-uniform quality evaluation for omnidirectional video. IEEE signal processing letters, Vol. 24, No. 9, pp. 1408–1412. doi: https://doi.org/10.1109/LSP.2017.2720693
[43] C. G. Bampis, A. C. Bovik, and Z. Li. 2018. A simple prediction fusion improves data-driven full-reference video quality assessment models. Picture Coding Symposium 2018 (PCS2018, pp. 298–302. doi: https://doi.org/10.1109/PCS.2018.8456293
[44] A. Dziembowski, D. Mieloch, J. Stankowski. A. Grzelka. 2022. IV-PSNR–the objective quality metric for immersive video applications. IEEE Transactions on Circuits and Systems for Video Technology. doi: https://doi.org/10.1109/TCSVT.2022.3179575
[45] J. -B. Jeong, S. Kim, S. Lee, I. Kim, E. -S. Ryu. 2021. Super Metric: Quality Assessment Methods for Immersive Video. Journal of Internet Computing and Services (JICS), Vol. 22, No. 2, pp. 51-58. doi: https://doi.org/10.7472/jksii.2021.22.2.51
[46] J. Jung, B. Kroon. 2021. Common Test Conditions for MPEG Immersive Video. 134th MPEG meeting of ISO/IEC JTC 1/SC 29/WG 4, MPEG/n00085.
[47] Versatile video encoder (VVenC), [Online]. Available at: https:// github.com/fraunhoferhhi/vvenc/.
[48] F. Bossen, K. Sühring, A. Wieckowski, S. Liu. 2021. VVC Complexity and Software Implementation Analysis. IEEE Transactions on Circuits and Systems for Video Technology, Vol. 31, No. 10, pp. 3765-3778. doi: https://doi.org/10.1109/TCSVT.2021.3072204

Comment


Editorial Office
1108, New building, 22, Teheran-ro 7-gil, Gangnam-gu, Seoul, Korea
Homepage: www.kibme.org TEL: +82-2-568-3556 FAX: +82-2-568-3557
Copyrightⓒ 2012 The Korean Institute of Broadcast and Media Engineers
All Rights Reserved