Search by item HOME > Access full text > Search by item

JBE, vol. 25, no. 5, pp.672-684, September, 2020

DOI: https://doi.org/10.5909/JBE.2020.25.5.672

Adversarial Framework for Joint Light Field Super-resolution and Deblurring

Jonathan Samuel Lumentut, Hyungsun Baek, and In Kyu Park

C.A E-mail: pik@inha.ac.kr

Abstract:

Restoring a low resolution and motion blurred light field has become essential due to the growing works on parallax-based image processing. These tasks are known as light-field enhancement process. Unfortunately, only a few state-of-the-art methods are introduced to solve the multiple problems jointly. In this work, we design a framework that jointly solves light field spatial super-resolution and motion deblurring tasks. Particularly, we generate a straight-forward neural network that is trained under low-resolution and 6-degree-of-freedom (6-DOF) motion-blurred light field dataset. Furthermore, we propose the strategy of local region optimization on the adversarial network to boost the performance. We evaluate our method through both quantitative and qualitative measurements and exhibit superior performance compared to the state-of-the-art methods.



Keyword: Deep learning, light field, deblurring, super-resolution

Reference:
[1] Y. Bok, H. G. Jeon, and I. S. Kweon, “Geometric calibration of micro-lense-based light field cameras using line features,” IEEE Trans. on Pattern Analysis and Machine Intelligence, 39(2):287–300, February 2017.
[2] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” In Proc. of Advances in Neural Information Processing Systems, pages 2672–2680, December 2014.
[3] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville, “Improved training of Wasserstein GANs,” In Proc. of Advances in Neural Information Processing Systems, pages 5767–5777, December 2017.
[4] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, June 2016.
[5] T. H. Kim, K. M. Lee, B. Scholkopf, and M. Hirsch, “Online video deblurring via dynamic temporal blending network,” In Proc. of IEEE International Conference on Computer Vision, pages 4058–4067, 2017.
[6] S. Iizuka, E. Simo-Serra, and H. Ishikawa, “Globally and locally consistent image completion,” ACM Transactions on Graphics, 36(4):107, Article No. 107, July 2017.
[7] P. Isola, J. Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 1125–1134, July 2017.
[8] H. G. Jeon, J. Park, G. Choe, J. Park, Y. Bok, Y. W. Tai, and I. S. Kweon, “Accurate depth map estimation from a lenslet light field camera,” In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 1547–1555, June 2015.
[9] N. K. Kalantari, T. C. Wang, and R. Ramamoorthi, “Learning-based view synthesis for light field cameras,” ACM Transactions on Graphics, 35(6), Article No. 193, November 2016.
[10] J. Kim, J. K. Lee, and K. M. Lee, “Accurate image super-resolution using very deep convolutional networks,” In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 1646–1654, June 2016.
[11] D. Krishnan, T. Tay, and R. Fergus, “Blind deconvolution using a normalized sparsity measure,” In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 233–240, June 2011.
[12] O. Kupyn, V. Budzan, M. Mykhailych, D. Mishkin, and J. Matas, “DeblurGAN: Blind motion deblurring using conditional adversarial networks,” In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 8183–8192, June 2018.
[13] W. S. Lai, J. B. Huang, Z. Hu, N. Ahuja, and M. H. Yang, “A comparative study for single image blind deblurring,” In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 1701–1709, June 2016.
[14] D. Lee, H. Park, I. K. Park, and K. M. Lee, “Joint blind motion deblurring and depth estimation of light field,” In Proc. of European Conference on Computer Vision, pp. 300-316, September 2018.
[15] M. Levoy and P. Hanrahan, “Light field rendering,” In Proc. of SIGGRAPH, pages 31-42, August 1996.
[16] J. S. Lumentut, T. H. Kim, R. Ramamoorthi, and I. K. Park, “Fast and full-resolution light field deblurring using a deep neural network,” IEEE Signal of Processing Letters, 26(12), pages 1788-1792, December 2019.
[17] J.S. Lumentut, Williem, and I. K. Park, “6-DOF Motion Blur Synthesis and Performance Evaluation of Light Field Deblurring,” Multimedia Tools and Application, 78(23), pages 33723-33746, December 2019.
[18] M. R. Mahesh Mohan and A. N. Rajagopalan, “Divide and conquer for full-resolution light field deblurring,” In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6421–6429, June 2018.
[19] S. Nah, T. H. Kim, and K. M. Lee, “Deep multi-scale convolutional neural network for dynamic scene deblurring,” In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, volume 1, pages 3883-3891, July 2017.
[20] R. Ng, “Fourier slice photography,” ACM Trans. on Graphics, 24(3):735–744, July 2005.
[21] J. Pan, Z. Hu, Z. Su, and M. H. Yang, “Deblurring text images via L0-regularized intensity and gradient prior,” In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 2901–2908, June 2014.
[22] W. Qiu, F. Zhong, Y. Zhang, S. Qiao, Z. Xiao, T. S. Kim, and Y. Wang, “UnrealCV: Virtual worlds for computer vision.” In Proc. of ACM International Conference on Multimedia, pages 1221–1224, October 2017.
[23] Z. Shen, W. S. Lai, T. Xu, J. Kautz, and M. H. Yang, “Deep semantic face deblurring,” In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 8260–8269, June 2018.
[24] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” In Proc. of International Conference on Learning Representations, April 2014.
[25] P. P. Srinivasan, R. Ng, and R. Ramamoorthi, “Light field blind motion deblurring,” In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 3958–3966, July 2017.
[26] P. P. Srinivasan, R. Tucker, J. T. Barron, R. Ramamoorthi, R. Ng, and N. Snavely, “Pushing the boundaries of view extrapolation with multiplane images,” In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 175–184, June 2019.
[27] P. P. Srinivasan, T. Wang, A. Sreelal, R. Ramamoorthi, and R. Ng, “Learning to synthesize a 4D RGBD light field from a single image,” In Proc. of IEEE International Conference on Computer Vision, pages 2262–2270, October 2017.
[28] S. Su, M. Delbracio, J. Wang, G. Sapiro, W. Heidrich, and O. Wang, “Deep video deblurring for hand-held cameras,” In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, volume 2, pages 237-246, July 2017.
[29] Y. W. Tai, P. Tan, and M. S. Brown, “Richardson-Lucy deblurring for scenes under a projective motion path,” IEEE Trans. on Pattern Analysis and Machine Intelligence, 33(8):1603–1618, August 2011.
[30] M. W. Tao, S. Hadap, J. Malik, and R. Ramamoorthi, “Depth from combining defocus and correspondence using light-field cameras,” In Proc. of the IEEE International Conference on Computer Vision, pages 673–680, December 2013.
[31] M. W. Tao, P. P Srinivasan, J. Malik, S. Rusinkiewicz, and R. Ramamoorthi, “Depth from shading, defocus, and correspondence using light-field angular coherence,” In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1940–1948, June 2015.
[32] D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Instance normalization: The missing ingredient for fast stylization,” arXiv preprint arXiv:1607.08022, 2016.
[33] T. C Wang, A. A. Efros, and Ravi Ramamoorthi, “Depth estimation with occlusion modeling using light-field cameras,” IEEE Trans. on Pattern Analysis and Machine Intelligence, 38(11):2170–2181, November 2016.
[34] Williem, I. K. Park, and K. M. Lee, “Robust light field depth estimation using occlusion-noise aware data costs,” IEEE Trans. on Pattern Analysis and Machine Intelligence, 40(10):2484–2497, October 2018.
[35] Y. Yoon, H. G. Jeon, D. Yoo, J. Y. Lee, and I. S. Kweon, “Learning a deep convolutional network for light-field image super-resolution,” In Proc. of IEEE International Conference on Computer Vision Workshop, pages 57–65, June 2015.
[36] S. Zhang, Y. Lin, and H. Sheng, “Residual networks for light field image super-resolution,” In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 11046–11055, June 2019.
[37] T. Zhou, R. Tucker, J. Flynn, G. Fyffe, and N. Snavely, “Stereo magnification: Learning view synthesis using multiplane images,” ACM Trans. on Graphics, 37(4), Article No. 65, July 2018.

Comment


Editorial Office
1108, New building, 22, Teheran-ro 7-gil, Gangnam-gu, Seoul, Korea
Homepage: www.kibme.org TEL: +82-2-568-3556 FAX: +82-2-568-3557
Copyrightⓒ 2012 The Korean Institute of Broadcast and Media Engineers
All Rights Reserved