Search by item HOME > Access full text > Search by item

JBE, vol. 26, no. 1, pp.61-69, January, 2021

DOI: https://doi.org/10.5909/JBE.2021.26.1.61

A Review on Deep Learning-based Image Outpainting

Kyunghun Kim, Kyeongbo Kong, and Suk-ju Kang

C.A E-mail: sjkang@sogang.ac.kr

Abstract:

Image outpainting is a very interesting problem in that it can continuously fill the outside of a given image by considering the context of the image. There are two main challenges in this work. The first is to maintain the spatial consistency of the content of the generated area and the original input. The second is to generate high quality large image with a small amount of adjacent information. Existing image outpainting methods have difficulties such as generating inconsistent, blurry, and repetitive pixels. However, thanks to the recent development of deep learning technology, deep learning-based algorithms that show high performance compared to existing traditional techniques have been introduced. Deep learning-based image outpainting has been actively researched with various networks proposed until now. In this paper, we would like to introduce the latest technology and trends in the field of outpainting. This study compared recent techniques by analyzing representative networks among deep learning-based outpainting algorithms and showed experimental results through various data sets and comparison methods.



Keyword: image completion, image restoration, image outpainting, extrapolation

Reference:
[1] Ballester, C., Bertalmio, M., Caselles, V., Sapiro, G., & Verdera, J. (2001). Filling-in by joint interpolation of vector fields and gray levels. IEEE transactions on image processing, 10(8), 1200-1211.
[2] Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. Image inpainting. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 417–424. ACM Press/AddisonWesley Publishing Co., 2000.
[3] Alexei A Efros and William T Freeman. Image quilting for texture synthesis and transfer. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 341–346. ACM, 2001
[4] Alexei A Efros and Thomas K Leung. Texture synthesis by non-parametric sampling. In Proceedings of the seventh IEEE international conference on computer vision, volume 2, pages 1033–1038. IEEE, 1999
[5] Goodfellow, Ian and Pouget-Abadie, Jean and Mirza, Mehdi and Xu, Bing and Warde-Farley, David and Ozair, Sherjil and Courville, Aaron and Bengio, Yoshua. Generative adversarial nets. In Advances in neural information processing systems, pages 2672–2680, 2014.
[6] Iizuka, Satoshi, Edgar Simo-Serra, and Hiroshi Ishikawa. "Globally and locally consistent image completion." ACM Transactions on Graphics (ToG) 36.4 (2017): 1-14.
[7] Liu, Guilin, et al. "Image inpainting for irregular holes using partial convolutions." Proceedings of the European Conference on Computer Vision (ECCV). 2018.
[8] Yu, Jiahui, et al. "Generative image inpainting with contextual attention." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.
[9] Johannes Kopf, Wolf Kienzle, Steven Drucker, and Sing Bing Kang. Quality prediction for image completion. ACM Transactions on Graphics (TOG), 31(6):131, 2012.
[10] Josef Sivic, Biliana Kaneva, Antonio Torralba, Shai Avidan, and William T Freeman. Creating and exploring a large photorealistic virtual space. 2008.
[11] Miao Wang, Yu-Kun Lai, Yuan Liang, Ralph R. Martin, and Shi-Min Hu. Biggerpicture: Data-driven image extrapolation using graph matching. ACM Trans. Graph., 33(6):173:1– 173:13, Nov. 2014.
[12] Yinda Zhang, Jianxiong Xiao, James Hays, and Ping Tan. Framebreak: Dramatic image extrapolation by guided shiftmaps. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1171–1178, 2013.
[13] Yang, Zongxin, et al. "Very Long Natural Scenery Image Prediction by Outpainting." Proceedings of the IEEE International Conference on Computer Vision. 2019.
[14] Teterwak, Piotr and Sarna, Aaron and Krishnan, Dilip and Maschinot, Aaron and Belanger, David and Liu, Ce and Freeman, William T. Boundless: Generative adversarial networks for image extension. In Proceedings of the IEEE International Conference on Computer Vision, pages 10521–10530, 2019.
[15] Yi Wang, Xin Tao, Xiaoyong Shen, and Jiaya Jia. Wide context semantic image extrapolation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1399–1408, 2019.
[16] Zhang, X., Chen, F., Wang, C., Tao, M., & Jiang, G. P. (2020). Sienet: Siamese expansion network for image extrapolation. IEEE Signal Processing Letters, 27, 1590-1594.
[17] Gatys, Leon A and Ecker, Alexander S and Bethge, Matthias. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2414–2423, 2016.
[18] Johnson, Justin and Alahi, Alexandre and Fei-Fei, Li. Perceptual losses for real-time style transfer and superresolution. In European conference on computer vision, pages 694–711. Springer, 2016.
[19] Kim, K., Yun, Y., Kang, K. W., Kong, K., Lee, S., & Kang, S. J. (2021). Painting Outside As Inside: Edge Guided Image Outpainting via Bidirectional Rearrangement With Progressive Step Learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 2122-2130).
[20] Xiao, Jianxiong and Hays, James and Ehinger, Krista A and Oliva, Aude and Torralba, Antonio. Sun database: Largescale scene recognition from abbey to zoo. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 3485–3492. IEEE, 2010.
[21] Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik. No-reference image quality assessment in the spatial domain. IEEE Transactions on image processing, 21(12):4695–4708, 2012.
[22] Salimans, Tim and Goodfellow, Ian and Zaremba, Wojciech and Cheung, Vicki and Radford, Alec and Chen, Xi. Improved techniques for training gans. In Advances in neural information processing systems, pages 2234–2242, 2016.
[23] Heusel, Martin and Ramsauer, Hubert and Unterthiner, Thomas and Nessler, Bernhard and Hochreiter, Sepp. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in neural information processing systems, pages 6626–6637, 2017.
[24] Mark Sabini and Gili Rusak. Painting outside the box: Image outpainting with gans. arXiv preprint arXiv:1808.08483, 2018.

Comment


Editorial Office
1108, New building, 22, Teheran-ro 7-gil, Gangnam-gu, Seoul, Korea
Homepage: www.kibme.org TEL: +82-2-568-3556 FAX: +82-2-568-3557
Copyrightⓒ 2012 The Korean Institute of Broadcast and Media Engineers
All Rights Reserved