Due to the advances made in recent years, methods based on deep neural networks have been able to achieve a state-of-the-art performance in various computer vision problems. In some tasks, such as image recognition, neural-based approaches have even been able to surpass human performance. However, the benchmarks on which neural networks achieve these impressive results usually consist of fairly high quality data. On the other hand, in practical applications we are often faced with images of low quality, affected by factors such as low resolution, presence of noise or a small dynamic range. It is unclear how resilient deep neural networks are to the presence of such factors. In this paper we experimentally evaluate the impact of low resolution on the classification accuracy of several notable neural architectures of recent years. Furthermore, we examine the possibility of improving neural networks’ performance in the task of low resolution image recognition by applying super-resolution prior to classification. The results of our experiments indicate that contemporary neural architectures remain significantly affected by low image resolution. By applying super-resolution prior to classification we were able to alleviate this issue to a large extent as long as the resolution of the images did not decrease too severely. However, in the case of very low resolution images the classification accuracy remained considerably affected.
If the inline PDF is not rendering correctly, you can download the PDF file here.
Abadi M. Agarwal A. Barham P. Brevdo E. Chen Z. Citro C. Corrado G. S. Davis A. Dean J. Devin M. Ghemawat S. Goodfellow I. Harp A. Irving G. Isard M. Jia Y. Jozefowicz R. Kaiser L. Kudlur M. Levenberg J. Mane D. Monga R. Moore S. Murray D. Olah C. Schuster M. Shlens J. Steiner B. Sutskever I. Talwar K. Tucker P. Vanhoucke V. Vasudevan V. Viegas F. Vinyals O. Warden P. Wattenberg M. Wicke M. Yu Y. and Zheng X. (2016). TensorFlow: Large-scale machine learning on heterogeneous distributed systems arXiv: 1603.04467.
Bevilacqua M. Roumy A. Guillemot C. and Alberi-Morel M.L. (2012). Low-complexity single-image super-resolution based on nonnegative neighbor embedding British Machine Vision Conference (BMVC) Guildford UK.
Chang H. Yeung D.-Y. and Xiong Y. (2004). Super-resolution through neighbor embedding Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition CVPR Washington DC USA Vol. 1 pp. I-I.
da Costa G.B.P. Contato W.A. Nazare T.S. Neto J.E. and Ponti M. (2016). An empirical study on the effects of different types of noise in image classification tasks arXiv: 1609.02781.
Dodge S. and Karam L. (2016). Understanding how image quality affects deep neural networks 8th International Conference on Quality of Multimedia Experience (QoMEX) Lisbon Portugal pp. 1-6.
Dong C. Loy C.C. He K. and Tang X. (2014). Learning a deep convolutional network for image super-resolution European Conference on Computer Vision Zurich Switzerland pp. 184-199.
Dong C. Loy C.C. and Tang X. (2016). Accelerating the super-resolution convolutional neural network European Conference on Computer Vision Amsterdam The Netherlands pp. 391-407.
Dutta A. Veldhuis R.N. and Spreeuwers L.J. (2012). The impact of image quality on the performance of face recognition 33rd WIC Symposium on Information Theory in the Benelux Boekelo The Netherlands pp. 141-148.
Freeman W.T. Jones T.R. and Pasztor E.C. (2002). Example-based super-resolution IEEE Computer Graphics and Applications 22(2): 56-65.
Goodfellow I. Bengio Y. and Courville A. (2016). Deep Learning MIT Press Cambridge MA.
Goodfellow I. Pouget-Abadie J. Mirza M. Xu B. Warde-Farley D. Ozair S. Courville A. and Bengio Y. (2014). Generative adversarial nets in Z. Ghahramani et al. (Eds.) Advances in Neural Information Processing Systems Curran Associates Inc. Red Hook NY pp. 2672-2680.
He K. Zhang X. Ren S. and Sun J. (2016). Deep residual learning for image recognition Proceedings of the IEEE
Huang J.-B. Singh A. and Ahuja N. (2015). Single image super-resolution from transformed self-exemplars Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Boston MA USA pp. 5197-5206.
Johnson J. Karpathy A. and Fei-Fei L. (2016). DenseCap: Fully convolutional localization networks for dense captioning Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Las Vegas VN USA pp. 4565-4574.
Karahan S. Yildirum M.K. Kirtac K. Rende F.S. Butun G. and Ekenel H.K. (2016). How image degradations affect deep CNN-based face recognition International Conference of the Biometrics Special Interest Group (BIOSIG) Darmstadt Germany pp. 1-5.
Kim J. Kwon Lee J. and Mu Lee K. (2016). Accurate image super-resolution using very deep convolutional networks Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Las Vegas NV USA pp. 1646-1654.
Kingma D. and Ba J. (2014). Adam: A method for stochastic optimization arXiv: 1412.6980.
Koziarski M. and Cyganek B. (2017). Image recognition with deep neural networks in presence of noise-dealing with and taking advantage of distortions Integrated Computer- Aided Engineering 24(4): 337-349.
Krizhevsky A. Sutskever I. and Hinton G.E. (2012). ImageNet classification with deep convolutional neural networks Neural Information Processing Systems Lake Tahoe CA USA pp. 1097-1105.
Ledig C. Theis L. Huszár F. Caballero J. Cunningham A. Acosta A. Aitken A. Tejani A. Totz J. Wang Z. and Shi W. (2016). Photo-realistic single image super-resolution using a generative adversarial network arXiv: 1609.04802.
Lim B. Son S. Kim H. Nah S. and Lee K.M. (2017). Enhanced deep residual networks for single image super-resolution IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshop Honolulu HI USA Vol. 1 p. 3.
Long J. Shelhamer E. and Darrell T. (2015). Fully convolutional networks for semantic segmentation Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Boston MA USA pp. 3431-3440.
Mao X. Shen C. and Yang Y.-B. (2016). Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections Neural Information Processing Systems Barcelona Spain pp. 2802-2810.
Martin D. Fowlkes C. Tal D. and Malik J. (2001). A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics 8th IEEE International Conference on Computer Vision ICCV 2001 Vancouver Canada Vol. 2 pp. 416-423.
Peng X. Hoffman J. Stella X.Y. and Saenko K. (2016). Fine-to-coarse knowledge transfer for low-res image classification IEEE International Conference on Image Processing (ICIP) Phoenix AZ USA pp. 3683-3687.
Ronneberger O. Fischer P. and Brox T. (2015). U-net: Convolutional networks for biomedical image segmentation International Conference on Medical Image Computing and Computer-Assisted Intervention Munich Germany pp. 234-241.
Russakovsky O. Deng J. Su H. Krause J. Satheesh S. Ma S. Huang Z. Karpathy A. Khosla A. Bernstein M. Berg A.C. and Fei-Fei L. (2015). ImageNet large scale visual recognition challenge International Journal of Computer Vision 115(3): 211-252.
Sanchez A. Moreno A.B. Velez D. and V´elez J.F. (2016). Analyzing the influence of contrast in large-scale recognition of natural images Integrated Computer-Aided Engineering 23(3): 221-235.
Schmidhuber J. (2015). Deep learning in neural networks: An overview Neural Networks 61(1): 85-117.
Schulter S. Leistner C. and Bischof H. (2015). Fast and accurate image upscaling with super-resolution forests Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Boston MA USA pp. 3791-3799.
Shi W. Caballero J. Huszár F. Totz J. Aitken A.P. Bishop R. Rueckert D. and Wang Z. (2016). Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Las Vegas NV USA pp. 1874-1883.
Simonyan K. and Zisserman A. (2014). Very deep convolutional networks for large-scale image recognition arXiv: 1409.1556.
Sun Y. Wang X. and Tang X. (2013). Deep convolutional network cascade for facial point detection Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Portland OR USA pp. 3476-3483.
Tai Y. Yang J. and Liu X. (2017). Image super-resolution via deep recursive residual network 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Honolulu HI USA pp. 2790-2798.
Timofte R. De Smet V. and Van Gool L. (2014). A+: Adjusted anchored neighborhood regression for fast super-resolution Asian Conference on Computer Vision Singapore Singapore pp. 111-126.
Vasiljevic I. Chakrabarti A. and Shakhnarovich G. (2016). Examining the impact of blur on recognition by convolutional networks arXiv: 1611.05760.
Yang J. Wright J. Huang T.S. and Ma Y. (2010). Image super-resolution via sparse representation IEEE Transactions on Image Processing 19(11): 2861-2873.
Zeyde R. Elad M. and Protter M. (2010). On single image scale-up using sparse-representations International Conference on Curves and Surfaces Paris France pp. 711-730.
Zou W.W. and Yuen P.C. (2012). Very low resolution face recognition problem IEEE Transactions on Image Processing 21(1): 327-340.