Joint Multi-scale Convolution Neural Network for Scene Classification of High Resolution Remote Sensing Imagery

doi:10.11947/j.AGCS.2018.20170191

Abstract

Abstract: High resolution remote sensing imagery scene classification is important for automatic complex scene recognition, which is the key technology for military and disaster relief, etc. In this paper, we propose a novel joint multi-scale convolution neural network (JMCNN) method using a limited amount of image data for high resolution remote sensing imagery scene classification. Different from traditional convolutional neural network, the proposed JMCNN is an end-to-end training model with joint enhanced high-level feature representation, which includes multi-channel feature extractor, joint multi-scale feature fusion and Softmax classifier. Multi-channel and scale convolutional extractors are used to extract scene middle features, firstly. Then, in order to achieve enhanced high-level feature representation in a limit dataset, joint multi-scale feature fusion is proposed to combine multi-channel and scale features using two feature fusions. Finally, enhanced high-level feature representation can be used for classification by Softmax. Experiments were conducted using two limit public UCM and SIRI datasets. Compared to state-of-the-art methods, the JMCNN achieved improved performance and great robustness with average accuracies of 89.3% and 88.3% on the two datasets.

Key words: high resolution remote sensing imagery, scene classification, joint multi-scale convolution neural network, enhanced high-level feature representation, limit datasets

CLC Number:

P237

ZHENG Zhuo, FANG Fang, LIU Yuanyuan, GONG Xi, GUO Mingqiang, LUO Zhongwen. Joint Multi-scale Convolution Neural Network for Scene Classification of High Resolution Remote Sensing Imagery[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(5): 620-630.

References

[1] CHERIYADAT A M. Unsupervised Feature Learning for Aerial Scene Classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2014, 52(1):439-451.
[2] SERRANO N, SAVAKIS A E, LUO J B. Improved Scene Classification Using Efficient Low-Level Features and Semantic Cues[J]. Pattern Recognition, 2004, 37(9):1773-1784.
[3] 殷慧, 曹永锋, 孙洪. 基于多维金字塔表达和AdaBoost的高分辨率SAR图像城区场景分类算法[J]. 自动化学报, 2010, 36(8):1099-1106. YIN Hui, CAO Yongfeng, SUN Hong. Urban Scene Classification Based on Multi-dimensional Pyramid Representation and AdaBoost Using High Resolution SAR Images[J]. Acta Automatica Sinica, 2010, 36(8):1099-1106.
[4] LAZEBNIK S, SCHMID C, PONCE J. Beyond Bags of Features:Spatial Pyramid Matching for Recognizing Natural Scene Categories[C]//Proceedings of 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. New York, NY:IEEE, 2006:2169-2178.
[5] YANG Yi, NEWSAM S. Spatial Pyramid Co-occurrence for Image Classification[C]//IEEE International Conference on Computer Vision. Barcelona, Spain:IEEE, 2011:1465-1472.
[6] ZHAO Bei, ZHONG Yanfei, ZHANG Liangpei. Scene Classification via Latent Dirichlet Allocation Using a Hybrid Generative/Discriminative Strategy for High Spatial Resolution Remote Sensing Imagery[J]. Remote Sensing Letters, 2013, 4(12):1204-1213.
[7] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet Classification with Deep Convolutional Neural Networks[C]//Proceedings of the 25th International Conference on Neural Information Processing Systems. Lake Tahoe, Nevada:ACM, 2012:1097-1105.
[8] HECHT-NIELSEN R. Theory of the Backpropagation Neural Network[C]//International Joint Conference on Neural Networks. Washington, DC:IEEE, 1989(1):593-605.
[9] 何小飞, 邹峥嵘, 陶超, 等. 联合显著性和多层卷积神经网络的高分影像场景分类[J]. 测绘学报, 2016, 45(9):1073-1080. DOI:10.11947/j.AGCS.2016.20150612. HE Xiaofei, ZOU Zhengrong, TAO Chao, et al. Combined Saliency with Multi-Convolutional Neural Network for High Resolution Remote Sensing Scene Classification[J]. Acta Geodaetica et Cartographica Sinica, 2016, 45(9):1073-1080. DOI:10.11947/j.AGCS.2016.20150612.
[10] CASTELLUCCIO M, POGGI G, SANSONE C, et al. Land Use Classification in Remote Sensing Images by Convolutional Neural Networks[J]. Acta Ecologica Sinica, 2015, 28(2):627-635.
[11] PENATTI O A B, NOGUEIRA K, SANTOS J A D. Do Deep Features Generalize from Everyday Objects to Remote Sensing and Aerial Scenes Domains?[C]//IEEE Conference on Computer Vision and Pattern Recognition Workshops. Boston, MA:IEEE, 2015:44-51.
[12] 李学龙, 史建华, 董永生, 等. 场景图像分类技术综述[J]. 中国科学(信息科学), 2015, 45(7):827-848. LI Xuelong, SHI Jianhua, DONG Yongsheng, et al. A Survey on Scene Image Classification[J]. Scientia Sinica (Informationis), 2015, 45(7):827-848.
[13] LI Haoxiang, LIN Zhe, SHEN Xiaohui, et al. A Convolutional Neural Network Cascade for Face Detection[C]//IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA:IEEE, 2015:5325-5334.
[14] GLOROT X, BORDES A, BENGIO Y. Deep Sparse Rectifier Neural Networks[C]//Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics. Fort Lauderdale, Florida:HAL, 2011:315-323.
[15] SZEGEDY C, LIU Wei, JIA Yangqing, et al. Going Deeper with Convolutions[C]//IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA:IEEE, 2015:1-9.
[16] SRIVASTAVA N, HINTON G, KRIZHEVSKY A, et al. Dropout:A Simple Way to Prevent Neural Networks from Overfitting[J]. Journal of Machine Learning Research, 2014, 15(1):1929-1958.
[17] ZHAO Bei, ZHONG Yanfei, XIA Guisong, et al. Dirichlet-Derived Multiple Topic Scene Classification Model for High Spatial Resolution Remote Sensing Imagery[J]. IEEE Transactions on Geoscience and Remote Sensing, 2016, 54(4):2108-2123.
[18] ZHAO Bei, ZHONG Yanfei, ZHANG Liangpei, et al. The Fisher Kernel Coding Framework for High Spatial Resolution Scene Classification[J]. Remote Sensing, 2016, 8(2):157.
[19] ZHU Qiqi, ZHONG Yanfei, ZHAO Bei, et al. Bag-of-Visual-Words Scene Classifier with Local and Global Features for High Spatial Resolution Remote Sensing Imagery[J]. IEEE Geoscience and Remote Sensing Letters, 2016, 13(6):747-751.
[20] ABADI M, BARHAM P, CHEN Jianmin, et al. TensorFlow:A System for Large-scale Machine Learning[C]//Proceedings of the 12th Usenix Conference on Operating Systems Design and Implementation. Berkeley, CA:USENIX Association, 2016.
[21] ZHANG Fan, DU Bo, ZHANG Liangpei. Saliency-Guided Unsupervised Feature Learning for Scene Classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2015, 53(4):2175-2184.
[22] LIENOU M, MAITRE H, DATCU M. Semantic Annotation of Satellite Images Using Latent Dirichlet Allocation[J]. IEEE Geoscience and Remote Sensing Letters, 2010, 7(1):28-32.
[23] CHEN Shizhi, TIAN YingLi. Pyramid of Spatial Relations for Scene-level Land Use Classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2015, 53(4):1947-1957.
[24] BREIMAN L. Random Forests[J]. Machine Learning, 2001, 45(1):5-32.
[25] ADANKON M M, CHERIET M. Support Vector Machine[J]. Computer Science, 2002, 1(4):1-28.