Remote sensing image intelligent interpretation: from supervised learning to self-supervised learning

doi:10.11947/j.AGCS.2021.20210089

Abstract

Abstract: Accurate interpretation of remote sensing image (RSI) plays a vital role in the implementation of remote sensing applications. In recent years, deep supervised learning has achieved great success in the field of RSI interpretation by its soaring performance on representation learning. However, this method heavily relies on large-scale and high-quality labeled data, while building a big remote sensing data set is extremely expensive because of the unique spatial-temporal heterogeneity of remote sensing data. This contradiction seriously restricts the performance of deep supervised learning in large areas and complicated remote sensing scenes. How to solve the last mile problem in the field of RSI accurate interpretation becomes urgent. This paper first systematically reviews the main research progress of supervised learning methods in the field of RSI interpretation, and then analyzes its limitations. Afterward, we introduce the concept of self-supervised learning and detail how it works for unsupervised feature learning. Finally, we briefly discuss open problems and future directions of self-supervised learning if it is applied in the field of RSI interpretation, with the aim of providing a new perspective for RSI interpretation with the adoption of huge unlabeled data.

Key words: remote sensing image intelligent interpretation, supervised learning, deep learning, self-supervised learning

CLC Number:

P231

TAO Chao, YIN Ziwei, ZHU Qing, LI Haifeng. Remote sensing image intelligent interpretation: from supervised learning to self-supervised learning[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(8): 1122-1134.

References

[1] 国家对地观测科学数据中心. 中国对地观测数据资源发展报告2019[R]. 北京:中国科学院空天信息创新研究院, 2020:1-34. National Earth Observation Data Center. Executive summary of the report on earth observation data resources of China (2019)[R]. Beijing:Aerospace Information Research Institute, Chinese Academy of Sciences, 2020:1-34.
[2] 何国金, 王力哲, 马艳, 等. 对地观测大数据处理:挑战与思考[J]. 科学通报, 2015, 60(Z1):470-478. HE Guojin, WANG Lizhe, MA Yan, et al. Processing of earth observation big data:challenges and countermeasures[J]. Chinese Science Bulletin, 2015, 60(Z1):470-478.
[3] 李德仁, 王密, 沈欣, 等. 从对地观测卫星到对地观测脑[J]. 武汉大学学报(信息科学版), 2017, 42(2):143-149. LI Deren, WANG Mi, SHEN Xin, et al. From earth observation satellite to earth observation brain[J]. Geomatics and Information Science of Wuhan University, 2017, 42(2):143-149.
[4] 李德仁, 张良培, 夏桂松. 遥感大数据自动分析与数据挖掘[J]. 测绘学报, 2014, 43(12):1211-1216. LI Deren, ZHANG Liangpei, XIA Guisong. Automatic analysis and mining of remote sensing big data[J]. Acta Geodaetica et Cartographica Sinica, 2014, 43(12):1211-1216.
[5] 张兵.遥感大数据时代与智能信息提取[J]. 武汉大学学报(信息科学版), 2018, 43(12):1861-1871. ZHANG Bing. Remotely sensed big data era and intelligent information extraction[J]. Geomatics and Information Science of Wuhan University, 2018, 43(12):1861-1871.
[6] CLERY D, VOSS D. All for one and one for all[J]. Science, 2005, 308(5723):809-810.
[7] 杜培军, 夏俊士, 薛朝辉, 等. 高光谱遥感影像分类研究进展[J]. 遥感学报, 2016, 20(2):236-256. DU Peijun, XIA Junshi, XUE Zhaohui, et al. Review of hyperspectral remote sensing image classification[J]. Journal of Remote Sensing, 2016, 20(2):236-256.
[8] CHENG Gong, HAN Junwei. A survey on object detection in optical remote sensing images[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2016, 117:11-28.
[9] CHENG Gong, HAN Junwei, LU Xiaoqiang. Remote sensing image scene classification:benchmark and state of the art[J]. Proceedings of the IEEE, 2017, 105(10):1865-1883.
[10] HINTON G E, SALAKHUTDINOV R R. Reducing the dimensionality of data with neural networks[J]. Science, 2006, 313(5786):504-507.
[11] ZHU Xiaoxiang, TUIA D, MOU Lichao, et al. Deep learning in remote sensing:a comprehensive review and list of resources[J]. IEEE Geoscience and Remote Sensing Magazine, 2017, 5(4):8-36.
[12] MA Lei, LIU Yu, ZHANG Xueliang, et al. Deep learning in remote sensing applications:a meta-analysis and review[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2019, 152:166-177.[Linkout]
[13] CHENG Gong, XIE Xingxing, HAN Junwei, et al. Remote sensing image scene classification meets deep learning:challenges, methods, benchmarks, and opportunities[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13:3735-3756.
[14] 陈军,陈晋. GlobeLand30遥感制图创新与大数据分析[J].中国科学:地球科学, 2018, 48:1391-1392. CHEN Jun, CHEN Jin. 2018. GlobeLand30:Operational global land cover mapping and big-data analysis[J]. Science China Earth Sciences, 2018, 48:1391-1392.
[15] JING Longlong, TIAN Yingli. Self-supervised visual feature learning with deep neural networks:a survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2393, PP(99):1.
[16] DEVLIN J, CHANG M W, LEE K, et al. Bert:Pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies. Minneapolis, Minnesota:NAACL, 2019:4171-4186.
[17] CARON M, MISRA I, MAIRAL J, et al. Unsupervised learning of visual features by contrasting cluster assignments[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems. Curran Associates, Inc, 2020:9912-9924.
[18] LAI Z, LU E, XIE W. MAST:A memory-augmented self-supervised tracker[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, WA:IEEE, 2020:6479-6488.
[18] LAI Zihang, LU E, XIE Weidi. MAST:a memory-augmented self-supervised tracker[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 13-19, 2020, Seattle, WA, USA. IEEE, 2020:6478-6487.
[19] HAGHIGHI F, TAHER M R H, ZHOU Z, et al. Learning semantics-enriched representation via self-discovery, self-classification, and self-restoration[C]//Proceedings of 2020 International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham, 2020:137-147.
[20] 周培诚, 程塨, 姚西文,等. 高分辨率遥感影像解译中的机器学习范式[J]. 遥感学报,2021,25(1):182-197 ZHOU Peicheng, CHENG Gong, YAO Xiwen, et al. Machine learning paradigms in high-resolution remote sensing image interpretation[J]. Journal of Remote Sensing, 2021, 25(1):182-197.
[21] ROUSE J W, HAAS R H, SCHELL J A, et al. Monitoring vegetation systems in the Great Plains with ERTS[J]. NASA special publication, 1974, 351(1974):309-317.
[22] MANJUNATH B S, MA W Y. Texture features for browsing and retrieval of image data[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1996, 18(8):837-842.
[23] PESARESI M, BENEDIKTSSON J A. A new approach for the morphological segmentation of high-resolution satellite imagery[J]. IEEE Transactions on Geoscience and Remote Sensing, 2001, 39(2):309-320.
[24] HUANG Xin, ZHANG Liangpei. Morphological building/shadow index for building extraction from high-resolution imagery over urban areas[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2012, 5(1):161-172.
[25] HARRIS C, STEPHENS M. A combined corner and edge detector[C]//Procedings of 1988 Alvey Vision Conference 1988. Manchester. Alvey Vision Club, 1988:10-5244.
[26] LOWE D G. Distinctive image features from scale-invariant keypoints[J]. International Journal of Computer Vision, 2004, 60(2):91-110.
[27] YANG Y, NEWSAM S. Spatial pyramid co-occurrence for image classification[C]//Proceedings of 2011 IEEE International Conference on Computer Vision. Barcelona, Spain:IEEE, 2011:1465-1472.
[28] ZHAO Bei, ZHONG Yanfei, XIA Guisong, et al. Dirichlet-derived multiple topic scene classification model for high spatial resolution remote sensing imagery[J]. IEEE Transactions on Geoscience and Remote Sensing, 2016, 54(4):2108-2123.
[29] ZHAO Bei, ZHONG Yanfei, ZHANG Liangpei. Scene classification via latent Dirichlet allocation using a hybrid generative/discriminative strategy for high spatial resolution remote sensing imagery[J]. Remote Sensing Letters, 2013, 4(12):1204-1213.
[30] ZHANG Xiuyuan, DU Shihong, WANG Qiao. Integrating bottom-up classification and top-down feedback for improving urban land-cover and functional-zone mapping[J]. Remote Sensing of Environment, 2018, 212:231-248.
[31] MOUNTRAKIS G, IM J, OGOLE C. Support vector machines in remote sensing:a review[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2011, 66(3):247-259.
[32] PAL M. Random forest classifier for remote sensing classification[J]. International Journal of Remote Sensing, 2005, 26(1):217-222.
[33] ZHENG Chen, ZHANG Yun, WANG Leiguang. Semantic segmentation of remote sensing imagery using an object-based Markov random field model with auxiliary label fields[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(5):3015-3028.
[34] DENG Jia, DONG Wei, SOCHER R, et al.ImageNet:a large-scale hierarchical image database[C]//Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami:IEEE, 2009:248-255.
[35] PENATTI O A B, NOGUEIRA K, DOS SANTOS J A. Do deep features generalize from everyday objects to remote sensing and aerial scenes domains?[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognitionworkshops. Boston, MA:IEEE, 2015:44-51.
[36] MARMANIS D, DATCU M, ESCH T, et al. Deep learning earth observation classification using ImageNet pretrained networks[J]. IEEE Geoscience and Remote Sensing Letters, 2016, 13(1):105-109.
[37] LIU Yanfei, ZHONG Yanfei, QIN Qianqing. Scene classification based on multiscale convolutional neural network[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018, 56(12):7109-7121.
[38] WANG Q, LIU S, CHANUSSOT J, et al. Scene classification with recurrent attention of VHR remote sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018, 57(2):1155-1167.
[38] WANG Qi, LIU Shaoteng, CHANUSSOT J, et al. Scene classification with recurrent attention of VHR remote sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(2):1155-1167.
[39] HELBER P, BISCHKE B, DENGEL A, et al. EuroSAT:a novel dataset and deep learning benchmark for land use and land cover classification[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019, 12(7):2217-2226.
[40] 叶利华, 王磊, 张文文, 等. 高分辨率光学遥感场景分类的深度度量学习方法[J]. 测绘学报, 2019, 48(6):698-707. YE Lihua, WANG Lei, ZHANG Wenwen, et al. Deep metric learning method for high resolution remote sensing image scene classification[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(6):698-707.
[41] LONG J, SHELHAMERE, DARRELL T. Fullyconvolutional networks for semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA:IEEE, 2015:3431-3440.
[42] LIU Yu, MINH NGUYEN D, DELIGIANNIS N, et al. Hourglass-shapenetwork based semantic segmentation for high resolution aerial imagery[J]. Remote Sensing, 2017, 9(6):522.
[43] HAMAGUCHI R, FUJITA A, NEMOTO K, et al. Effective Use of Dilated Convolutions for Segmenting Small Object Instances in Remote Sensing Imagery[C]//Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). Lake Tahoe, Nevada:IEEE, 2018:1442-1450.
[44] WANG Hongzhen, WANG Ying, ZHANG Qian, et al. Gated convolutional neural network for semantic segmentation in high-resolution images[J]. Remote Sensing, 2017, 9(5):446.
[45] VALADA A, MOHAN R, BURGARD W. Self-supervised model adaptation for multimodal semantic segmentation[J]. International Journal of Computer Vision, 2020, 128(5):1239-1285.
[46] SHANG Ronghua, ZHANG Jiyu, JIAO Licheng, et al. Multi-scale adaptive feature fusion network for semantic segmentation in remote sensing images[J]. Remote Sensing, 2020, 12(5):872.
[47] SHUAI Bing, ZUO Zhen, WANG Bing, et al. Scene segmentation with DAG-recurrent neural networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(6):1480-1493.
[48] 李道纪,郭海涛,卢俊, 等. 遥感影像地物分类多注意力融和U型网络法[J]. 测绘学报, 2020, 49(8):1051-1064. LI Daoji, GUO Haitao, LU Jun, et al. A remote sensing image classification procedure based on multilevel attention fusion U-Net[J]. Acta Geodaetica et Cartographica Sinica, 2020, 49(8):1051-1064.
[49] TAO Chao, QI Ji, LI Yansheng, et al. Spatial information inference net:Road extraction using road-specific contextual information[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2019, 158:155-166.
[50] GIRSHICK R, DONAHUE J, DARRELL T, et al. Rich feature hierarchies for accurate object detection and semantic segmentation[C]//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, Ohio:IEEE, 2014:580-587.
[51] GIRSHICK R. Fast R-CNN[C]//Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago, Chile:IEEE, 2015:1440-1448.
[52] REN Shaoqing, HE Kaiming, GIRSHICK R, et al. Faster R-CNN:towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6):1137-1149.
[53] YANG Xue, YANG Jirui, YAN Junchi, et al. SCRDet:towards more robust detection for small, cluttered and rotated objects[C]//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision.Seoul, Korea (South):IEEE, 2019:8231-8240.
[54] LI Ke, CHENG Gong, BU Shuhui, et al. Rotation-insensitive and context-augmented object detection in remote sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018, 56(4):2337-2348.
[55] REDMON J, DIVVALA S, GIRSHICK R, et al. You only look once:unified, real-time object detection[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas, USA:IEEE, 2016:779-788.
[56] XIANG W, ZHANG D Q, YU H, et al. Context-aware single-shot detector[C]//Proceedings of 2018 IEEE Winter Conference on Applications of Computer Vision. Lake Tahoe, Nevada:IEEE, 2018:1784-1793.
[57] YU Yongtao, GUAN Haiyan, LI Dilong, et al. Orientation guided anchoring for geospatial object detection from remote sensing imagery[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 160:67-82.
[58] VAN E A. Satellite imagery multiscale rapid detection with windowed networks[C]//Proceedings of 2019 IEEE Winter Conference on Applications of Computer Vision. Hilton Waikoloa Village, Hawaii:IEEE, 2019:735-743.
[59] HE, H. AND GARCIA, E.A. Learning from imbalanced data[J]. IEEE Transactions on Knowledge And and Data Engineering, 2009, 21(9):1263-1284.
[60] JOHNSON J M, KHOSHGOFTAAR T M. Survey on deep learning with class imbalance[J]. Journal of Big Data, 2019, 6(1):1-54.
[61] JAISWAL A, BABU A R, ZADEH M Z, et al. A survey on contrastive self-supervised learning[J]. Technologies, 2021, 9(1):2.
[62] IIZUKA S, SIMO-SERRA E, ISHIKAWA H. Globally and locally consistent image completion[J]. ACM Transactions on Graphics, 2017, 36(4):1-14.
[63] PATHAK D, KRAHENBUHL P, DONAHUE J, et al. Context encoders:Feature learning by inpainting[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV:IEEE, 2016:2536-2544.
[64] ZHANG R, ISOLA P, EFROS A A. Colorful image colorization[C]//Proceedings of 2016 European Conference on Computer Vision (ECCV). Amsterdam, the Netherlands:Springer, 2016:649-666.
[65] LARSSON G, MAIRE M, SHAKHNAROVICH G. Colorization as a proxy task for visual understanding[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI:IEEE, 2017:840-849.
[66] HE Kaiming, FAN Haoqi, WU Yuxin, et al. Momentum contrast for unsupervised visual representation learning[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, WA:IEEE, 2020:9726-9735.
[67] KALANTIDIS Y, SARIYILDIZ M B, PION N, et al. Hard negative mixing for contrastive learning[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems. Curran Associates, Inc, 2020:21798-21809.
[68] CHEN TING, KORNBLITH S, NOROUZI M, et al. A simple framework for contrastive learning of visual representations[C]//Proceedings of the 37th International Conference on Machine Learning.[S.l.]:PMLR, 2020:1597-1607. Virtual Event.
[69] YANG Xingyi, HE Xuehai, LIANG Yuxiao, et al. Transfer learning or self-supervised learning? A tale of two pretraining paradigms[EB/OL].[2020-12-16].. https://arxiv.org/abs/2007.04234.
[70] GONG Peng, LIU Han, ZHANG Meinan, et al. Stable classification with limited sample:transferring a 30m resolution sample set collected in 2015 to mapping 10m resolution global land cover in 2017[J]. Science Bulletin, 2019, 64(6):370-373.
[71] LONG Yang, XIA Guisong, LI Shengyang, et al. DiRS:On creating benchmark datasets for remote sensing image interpretation[EB/OL].[2019-11-06]. https://arxiv.org/abs/2006.12485v1.
[72] XIA Guisong, BAI Xiang, DING Jian, et al. DOTA:a large-scale dataset for object detection in aerial images[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT:IEEE, 2018:3974-3983.
[73] TAO Chao, QI Ji, LU Weipeng, et al. Remote sensing image scene classificationwith self-supervised paradigm under limited labeled samples[J]. IEEE Geoscience and Remote Sensing Letters, 2020(99):1-5.[Linkan]
[74] XIE E, DING J, WANG W, et al. Detco:unsupervised contrastive learning for object detection[C]//Proceedings of 2021 IEEE International Conference on Computer Vision.[S.l.]:IEEE, 2021.
[75] ZHU Linchao, YANG Yi. ActBERT:learning global-local video-text representations[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, USA:IEEE, 2020:8743-8752.
[76] HINTON G, VINYALS O, DEAN J. Distilling the knowledge in a neural network[C]//Proceedings of 2014 International Conference on Neural Information Processing SystemsDeep Learning workshop. Montreal, Quebec:MIT Press. 2014.
[77] TUNG F, MORI G. Similarity-preserving knowledge distillation[C]//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision.Seoul, Korea (South):IEEE, 2019:1365-1374.
[78] JIN Pu, XIA Guisong, HU Fan, et al. Aid++:an updated version of aid on scene classification[C]//Proceedings of IEEE 2018 International Geoscience and Remote Sensing Symposium. Valencia, Spain:IEEE, 2018:4721-4724.
[79] LI Ke, WAN Gang, CHENG Gong, et al. Object detection in optical remote sensing images:a survey and a new benchmark[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 159:296-307.
[80] SUMBUL G, CHARFUELAN M, DEMIR B, et al. Bigearthnet:a large-scale benchmark archive for remote sensing image understanding[C]//Proceedings of 2019 IEEE International Geoscience and Remote Sensing Symposium. Yokohama,Japan:IEEE, 2019:5901-5904.