A template matching method of multimodal remote sensing images based on deep convolutional feature representation

doi:10.11947/j.AGCS.2019.20180432

Abstract

Abstract: Due to significant non-linear radiometric differences between multimodal remote sensing images (e.g., optical, infrared, and SAR), traditional methods cannot efficiently extract common features between such images, and are vulnerable for image matching. To address that, the deep learning technique is introduced into the present study to design a matching method based on Siamese network, which aims to extract common features between multimodal images. The network is first optimized by removing the pooling layer and extracting the feature layer from Siamese network to maintain the integrity and positional accuracy of the feature information, making it possible the effective extraction of common features between multimodal images. Then, the template matching strategy is adopted to achieve high-precision matching of multimodal images. The proposed method is evaluated by using multiple multimodal remote sensing images. The results show that the proposed method outperforms traditional template-matching methods in both the matching correct ratio and matching accuracy.

Key words: multimodal image, image matching, deep learning, Siamese network

CLC Number:

P231

NAN Ke, QI Hua, YE Yuanxin. A template matching method of multimodal remote sensing images based on deep convolutional feature representation[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(6): 727-736.

References

[1] 闫利, 王紫琦, 叶志云. 顾及灰度和梯度信息的多模态影像配准算法[J]. 测绘学报, 2018, 47(1):71-81. DOI:10.11947/j.AGCS.2018.20170368. YAN Li, WANG Ziqi, YE Zhiyun. Multimodal image registration algorithm considering grayscale and gradient information[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(1):71-81. DOI:10.11947/j.AGCS.2018.20170368.
[2] YE Yuanxin, SHAN Jie, BRUZZONE L, et al. Robust registration of multimodal remote sensing images based on structural similarity[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(5):2941-2958.
[3] ZITOVÁ B, FLUSSER J. Image registration methods:a survey[J]. Image and Vision Computing, 2003, 21(11):977-1000.
[4] 余先川, 吕中华, 胡丹. 遥感图像配准技术综述[J]. 光学精密工程, 2013, 21(11):2960-2972. YU Xianchuan, LÜ Zhonghua, HU Dan. Review of remote sensing image registration techniques[J]. Optics and Precision Engineering, 2013, 21(11):2960-2972.
[5] MORAVEC H P. Towards automatic visual obstacle avoidance[C]//Proceedings of the 5th International Joint Conference on Artificial Intelligence. Massachusetts US:MIT, 1977.
[6] HARRIS C, STEPHENS M. A combined corner and edge detector[C]//Proceedings of the 4th Alvey Vision Conference. Manchester, UK:Organising Committee AVC, 1988:147-151.
[7] ZIOU D, TABBONE S. Edge detection techniques:an overview[J]. International Journal of Pattern Recognition and Image Analysis, 1998(8):537-559.
[8] ZHANG Dengrong, YU Le, CAI Zhigang. Automatic registration for ASAR and TM images based on region features[C]//Proceedings of SPIE 6752, Geoinformatics 2007:Remotely Sensed Data and Information. Nanjing:SPIE, 2007:6752.
[9] 张宝尚, 田铮, 延伟东. 基于分割区域的SAR图像配准方法研究[J]. 工程数学学报, 2011, 28(1):7-14. ZHANG Baoshang, TIAN Zheng, YAN Weidong. An SAR image registration algorithm based on segmentation-derived regions[J]. Chinese Journal of Engineering Mathematics, 2011, 28(1):7-14.
[10] LOWE D G. Object recognition from local scale-invariant features[C]//Proceeding of the 7th International Conference on Computer Vision. Kerkyra, Corfu, Greece:IEEE, 1999:1150-1157.
[11] LOWE D G. Distinctive image features from scale-invariant keypoints[J]. International Journal of Computer Vision, 2004, 60(2):91-110.
[12] BELONGIE S, MALIK J, PUZICHA J. Shape matching and object recognition using shape contexts[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002, 24(4):509-522.
[13] SURI S, REINARTZ P. Mutual-information-based registration of TerraSAR-X and Ikonos imagery in urban areas[J]. IEEE Transactions on Geoscience and Remote Sensing, 2010, 48(2):939-949.
[14] MARTINEZ A, GARCIA-CONSUEGRA J, ABAD F. A correlation-symbolic approach to automatic remotely sensed image rectification[C]//Proceedings of IEEE 1999 International Geoscience and Remote Sensing Symposium. Hamburg, Germany:IEEE, 1999:336-338.
[15] JOHNSON K, COLE-RHODES A, ZAVORIN I, et al. Mutual information as a similarity measure for remote sensing image registration[C]//Proceedings of SPIE 4383, Geo-Spatial Image and Data Exploitation Ⅱ. Orlando, FL, United States:SPIE, 2001:51-61.
[16] YE Yuanxin, SHAN Jie, HAO Siyuan, et al. A local phase based invariant feature for remote sensing image matching[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018(142):205-221.
[17] ZAGORUYKO S, KOMODAKIS N. Learning to compare image patches via convolutional neural networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Boston:IEEE, 2015:4353-4361.
[18] 范大昭, 董杨, 张永生. 卫星影像匹配的深度卷积神经网络方法[J]. 测绘学报, 2018, 47(6):844-853. DOI:10.11947/j.AGCS.2018.20170627. FAN Dazhao, DONG Yang, ZHANG Yongsheng. Satellite image matching method based on deep convolution neural network[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(6):844-853. DOI:10.11947/j.AGCS.2018.20170627.
[19] HE Haiqing, CHEN Min, CHEN Ting, et al. Matching of remote sensing images with complex background variations via Siamese convolutional neural network[J]. Remote Sensing, 2018, 10(3):355.
[20] BROMLEY J, GUYON I, LECUN Y, et al. Signature verification using a "Siamese" time delay neural network[C]//Proceedings of the 6th International Conference on Neural Information Processing Systems. Denver, Colorado:Morgan Kaufmann Publishers Inc, 1993.
[21] CHOPRA S, HADSELL R, LECUN Y. Learning a similarity metric discriminatively, with Application to Face Verification//Proceeding of 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. San Diego, CA:IEEE, 2005.
[22] 许慧敏. 基于深度学习U-Net模型的高分辨率遥感影像分类方法研究[D]. 成都:西南交通大学, 2018. XU Huimin. Method research of high resolution remote sensing imagery classification based on U-Net model of deep learning[D]. Chengdu:Southwest Jiaotong University, 2018.
[23] 周飞燕, 金林鹏, 董军. 卷积神经网络研究综述[J]. 计算机学报, 2017, 40(6):1229-1251. ZHOU Feiyan, JIN Linpeng, DONG Jun. Review of convolutional neural network[J]. Chinese Journal of Computers, 2017,40(6):1229-1251.
[24] HADSELL R, CHOPRA S, LECUN Y. Dimensionality reduction by learning an invariant mapping[C]//Proceedins of 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. New York:IEEE, 2006.
[25] TRAPPEY A J C, HSU F C, TRAPPEY C V, et al. Development of a patent document classification and search platform using a back-propagation network[J]. Expert Systems with Applications, 2006, 31(4):755-765.
[26] YE Yuanxin, SHAN Jie. A local descriptor based registration method for multispectral remote sensing images with non-linear intensity differences[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2014(90):83-95.
[27] 林善明, 朱小艳, 周建华, 等. 一种结合互信息和模板匹配的配准方法[J]. 计算机工程, 2010, 36(14):198-200. LIN Shanming, ZHU Xiaoyan, ZHOU Jianhua, et al. Registration method with mutual information and template matching[J]. Computer Engineering, 2010, 36(14):198-200.