Panoramic SLAM for multi-camera rig

doi:10.11947/j.AGCS.2019.20180443

Abstract

Abstract: Simultaneous localization and mapping (SLAM) is a research hotspot in fields of photogrammetry, computer vision and robotics, and has been widely applied in mobile mapping system, robots, driverless car, etc. This paper presents a fully automated feature based SLAM solution for a panoramic imaging system consisted of multi-camera rig. First, we developed a fisheye camera calibration model for guaranteeing high accurate coordinate transformation between the fisheye camera and the panoramic camera. Second, we imbedded the panoramic camera model into the SLAM process including initialization, local map building, key frame selection, graph optimization and bundle adjustment. In addition, we developed the algorithm in the processes of feature matching, bundle adjustment, frame tracking considering the disadvantages from the large image distortion and long baseline of the panoramic camera system. Experiments are executed on two data sets with more than 8000 panoramic images. Results show that the proposed panoramic SLAM solution achieves automatic camera localization and map construction, and the localization accuracy approaches the GPS reference. With respect to the mainstream SLAM systems based on conventional cameras, such as Mono-SLAM, Stereo-SLAM and RGB-D SLAM, our proposed panoramic SLAM system could serve as a beneficial supplement and supplies a cheap solution for GPS denied localization problem.

Key words: SLAM, panoramic camera, fisheye camera, camera calibration

CLC Number:

P237

JI Shunping, QIN Zijie. Panoramic SLAM for multi-camera rig[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(10): 1254-1265.

References 34

[1]	龚健雅, 季顺平. 摄影测量与深度学习[J]. 测绘学报, 2018, 47(6):693-704. DOI:10.11947/j.AGCS.2018.20170640. GONG Jianya, JI Shunping. Photogrammetry and deep learning[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(6):693-704. DOI:10.11947/j.AGCS.2018.20170640.
[2]	邸凯昌, 万文辉, 赵红颖, 等. 视觉SLAM技术的进展与应用[J]. 测绘学报, 2018, 47(6):770-779. DOI:10.11947/j.AGCS.2018.20170652. DI Kaichang, WANG Wenhui, ZHAO hongying, et al. Progress and application of visual SLAM[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(6):770-779.DOI:10.11947/j.AGCS.2018. 20170652.
[3]	LIU H M, ZHANG G F, BAO H J. A survey of monocular simultaneous localization and mapping[J]. Journal of Computer-Aided Design & Computer Graphics, 2016, 28(6):855-868.
[4]	DAVISON A J, REID I D, MOLTON N D, et al. MonoSLAM:real-time single camera SLAM[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(6):1052-1067.
[5]	MUR-ARTAL R, MONTIEL J M M, TARDÓS J D. ORB-SLAM:A versatile and accurate monocular SLAM system[J]. IEEE Transactions on Robotics, 2017, 31(5):1147-1163.
[6]	ENGEL J, SCHÖPS T, CREMERS D. LSD-SLAM:large-scale direct monocular SLAM[M]//FLEET D, PAJDLA T, SCHIELE B, et al. Computer Vision-ECCV 2014. Zurich:Cham- Springer, 2014:834-849.
[7]	FORSTER C, PIZZOLI M, SCARAMUZZA D. SVO:Fast semi-direct monocular visual odometry[C]//Proceedings of IEEE International Conference on Robotics and Automation. Hong Kong:IEEE, 2014:15-22.
[8]	FORSTER C, ZHANG Z C, GASSNER M, et al. SVO:semidirect visual odometry for monocular and multicamera systems[J]. IEEE Transactions on Robotics, 2017, 33(2):249-265.
[9]	GAO X, ZHANG T. Robust RGB-D simultaneous localization and mapping using planar point features[J]. Robotics and Autonomous Systems, 2015, 72(10):1-14.
[10]	NEWMAN P, HO K. SLAM-loop closing with visually salient features[C]//Proceedings of 2005 IEEE International Conference on Robotics and Automation. Barcelona:IEEE, 2005:635-642.
[11]	RUBLEE E, RABAUD V, KONOLIGE K, et al. ORB:An efficient alternative to SIFT or SURF[C]//Proceedings of 2011 International Conference on Computer Vision. Barcelona:IEEE, 2011:2564-2571.
[12]	季顺平, 史云. 多镜头组合型全景相机两种成像模型的定位精度比较[J]. 测绘学报, 2014, 43(12):1252-1258. DOI:10.13485/j.cnki.11-2089.2014.0169. JI Shunping, SHI Yun. Comparison of two sensor models for multi-camera rig system in measurements[J]. Acta Geodaetica et Cartographica Sinica, 2014, 43(12):1252-1258. DOI:10.13485/j.cnki.11-2089.2014.0169.
[13]	季顺平, 史云. 高噪声环境下基于参考影像的车载序列影像定位方法[J]. 测绘学报, 2014, 43(11):1174-1181. DOI:10.13485/j.cnki.11-2089.2014.0181. JI Shunping, SHI Yun. Georegistration of ground sequential imagery with geo-referenced aerial images in high noise environments[J]. Acta Geodaetica et Cartographica Sinica, 2014, 43(11):1174-1181. DOI:10.13485/j.cnki.11-2089.2014.0181.
[14]	RITUERTO A, PUIG L, GUERRERO J J. Visual SLAM with an omnidirectional camera[C]//Proceedings of the 20th International Conference on Pattern Recognition. Istanbul, Turkey:IEEE, 2010:348-351.
[15]	LEMAIRE T, LACROIX S. SLAM with panoramic vision[J]. Journal of Field Robotics, 2010, 24(1-2):91-111.
[16]	JI Shunping, SHI Yun, SHAN Jie, et al. Particle filtering methods for georeferencing panoramic image sequence in complex urban scenes[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2015, 105(7):1-12.
[17]	CARUSO D, ENGEL J, CREMERS D. Large-scale direct SLAM for omnidirectional cameras[C]//Proceedings of 2015 IEEE/RSJ IEEE International Conference on Intelligent Robots and Systems. Hamburg:IEEE, 2015:141-148.
[18]	谢东海, 钟若飞, 吴俣, 等. 球面全景影像相对定向与精度验证[J]. 测绘学报, 2017, 46(11):1822-1829. DOI:10.11947/j.AGCS.2017.20160645. XIE Donghai, ZHONG Ruofei, WU Yu, et al. Relative pose estimation and accuracy verification of spherical panoramic image[J]. Acta Geodaetica et Cartographica Sinica, 2017, 46(11):1822-1829. DOI:10.11947/j.AGCS.2017.20160645.
[19]	闫利, 王奕丹. 扩展共线方程结合车载组合全景相机影像区域网平差[J]. 测绘学报, 2017, 46(4):460-467. DOI:10.11947/j.AGCS.2017.20160464. YAN Li, WANG Yidan. Block adjustment of vehicle-borne multi-camera rig images using extended collinearity equations[J]. Acta Geodaetica et Cartographica Sinica, 2017, 46(4):460-467. DOI:10.11947/j.AGCS.2017.20160464.
[20]	石丽梅, 赵红蕊, 李明海, 等. 车载移动测图系统外方位元素标定方法[J]. 测绘学报, 2015, 44(1):52-58. DOI:10.11947/j.AGCS.2015.20130203. SHI Limei, ZHAO Hongrui, LI Minghai, et al. Extrinsic calibration for vehicle-based mobile mapping system[J]. Acta Geodaetica et Cartographica Sinica, 2015, 44(1):52-58. DOI:10.11947/j.AGCS.2015.20130203.
[21]	GEYER C, DANIILIDIS K. Catadioptric camera calibration[C]//Proceedings of the 7th IEEE International Conference on Computer Vision. Kerkyra, Greece, Greece:IEEE, 1999:398-404.
[22]	SATO T, IKEDA S, YOKOYA N. Extrinsic camera parameter recovery from multiple image sequences captured by an omni-directional multi-camera system[C]//Proceedings of European Conference on Computer Vision. Prague, Czech Republic:Springer, 2004:326-340.
[23]	MUR-ARTAL R, TARDÓS J D. ORB-SLAM2:An open-source SLAM system for monocular, stereo, and RGB-D cameras[J]. IEEE Transactions on Robotics, 2017, 33(5):1255-1262.
[24]	IKEDA S, SATO T, YOKOYA N. High-resolution panoramic movie generation from video streams acquired by an omnidirectional multi-camera system[C]//Proceedings of IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, MFI2003. Tokyo, Japan:IEEE, 2003.
[25]	GUTIERREZ D, RITUERTO A, MONTIEL J M M, et al. Adapting a real-time monocular visual SLAM from conventional to omnidirectional cameras[C]//Proceedings of 2011 IEEE International Conference on Computer Vision Workshops. Barcelona, Spain:IEEE, 2011:343-350.
[26]	PAYÁ L, FERNÁNDEZ L, GIL A, et al. Map building and monte carlo localization using global appearance of omnidirectional images[J]. Sensors, 2010, 10(12):11468-11497.
[27]	CUMMINS M, NEWMAN P. FAB-MAP:appearance-based place recognition and mapping using a learned visual vocabulary model[C]//Proceedings of the 27th International Conference on Machine Learning.Haifa, Israel:[s.n.], 2010.
[28]	SATO T, YOKOYA N. Efficient hundreds-baseline stereo by counting interest points for moving omni-directional multi-camera system[J]. Journal of Visual Communication and Image Representation, 2010, 21(5-6):416-426.
[29]	LEPETIT V, MORENO-NOGUER F, FUA P. EPnP:an accurate O(n) solution to the PnP problem[J]. International Journal of Computer Vision, 2009, 81(2):155-166.
[30]	KÜMMERLE R, GRISETTI G, STRASDAT H, et al. G2o:a general framework for graph optimization[C]//Proceedings of 2011 IEEE International Conference on Robotics and Automation. Shanghai, China:IEEE, 2011:3607-3613.
[31]	STRASDAT H, MONTIEL J M M, DAVISON A J. Scale drift-aware large scale monocular SLAM[M]//Robotics:Science and Systems. Zurich:The MIT Press, 2010.
[32]	HORN B K P, HILDENH M, NEGAHDARIPOUR S. Closed-form solution of absolute orientation using orthonormal matrices[J]. Journal of the Optical Society of America A, 1998, 5(7):1127-1135.
[33]	KANNALA J, BRANDT S S. A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006, 28(8):1335-1340.
[34]	ZHANG Guofeng, LIU Haomin, DONG Zilong, et al. Efficient non-consecutive feature tracking for robust structure-from-motion[J]. IEEE Transactions on Image Processing, 2016, 25(12):5957-5970.