移动测量系统中 RGB 颜色赋给点云的错位修正方法

With the continuous development of three-dimensional city modeling, traditional close-range photogrammetry is limited by complex processing procedures and incomplete 3D depth information, making it unable to meet high-precision modeling requirements. In contrast, the integration of light detection and ranging and cameras in mobile measurement systems provides a new and highly effective solution. Currently, integrated mobile measurement systems commonly require cameras, lasers, position and orientation system and inertial measurement units; thus, the hardware cost is relatively expensive, and the system integration is complex. Therefore, in this paper, we propose a ground mobile measurement system only composed of a LiDAR and a GoPro camera, providing a more convenient and reliable way to automatically obtain 3D point cloud data with spectral information. The automatic point cloud coloring based on video images mainly includes four aspects: (1) Establishing models for radial distortion and tangential distortion to correct video images. (2) Establishing a registration method based on normalized Zernike moments to obtain the exterior orientation elements. The error of the result is only 0.5–1 pixel, which is far higher than registration based on a collinearity equation. (3) Establishing relative orientation based on essential matrix decomposition and nonlinear optimization. This involves uniformly using the speeded-up robust features algorithm with distance restriction and random sample consensus to select corresponding points. The vertical parallax of the stereo image pair model is less than one pixel, indicating that the accuracy is high. (4) A point cloud coloring method based on Gaussian distribution with central region restriction is adopted. Only pixels within the central region are considered valid for coloring. Then, the point cloud is colored based on the mean of the Gaussian distribution of the color set. In the colored point cloud, the textures of the buildings are clear, and targets such as windows, grass, trees, and vehicles can be clearly distinguished. Overall, the result meets the accuracy requirements of applications such as tunnel detection, street-view modeling and 3D urban modeling.

Reconstruction of geometric structures from images using supervised learning suffers from limited available amount of accurate data. One type of such data is accurate real-world RGB-D images. A major challenge in acquiring such ground truth data is the accurate alignment between RGB images and the point cloud measured by a depth scanner. To overcome this difficulty, we consider a differential optimization method that aligns a colored point cloud with a given color image through iterative geometric and color matching. In the proposed framework, the optimization minimizes the photometric difference between the colors of the point cloud and the corresponding colors of the image pixels. Unlike other methods that try to reduce this photometric error, we analyze the computation of the gradient on the image plane and propose a different direct scheme. We assume that the colors produced by the geometric scanner camera and the color camera sensor are different and therefore characterized by different chromatic acquisition properties. Under these multimodal conditions, we find the transformation between the camera image and the point cloud colors. We alternately optimize for aligning the position of the point cloud and matching the different color spaces. The alignments produced by the proposed method are demonstrated on both synthetic data with quantitative evaluation and real scenes with qualitative results.

… Coloured point clouds are … of point clouds acquired with Mobile Laser Scanning are due to perspective in the camera image, different resolution or poor calibration between the LiDAR …

Registration of 3D lidar point clouds with optical images is critical in the combination of multisource data. Geometric misalignment originally exists in the pose data between lidar point clouds and optical images. To improve the accuracy of the initial pose and the applicability of the integration of 3D points and image data, we develop a simple but efficient registration method. We first extract point features from lidar point clouds and images: point features are extracted from single-frame lidar and point features are extracted from images using a classical Canny operator. The cost map is subsequently built based on Canny image edge detection. The optimization direction is guided by the cost map, where low cost represents the desired direction, and loss function is also considered to improve the robustness of the proposed method. Experiments show positive results.

… Here, we call misalignment that can not be removed by RGB-… ; c is the colorized depth map for better visualization; d … point cloud that was automatically generated from the depth …

… , addressing the dimensional mismatch between LiDAR and visual measurements to … real-time fusion coloring strategy to color the LiDAR point cloud. Algorithm 1 clearly outlines …

Highlights What are the main findings? We developed an end-to-end pipeline combining multi-camera fusion with automated and targetless LiDAR–camera calibration to achieve full 360° real-time colourisation. Added colour correction and low-light enhancement modules recovered scene details at illumination as low as 0.5 lx, comparable to well-lit conditions. What is the implication of the main finding? A hardware-agnostic solution is provided for reliable monitoring and mapping in low-light environments, including underground mines and night-time navigation. Deployment is simplified with improvements for LiDAR interpretability for applications such as autonomous navigation, geological surveys, and vegetation analysis. Abstract In recent years, the fusion of camera data with LiDAR measurements has emerged as a powerful approach to enhance spatial understanding. This study introduces a novel, hardware-agnostic methodology that generates colourised point clouds from mechanical LiDAR using multiple camera inputs, providing complete 360-degree coverage. The primary innovation lies in its robustness under low-light conditions, achieved through the integration of a low-light image enhancement module within the fusion pipeline. The system requires initial calibration to determine intrinsic camera parameters, followed by automatic computation of the geometric transformation between the LiDAR and cameras—removing the need for specialised calibration targets and streamlining the setup. The data processing framework uses colour correction to ensure uniformity across camera feeds before fusion. The algorithm was tested using a Velodyne Puck Hi-Res LiDAR and a four-camera configuration. The optimised software achieved real-time performance and reliable colourisation even under very low illumination, successfully recovering scene details that would otherwise remain undetectable.

Even given the initial extrinsic of LiDAR and camera, high-precision automatic alignment between the Lidar scans and pictures acquired by mobile measurement systems (MMS) remains challenging due to clock asynchrony between sensors during data acquisition. In addition, the majority of images and mobile LiDAR Data registration methods are proposed for panoramic image sequences. A few research works have been focused on multi-lens combined images. We proposed an automated alignment approach for mobile lidar scans and multi-camera combined pictures using the initial poses of the images. First, we calibrate the extrinsic of the multi-lens combined camera and LiDAR sensor. Then, the odometry is obtained by the LiDAR SLAM algorithm, and the initial pose of each image is obtained. Then the sparse feature point cloud with the same scale as the lidar scan is obtained through the SfM algorithm with image initial poses. Finally, the iterative closest point (ICP) method is used to register the LiDAR scene points set with the image feature points set. The method is qualitatively and quantitatively evaluated on the indoor and outdoor data sets gathered from the self-developed mobile measurement platform. The alignment error of the approach suggested in this work is less than 1 pixel.

… to detect indications of boresighting misalignment, and tall walls … of point clouds using the SBET files, colorization of point clouds using imagery, evaluation and correction of point clouds …

Large-scale terrestrial laser scanning (TLS) point clouds are increasingly used for applications such as digital twins and cultural heritage documentation; however, removing unwanted human points captured during acquisition remains a largely manual and time-consuming process. This study proposes a geometry-aware framework for automatically removing human noise from TLS point clouds by projecting 2D instance segmentation masks (obtained using You Only Look Once (YOLO) v8 with an instance segmentation head) into 3D space and validating candidates through multi-stage geometric filtering. To suppress false positives induced by reprojection misalignment and planar background structures (e.g., walls and ground), we introduce projection-followed geometric validation (or “geometric gating”) using Density-Based Spatial Clustering of Applications with Noise (DBSCAN) and principal component analysis (PCA)-based planarity analysis, followed by cluster-level plausibility checks. Experiments were conducted on two real-world outdoor TLS datasets—(i) Osaka Metropolitan University Sugimoto Campus (OMU) (82 scenes) and (ii) Jinaimachi historic district in Tondabayashi (JM) (68 scenes). The results demonstrate that the proposed method achieves high noise removal accuracy, obtaining precision/recall/intersection over union (IoU) of 0.9502/0.9014/0.8607 on OMU and 0.8912/0.9028/0.8132 on JM. Additional experiments on mobile mapping system (MMS) data from the Waymo Open Dataset demonstrate stable performance without parameter recalibration. Furthermore, quantitative and qualitative comparisons with representative time-series geometric dynamic object removal methods, including DUFOMap and BeautyMap, show that the proposed approach maintains competitive recall under a human-only ground-truth definition while reducing over-removal of static structures in TLS scenes, particularly when humans are observed in only one or a few scans due to limited revisit frequency. The end-to-end processing time with YOLOv8 was 935.62 s for 82 scenes (11.4 s/scene) on OMU and 571.58 s for 68 scenes (8.4 s/scene) on JM, supporting practical efficiency on high-resolution TLS imagery. Ablation studies further clarify the role of each stage and indicate stable performance under the observed reprojection errors. The annotated human point cloud dataset used in this study has been publicly released to facilitate reproducibility and further research on human noise removal in large-scale TLS scenes.

… In the end, our point cloud coloring effect fits well with the actual image color. The reprojection error … is also only 0.135, which provides better visualization of point cloud data processing. …

Common single-line 2D LiDAR sensors and cameras have become core components in the field of robotic perception due to their low cost, compact size, and practicality. However, during the data fusion process, the randomness and complexity of real industrial scenes pose challenges. Traditional calibration methods for LiDAR and cameras often rely on precise targets and can accumulate errors, leading to significant limitations. Additionally, the semantic fusion of LiDAR and camera data typically requires extensive projection calculations, complex clustering algorithms, or sophisticated data fusion techniques, resulting in low real-time performance when handling large volumes of data points in dynamic environments. To address these issues, this paper proposes a semantic fusion algorithm for LiDAR and camera data based on contour and inverse projection. The method has two remarkable features: (1) Combined with the ellipse extraction algorithm of the arc support line segment, a LiDAR and camera calibration algorithm based on various regular shapes of an environmental target is proposed, which improves the adaptability of the calibration algorithm to the environment. (2) This paper proposes a semantic segmentation algorithm based on the inverse projection of target contours. It is specifically designed to be versatile and applicable to both linear and arc features, significantly broadening the range of features that can be utilized in various tasks. This flexibility is a key advantage, as it allows the method to adapt to a wider variety of real-world scenarios where both types of features are commonly encountered. Compared with existing LiDAR point cloud semantic segmentation methods, this algorithm eliminates the need for complex clustering algorithms, data fusion techniques, and extensive laser point reprojection calculations. When handling a large number of laser points, the proposed method requires only one or two inverse projections of the contour to filter the range of laser points that intersect with specific targets. This approach enhances both the accuracy of point cloud searches and the speed of semantic processing. Finally, the validity of the semantic fusion algorithm is proven by field experiments.

… laser transmitter, and laser reflected by the object are received and imaged by the camera, then pixels of laser are detected though image processing … model and laser plane function, …

This article presents a multi-sensor calibration and fusion work for a dense reconstruction and colorization system. Unlike previous research, it explicitly addresses the edge effect in LiDAR measurements in both calibration and sensor fusion. The proposed calibration method can accomplish LiDAR-motor and LiDAR-camera calibration at a one-time experiment. The feature extraction algorithm for the point cloud automatically eliminates the edge-effect-induced noises in the calibration process. The experiment results show that it can achieve even better results than work without edge effect elimination in terms of calibration errors. As a result, it has achieved a calibration accuracy superior to previous researches. The system is testified by an indoor scene, which shows a desirable color point cloud generation with dense and clean data.

In autonomous driving systems, sensor-based environmental perception is paramount. However, in long-distance perception for rail transit, the extrinsic calibration of LiDAR and telephoto cameras is hindered by sparse point clouds and intrinsic parameter inaccuracies. To address these challenges, we propose a novel calibration board design and a corresponding joint extrinsic calibration method. Inspired by engineering positioning principles, this calibration board builds upon the traditional checkerboard by integrating circular positioning holes. By coupling spatial re-projection constraints with geometric feature alignment, the proposed approach markedly improves feature point extraction and 2D–3D correspondences. Experimental results demonstrate that the method substantially enhances both calibration accuracy and efficiency, offering solid technical support for environmental perception in rail transit.

Accurate fusion of three-dimensional (3D) LiDAR point clouds with two-dimensional (2D) RGB images is essential for reliable perception in autonomous systems and mobile robotics. This paper presents a robust and generalizable LiDAR–camera extrinsic calibration framework based on probabilistic modeling and optimization on SE(3)(3). Camera intrinsics are estimated using checkerboard-driven calibration, while extrinsics are computed from 3D–2D correspondences using a PnP-initialized Expectation–Maximization (EM) formulation combined with robust Gauss–Newton refinement. The method supports heterogeneous LiDAR and camera configurations, including Velodyne VLP-16/32/64 and Intel RealSense D435/D455, without relying on fixed sensor baselines. Quantitative evaluation on the KITTI dataset and a custom ROS-based platform shows average reprojection errors between 0.42–0.78 px and depth alignment errors of 2.1–4.8 cm across multiple setups. The projection-based fusion yields geometrically consistent depth-enhanced overlays, demonstrating robustness, adaptability, and suitability for real-time deployment in multi-sensor perception systems.

Extrinsic calibration of multiview camera systems typically uses limited local simultaneous co-observations to achieve global optimization. Conventional approaches face two key shortcomings: when local simultaneous co-observations involve multiple viewpoints, the inability to exploit co-observation relationships leads to modeling gaps and decreased optimization stability; when explicit loop-closure constraints are insufficient, the effectiveness of global optimization is limited, resulting in reduced overall calibration accuracy. To address these issues, this article presents a sparse graph reprojection-based extrinsic calibration method for multicamera systems. The method first performs local optimization based on the hand–eye calibration principle, followed by global optimization by integrating a pose graph with bundle adjustment (BA). By uniformly modeling the sparse simultaneous co-observation relationships in local measurements and concurrently establishing reprojection constraints within the sparse pose graph, overall robustness increases. Additionally, a target pair reuse mechanism is introduced to explicitly compensate for missing loop-closure constraints in chain-like configurations, ensuring globally consistent solutions through graph optimization. Simulation experiments evaluated the calibration accuracy and stability of the proposed method for all cameras, while real-world tests assessed its calibration performance for the camera without loop closures, demonstrating its feasibility and effectiveness.

… ” scanning pattern of MLS provides 360 scanning for seamless and complete point clouds data … be speed up by multithreading processing or simultaneous processing of separated road …

We present a new deep point cloud rendering pipeline through multi-plane projections. The input to the network is the raw point cloud of a scene and the output are image or image sequences from a novel view or along a novel camera trajectory. Unlike previous approaches that directly project features from 3D points onto 2D image domain, we propose to project these features into a layered volume of camera frustum. In this way, the visibility of 3D points can be automatically learnt by the network, such that ghosting effects due to false visibility check as well as occlusions caused by noise interferences are both avoided successfully. Next, the 3D feature volume is fed into a 3D CNN to produce multiple planes of images w.r.t. the space division in the depth directions. The multi-plane images are then blended based on learned weights to produce the final rendering results. Experiments show that our network produces more stable renderings compared to previous methods, especially near the object boundaries. Moreover, our pipeline is robust to noisy and relatively sparse point cloud for a variety of challenging scenes.

… refine occlusion handling, by creating detailed depth maps … pre-processing to get very pleasing point clouds renderings. … texture mapping of point sprites. Similarly, we approximate the …

… with ambient occlusion in the middle and textured on the right… Finally a texture map is computed from the input images (see … not considered in the occlusion handling. A quite advanced …

Mobile mapping systems integrate multiple sensors to collect large volumes of geospatial data in motion. With the growing demand for mapping enclosed and hard-to-reach areas, there has been significant advancement in handheld mobile mapping systems utilizing SLAM (Simultaneous Localization and Mapping) technology. To ensure theirdata is efficiently usable, these systems should produce oriented images, which are essential for visualization, point cloud colorization, and monoplotting. Achieving so requiresprecise extrinsic camera calibration. This research provides a comprehensive overview of existing extrinsic camera calibration methods and evaluates their suitability for applicationin handheld mobile mapping systems. The goal is to identify a method that meets the accuracy and practical needs of these systems, facilitating more effective data processingand utilization in challenging environments.

Mobile Mapping is an efficient technology to acquire spatial data of the environment. The spatial data is fundamental for applications in crisis management, civil engineering or autonomous driving. The extrinsic calibration of the Mobile Mapping System is a decisive factor that affects the quality of the spatial data. Many existing extrinsic calibration approaches require the use of artificial targets in a time-consuming calibration procedure. Moreover, they are usually designed for a specific combination of sensors and are, thus, not universally applicable. We introduce a novel extrinsic self-calibration algorithm, which is fully automatic and completely data-driven. The fundamental assumption of the self-calibration is that the calibration parameters are estimated the best when the derived point cloud represents the real physical circumstances the best. The cost function we use to evaluate this is based on geometric features which rely on the 3D structure tensor derived from the local neighborhood of each point. We compare different cost functions based on geometric features and a cost function based on the Rényi quadratic entropy to evaluate the suitability for the self-calibration. Furthermore, we perform tests of the self-calibration on synthetic and two different real datasets. The real datasets differ in terms of the environment, the scale and the utilized sensors. We show that the self-calibration is able to extrinsically calibrate Mobile Mapping Systems with different combinations of mapping and pose estimation sensors such as a 2D laser scanner to a Motion Capture System and a 3D laser scanner to a stereo camera and ORB-SLAM2. For the first dataset, the parameters estimated by our self-calibration lead to a more accurate point cloud than two comparative approaches. For the second dataset, which has been acquired via a vehicle-based mobile mapping, our self-calibration achieves comparable results to a manually refined reference calibration, while it is universally applicable and fully automated.

… technique is studied for automatic alignment of the ranging pole. … the extrinsic calibration process of a 2 MP Scorpion Pt Grey color camera mounted on a high accuracy mobile mapping …

With the rapid development of autonomous driving, robotics, and intelligent transportation, multi-sensor-based environment sensing technology for intelligent vehicles has become a popular research direction. In order to better fuse the data acquired by multi-sensors, accurate external parameter calibration becomes one of the critical issues. According to the method of external parameter calibration, this paper first introduces the offline calibration technology based on target and targetless methods. However, once these two methods change the relative position between the camera and the LiDAR, it can only be returned to the field to re-calibrate. The computational complexity is high, which makes it necessary to use the online calibration directly. Hence, this paper follows up with the introduction of online calibration technology based on deep learning. Unlike previous methods that need to extract features from calibration boards or environments, various types of networks can directly learn the mapping relationship between images and point clouds, From the calibration results, the average error of translation and rotation of traditional methods can reach 0.34cm and 0.45°, the average error of using deep learning networks such as LCCNet, which is the most widely used in existing networks and has good calibration effect, can reach 0.297cm and 0.017°. Compared with the traditional method, the accuracy of online calibration technology is respectively improved by 12.6% and 96.2%, which shows the results of online calibration technology are better than the traditional offline method, and there are some recently proposed methods incorporate an attention mechanism and use an optimization algorithm instead of a loss function to refine the outer parameters. From the review, learning the relative relationships between sensors through neural networks works best, and the process is relatively free of human intervention. Contrary to the existing reviews, this paper provides a general structure of calibration methods universally used in various environments and compares various methods based on this general structure.

… and combined sparse map at each stage of the extrinsic calibration process in figure 8. We observe that the final sparse maps are much more well-defined and well-aligned to one …

… extrinsic calibration of the sensors is obtained. Our approach extends standard techniques for motion-based calibration … it now becomes possible to align lidar and cameras using only …

Line scanning cameras, which capture only a single line of pixels, have been increasingly used in ground based mobile or robotic platforms. In applications where it is advantageous to directly georeference the camera data to world coordinates, an accurate estimate of the camera’s 6D pose is required. This paper focuses on the common case where a mobile platform is equipped with a rigidly mounted line scanning camera, whose pose is unknown, and a navigation system providing vehicle body pose estimates. We propose a novel method that estimates the camera’s pose relative to the navigation system. The approach involves imaging and manually labelling a calibration pattern with distinctly identifiable points, triangulating these points from camera and navigation system data and reprojecting them in order to compute a likelihood, which is maximised to estimate the 6D camera pose. Additionally, a Markov Chain Monte Carlo (MCMC) algorithm is used to estimate the uncertainty of the offset. Tested on two different platforms, the method was able to estimate the pose to within 0.06 m/1.05° and 0.18 m/2.39°. We also propose several approaches to displaying and interpreting the 6D results in a human readable way.

… We illustrate the convexity of the objective function and discuss the convergence of … extrinsic calibration method that aligns edges in the thermal/RGB images with the edges in the LIDAR…

… georeferenced mobile systems and it can estimate the extrinsic … do not address the extrinsic calibration of the sensors with … evaluated in the mapping frame (3-D alignment) as well as …

… parameters by finding the geometric constraint that align the … of alignment of manually extracted feature correspondences: … often used calibration object for extrinsic calibration of LRF …

… In order to correctly combine measurements from both sensors, it is required to know their relative pose, that is, to solve their extrinsic calibration. In this paper we present a new …

… In this paper, we introduce a two-step method to register images to large-… registration by iteratively registering the real image to a synthetic image obtained from the estimated camera …

Image and point cloud registration (2D-3D registration) is an essential prerequisite for multi-modal feature fusion. However, due to the significant feature difference of point cloud and image, it is challenging to establish 2D-3D correspondences. Targeting for the background of autonomous driving, we propose 2D-3D registration method with object-level correspondence (OL-Reg) in this paper. Object-level correspondence consists of object bounding box and object contour in 2D image and 3D space. The first step is to match 2D-3D objects. Due to sensor pose and field of view (FoV) difference, object shape and occlusion is different in image and point cloud, causing the difficulty of object matching. To solve this issue, we represent object as 3D bounding box, and design 2D-3D object matching with 3D box projection (Box-Proj) constraint. It aligns object 3D bounding box in image and point cloud. After that, the next step is to build 2D-3D correspondence from the matched objects. To extract correspondence from object with irregular shape, we notice the distance constraint of object surface and rays back-projected from object contour, and present projection based iterative closest point (Proj-ICP). Towards the stability of Proj-ICP, object-level regularization term is designed. Experiment is conducted in KITTI object and odometry dataset. With the pre-trained 3D object detector, results suggest that OL-Reg has the better performance than current approaches in tasks of re-localization and extrinsic calibration. Source code will be released at https://github.com/anpei96/ol-reg-demo.

The fusion of light detection and ranging (LiDAR) and camera data is a promising approach to improve the environmental perception and recognition for intelligent vehicles because of the combination of depth and color information. One of the difficulties in achieving the fusion is the accurate alignment of the 3-D points with the image pixels. Current methods of data alignment involve the steps of estimating the camera intrinsic parameters and developing a transformation matrix between the camera and LiDAR frame. The drawback of these methods is the accumulation of errors during the calculation of the camera intrinsic parameters and the transformation matrix. In order to improve the data alignment accuracy, we propose a novel algorithm that directly calculates the alignment between the 3-D points and the pixels without the need for camera parameters and calibration of the coordinate transformation matrix. We call the proposed method the pixel and 3-D point alignment (PPA) method. The alignment procedure is achieved by using the extracted corresponding points. First, we calculate a linear alignment matrix without considering the image distortion; and second, we optimize the parameters using the maximum likelihood estimation to consider the camera distortion. Simulation and experimental results indicate that the PPA method is able to align the 3-D points in LiDAR frame with the pixels in image frame with higher accuracy and increased robustness against noise in calibration process than comparable state-of-the-art methods.

To assist in the implementation of a fine 3D terrain reconstruction of the scene in remote sensing applications, an automatic joint calibration method between light detection and ranging (LiDAR) and visible light camera based on edge points refinement and virtual mask matching is proposed in this paper. The proposed method is used to solve the problem of inaccurate edge estimation of LiDAR with different horizontal angle resolutions and low calibration efficiency. First, we design a novel calibration target, adding four hollow rectangles for fully automatic locating of the calibration target and increasing the number of corner points. Second, an edge refinement strategy based on background point clouds is proposed to estimate the target edge more accurately. Third, a two-step method of automatically matching between the calibration target in 3D point clouds and the 2D image is proposed. Through this method, i.e., locating firstly and then fine processing, corner points can be automatically obtained, which can greatly reduce the manual operation. Finally, a joint optimization equation is established to optimize the camera’s intrinsic and extrinsic parameters of LiDAR and camera. According to our experiments, we prove the accuracy and robustness of the proposed method through projection and data consistency verifications. The accuracy can be improved by at least 15.0% when testing on the comparable traditional methods. The final results verify that our method is applicable to LiDAR with large horizontal angle resolutions.

Non-repetitive scanning Light Detection And Ranging(LiDAR)-Camera systems are commonly used in autonomous navigation industries, benefiting from their low-cost and high-perception characteristics. However, due to the irregular scanning pattern of LiDAR, feature extraction on point cloud encounters the problem of non-uniformity distribution of density and reflectance intensity, accurate extrinsic calibration remains a challenging task. To solve this problem, this paper presented an open-source calibration method using only a printed chessboard. We designed a two-stage coarse-to-fine pipeline for 3D corner extraction. Firstly, a Gaussian Mixture Model(GMM)-based intensity cluster approach is proposed to adaptively identify point segments in different color blocks of the chessboard. Secondly, a novel Iterative Lowest-cost Pose(ILP) algorithm is designed to fit the chessboard grid and refine the 3D corner iteratively. This scheme is unique for turning the corner feature extraction problem into a grid align problem. After the corresponding 3D-2D points are solved, by applying the PnP(Perspective-n-Point) method, along with nonlinear-optimization refinement, the extrinsic parameters are obtained. Extensive simulation and real-world experimental results show that our method achieved subpixel-level precision in terms of reprojection error. The comparison demonstrated that the effectiveness and accuracy of the proposed method outperformed existing methods.

… laser beam. This, however, is not the same as the perceived color of an object as acquired by a regular digital 2D camera… points and 3D directions, a refined focal length and a rotation. …

… We assume the intrinsic camera … the camera registration problem in two steps as depicted on Fig.1. In the first step, we obtain coarse camera parameters which are further refined in the …

Calibration between color camera and 3D Light Detection And Ranging (LIDAR) equipment is an essential process for data fusion. The goal of this paper is to improve the calibration accuracy between a camera and a 3D LIDAR. In particular, we are interested in calibrating a low resolution 3D LIDAR with a relatively small number of vertical sensors. Our goal is achieved by employing a new methodology for the calibration board, which exploits 2D-3D correspondences. The 3D corresponding points are estimated from the scanned laser points on the polygonal planar board with adjacent sides. Since the lengths of adjacent sides are known, we can estimate the vertices of the board as a meeting point of two projected sides of the polygonal board. The estimated vertices from the range data and those detected from the color image serve as the corresponding points for the calibration. Experiments using a low-resolution LIDAR with 32 sensors show robust results.

A precise RGB-colored point cloud map serves as a reliable data foundation and a compact multi-sensor representation for tasks such as 3D perception, localization, and navigation. Therefore, efficient and accurate alignment between camera and LiDAR is essential. However, most existing color mapping methods require precise time synchronization and well-calibrated extrinsic value, and these colorizing frameworks are often hard to use. In this paper, we propose EasyColor, a precise colorization method that jointly leverages LiDAR reflectivity and visual constraint, effectively removing the need for time synchronization and extrinsic calibration between the two sensors. Our method follows a plug-and-play framework. Given an image sequence and a pre-built point cloud map, we first generate 2D reflectivity image sequence by exploiting variations in LiDAR reflectivity intensity. A cross-domain matching network, combined with a Perspective-n-Point (PnP) solver, is employed to align the reflectivity and camera image sequence. To further optimize camera poses and ensure photometric consistency, we fuse visual epipolar geometric constraint and PnP constraint. Our joint optimization produces both accurate and photometrically consistent colored map, along with optimized camera poses. Extensive experiments demonstrate that EasyColor achieves precise point cloud colorization without requiring synchronized LiDAR and camera data and extrinsic values. To facilitate future research, we will release EasyColor and dataset after review stage.

In recent years, multi-sensor fusion technology has made enormous progress in 3D reconstruction, surveying and mapping, autonomous driving, and other related fields, and extrinsic calibration is a necessary condition for multi-sensor fusion applications. This paper proposes a 3D LIDAR-to-camera automatic calibration framework based on graph optimization. The system can automatically identify the position of the pattern and build a set of virtual feature point clouds, and can simultaneously complete the calibration of the LIDAR and multiple cameras. To test this framework, a multi-sensor system is formed using a mobile robot equipped with LIDAR, monocular and binocular cameras, and the pairwise calibration of LIDAR with two cameras is evaluated quantitatively and qualitatively. The results show that this method can produce more accurate calibration results than the state-of-the-art method. The average error on the camera normalization plane is 0.161 mm, which outperforms existing calibration methods. Due to the introduction of graph optimization, the original point cloud is also optimized while optimizing the external parameters between the sensors, which can effectively correct the errors caused during data collection, so it is also robust to bad data.

… LiDAR sensing system … mobile mapping systems or civil LiDAR-equipped cars can be calibrated on a regular basis using a dedicated calibration setup, we aim at a method for automatic …

Abstract. The representation of 3D geometric and photometric information of the real world is one of the most challenging and extensively studied research topics in the photogrammetry and robotics communities. In this paper, we present a fully automatic framework for 3D high quality large scale urban texture mapping using oriented images and LiDAR scans acquired by a terrestrial Mobile Mapping System (MMS). First, the acquired points and images are sliced into temporal chunks ensuring a reasonable size and time consistency between geometry (points) and photometry (images). Then, a simple, fast and scalable 3D surface reconstruction relying on the sensor space topology is performed on each chunk after an isotropic sampling of the point cloud obtained from the raw LiDAR scans. Finally, the algorithm proposed in (Waechter et al., 2014) is adapted to texture the reconstructed surface with the images acquired simultaneously, ensuring a high quality texture with no seams and global color adjustment. We evaluate our full pipeline on a dataset of 17 km of acquisition in Rouen, France resulting in nearly 2 billion points and 40000 full HD images. We are able to reconstruct and texture the whole acquisition in less than 30 computing hours, the entire process being highly parallel as each chunk can be processed independently in a separate thread or computer.

Handheld mobile scanning systems usually do not integrate high-precision position and orientation system (POS), and the traditional 3D point cloud and 2D image coloring lack a unified spatial reference. This letter proposes a method for registration and coloring of 3D point cloud and panoramic image obtained by handheld simultaneous localization and mapping (SLAM) system. Firstly, the initial registration parameters are generated by slam trajectory coordinate and pose interpolation. Secondly, the point cloud is converted into depth map according to the spherical 3D-2D imaging model, and then the optimal registration parameters between depth map and panorama are calculated based on the mutual information maximization. Finally, the multi segment point cloud is colored in batch by using optimal parameters, generating the color point cloud of the complete scene. The experimental results show that our method can realize the coloring of point clouds in different scenes such as indoor and outdoor without POS, and is suitable for the data fusion application of handheld scanning point clouds and panoramic images.

… the image … color point-cloud with panoramic image and laser point-cloud, and deduce the equation of the correspondence between points in panoramic images and laser point-clouds .…

… panoramic images and MLS point clouds based on the matching of pole objects. Firstly, 2D pole instances in the panoramic images … Secondly, every corresponding frustum point cloud …

The Remote sensing application plays a major role in real-world critical application projects. The research introduces a novel approach, "3D Point-Cloud Processing Using Panoramic Images for Object Detection," aimed at enhancing the interpretability of laser point clouds through the integration of color information derived from panoramic images. Focusing on the context of Mobile Measurement Systems (MMS), where various digital cameras are utilized, the work addresses the challenges associated with processing panoramic images offering a 360-degree view angle. The core objective is to develop a robust method for generating color point clouds by establishing a mathematical correspondence between panoramic images and laser point clouds. The collinear principle of three points guides the fusion process, involving the center of the omnidirectional multi-camera system, the image point on the sphere, and the object point. Through comprehensive experimental validation, the work confirms the accuracy of the proposed algorithm and formulas, showcasing its effectiveness in generating color point clouds within MMS. This research contributes to the present development of 3D point-cloud processing, introducing a contemporary methodology for improved object detection through the fusion of panoramic images and laser point clouds.

As the combination of panoramic images and laser point clouds becomes more and more widely used as a technique, the accurate determination of external parameters has become essential. However, due to the relative position change of the sensor and the time synchronization error, the automatic and accurate matching of the panoramic image and the point cloud is very challenging. In order to solve this problem, this paper proposes an automatic and accurate registration method for panoramic images and point clouds of urban large scenes based on line features. Firstly, the multi-modal point cloud line feature extraction algorithm is used to extract the edge of the point cloud. Based on the point cloud intensity orthoimage (an orthogonal image based on the point cloud’s intensity values), the edge of the road markings is extracted, and the geometric feature edge is extracted by the 3D voxel method. Using the established virtual projection correspondence for the panoramic image, the panoramic image is projected onto the virtual plane for edge extraction. Secondly, the accurate matching relationship is constructed by using the feature constraint of the direction vector, and the edge features from both sensors are refined and aligned to realize the accurate calculation of the registration parameters. The experimental results show that the proposed method shows excellent registration results in challenging urban scenes. The average registration error is better than 3 pixels, and the root mean square error (RMSE) is less than 1.4 pixels. Compared with the mainstream methods, it has advantages and can promote the further research and application of panoramic images and laser point clouds.

This paper presents a deterministic system that combines textual semantic data from panoramic images with LiDAR point clouds in a mobile mapping setup. Urban scenes often include textual elements, such as signs and business names, that provide key details typically missing from LiDAR-based urban digital twins. The presented method uses deep learning-based OCR to extract text from street panoramas and then categorizes it into urban types using a rule-based classifier. Text regions are geometrically projected into the LiDAR environment by converting image coordinates into viewing rays that intersect LiDAR surfaces, such as facades. Data from multiple panoramas are merged with confidence-weighted spatial clustering to produce consistent semantic markers for urban features. Extracted business names enable text-based searches of the LiDAR point cloud, allowing facility location by category, keyword, or brand. Tests on datasets from European and U.S. cities support plausible facade-level localization and demonstrate the framework’s ability to enhance LiDAR point clouds with searchable semantic information. The main contribution is not a new standalone OCR or LiDAR-processing algorithm, but a deterministic multimodal integration framework that combines deep-learning OCR, geometric back-projection, and cross-view spatial fusion to convert street-level textual cues into reliable, queryable 3D semantic markers within mobile-mapping LiDAR data.

Registration between terrestrial LiDAR and optical imagery plays a crucial role in information fusion. However, it is difficult to find reliable correlations among the different feature information of optical imagery and LiDAR point clouds. Therefore, in order to achieve high-precision registration of heterogeneous sensors, a method based on spherical epipolar line and spherical absolute orientation is proposed in this paper. The method firstly projects the LiDAR point clouds into spherical images based on the spherical imaging model and derives the spherical epipolar line equation. Then the relative and absolute orientations of the spherical LiDAR images and the optical images are performed based on manually selected control points. Finally, based on Harris corner extraction, combined with the geometric constraints of the spherical epipolar line and absolute orientation, dense matching between optical and LIDAR images are achieved, and all matching points are used as control points for registration to improve the accuracy of manually selected points registration. Multiple sets of test data are acquired outdoors using a FARO Focus S laser scanner, a Z + F IMAGER 5010C laser scanner, and a Ladybug5+ panoramic camera. The experimental results show that the method in this paper is practical and improves the accuracy of manual points selection registration, and the degree of improvement is related to the number of successfully matched corner points.

We present two algorithms for aligning two colored point clouds. The two algorithms are designed to minimize a probabilistic cost based on the color-supported soft matching of points in a point cloud to their K-closest points in the other point cloud. The first algorithm, like prior iterative closest point algorithms, refines the pose parameters to minimize the cost. Assuming that the point clouds are obtained from RGB-depth images, our second algorithm regards the measured depth values as variables and minimizes the cost to obtain refined depth values. Experiments with our synthetic dataset show that our pose refinement algorithm gives better results compared to the existing algorithms. Our depth refinement algorithm is shown to achieve more accurate alignments from the outputs of the pose refinement step. Our algorithms are applied to a real-world dataset, providing accurate and visually improved results.

In the last stage of colored point cloud registration, depth measurement errors hinder the achievement of accurate and visually plausible alignments. Recently, an algorithm has been proposed to extend the Iterative Closest Point (ICP) algorithm to refine the measured depth values instead of the pose between point clouds. However, the algorithm suffers from numerical instability, so a postprocessing step is needed to restrict erroneous output depth values. In this paper, we present a new algorithm with improved numerical stability. Unlike the previous algorithm heavily relying on point-to-plane distances, our algorithm constructs a cost function based on an adaptive combination of two different projected distances to prevent numerical instability. We address the problem of registering a source point cloud to the union of the source and reference point clouds. This extension allows all source points to be processed in a unified filtering framework, irrespective of the existence of their corresponding points in the reference point cloud. The extension also improves the numerical stability of using the point-to-plane distances. The experiments show that the proposed algorithm improves the registration accuracy and provides high-quality alignments of colored point clouds.

We present a “3D environment detection using multi-view color images and LiDAR point clouds” in this paper. Combining “multi-view color images” and “LiDAR point clouds” to solve the problem of insufficient resolution of traditional LiDAR point clouds to further realize accurate three-dimensional environment detection and recognition. The system adopts eight cameras to capture multi-view panorama images and combines with the depth information of the LiDAR point clouds to establish high-resolution 3D color images and depth images for subsequent objects detection and segmentation. According to the 3D point clouds information and multi-view images, 3D dynamic environment preprocessing, data processing and data format conversion are performed, the neural network input signal is extracted. Then through the neural network training, to determine the three-dimensional environment, and cutting out obstacles such as cars and pedestrians.