A Convenient Calibration Method for LRF-Camera Combination Systems Based on a Checkerboard

Abstract

In this paper, a simple and easy high-precision calibration method is proposed for the LRF-camera combined measurement system which is widely used at present. This method can be applied not only to mainstream 2D and 3D LRF-cameras, but also to calibrate newly developed 1D LRF-camera combined systems. It only needs a calibration board to record at least three sets of data. First, the camera parameters and distortion coefficients are decoupled by the distortion center. Then, the spatial coordinates of laser spots are solved using line and plane constraints, and the estimation of LRF-camera extrinsic parameters is realized. In addition, we establish a cost function for optimizing the system. Finally, the calibration accuracy and characteristics of the method are analyzed through simulation experiments, and the validity of the method is verified through the calibration of a real system.

1. Introduction

In the field of measurements, a single sensor is seldom able to perform high-precision measurements by itself. Combined multi-sensor measurement schemes can effectively combine the characteristics of each sensor, leveraging the complementary advantages of sensors, and improving the accuracy and robustness of the measurement system. As [1] shows, laser range finders (LRFs) provide high-precision distance information, while camera can provide rich image information. The combination of LRFs and cameras has attracted wide attention, with interesting applications in navigation [2], human detection [3] and 3D texture reconstruction [4].

Compared with the current mainstream schemes combining scanning lasers and vision, the more challenging combination of 1-D laser ranging and vision has attracted the attention of researchers due to its low cost and wide applicability. The Shuttle Radar Topography Mission (SRTM) [5] realizes high-precision measurements of Interferometric Synthetic Aperture Radar (IFSAR) on long-range cooperative targets. Ordez [6] proposed a combination of camera and Laser Distance Meter (LDM) to estimate the length of a line segment in an unknown plane. Wu [7] applied this method to a visual odometry (VO) system and realized the application in a quasi-plane scene. In our previous work, we further extended this method and constructed a complete SLAM method based on laser-vision fusion [1].

Sensor calibration is the premise of data fusion, including the calibration of each sensor’s own parameters and the relationship of relative data between each sensor [8]. However, as a necessary prerequisite for high-precision measurements, the calibration technology of 1D laser-camera systems evolves seldom. The existing calibration algorithm based on scanning laser ranging has been unable to apply, but the traditional one-dimensional laser calibration algorithms require a high-precision manipulator laser interferometer and other complex equipment.

In this paper, a simple and feasible high-precision laser and visual calibration algorithm is proposed, which can calibrate the parameters of laser and camera sensors through only simple data processing. Firstly, the camera parameters and distortion coefficients are determined using a non-iterative method. Then, the coordinates of the laser spot in the camera coordinate system are obtained by inversion of the laser image points in the image, and the initial values of external camera and laser ranging parameters are estimated. Finally, the parameters are optimized through the parameterization of the rotation matrix [9] and the Gröbner basis method [10]. Compared with the existing methods, the main contributions of this paper are as follows:

(1)

The method proposed in this paper has wider applicability. It can be used for joint calibration of vision sensors and LRF from 1D to 3D.

(2)

Compared with existing 1D laser-vision calibration methods, the proposed method can be realized using a simple chessboard lattice, without complicated customized targets and high-precision mechanical structures.

(3)

The accuracy and usability of the proposed method are verified by simulation and observation experiments.

This paper is organized as follows: the existing methods related to our work are outlined in the following section. Section 3 and Section 4 describe the mathematical model and illustrate the proposed algorithm. In Section 5, we evaluate the solution of the simulation and observation experiments. Finally, conclusions and future are provided in Section 6.

2. Related Work

For the extrinsic parameters between LRF and vision sensors, it is helpful to combine the high-precision distance information of laser ranging with the high lateral resolution of vision to achieve high-precision pose estimation. However, this method is mostly used to calibrate 2D or 3D LRFs and cameras.

Vasconcelos [11] calibrated the camera-laser extrinsic parameters by moving a checkerboard freely. This method assumes that the internal parameters are known and accurate, and converts the external parameter calibration problem into a plane-coplanar alignment problem to reach an exact solution. Similar work includes Scaramuzza [12] and Ha [13]. Ranjith [14] realized the correlation and calibration of 3D LiDAR data and image data through feature point retrieval. Zhang [15] uses mobile LRF and visual camera to achieve self-calibration of their external parameters through motion constraints. Viejo [16] realized the correlation of the two sets of data by arranging control points, and calibrated the external parameters of 3D LiDAR and a monocular camera.

However, the above algorithms are mostly used to calibrate the external parameters of 2D or 3D scanning laser and vision systems, and cannot be used for 1D laser ranging without a scanning mechanism due to the lack of constraints. For the calibration of 1D LRF, the traditional method mostly realizes the correlation between the two by means of complex a manipulator or specific calibration target. For example, Zhu’s [17] calibration algorithm is used to calibrate the direction and position parameters of a laser range finder based on spherical fitting. The calibration accuracy is high, but the solution is highly customized and not universal. Lu [18] designed a multi-directional calibration block to calibrate the laser beam direction of a point laser probe on the platform of a coordinate measuring machine. Zhou [19] proposed a new calibration algorithm for serial coordinate measuring machines (CMMs) with cylindrical and conical surfaces as calibration objects. Similar calibration methods are used in the implementation of the LFR-camera slam method [1]. The relative rotation and translation of the two sensors’ coordinate systems are estimated through a high-precision laser tracker.

Although this method can achieve high accuracy, it requires the installation of sensors on precision measuring equipment, which has high calibration cost and complex operation, and cannot meet the needs of low-cost and fast landing scenarios such as existing robots. In 2010, Ordez [6] proposed a set method of cameras and LRF to measure short distances in the plane. In another study [20], the author introduces a preliminary calibration method for a digital camera and a laser rangefinder. The experiment involves the artificial adjustment of the projection center of the laser pointer, and only two laser projections are used. The accuracy and robustness of the calibration method are both problematic. After that, Wu et al. [7] proposed a two-part calibration method based on the Ransac scheme, and solved the corresponding linear equation in the image by creating the index table of laser spot. However, this method cannot be well applied to the case where the laser light is close to the optical axis of the camera, and the final accuracy evaluation criteria are not given.

Zhang [21] proposed a simple calibration method for camera intrinsic parameters, where the parameters were determined using a non-linear method, and high accuracy was achieved. Afterwards, based on Zhang’s framework, researchers improved accuracy and scene expansion by designing different forms of targets [22,23,24] and improving the calibration of the algorithm [25,26,27]. Hartly [28] introduced the distortion division model to correct the imaging distortion. On this basis, Hong [29] further explored the calibration method of large distortion cameras.

Currently, the calibration of omnidirectional cameras has attracted wide attention in order to improve the user’s degree of freedom and immersion in the virtual reality and autopilot. Li et al. [30] proposed a multi-phase camera calibration scheme based on random pattern calibration board. Their method supports the calibration of a camera system which comprise normal pinhole cameras. Gwon Hwan [31] proposed a new intrinsic calibration and extrinsic calibration method of omnidirectional cameras based on the Aruco marker and a Charuco board. The calibration structure and method can solve the problem of suing overly complicated procedures to accurately calibrate multiple cameras.

At the same time, the calibration board also plays an important role in the other calibration processs. Liu [32] studied different applications of lasers and cameras. The calibration method of multiple non-common-view cameras by scanning a laser rangefinder is proposed. In the literature, the correlation between laser distance information and camera images is established through a specific calibration plate, so as to realize the relative pose estimation between cameras. Inspired by Liu’s work [32], we establish a constraint of 1D laser and monocular vision by combining planar and coplanar constraints, so as to determine related external parameters. Considering that the camera imaging model has a direct impact on the calibration accuracy, we have improved Zhang‘s method [21] used in camera calibration by replacing the traditional polynomial model with the division distortion model, and solved the linear solution of the iterative optimization using variable least squares on the basis of Hartly [28] and Hong [29]. Thus, the problem of falling into local optimal solutions is avoided, and the calibration speed is greatly improved. Combining the above innovations, a convenient method for calibrating the parameters of the camera-laser measurement system is realized, which can complete the calibration of measurement systems, including camera internal parameters, distortion coefficients and camera-laser external parameters, in one operation.

3. Measurement Model

Previous researchers established relatively mature camera imaging and laser measurement models. We integrate the two mathematical models and construct a complete mathematical description of the coordinate system.

As shown in the Figure 1, $O_C$ is the camera coordinate system, $\overrightarrow{O_C{Z}_C}$ is the optical axis direction of the camera, $O-uv$ is the image plane of the camera, $O_T$ is the coordinate system of the target itself, point $P_l$ is the spatial position of the laser spot, point $P_w$ is the spatial coordinate of the target control point and $O_l$ represents the coordinate system of 1D laser ranging. We set the camera coordinate system $O_c$ as the measurement coordinate system $O_M$ of the system. In the next part, we introduce the imaging model of the monocular camera and the 1D laser ranging model, and convert and fuse the data through extrinsic parameters $[\begin{array}{cc}{R}_{l2C}& T_{l2C}\end{array}]$.

3.1. Camera Imaging Model

In order to describe the imaging process of a monocular camera more accurately, we combine the lens distortion model with the aperture imaging model and introduce the shift of the distortion center $e$ relative to the image center $O_P$ [28]. In the camera coordinate system, $O_p-xy$ is the physical coordinate system of the phase plane and $O-uv$ represents the image coordinate system. Image center Op denotes the intersection of the optical axis and the image plane.

The ideal imaging process can be described as the process of transforming a point $P_i^T$ ($X_i^T={[\begin{array}{cccc}{X}_i^T& Y_i^T& Z_i^T& 1\end{array}]}^T$) in the world coordinate system to the image plane imaging point $P_i^u$ ($x_i^u={[\begin{array}{ccc}{u}_i^u& v_i^u& 1\end{array}]}^T$) through a projection relationship. The mathematical expression is as follows:

$${\rho }_i{x}_i^u=A_C{T}_{T2C}{X}_i^T=\left[\begin{array}{ccc}{f}_u& s& u_0\\ 0& f_v& v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{cccc}{r}_{11}& r_{12}& r_{13}& t_1\\ r_{21}& r_{22}& r_{23}& t_2\\ r_{31}& r_{32}& r_{33}& t_3\end{array}\right]\left[\begin{array}{c}{X}_i^T\\ Y_i^T\\ Z_i^T\\ 1\end{array}\right]$$

(1)

where ${\rho }_i$ is a named depth scale factor, the intrinsic matrix and $A_C$ is described by a five-parameter model; $f_u,f_v$ are the focal lengths, ${\left[\begin{array}{cc}{u}_0& v_0\end{array}\right]}^T$ is the coordinate of the image center $O_P$, and $s$ is the skew coefficient. $T_{T2C}$ is the transformation matrix relating $O_T$ to $O_C$ and it can be expressed as a rotation matrix $R_{T2C}$ combined with the translation vector $t_{T2C}$.

Due to lens design and processing, the actual imaging process is distorted. We introduce a division distortion model to improve our imaging. The mathematical expressions are as follows:

$$x_i^u-e=\frac{x_i^d-e}{1+{\lambda }_1{\left[r_i^d\right]}^2+{\lambda }_2{\left[r_i^d\right]}^4+\dots }$$

(2)

$x_i^d$ represents the actual position of projection point $P_i^T$, and its coordinates are $x_i^d={[\begin{array}{ccc}{u}_i^d& v_i^d& 1\end{array}]}^T$; ${\lambda }_1$ and ${\lambda }_2$ are the distortion coefficients and $r_d$ represents the distance from point $x_i^d$ to the distortion center $e$, expressed as $r_i^d=\sqrt{{(u_i^d-d{u}_0)}^2+{(v_i^d-d{v}_0)}^2}$.

In order to illustrate the method more clearly, the most important parameters used in this paper and their meaning are shown in

Table 1. The parameter statement of the system.

	Parameter	Mean
Coordinats	$O_M$/$O_T$/${\mathrm{O}}_C$/$O_l$	measurement/target/camera/laser-ranging coordinate system
	$O-uv$	image plane coordinate system of camera
	$T_{T2C}$	transformation matrix relating $O_T$ to $O_C$
	$R_{l2M}$, $t_{l2M}$	Extrinsic matrix of $O_l$ to $O_C$
	$R_{T2C}$, $t_{T2C}$	Extrinsic matrix of $O_T$ to $O_C$
	$T_{e2\widehat{e}}$	transformation matrix relating the distortion center $e$ to new distortion center $\widehat{e}$
Imaging Geometry	$A_C$	intrinsic matrix
	${\lambda }_1$, ${\lambda }_2$	distortion coefficients
	$e$	distortion center, expressed as ${\left[\begin{array}{cc}d{u}_0& d{v}_0\end{array}\right]}^T$
	${\rho }_i$	depth scale factor
	$H$	homography matrix
	$\widehat{H}$	transformed homography matrix $H$
	$F_H$	fundamental matrix of distortion
	${\widehat{F}}_H$	transformed fundamental matrix of distortion
Variable	$X_i^T$	position of the corner in the target coordinate system,
	$x_i^u$	ideal position of projection point in image plane
	$x_i^d$	actual position of projection point in image plane
	${\widehat{x}}_i^d$	transformed image coordinates $x_i^d$
	$\mathbb{Q}(X^C,Y^C,Z^C)$/$\mathbb{Q}(\begin{array}{cc}{R}_{T2M},& t_{T2M}\end{array})$	plane equation of the target in camera coordinate system
	$ℂ(X^C,Y^C,Z^C)$	linear equation of the laser beam in the camera coordinate system
	$E(A_C,{\lambda }_1,{\lambda }_2,R_{T2M},t_{T2M})$	objective functions to be optimized
	$E_{dr}$	re-projection error

.

3.2. LRF Model

The mathematical model of the 1D laser ranging module is relatively simple. The laser ranging module can output single point laser distance information by observing the reflected signal and calculating the optical path using image coherence [33]. The mathematical determination of the origin coordinate and laser direction of the laser ranging module allows the coordinate of the laser in the measurement coordinate system. In order to better represent the measurement results in the system measurement coordinate system $O_M$, we set up the European three-dimensional coordinate system $O_l$ for the LRF module. The laser emission direction is $\overrightarrow{O_l{Z}_l}$, the directions of $\overrightarrow{O_l{X}_l}$ are perpendicular and parallel to the $O_C-xy$ plane, and the directions of $\overrightarrow{O_l{Y}_l}$ are determined by the right-hand rule, as shown in Figure 1. The measured distance information $d_i^l$ represents the distance from the origin $O_l$ to the laser spot $P_i^l$.

In the process of extrinsic parameter calibration, the coordinate origin $O_l$ of the laser ranging coordinate system and the laser emission direction $\overrightarrow{O_l{Z}_l}$ need to be calculated. Finally, the conversion relations between camera measurement system $O_M$ and the LRF coordinate system $O_l$ are estimated, the rotation matrix $R_{l2M}$ and the translation vector $t_{l2M}$ are determined.

4. Methodology

Calibration of the measurement system is the process of determining the model parameters of the measurement system. For our system, through the measurement and imaging of a specific target, the model parameters of the measurement system are determined using the corresponding relationship between the coordinates of the control points and the image coordinates. The main parameters are the intrinsic parameters of the camera and the extrinsic parameters of between LRF and camera.

The specific calibration process is divided into three main steps: (1) the estimation of the camera distortion center $e$; (2) the intrinsic parameters $A_C$ and distortion coefficients ${\lambda }_1{\lambda }_2$ are decoupled and determined independently; (3) finding the extrinsic parameters $\left[\begin{array}{cc}{R}_{l2M}& t_{l2M}\end{array}\right]$ for translating the laser-vision coordinate system to the measurement coordinate system; (4) determining the optimal solution $\left(A_C,{\lambda }_1,{\lambda }_2,\left[\begin{array}{cc}{R}_{l2M}& t_{l2M}\end{array}\right]\right)$ using the Gröbner basis method. In this section, we elaborate on the above.

4.1. The Center of Distortion

In many studies, it is usually assumed that the distortion center and the main point are in the same position, but Hartley [28] determined experimentally that there is a certain deviation between them. During the calibration process, we use a checkerboard as the calibration object, and extract the corners $P_i^T$ of the checkerboard as the control points for camera calibration. Since the corners are distributed on a plane, we set the $Z_i^T=0$ in the target coordinate system, in which case the imaging model can be expressed as:

$${\rho }_i{x}_i^u=P{X}_i^T=A[\begin{array}{cccc}{r}_1& r_2& r_3& t_1\end{array}]\left[\begin{array}{c}{X}_i^T\\ Y_i^T\\ 0\\ 1\end{array}\right]$$

(3)

where $\left[\begin{array}{ccc}{r}_1& r_2& r_3\end{array}\right]$ is the column vector of rotation matrix $R_{T2C}$. The above equation can be simplified as:

$${\rho }_i{x}_i^u=H{X}_i^T=A_C[\begin{array}{ccc} r_1& r_2& t\end{array}]\left[\begin{array}{c}{X}_i^T\\ Y_i^T\\ 1\end{array}\right]$$

(4)

Matrix $H$ called the homography matrix, and expresses the mapping relation between the corner of the checkerboard and the image points. The coordinates of $P_i^T$ are abbreviated as $X_i^T={\left[\begin{array}{ccc}{X}_i^T& Y_i^T& 1\end{array}\right]}^T$.

From the division model of Equation (2), we obtain:

$$x_i^d=e+k_i\left(x_i^u-e\right),k_i=1+{\lambda }_1{\left[r_i^d\right]}^2+{\lambda }_2{\left[r_i^d\right]}^4+\dots$$

(5)

We multiply the left side of the equations by ${\left[e\right]}_{\times }$ and combine it with Equation (4). In consideration of ${\left[e\right]}_{\times }e=0$:

$${\left[e\right]}_{\times }{x}_i^d=k_i{\left[e\right]}_{\times }H{X}_i^T,{\left[e\right]}_{\times }=\left[\begin{array}{ccc}0& -1& d{v}_0\\ 1& 0& -d{u}_0\\ -d{v}_0& d{u}_0& 0\end{array}\right]$$

(6)

We then multiply the left sides of the equations by ${\left[x_i^d\right]}^T$ and obtain:

$${\left[x_i^d\right]}^T{\left[e\right]}_{\times }H{X}_i^T=0$$

(7)

Let $F_H={\left[e\right]}_{\times }H$. $F_H$ is called the fundamental matrix of distortion and is expressed as follows:

$${\left[x_i^d\right]}^T{F}_H{X}_i^T=0, F_H=\left[\begin{array}{ccc}{F}_{11}& F_{12}& F_{13}\\ F_{21}& F_{22}& F_{23}\\ F_{31}& F_{32}& F_{33}\end{array}\right]$$

(8)

We can solve the values of the fundamental matrix $F_H$ using 8 pairs of corresponding corner points. The equation can be formulated as:

$$A{f}_H=0$$

(9)

where:

$$\begin{array}{l}A=\left[\begin{array}{ccccccccc}{x}_1^d{X}_1^T& x_1^d{Y}_1^T& x_1^d& y_1^d{X}_1^T& y_1^d{Y}_1^T& y_1^d& X_1^T& Y_1^T& 1\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ x_n^d{X}_n^T& x_n^d{Y}_n^T& x_n^d& y_n^d{X}_n^T& y_n^d{Y}_n^T& y_n^d& X_n^T& Y_n^T& 1\end{array}\right]\\ f_H=[\begin{array}{ccccccccc}{F}_{11}& F_{12}& F_{13}& F_{21}& F_{22}& F_{23}& F_{31}& F_{32}& F_{33}\end{array}]\end{array}$$

(10)

The corresponding equations are solvable using least square when the number of points is greater than 8 points. The corresponding distortion center $e$ is the left null vector of $F_H$:

$$e^T{\left[e\right]}_{\times }=0\iff e^T{F}_H=e^T{\left[e\right]}_{\times }H=0$$

(11)

So far, we have obtained the image coordinates of the distorted center $e$. The corresponding homography matrix $H$ can be obtained using the fundamental matrix $F_H$.

4.2. Decoupling Camera Parameters

If the image coordinate origin $O_P$ is moved to the distortion center e, the new distortion center after translation is expressed as $\widehat{e}={\left[\begin{array}{ccc}0& 0& 1\end{array}\right]}^T$. In the new coordinate system, Equation (8) is expressed as:

$${\left[{\widehat{x}}_i^d\right]}^T{\widehat{F}}_H{X}_i^T=0$$

(12)

where ${\widehat{x}}_i^d$ and ${\widehat{F}}_H$ represent the transformed image coordinates $x_i^d$ and the fundamental matrix $F_H$. The transformation relationship is as follows:

$${\widehat{x}}_i^d=T_{e2\widehat{e}}{x}_i^d , {\widehat{F}}_H=T_{e2\widehat{e}}{F}_H, \mathrm{where} T_{e2\widehat{e}}=\left[\begin{array}{ccc}1& 0& -d{u}_0\\ 0& 1& -d{v}_0\\ 0& 0& 1\end{array}\right]$$

(13)

From the definition of ${\widehat{F}}_H$:

$${\widehat{F}}_H={\left[e\right]}_{\times }\widehat{H}=\left[\begin{array}{ccc}0& -1& 0\\ 1& 0& 0\\ 0& 0& 0\end{array}\right]\widehat{H}$$

(14)

Let:

$${\widehat{F}}_H=\left[\begin{array}{c}{\widehat{F}}_1\\ {\widehat{F}}_2\\ {\widehat{F}}_3\end{array}\right]\left[\begin{array}{ccc}{\widehat{F}}_{11}& {\widehat{F}}_{12}& {\widehat{F}}_{13}\\ {\widehat{F}}_{21}& {\widehat{F}}_{22}& {\widehat{F}}_{23}\\ {\widehat{F}}_{31}& {\widehat{F}}_{32}& {\widehat{F}}_{33}\end{array}\right]$$

(15)

and:

$$\widehat{H}=\left[\begin{array}{c}{\widehat{H}}_1\\ {\widehat{H}}_2\\ {\widehat{H}}_3\end{array}\right]\left[\begin{array}{ccc}{\widehat{H}}_{11}& {\widehat{H}}_{12}& {\widehat{H}}_{13}\\ {\widehat{F}}_{21}& {\widehat{F}}_{22}& {\widehat{H}}_{23}\\ {\widehat{H}}_{31}& {\widehat{H}}_{32}& {\widehat{H}}_{33}\end{array}\right]$$

(16)

Equations (15) and (16) are then introduced into Equation (14):

$${\widehat{H}}_1={\widehat{F}}_2,{\widehat{H}}_2=-{\widehat{F}}_1$$

(17)

So far, the first two rows ${\widehat{H}}_1 {\widehat{H}}_2$ of the homography matrix have been obtained. Referring to Equations (2) and (4), the image distortion after translation can be expressed as follows:

$${\rho }_i\frac{x_i^d}{1+{\lambda }_1{\left[r_i^d\right]}^2+{\lambda }_2{\left[r_i^d\right]}^4+\dots }=H{X}_i^T$$

(18)

An equation set can be obtained after sorting out:

$$\left[\begin{array}{c}{\widehat{x}}_i^d{\left[X_i^T\right]}^T\left(-{\widehat{F}}_2^{}{X}_i^T\right)\left[\begin{array}{ccc}{\left[{\widehat{r}}_i^d\right]}^2& {\left[{\widehat{r}}_i^d\right]}^4& \cdots \end{array}\right]\\ {\widehat{y}}_i^d{\left[X_i^T\right]}^T\left({\widehat{F}}_1^{}{X}_i^T\right)\left[\begin{array}{ccc}{\left[{\widehat{r}}_i^d\right]}^2& {\left[{\widehat{r}}_i^d\right]}^4& \cdots \end{array}\right]\end{array}\right]\left[\begin{array}{c}{[{\widehat{H}}_3]}^T\\ {\lambda }_1\\ {\lambda }_2\\ ⋮\end{array}\right]=\left[\begin{array}{c}{\widehat{F}}_2^{}{X}_i^T\\ -{\widehat{F}}_1^{}{X}_i^T\end{array}\right]$$

(19)

For Equation (19) and the combined Equation (17), two equations can be obtained for each pair of corner points. When the number of corresponding points $N>=n+3$ (where n denotes the number of distortion parameters), an overdetermined equation is obtained. This can be achieved by moving the target, as shown in Figure 2. The homography matrix $\widehat{H}$ and the distortion coefficients ${\lambda }_1{\lambda }_2$ can be obtained by using the least square method.

4.3. Parameter Solution

From the perspective projection model, the imaging relationship of the translation sequence can be expressed as follows:

$${\rho }_i{x}_i^u={\rho }_i{\left[T_{e2\widehat{e}}\right]}^{-1}{\widehat{x}}_i^u={\left[T_{e2\widehat{e}}\right]}^{-1}\widehat{H}{X}_i^T=H{X}_i^T$$

(20)

It is known that:

$$H={\left[T_{e2\widehat{e}}\right]}^{-1}\widehat{H}=\left[\begin{array}{ccc}1& 0& d{u}_0\\ 0& 1& d{v}_0\\ 0& 0& 1\end{array}\right]\widehat{H}$$

(21)

Equation (19) can been solved to obtain $\widehat{H}$. The initial homography matrix $H$ can be calculated by substituting Equation (21). We set $[\begin{array}{ccc} r_1& r_2& t\end{array}]$$H=\left[\begin{array}{ccc}{H}_1& H_2& H_3\end{array}\right]=A_C[\begin{array}{ccc} r_1& r_2& t\end{array}]$. By using the orthogonality and normality of rotation matrix $[\begin{array}{ccc} r_1& r_2& r_3\end{array}]$, we obtain:

$$\left\{\begin{array}{c}{r}_1{r}_2=0\\ r_1{r}_1=r_2{r}_2\end{array}\right.\iff \left\{\begin{array}{c}{\left[H_1\right]}^T{\left[A_C\right]}^{-T}{\left[A_C\right]}^{-1}{H}_2=0\\ {\left[H_1\right]}^T{\left[A_C\right]}^{-T}{\left[A_C\right]}^{-1}{H}_1={\left[H_2\right]}^T{\left[A_C\right]}^{-T}{\left[A_C\right]}^{-1}{H}_2\end{array}\right.$$

(22)

Therefore, three images are needed to find five unknowns in the camera intrinsic parameter matrix $A_C$. If the camera collects $n$ images from different directions for calibration, a set of linear equations containing $2n$ constrained equations can be established, which can be written in matrix form as follows:

$$Vb=0$$

(23)

where $V$ is the coefficient matrix and $b$ is the variable to be solved, with:

$$b=[\begin{array}{cccccc}{B}_{11}& B_{12}& B_{13}& B_{22}& B_{23}& B_{33}\end{array}]$$

(24)

$$B={\left[A_C\right]}^{-T}{\left[A_C\right]}^{-1}=\left(\begin{array}{ccc}{B}_{11}& B_{12}& B_{13}\\ B_{21}& B_{22}& B_{23}\\ B_{31}& B_{32}& B_{33}\end{array}\right)$$

(25)

The solvable camera intrinsic parameters are:

$$\begin{array}{l}{v}_0=(B_{12}{B}_{13}-B_{11}{B}_{23})/(B_{11}{B}_{22}-B_{12}^2)\\ \lambda =B_{33}-[B_{13}^2+v_0(B_{12}{B}_{13}-B_{11}{B}_{23})]/B_{11}\\ f_u=\sqrt{\lambda /B_{11}}\\ f_v=\sqrt{\lambda B_{11}/(B_{11}{B}_{22}-B_{12}^2)}\\ s=-B_{12}{f}_u^2{f}_v/\lambda \\ u_0=k{v}_0/f_v-B_{13}{f}_u^2/\lambda \end{array}$$

(26)

Similarly, the camera parameters can be obtained:

$$[\begin{array}{cccc}{r}_1& r_2& r_3& t_1\end{array}]=\left[\begin{array}{cccc}\frac{{\left[A_C\right]}^{-1}{H}_1}{\left|{\left[A_C\right]}^{-1}{H}_1\right|}& \frac{{\left[A_C\right]}^{-1}{H}_2}{\left|{\left[A_C\right]}^{-1}{H}_3\right|}& r_1\times r_2& \frac{{\left[A_C\right]}^{-1}{H}_3}{\left|{\left[A_C\right]}^{-1}{H}_3\right|}\end{array}\right]$$

(27)

In the case of obtaining the parameters outside the target, the spatial coordinate $X_l^C={\left[\begin{array}{cccc}{X}_l^C& Y_l^C& Z_l^C& 1\end{array}\right]}^T$ of the laser spot $P_l$ in the camera coordinate system can be found by solving the known plane equation $\mathbb{Q}(X^C,Y^C,Z^C)$ in the direction obtained by connecting the ray and the target from the camera optical center $O_C$ to the ideal image point coordinate $x_l^u={\left[\begin{array}{ccc}{u}_l^u& v_l^u& 1\end{array}\right]}^T$. Moving the calibration board along the laser direction, the spatial position of laser spot can be obtained at different distances after multiple acquisitions. By processing the data, the laser beam can be straight in the camera coordinate system.

Through data processing, the linear equation $ℂ(X^C,Y^C,Z^C)$ of the laser beam in the camera coordinate system can be obtained in the form of Equation (29). By combining the distance information DL obtained through laser ranging, the spatial coordinates of the laser origin in the camera coordinate system can be obtained, and the transformation relationship $\left[\begin{array}{cc}{R}_{l2M}& t_{l2M}\end{array}\right]$ between the laser system and camera system can be estimated:

$$\mathbb{Q}(X^C,Y^C,Z^C):\left[\begin{array}{cccc}{A}_Q& B_Q& C_Q& D_Q\end{array}\right]{\left[\begin{array}{cccc}{X}^C& Y^C& Z^C& 1\end{array}\right]}^T=0$$

(28)

$$ℂ(X^C,Y^C,Z^C):\frac{X_{li}^C-X_{l0}^C}{A_l}=\frac{Y_{li}^C-Y_{l0}^C}{B_l}=\frac{Z_{li}^C-Z_{l0}^C}{C_l}$$

(29)

where $X_{l0}^C=\frac{1}{n}{\displaystyle \sum X_{li}^C}$, $Y_{l0}^C=\frac{1}{n}{\displaystyle \sum Y_{li}^C}$, $Z_{l0}^C=\frac{1}{n}{\displaystyle \sum Z_{li}^C}$. Combining with Equation (27), we have:

$$\mathbb{Q}(\begin{array}{cc}{R}_{T2M},& t_{T2M}\end{array})=\left[\begin{array}{cccc}{A}_Q& B_Q& C_Q& D_Q\end{array}\right][\begin{array}{cccc}{r}_1& r_2& r_3& t_1\end{array}]{\left[\begin{array}{cccc}{X}^T& Y^T& Z^T& 1\end{array}\right]}^T$$

(30)

By combining with the imaging model, the linear equation between laser spot and camera light center can be expressed in two-point form:

$$\frac{X^C}{u_l^u}=\frac{Y^C}{v_l^u}=\frac{2\cdot Z^C}{f_u+f_v}$$

(31)

Equations (30) and (31) are solved simultaneously, the only solution of which $X_l^C={\left[\begin{array}{cccc}{X}_l^C& Y_l^C& Z_l^C& 1\end{array}\right]}^T$ is the coordinate of the laser spot on the target in the camera coordinate system.

After many measurements, the linear equation can be expressed as a series of spatial point sets $\left\{\left.X_{li}^C\right| X_{li}^C={\left[\begin{array}{cccc}{X}_{li}^C& Y_{li}^C& Z_{li}^C& 1\end{array}\right]}^T,i=1,2,3,\cdots \right\}$, as shown in Figure 2. Constraints can be applied using a point-line relationship to solve the linear equation $ℂ(X^C,Y^C,Z^C)$ corresponding to laser rays, such as:

$$\left[\begin{array}{ccc}{Y}_{li}^C-Y_{l0}^C& -\left(X_{li}^C-X_{l0}^C\right)& 0\\ 0& -\left(Z_{li}^C-Z_{l0}^C\right)& Y_{li}^C-Y_{l0}^C\end{array}\right]\left[\begin{array}{cc}{A}_l& 0\\ B_l& B_l\\ 0& C_l\end{array}\right]=0$$

(32)

A space point can provide two constraints, and we need at least two space points to solve the equation and estimate the linear equation $ℂ(X^C,Y^C,Z^C)$. Finally, the laser origin position is determined on the line by calculating the distance information obtained by ranging according to the coordinate system established before and using the relative transformation matrix of laser-camera $\left[\begin{array}{cc}{R}_{T2M}& t_{T2M}\end{array}\right]$.

4.4. Optimization of Solution

The above process does not involve any iteration. The camera internal parameters and laser-camera external parameters can be found using least squares. The calculation speed is fast and local minima can be effectively avoided effectively. If we want to obtain higher accuracy, we can take the calculated value as the initial value, and further improve the calibration accuracy of the system through the non-linear optimization method.

Given $n$ calibrated images, each image has $m$ corners $x_i^d$ and one laser projection point $x_l^d$. The following objective functions are then constructed:

$$E(A_C,{\lambda }_1,{\lambda }_2,R_{T2M},t_{T2M})={\displaystyle \sum _{i=1}^n\left({\displaystyle \sum _{j=1}^m\left|x_{i,j}^d-Pro(X_j^T)\right|+\gamma \left|x_{l,j}^d-Pro(d_i^l)\right|}\right)}$$

(33)

where $Pro(X_j^T)$ and $Pro(d_i^l)$ represent the projection functions of corner points $X_j^T$ and laser spot $X_l^C$ under the division distortion model, and $\gamma$ is a named weight coefficient that denotes the contribution of corner and laser points to errors, generally speaking $\gamma =5$.

Using Cayley-Gibbs-Rodriguez (CGR) [9] to parameterize the rotation matrix $R$, the latter can be expressed as a function of the CGR parameters $s=\left[\begin{array}{ccc}{s}_1& s_2& s_3\end{array}\right]$:

$$R=\frac{1}{1+s_1{}^2+s_2{}^2+s_3{}^2}\left[\begin{array}{ccc}1+s_1{}^2-s_2{}^2-s_3{}^2& 2s_1{s}_2-2s_3& 2s_1{s}_3+2s_2\\ 2s_1{s}_2+2s_3& 1-s_1{}^2+s_2{}^2-s_3{}^2& 2s_2{s}_3-2s_1\\ 2s_1{s}_3-2s_2& 2s_2{s}_3+2s_1& 1-s_1{}^2-s_2{}^2+s_3{}^2\end{array}\right]$$

(34)

The problem is then transformed into an unconstrained optimization problem. The automatic Gröbner basis method [10] is used to solve Equation (32), and the minimum solution $E_{\min }({\tilde{A}}_C,{\tilde{\lambda }}_1,{\tilde{\lambda }}_2,{\tilde{R}}_{l2M},{\tilde{t}}_{l2M})$ can be obtained. A nonlinear optimization method is used to further improve the accuracy and stability of the solution.

In this part, we have completed the estimation of the optimal solution of all parameters, including the camera intrinsic parameter matrix ${\tilde{A}}_C$, distortion coefficient ${\tilde{\lambda }}_1{\tilde{\lambda }}_2$ and laser-camera external parameters $\left[\begin{array}{cc}{\tilde{R}}_{l2M}& {\tilde{t}}_{l2M}\end{array}\right]$.

5. Experiment and Analysis

In this part, we evaluate the calibration methods of the camera internal parameters and camera-laser external parameters. The effectiveness and influencing factors of the proposed system calibration algorithm are analyzed through computer simulation experiments, while the measurement system is calibrated through observation experiments. In order to better evaluate the calibration results, we refer to the re-projection error [34] evaluation method in the camera calibration process, and unify the laser spot and target corner to establish the following error evaluation function:

$$E_{dr}=\frac{1}{m\cdot n}{\displaystyle \sum _{i=1}^n\left({\displaystyle \sum _{j=1}^m\left|x_{i,j}^d-Pro(X_j^T)\right|+\gamma \left|x_{l,i}^d-Pro(d_i^l)\right|}\right)}$$

(35)

The re-projection error E_dr is an important metric of the calibration results: the smaller E_dr is, the better the calibration results are.

5.1. Simulation Result

For the simulation experiment, we used the MATLAB R2016a software for Windows 10. The relevant parameters of the simulation system are shown in

Table 2. System parameters of simulation.

Parameter	$A_C$	$e$	$({\lambda }_1,{\lambda }_2)$	$R_{l2M}$	$t_{l2M}$
Set value	$\left[\begin{array}{ccc}850& s& 512\\ 0& 850& 384\\ 0& 0& 1\end{array}\right]$	$\left[\begin{array}{c}509\\ 380\end{array}\right]$	$\left(\begin{array}{c}6.15\times {10}^{-7}\\ 1.6\times {10}^{-13}\end{array}\right)$	$\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right]$	$\left[\begin{array}{c}50\\ 0\\ 0\end{array}\right]$
Unit	$\mathrm{pixel}$	$\mathrm{pixel}$	$\left(\begin{array}{c}{\mathrm{pixel}}^{-2}\\ {\mathrm{pixel}}^{-4}\end{array}\right)$	-	$\mathrm{mm}$

. In the measurement system, the laser direction is parallel to the optical axis of the camera and a 50 mm offset in the $\overrightarrow{O_M{X}_M}$ direction is arranged.

The target is shown in the Figure 3, where the blue dots represent the corners of the checkerboard lattice, evenly distributed in the plane, and the adjacent corners are $15 \mathrm{mm}$ apart. The relative position between the target and the system is randomly generated by the system within a given range.

Throughout the experiment, we compare the estimated values from each calculation with the real values set by simulation, and evaluate the accuracy of the algorithm by calculating the deviation between the two. The error is expressed as follows:

$$\left\{\begin{array}{l}{E}_{fu}=\frac{\left|{\tilde{f}}_u-f_u\right|}{f_u},E_{fv}=\frac{\left|{\tilde{f}}_v-f_v\right|}{f_v}\\ E_{u0}=\frac{\left|{\tilde{u}}_0-u_0\right|}{u_0},E_{u0}=\frac{\left|{\tilde{v}}_0-v_0\right|}{v_0}\\ E_R={\max }_{i=1}^3\left|\arccos \left({\tilde{r}}_i\cdot r_i\right)\right|\\ E_t=\left|{\tilde{t}}_{l2M}-t_{l2M}\right|\end{array}\right.$$

(36)

Kopparapu et al., confirmed [32] that noise has a significant impact on calibration accuracy. We add ${\omega }_{noise}\sim Gauss\left(0,{\Sigma }_{noise}\right)$ Gaussian noise to the simulated projection image, where ${\Sigma }_{noise}$ is the standard deviation of the Gaussian distribution. In the simulation, the standard deviation ${\Sigma }_{noise}$ of noise increases gradually in the range of $0.1$ to $1.5 \mathrm{pixel}$. For each ${\omega }_{noise}$ X distribution, we performed $100$ independent experiments, and obtained the average value of calibration error as the statistical result.

The results are shown in Figure 4. It can be seen that with the increase of noise, the deviation between the calibration parameters and the true value increases linearly. When the corner extraction noise is 0.5 pixels, the system calibration error is about 0.2, the focal length deviation is 0.1%, and the main point deviation is about 0.8%. In terms of extrinsic parameters, the translation error also follows a linear distribution, but the fluctuation is more obvious. It can be seen that the system is sensitive to the internal parameters. At the same time, under the corner extraction error of 0.5 pixels, the translation error is about 1 mm and the rotation error is 0.02 degrees.

In addition, we analyzed the impact of the number of collected data on the calibration accuracy, and set the calibration data to gradually increase from the minimum of three groups of image distance data to 15 groups. The results are shown in the Figure 5. With the increase of calibration data, the re-projection error remains almost stable, but the accuracy of the estimated system variables is significantly improved. When the number of data increases to 8, the decline of the correlation error slows down. Therefore, sufficient calibration data collected in a certain range can help to improve the accuracy of system calibration. However, after reaching a certain number, the effect gradually decreases, and so 8–10 groups of data are appropriate.

We also analyzed the influence of the measurement error of the laser ranging system on the calibration accuracy. The Gaussian-distributed noise ${\omega }_d\sim Gauss\left(0,{\Sigma }_d\right)$ was added to the ranging error, and the standard deviation ${\Sigma }_d$ was changed gradually from 1 mm to 15 mm. We calculated the calibration errors of the parameters of the system at each noise level. As shown in Figure 6, except for the linear relationship between translation vector and distance error, the other parameters hardly change with the increase of error.

5.2. Real Experiment

In the actual experiment, we built a measurement system with a 1D laser-camera combination, and calibrated the system with the method proposed in this paper. As shown in Figure 7, the system is composed of a MER-131-210U3C camera and a SKD-100 laser ranging system. The related parameters are shown in

Table 3. The system parameters of 1D LRF and camera.

Sensors	Parameter	Value
MER-131-210U3C (camera)	Sensor Size	${1/2}^{″}$
	Resolution	$1280 (\mathrm{H}) \times 1024 (\mathrm{V})$
	Frame Rate	$210 FPS$
	Pixel Size	$4.8 \mathsf{\mu }\mathrm{m}\times 4.8 \mathsf{\mu }\mathrm{m}$
	Focuses	$5 \mathrm{mm}$
	F (Relative Aperture)	$1.4\sim 16$
SKD-100 (LRF)	Wavelength	$635 \mathrm{nm}$
	Range	$1\sim 1000 \mathrm{mm}$
	Accuracy	$2 \mathrm{mm}$

.

The system was calibrated using a calibration board composed of $11\times 8$ square chessboard lattices with a distance of 15 mm between corners. The iterative Harris algorithm was used to extract the checkerboard corner coordinates (red +) from the calibrated image accurately, and the centroid method was used to extract the image coordinates (green $\times$) of the laser spot. The accuracy can reach sub-pixel level. The results of 12 images collected at different distances from $150 \mathrm{mm}$ to $1500 \mathrm{mm}$ are shown in Figure 8.

In order to verify the accuracy of our calibration method, we compared the internal parameters obtained with the classical Zhang [21] calibration method and the Li’s method [30]. In the calibration process of Bo‘s method, we replaced the original random corner matching process by directly inputting the coordinates of checkerboard lattices into the program, but still retain the complete algorithm for camera parameter determination. The results are shown in

Table 4. System parameters of simulation.

Method	$A_C$	$e$	$({\lambda }_1,{\lambda }_2)$	Mean $E_{rp}$
Zhang [21]	$\left[\begin{array}{ccc}1053.3& 0& 643.5\\ 0& 1048.0& 539.7\\ 0& 0& 1\end{array}\right] \mathrm{pixel}$	-	$\left(\begin{array}{c}0.1348{ \mathrm{mm}}^{-2}\\ 0.01661{ \mathrm{mm}}^{-4}\end{array}\right)$	$0.09337 \mathrm{pixel}$
Li [30]	$\left[\begin{array}{ccc}1059.7& 0.1604& 649.5\\ 0& 1058.9& 539.2\\ 0& 0& 1\end{array}\right] \mathrm{pixel}$	-	$\left(\begin{array}{c}0.1372{ \mathrm{mm}}^{-2}\\ 0.1993{ \mathrm{mm}}^{-4}\end{array}\right)$	$0.07974 \mathrm{pixel}$
proposed	$\left[\begin{array}{ccc}1055.2& 0& 647.2\\ 0& 1054.9& 538.1\\ 0& 0& 1\end{array}\right] \mathrm{pixel}$	$\left[\begin{array}{c}509\\ 380\end{array}\right]$$\mathrm{pixel}$	$\left(\begin{array}{c}6.15\times {10}^{-7}\\ 1.6\times {10}^{-13}\end{array}\begin{array}{c}{\mathrm{pixel}}^{-2}\\ {\mathrm{pixel}}^{-4}\end{array}\right)$	$0.07725 \mathrm{pixel}$

, where the accuracy of the intrinsic parameters obtained by the calibration methods are compared. The calibration accuracy is evaluated using the re-projection error [34] and expressed as:

$$E_{rp}=\frac{1}{m}{\displaystyle \sum _{j=1}^m\sqrt{{\left|x_j^d-Pro(X_j^T)\right|}^2}}$$

(37)

From the calibration results in Figure 9 and Table 4, we see that our method and Zhang’s method [21] have similar calibration results in camera intrinsic parameters. Because the distortion models used by the two methods are different, the physical meanings of the distortion coefficients are different, so it is meaningless to compare them. Judging from the re-projection error, our method is slightly better than Zhang’s calibration algorithm. This proves the effectiveness of our calibration algorithm.

At the same time, the extrinsic parameters $\left[\begin{array}{cc}{\tilde{R}}_{l2M}& {\tilde{t}}_{l2M}\end{array}\right]$ of the laser-camera combination of the measurement system are also calculated and the calibration results were evaluated using the evaluation function set Equation (37). The results are shown in

Table 5. System extrinsic parameters and evaluation error.

Parameter	$R_{l2M}$	$t_{l2M}$	$E_{dr}$
Set value	$\left[\begin{array}{ccc}0.9974& 0.0052& -0.0715\\ -0.0052& 0.9893& -0.1456\\ 0.0715& 0.1456& 1\end{array}\right]$	$\left[\begin{array}{c}0.5030\\ 32.6479\\ 1.859\end{array}\right]$	0.10173
Unit	-	mm	pixel

and Figure 10.

Ferrara et al. [35] mentioned that the position of the checkerboard has an effect on the accuracy of calibration. We supplemented a set of calibration data of checkerboard location on the edge of the image to verify the effect of the change of checkerboard location on the accuracy of the proposed method. The data are shown in Figure 11. The calibration results are shown in

Table 6. System parameters and evaluation error when checkerboard is close to edge.

Parameter	$A_C$	$e$	$({\lambda }_1,{\lambda }_2)$	Mean $E_{rp}$
Value	$\left[\begin{array}{ccc}1057.9& 0& 646.9\\ 0& 1057.2& 536.8\\ 0& 0& 1\end{array}\right] \mathrm{pixel}$	$\left[\begin{array}{c}507\\ 379\end{array}\right]$$\mathrm{pixel}$	$\left(\begin{array}{c}6.21\times {10}^{-7}\\ 1.53\times {10}^{-13} \end{array}\begin{array}{c}{\mathrm{pixel}}^{-2}\\ {\mathrm{pixel}}^{-4}\end{array}\right)$	$0.08371 \mathrm{pixel}$
Parameter	$R_{l2M}$	$t_{l2M}$	$E_{dr}$
Set value	$\left[\begin{array}{ccc}0.9962& 0.0101& -0.0865\\ -0.0101& 0.9902& -0.1381\\ 0.0855& 0.1388& 0.9999\end{array}\right]$	$\left[\begin{array}{c}0.8167\\ 31.5531\\ 1.6742\end{array}\right]$$\mathrm{mm}$	$0.1147 \mathrm{pixel}$

. When the image is in the edge position, the calibration results are basically consistent with the internal and external parameters obtained in Table 3 and Table 4, and the re-projection errors of the internal and external parameters are slightly increased, but the difference is small. It therefore shown that the method proposed in this paper is also applicable when the collected data lie on the edge of the image.

6. Conclusions

In this paper, we present a convenient and fast method for calibrating a combined 1D laser ranging and monocular camera measurement system, aiming to realize an accurate measurement system fusing laser and vision. The method is easy to implement and has high calibration accuracy. The fast robust determination of the camera imaging model parameters is achieved by introducing a division distortion model. Then, a linear-plane constraint is formulated to realize robust estimation of the initial value of the laser-vision parameters. Finally, an unconstrained optimization problem is formulated using the rotation matrix parameters, and the high precision calibration of the whole measurement system is realized. The factors affecting the calibration accuracy are analyzed through simulation experiments, and the effectiveness of the proposed method is verified through real scene experiments.