1. Introduction
3D optical measurement has been widely used in various applications like industrial inspection, inverse engineering, robot navigation, and human–machine interaction, etc. [
1,
2,
3,
4,
5]. Off-the-shelf commercial 3D scanners like structured light system [
6,
7,
8] and laser 3D scanners [
9] can perform well in the 3D measurement of diffuse surface or Lambertian surface. However, these 3D measurement techniques usually fail to deal with mirror-like specular surfaces subject to the strong reflectance. Instead of avoiding the specular reflection by coating the surface with powders, the phase measuring deflectometry (PMD) technique [
10,
11,
12] takes full advantage of the surface reflective property, which captures the virtual image reflected by the target surface. By analyzing the deformed strip patterns captured by the camera, precise discrete slope variations of the test surface can be calculated. By integrating the slope data, 3D topography of the target surface can be reconstructed precisely [
13,
14,
15].
In the PMD-based 3D measurement systems, the calculated surface slope and the integration step are highly sensitive to the system error, which is caused by the calibration accuracy [
16,
17]. Therefore, the calibration method of the PMD system is very important, which determines the system measurement precision directly. Mostly, the PMD system calibration performs by calibrating the camera and system geometric parameters respectively. The camera calibration is conducted to compute the term camera calibration parameters [
18,
19,
20]. The system geometry calibration refers to determining the relation between the screen, the reference mirror, and the camera. In detail, this step aims to find correspondence between screen pixels, the mirror reflected points, and the camera pixels. Since the screen that used to display the strip patterns does not lie within the field view of the camera in the PMD systems, it is difficult to obtain geometrical relation between the camera and screen without auxiliary setup.
To solve the geometry calibration problem in PMD systems, existing approaches can be generally classified into four categories. (i) Using a laser tracker or a coordinate measuring machine to specify the positions of the camera, the screen, and the mirror [
21,
22,
23]. Although the accuracy can be guaranteed by the usage of additional precise instruments, the operation is laborious and the equipment is very expensive. (ii) Using a reference planar mirror with markers printed on it [
10,
24,
25]. These methods indirectly calibrate the pose between the screen and the camera by the position of the reference mirror. The mirror should be custom-made with precise markers. Besides, markers on the mirror may interrupt the reflected patterns. (iii) Using a marker-less mirror by changing its pose at least three different positions [
26,
27,
28]. The pose of the screen can be estimated by optimizing all the pose parameters simultaneously with the application of bundle adjustment [
27] process or orthogonality constraint [
28]. (iv) By treating the system geometry calculation as an optimization problem [
29]. Given initial values of the system geometry parameters, and the iteration procedure is introduced to reach an optimal value [
30,
31,
32]. Such methods are usually sensitive to the initialization of system parameters and cannot guarantee global optimal results. Moreover, the optimization is based on the criterion of 2D image re-projection error, which cannot reflect final 3D reconstruction errors.
In this paper, a 3D criterion-based calibration method is proposed to improve the global calibration accuracy of the PMD system. The system is initially calibrated by traditional means with a flat mirror. Then, a planar mirror is used for parameter optimization. Based on the reference mirror, 3D metric error criteria are defined as the planarity error, the distance error, and the angular error. A multi-objective optimization problem is established by considering all the system parameters as variables. Using the primary calibration results as initial values, by minimizing the defined 3D measurement errors, the optimal calibration parameters can be computed finally. The results show, with a simple planar mirror and the proposed calibration method, system calibration accuracy can be greatly improved compared with conventional calibration methods.
2. Modeling and Initial Calibration of the PMD System
Figure 1 describes the basic principle of the PMD measurement system. The system is composed of a camera, a display screen, and the specular surface to be tested. Structured light patterns displayed by the screen are reflected by the targeted surface, and captured by the camera. The obtained images are the virtual images of the real screen. Then the surface x-slope and y-slope can be calculated by the system geometrical model and calibration parameters. Finally, the slope is integrated to obtain the 3D shape of the target surface. For simplicity, as shown in
Table 1, the coordinate frames of the screen and the camera are defined as {s} and {c}, respectively. {v} refers to the virtual screen, which is the image of the real screen. The definitions of the concerned coordinate systems are listed in
Table 1. In this paper, bold capital letters
R and
T are used to denote the rotation matrix and the translation vector. The subscript refers to coordinate transformation between the source and the target coordinate system, e.g.,
Rw2c means the rotation matrix from world coordinate system {w} to the camera coordinate system {c}, and so on in a similar fashion.
The fundamental principle of the PMD method is the law of reflection, which means the incident angle and reflected angle for any surface point should be exactly equal. For convenience, as shown in
Figure 1, the camera can be assumed as an active observer. A ray that emitted from point
P goes through the camera optical center, reflected by the specular surface at point
M, and hits the screen at point
Q. In the initial calibration procedure, the reference mirror is placed at more than three different positions. For each position, a pattern with asymmetric circular markers is displayed by the screen and captured by the camera to calculate the internal parameters as shown by
Figure 2a. Because the virtual screen has further distance to the camera compared with the real target surface, the captured images are usually blurred subject to the defocus of the camera. The blurred calibration images make the localization of dot center inaccurate as well as the calibration results of camera. That’s also one reason to apply a global optimization for all the system parameters.
The virtual fiducials
is mapped to camera pixel
P. Let
denotes the coordinate in the frame {c}, and image pixel
P is denoted as
in image plane. According to the pinhole model, the normalized form of
pc can be written as:
Considering the lens distortions
, where
refers to the radial coefficients and
refers to the tangential coefficients, the undistorted expression of
can be expressed as:
where
and
refers to the tangential distortion vector that is expressed as:
The homogeneous coordinate
of the undistorted point
P can be expressed as:
where
A is the camera intrinsic matrix, which can be expressed as:
For a complete model with lens distortions, there are 10 parameters to be estimated, i.e., . The parameter of α refers to the skewness of the sensor axes, which can be assumed to be 0 for most modern imaging sensors. Intrinsic parameters of the camera can be estimated by a lot of existing calibration methods.
The extrinsic parameters between frame {v} and {c} are expressed as
Rv2c and
Tv2c, which can be mathematically represented as:
where
s is a scale factor. Based on the maximum likelihood estimation algorithm [
19], the internal and external parameters can be estimated by assessing the re-projection of the fiducials in frame {v} onto frame {c}, denoting the nonlinear mathematical function as
, the minimization of the re-projection errors can be expressed as:
The structured light patterns captured by the camera are from the real screen frame {s}. Therefore, it is the real point
Q that transformed into the camera frame {c}. To estimate the position and orientation of the real screen, a transparent Polyethylene Terephthalate (PET) calibration board is attached to the mirror at its last position and imaged by the camera, as shown in
Figure 2b. Since the camera has been calibrated, the intrinsic parameters can be used to estimate the extrinsic parameters
Rw2c and
Tw2c by the following equation:
According to the geometrical relation [
33] and the Householder transformation [
34], the relation between {s} and {c} can be calculated as:
where
is a 3 × 3 identity matrix, and
,
is the normal vector of mirror respect to the camera frame {c},
refers to the distance between the calibration mirror and the camera optical center. They can be derived via
Rw2c and
Tw2c as:
With the above operation and calculation, the initial camera and system geometrical parameters can be obtained. However, subject to the camera defocus, feature detection errors, manufacturing error of the transparent checkboard pattern and operating errors, etc., the initial calibration results of the PMD system are usually inaccurate. The following section will show how it can be optimized with a simple planar mirror.
3. Global Optimization of PMD System Parameters
Optimization of calibration parameters with respect to the 2D re-projection error criterion has been a standard step in existing calibration methods [
16,
26,
29] for 3D optical measurement. However, the re-projection error cannot reflect real metric errors of the vision-based measurement system. More importantly, the 2D re-projection error based optimization procedure is only applied to the camera model parameters. But the factors of camera lens defocusing, structured light decoding error, and slope calculation errors, etc. are not considered. All these factors can make the final PMD calibration results inaccurate. That’s also the major motivation of this research, that a global optimization must be performed after the primary calibration so as to improve the system calibration accuracy. In this section, to improve the PMD system calibration accuracy, a practical 3D-based calibration procedure is introduced as follows. The underlying principle of the proposed method is to regard calibration of the PMD system as a global optimization problem. The primary calibration results are used as the initial values, and some objective functions are constructed to minimize the 3D metric deviation. By solving the nonlinear multi-objective optimization problem, the optimal parameters can be obtained. The workflow of the procedure is shown in
Figure 3.
The object used for calibration is rather simple as a square planar mirror, which is aluminum coated with λ/10 flatness to guarantee high planarity and precision. The first image displayed on the screen is a white pattern without any information, which is used to illuminate the mirror and make it visible for the camera. Based on this image, four corners (
cc) of the mirror can be detected with subpixel accuracy, and the mask of the mirror area can be located in the image. To improve efficiency, a liner uniform down-sampling is applied and gets a set of image points (
cp). Based on the reconstruction method and primary calibration parameters, 3D coordinates of
cp and
cc can be calculated and denoted as
Cp and
Cc, as shown in
Figure 4.
Based on the reconstructed 3D information of the mirror, three metric errors are defined so as to construct the multi-objective optimization problem for accurate system parameter estimation. The adopted three objective functions are defined as follows.
3.1. Planarity Error
The square mirror used for reconstruction can be treated as a perfect plane. Regardless of the calibration derivations and reconstruction derivations, the planarity error of the
Cp should be zero. In general, a least-square fitting approach is applied to minimize the distance between the reconstructed points on the mirror surface. Given a plane, for each surface point, it has the same slope value in a certain direction. So we can use the slope values instead of planar fitting to describe the planarity term, which also makes the calculation more efficient. In details, the slope value can represent the planarity error by the following expression:
where
N is the number of sampling points,
and
are slopes along the x and y-axis of the i-th sampling point,
and
are the corresponding mean value, respectively.
3.2. Distance Error
For each corner
cc, its average distance to clockwise adjacent corners can be calculated and denoted as
dj. There are four corners and the distance error is simply defined as:
where,
L is the nominal edge length of the mirror.
3.3. Angular Error
The planarity and distance error terms are still not enough to guarantee the reference plane will be well reconstructed. Considering the distortion caused by the affine transformation, the 3rd error term named angular error is defined. For each corner
, by connecting it with the adjacent corners, all angles
can be calculated, and the angular error can be defined as:
where, the ground-truth value of
is known to be 90 degree.
For the adopted PMD system, there are 19 parameters to be optimized totally, which can be expressed by a vector
K as:
Corresponding rotation vector is denoted as
by Rodrigues transformation to satisfy the orthogonality constraint, i.e.,
. The primary calibration results are used as the initial values of
K and denote it as
K0. Therefore, a multi-objective optimization problem can be established as:
where the boundary coefficient
lb and
ub are empirically set to [0.9, 1.1], the weighting factors
and
are used to balance the effects from three error criteria, which can be evaluated empirically to satisfy
, e.g.,
To solve this multi-objective optimization problem, a lot of mathematical tools can be used. In this work, the “fgoalattain” function provided in the MATLAB optimization toolbox is used, which is based on the goal attainment method as described in [
35]. Finally, the solved optimal system calibration parameters
are used for surface reconstruction. Since 3D metrics are used instead of 2D image re-projection errors, accurate measurement results can be promised as the experiments verified.
4. Experimental Results and Discussion
To verify the performance of the proposed calibration method, a PMD system is established as shown in
Figure 5a. Dimension and relative positions of the setup is as shown by
Figure 5b. The system consists of an LCD screen (HP 24 es 23.8-inch), which has a resolution of 1920 × 1080 and the pixel pitch is 0.2745 mm, an industrial camera (PointGray FL3-U3-32S2C-CS), which has a resolution of 2080 × 1552 pixel and the pixel size is 2.5 μm. The camera lens has a focal length of 12 mm. Distance from the object to the camera is about 360 mm. A sinusoidal phase-shifting pattern is adopted for the surface slope calculation as shown in
Figure 5a. Virtual images of the displayed fringe patterns can be reflected by the mirror and then captured by the camera.
In the primary calibration step, the asymmetric dots pattern is displayed by the screen. By changing the posture of the reference flat mirror, 10 different views of patterns are acquired for camera calibration. While the last image was captured, without changing the position of the mirror, a transparent PET calibration board is attached on the mirror surface and captured by the camera. Following the method described in
Section 2, the initial calibration parameters can be calculated. In the system parameter optimization step, a square mirror with a size of 60 × 60 mm is used as shown in
Figure 6. Four corners of the mirror are detected from the image firstly. Then, with the primary calibration results, the mirror surface can be reconstructed to get its 3D model. Based on the reconstructed dense 3D points, a uniform down-sampling is applied to reduce the point number. Finally, three metric errors can be calculated. By solving Equation (15), the optimized system parameters can be calculated.
Table 2 shows the parameters before and after optimization, namely
K0 and
respectively. The reference mirror is placed in the system as shown by
Figure 6a. Measurement results of the reference mirror plane before and after optimization are shown in
Figure 6b and
Figure 6c respectively. And the three metric errors computed with two groups of calibration parameters are given by
Table 3. We can see that the planarity by the optimized calibration parameters can be improved almost 10 times. The distance and angular errors also can be improved greatly after the optimization of system calibration parameters.
In the first experiment, a stainless gauge block with a polished surface is used for evaluation, which has a size of 26 × 6 × 10 mm as shown in
Figure 7a. The block surface is reconstructed twice based on the same structured light images with initial parameters and optimal parameters. So, the difference between the two measurement results can only be caused by the calibration error. 3D reconstruction results by the conventional and proposed method are as shown in
Figure 7b and
Figure 7c respectively. By fitting the reconstructed 3D points with an ideal plane, the peak and valley (PV) and root mean square (RMS) values are calculated as {5.7, 1.5} μm and {2.6, 0.52} μm respectively. To indicate the change of surface shape more clearly, a section line is selected and plotted as shown in
Figure 7d, the blue and red lines are the profile of
Figure 7b and
Figure 7c, respectively. With the above comparison results, we can see that measurement results by the optimal system calibration parameters can be improved distinctly.
A concave aluminized glass is also used for the experiment as shown in
Figure 8a. According to the specification provided by the manufacturer, its nominal aperture is 50 mm, and the radius of curvature of the concave mirror is 200 mm, with a tolerance of ±2%. Using the optimal calibration results, the reconstructed 3D model of the concave surface is as shown in
Figure 8b. To visualize the 3D shape of the model,
z axis is zoomed in. By fitting the reconstruction results with an ideal sphere as shown in
Figure 8c, radius of reconstructed 3D model is about 199.12 mm, which locates within the metric tolerance. The PV and RMS error are calculated as 14.6 μm and 6.9 μm, respectively as shown in
Figure 8d. In comparison, with the primary calibration results, the radius is calculated as 197.73 mm, the PV and RMS errors are 19.68 μm and 12.04 μm. Measurement accuracy improvement by the optimized system calibration results can be fully demonstrated.
To compare the 3D quality of the reconstructed shape with another 3D means, a commercial structured light 3D scanner (with nominal precision of 2.0 μ) was used to reconstruct the concave mirror and the result was shown in
Figure 9. Subject to the mirror reflective surface, the structured light scanner cannot scan the mirror surface directly. To make the structured light scanner work, a coating operation was applied to the concave mirror surface to make it not reflective as shown in
Figure 9a. The coating process is a usual procedure for the 3D scanners in industrial inspections, especially for the reflective or transparent surfaces. The reconstructed 3D model is as shown in
Figure 9b. With a similar sphere fitting operation as shown by
Figure 9c, the fitted radius is calculated as 206.9697 mm. In comparison, the result by the PMD system with optimized calibration parameters is much more close to the ground-truth data. In addition, compared with the 3D coordinate measuring method, the PMD method has the distinct advantage of non-contact and also much more efficient. Compared with structured light or laser 3D scanning techniques, the PMD method can work with a mirror-like surface directly. It has great potential in the online 3D inspection of mirror-like surfaces.