Slides and example codes are available:

https://github.com/sunglok/3dv_tutorial

An Invitation
Introduction to 3D Vision
: A Tutorial for Everyone

Sunglok Choi, Senior Researcher

[email protected] | http://sites.google.com/site/sunglok
Electronics and Telecommunication Research Institute (ETRI)
What is Computer Vision?

Computer vision is an interdisciplinary field that deals with how computers can be made to
gain high-level understanding from digital images or videos. From the perspective of
engineering, it seeks to automate tasks that the human visual system can do.[1][2][3]
"Computer vision is concerned with the automatic extraction, analysis and understanding of
useful information from a single image or a sequence of images. It involves the development
of a theoretical and algorithmic basis to achieve automatic visual understanding."[9] As a
scientific discipline, computer vision is concerned with the theory behind artificial systems that
extract information from images. The image data can take many forms, such as video sequences,
views from multiple cameras, or multi-dimensional data from a medical scanner.[10] As a
technological discipline, computer vision seeks to apply its theories and models for the
construction of computer vision systems.
Computer Vision by Wikipedia

3
What is Computer Vision?

Computer vision is an interdisciplinary field that deals with how computers can be made to
gain high-level understanding from digital images or videos. From the perspective of
engineering, it seeks to automate tasks that the human visual system can do.[1][2][3]
"Computer vision is concerned with the automatic extraction, analysis and understanding of
useful information from a single image or a sequence of images. It involves the development
of a theoretical and algorithmic basis to achieve automatic visual understanding."[9] As a
scientific discipline, computer vision is concerned with the theory behind artificial systems that
extract information from images. The image data can take many forms, such as video sequences,
views from multiple cameras, or multi-dimensional data from a medical scanner.[10] As a
technological discipline, computer vision seeks to apply its theories and models for the
construction of computer vision systems.
Computer Vision by Wikipedia

4
What is Computer Vision?
Image Understanding

Computer
Graphics

(Synthesized) Image inverse relationship

Tower of Pisa

Shape
Computer Face
Vision
Location

Human

Real World (Projected) Image Information

Image
Processing

(transformed) Image/Signal

5
What is Computer Vision?

Computer Vision

What is it? Where am I?

▪ Label (e.g. Tower of Pisa) ▪ Place (e.g. Piazza del Duomo, Pisa, Italy)
▪ Shape (e.g. ) ▪ Location (e.g. )

(84, 10, 18) [m]

6
What is 3D Vision?
Visual Geometry (Multiple View Geometry)
Geometric Vision

Computer Vision

What is it? Where am I?

▪ Label (e.g. Tower of Pisa) ▪ Place (e.g. Piazza del Duomo, Pisa, Italy)
▪ Shape (e.g. ) ▪ Location (e.g. )

(84, 10, 18) [m]

Recognition Problems v.s. Reconstruction Problems

YOLO v2 (2016) ORB-SLAM2 (2016) 7

What is 3D Vision?

▪ Reference Books

8
What is 3D Vision?

▪ OpenCV (Open Source Computer Vision)

– calib3d Module: Camera Calibration and 3D Reconstruction

cf. All examples in this tutorial are mostly less than 100 lines and based on recent OpenCV (> 3.0.0).
9
What is 3D Vision?

image depth image, range data, point cloud, polygon mesh, …

3D Vision v.s. 3D Data Processing

Perspective Camera RGB-D Camera

(Stereo, Structured Light, ToF, Light Field)

Range Sensor
Omni-directional Camera
(LiDAR, RADAR)

10
What is 3D Vision?

image depth image, range data, point cloud, polygon mesh, …

3D Vision v.s. 3D Data Processing

Perspective Camera RGB-D Camera

(Stereo, Structured Light, ToF, Light Field)

Range Sensor
Omni-directional Camera
(LiDAR, RADAR)

11
 Applications: Photo Browsing

▪ Photo Tourism (2006)

[Reference] Snavely et al., Photo Tourism: Exploring Photo Collections in 3D, SIGGRAPH, 2006 12
 Applications: 3D Reconstruction

▪ Building Rome in a Day (2009)

[Reference] Agarwal et al., Building Rome in a Day, ICCV, 2009 13

 Applications: 3D Reconstruction from Cellular Phones

▪ Structure from Small Motion (SfSM; 2015)

▪ Casual 3D Photography (2017)

[Reference] Im et al., High Quality Structure from Small Motion for Rolling Shutter Cameras, ICCV, 2015
14
[Reference] Hedman et al., Causual 3D Photography, SIGGRAPH Asia, 2017
 Applications: Real-time Visual SLAM

▪ ORB-SLAM (2014)

[Reference] Mur-Artal et al., ORB-SLAM: A Versatile and Accurate Monocular SLAM System, T-RO, 2015 15
 Applications: Augmented Reality

▪ PTAM: Parallel Tracking and Mapping (2007)

[Reference] Klein and Murray, Parallel Tracking and Mapping for Small AR Workspaces, ISMAR, 2007 16
 Applications: Mixed Reality

▪ Microsoft Hololens 2 (2019)

– Head tracking: 4 x visible light cameras

An image is from SlashGear (https://www.slashgear.com/microsoft-hololens-image-sensors-windows-10-update-18534584/) 17

 Applications: Virtual Reality

▪ Oculus Quest (2019)

An image is from TechSpot (https://www.techspot.com/products/audio-video/oculus-quest.202424/) 18

Table of Contents

▪ What is 3D Vision?
▪ Single-view Geometry
– Camera Projection Model
– General 2D-3D Geometry
▪ Two-view Geometry
– Planar 2D-2D Geometry (Projective Geometry)
– General 2D-2D Geometry (Epipolar Geometry)
▪ Correspondence Problem
▪ Multi-view Geometry
▪ Summary

19
Getting Started with 2D

▪ Similarity

20
Getting Started with 2D

▪ Similarity

21
Getting Started with 2D

▪ Point transformation

22
Getting Started with 2D

▪ Coordinate transformation

23
Getting Started with 2D

▪ Coordinate transformation is the inverse of point transformation.

24
Getting Started with 2D

▪ Image coordinate (unit: [pixel])

▪ Camera coordinate (unit: [meter])

25
Getting Started with 2D

▪ 2D rotation matrix
cf. Properties of a rotation matrix
• 𝑅 −1 = 𝑅 ⊺ (orthogonal matrix)
• det(𝑅) = 1

▪ 3D rotation matrix

Cameras Vehicles Airplanes Telescopes

𝜃𝑥 Tilt Pitch Attitude Elevation
𝜃𝑦 Pan Yaw Heading Azimuth
𝜃𝑧 Roll Roll Bank Horizon

26
Single-view Geometry
Camera Projection Model
- Pinhole Camera Model
- Geometric Distortion
General 2D-3D Geometry
- Camera Calibration
- Absolute Camera Pose Estimation

The Cyclops, gouache and oil by Odilon Redon

 Camera Projection Model

▪ Pinhole Camera Model

A large-scale camera obscura A modern-day camera obscura An Image in camera obscura

at San Francisco, California at Portslade, England

camera obscura

pinhole
image
image plane
object

Images are from Wikipedia (https://en.wikipedia.org/wiki/Camera_obscura). 28

Camera Projection Model

▪ Pinhole Camera Model cf. conclusion in advance

image plane image plane

camera center

principal axis
focal length focal length

camera obscura

pinhole
image
image plane
object

29
Camera Projection Model

▪ Pinhole Camera Model

image plane

principal axis
+

principal point
+

image plane

30
Camera Projection Model

▪ Pinhole Camera Model

image plane
+

principal axis
+

principal point
+

image plane

31
Camera Projection Model

▪ Example: Simple Camera Calibration

– Unknown: Focal length (𝑓) (unit: [pixel])
– Given: The observed object height (ℎ) on the image plane (unit: [pixel])
– Assumptions
• The object height (𝐻) and distance (𝑍) from the camera are known.
• The object is aligned with the image plane.

object

image plane

32
Camera Projection Model

▪ Example: Simple Distance Measurement

– Unknown: Object distance (𝑍) (unit: [m])
– Given: The observed object height (ℎ) on the image plane (unit: [pixel])
– Assumptions
• The object height (𝐻) and focal length (𝑓) are known.
• The object is aligned with the image plane.

object

image plane

33
Camera Projection Model

▪ Example: Simple Height Measurement

– Unknown: Object height (𝐻) (unit: [m])
– Given: The observed object height (ℎ) on the image plane (unit: [pixel])
– Assumptions
• The object distance (𝑍) from the camera and focal length (𝑓) are known.
• The object is aligned with the image plane.

object

image plane

camera

34
 Camera Projection Model vanishing point

▪ Vanishing Points
– A point on the image plane where mutually parallel lines in 3D space converge
• A vector to the vanishing point is parallel to the lines.
• A vector to the vanishing point is parallel to the reference plane made by the lines.

An images is from Wikipedia (https://en.wikipedia.org/wiki/Vanishing_point). 35

Camera Projection Model vanishing point

▪ Vanishing Points
– A point on the image plane where mutually parallel lines in 3D space converge
• A vector to the vanishing point is parallel to the lines.
• A vector to the vanishing point is parallel to the reference plane made by the lines.

top-view

parallel lines

vanishing point

side-view
lines

36
Camera Projection Model vanishing point

▪ Example: Simple Visual Compass + principal point

– Unknown: Tilt angle (𝜃𝑥 ) (unit: [rad])
– Given: A vanishing point (𝑥, 𝑦) from the reference plane
– Assumptions
• The focal length (𝑓) is known.
• The principal point (𝑐𝑥 , 𝑐𝑦 ) is known or selected as the center of images.
• The camera has no roll, 𝜃𝑧 = 0.

cf. The tilt angle in this page is defined as the opposite direction of the common notation.

vanishing point

reference plane

37
Camera Projection Model vanishing point

▪ Example: Simple Visual Compass + principal point

– Unknown: Tilt angle (𝜃𝑥 ) (unit: [rad])
– Given: A vanishing point (𝑥, 𝑦) from the reference plane
– Assumptions
• The focal length (𝑓) is known.
• The principal point (𝑐𝑥 , 𝑐𝑦 ) is known or selected as the center of images.
• The camera has no roll, 𝜃𝑧 = 0.

cf. Similarly, the pan angle (𝜃𝑦 ) w.r.t. rails also can be calculated using 𝑥 instead of 𝑦.

vanishing point

reference plane

38
 head point

Camera Projection Model

contact point

▪ Example: Object Localization and Measurement #1

– Unknown: Object position and height (unit: [m])
– Given: The object’s contact and head points on the image (unit: [pixel])
– Assumptions
• The focal length, principal points, and camera height, are known.
• The camera is aligned to the reference plane.
• The object is on the reference plane.

head point
+

contact point

39
 head point

Camera Projection Model

contact point

▪ Example: Object Localization and Measurement #2

– Unknown: Object position and height (unit: [m])
– Given: The object’s contact and head points on the image (unit: [pixel])
– Assumptions
• The focal length, principal points, and camera height, are known.
• The camera is aligned to the reference plane. The camera orientation is known.
• The object is on the reference plane.

∵ same camera center

head point
+

contact point

40
Camera Projection Model

▪ Example: Object Localization and Measurement #2 [object_localization.cpp]

41
1. #include "opencv2/opencv.hpp"

2. ...

3. int main()
4. {
5. const char* input = "data/daejeon_station.png";
6. double f = 810.5, cx = 480, cy = 270, L = 3.31;
7. cv::Point3d cam_ori(DEG2RAD(-18.7), DEG2RAD(-8.2), DEG2RAD(2.0));
8. cv::Range grid_x(-2, 3), grid_z(5, 35);

9. ...

10. while (true)

11. {
12. cv::Mat image_copy = image.clone();
13. if (drag.end.x > 0 && drag.end.y > 0)
14. {
15. // Calculate object location and height
16. cv::Point3d c = R.t() * cv::Point3d(drag.start.x - cx, drag.start.y - cy, f);
17. if (c.y < DBL_EPSILON) continue; // Skip the degenerate case (beyond the horizon)
18. cv::Point3d h = R.t() * cv::Point3d(drag.end.x - cx, drag.end.y - cy, f);
19. double Z = c.z / c.y * L, X = c.x / c.y * L, H = (c.y / c.z - h.y / h.z) * Z;

20. // Draw head/contact points and location/height

21. cv::line(image_copy, drag.start, drag.end, cv::Vec3b(0, 0, 255), 2);
22. cv::circle(image_copy, drag.end, 4, cv::Vec3b(255, 0, 0), -1);
23. cv::circle(image_copy, drag.start, 4, cv::Vec3b(0, 255, 0), -1);
24. cv::putText(image_copy, cv::format("X:%.2f, Z:%.2f, H:%.2f", X, Z, H), ...);
25. }

26. // Show the image

27. cv::imshow("3DV Tutorial: Object Localization and Measurement", image_copy);
28. int key = cv::waitKey(1);
29. if (key == 27) break; // 'ESC' key: Exit
30. }
31. return 0;
32. }

42
Camera Projection Model

▪ Camera Matrix K

specialized

+ generalized

image plane (unit: [px]) normalized image plane (unit: [m])

principal axis

43
Camera Projection Model

▪ Homogeneous Coordinates (a.k.a. projective coordinates)

– It describes 𝑛-dimensional project space as 𝑛 + 1-dimensional coordinate system. It hold non-
conventional equivalence relationship: 𝑥1 , 𝑥2 , … , 𝑥𝑛+1 ~ 𝜆𝑥1 , 𝜆𝑥2 , … , 𝜆𝑥𝑛+1 such that (0 ≠ 𝜆 ∈ ℝ)
• e.g. (5, 12) is written as (5, 12, 1) which is also equal to (10, 24, 2) or (30, 36, 3) or …

– On the previous slide,

image plane (unit: [px]) normalized image plane (unit: [m])

principal axis

44
Camera Projection Model

▪ Why Homogeneous Coordinates?

– An affine transformation (𝐲 = A𝐱 + 𝐛) is formulated by a single matrix multiplication.
– A light ray (line at the camera center) is observed as a point on the image plane.
• A plane at the camera center is observed as a line on the image plane.
• A conic whose peak is at the camera center is observed as a conic section on the image plane.
– A point at infinity (a.k.a. ideal point) is numerically represented by 𝑤 = 0.
– A point and line (𝑎𝑥 + 𝑏𝑦 + 𝑐 = 0) are described beautifully as like 𝐥⊺ 𝐱 = 0 or 𝐱 ⊺ 𝐥 = 0 (𝐥 = [𝑎, 𝑏, 𝑐]⊺ ).
• Intersection of two lines: 𝐱 = 𝐥1 × 𝐥2
Duality of a point and line
• A line by two points: 𝐥 = 𝐱1 × 𝐱 2
– …

45
Camera Projection Model

▪ Projection Matrix P
– If a 3D point is not based on the camera coordinate, the point should be transformed.

3x4 matrix
world coordinate

camera coordinate

– The camera pose can be derived from R⊺ and −R⊺ 𝐭. 3x4 matrix

▪ Camera Parameters ~ projection matrix

– Intrinsic parameters ~ camera matrix
• e.g. focal length, principle point, skew, distortion coefficient, …
– Extrinsic parameters ~ point transformation
• e.g. rotation and translation

46
 A point cloud: ”data/box.xyz”

Camera Projection Model

▪ Example: Image Formation [image_formation.cpp]

CloudCompare (https://danielgm.net/cc/)

Addictive Gaussian noise: 𝝈 = 1 [px]

47
Camera Projection Model

▪ Example: Image Formation [image_formation.cpp]

1. #include "opencv2/opencv.hpp"

2. #define Rx(rx) (cv::Mat_<double>(3, 3) << 1, 0, 0, 0, cos(rx), -sin(rx), 0, sin(rx), cos(rx))

3. #define Ry(ry) (cv::Mat_<double>(3, 3) << cos(ry), 0, sin(ry), 0, 1, 0, -sin(ry), 0, cos(ry))
4. #define Rz(rz) (cv::Mat_<double>(3, 3) << cos(rz), -sin(rz), 0, sin(rz), cos(rz), 0, 0, 0, 1)

5. int main()
6. {
7. // The given camera configuration: focal length, principal point, image resolution, position, and orientation
8. double f = 1000, cx = 320, cy = 240, noise_std = 1;
9. cv::Size img_res(640, 480);
10. std::vector<cv::Point3d> cam_pos = { cv::Point3d(0, 0, 0), cv::Point3d(-2, -2, 0), ... };
11. std::vector<cv::Point3d> cam_ori = { cv::Point3d(0, 0, 0), cv::Point3d(-CV_PI / 12, CV_PI / 12, 0), ... };

12. // Load a point cloud in the homogeneous coordinate

13. FILE* fin = fopen("data/box.xyz", "rt");
14. if (fin == NULL) return -1;
15. cv::Mat X;
16. while (!feof(fin))
17. {
18. double x, y, z;
19. if (fscanf(fin, "%lf %lf %lf", &x, &y, &z) == 3) X.push_back(cv::Vec4d(x, y, z, 1));
20. }
21. fclose(fin);
22. X = X.reshape(1).t(); // Convert to a 4 x N matrix
24. // Generate images for each camera pose
25. cv::Mat K = (cv::Mat_<double>(3, 3) << f, 0, cx, 0, f, cy, 0, 0, 1);
26. for (size_t i = 0; i < cam_pos.size(); i++)
27. {
28. // Derive a projection matrix
29. cv::Mat Rc = Rz(cam_ori[i].z) * Ry(cam_ori[i].y) * Rx(cam_ori[i].x);
30. cv::Mat tc(cam_pos[i]);
31. cv::Mat Rt;
32. cv::hconcat(Rc.t(), -Rc.t() * tc, Rt);
33. cv::Mat P = K * Rt;

34. // Project the points (cf. OpenCV provides 'cv::projectPoints()' with consideration of distortion.)
35. cv::Mat x = P * X;
36. x.row(0) = x.row(0) / x.row(2);
37. x.row(1) = x.row(1) / x.row(2);
38. x.row(2) = 1;

39. // Add Gaussian noise

40. cv::Mat noise(2, x.cols, x.type());
41. cv::randn(noise, cv::Scalar(0), cv::Scalar(noise_std));
42. x.rowRange(0, 2) = x.rowRange(0, 2) + noise;

43. // Show and store the points

44. cv::Mat image = cv::Mat::zeros(img_res, CV_8UC1);
45. for (int c = 0; c < x.cols; c++)
46. {
47. cv::Point p(x.col(c).rowRange(0, 2));
48. if (p.x >= 0 && p.x < img_res.width && p.y >= 0 && p.y < img_res.height)
49. cv::circle(image, p, 2, 255, -1);
50. }
51. cv::imshow(cv::format("3DV_Tutorial: Image Formation %d", i), image);

52. FILE* fout = fopen(cv::format("image_formation%d.xyz", i).c_str(), "wt");

53. if (fout == NULL) return -1;
54. for (int c = 0; c < x.cols; c++)
55. fprintf(fout, "%f %f 1\n", x.at<double>(0, c), x.at<double>(1, c));
56. fclose(fout);
57. }

58. cv::waitKey(0);
49
59. return 0;
60. }
 Camera Projection Model

▪ Geometric Distortion
– Geometric distortion models are usually defined on the normalized image plane.
– Polynomial distortion model (a.k.a. Brown-Conrady model)
• Radial distortion: 𝑘1 , 𝑘2 , …
• Tangential distortion: 𝑝1 , 𝑝2 , … (usually negligible)

barrel dist.

pincushion dist. Why? lens and sensor

are not parallel.

radial distortion mustache dist. tangential distortion

– Field-of-view (FOV) model
• Radial distortion is modeled by a tangent function (good for severe distortion; e.g. fisheye lens).

Images are from Camera Calibration Toolbox for MATLAB (http://www.vision.caltech.edu/bouguetj/calib_doc/htmls/example.html), 50

Wikipedia (https://en.wikipedia.org/wiki/Distortion_(optics)), and MathWorks (https://mathworks.com/help/vision/ug/camera-calibration.html)
 Camera Projection Model

▪ Geometric Distortion Correction

– Input: The original image
– Output: Its rectified image (w/o geometric distortion)
– Given: Its camera matrix and distortion coefficient
– Solutions for the polynomial distortion model
• OpenCV cv::undistort() and cv::undistortPoints() (cf. included in imgproc module)
↔ cv::projectPoints() (cf. included in calib3d module)
• Camera Distortion Correction: Theory and Practice, http://darkpgmr.tistory.com/31

distortion correction

K1: 1.105763E-01
K2: 1.886214E-02
K3: 1.473832E-02
P1:-8.448460E-03
P2:-7.356744E-03

Images are from Shawn Becker (http://alumni.media.mit.edu/~sbeck/) 51

Camera Projection Model

▪ Example: Geometric Distortion Correction [distortion_correction.cpp]

Original Image Rectified Image

52
1. #include "opencv2/opencv.hpp"

2. int main()
3. {
4. const char* input = "data/chessboard.avi";
5. cv::Matx33d K(432.7390364738057, 0, 476.0614994349778, 0, 431.2395555913084, 288.7602152621297, 0, 0, 1);
6. std::vector<double> dist_coeff = { -0.2852754904152874, 0.1016466459919075, ... };

7. // Open a video
8. cv::VideoCapture video;
9. if (!video.open(input)) return -1;

10. // Run distortion correction

11. bool show_rectify = true;
12. cv::Mat map1, map2;
13. while (true)
14. {
15. // Grab an image from the video
16. cv::Mat image;
17. video >> image;
18. if (image.empty()) break;

19. // Rectify geometric distortion (cf. 'cv::undistort()' can be applied for one-time remapping.)
20. if (show_rectify)
21. {
22. if (map1.empty() || map2.empty())
23. cv::initUndistortRectifyMap(K, dist_coeff, cv::Mat(), cv::Mat(), image.size(), CV_32FC1, map1, map2);
24. cv::remap(image, image, map1, map2, cv::InterpolationFlags::INTER_LINEAR);
25. }

26. // Show the image

27. cv::imshow("3DV Tutorial: Distortion Correction", image);
28. int key = cv::waitKey(1);
29. if (key == 27) break; // 'ESC' key: Exit
30. else if (key == 9) show_rectify = !show_rectify; // 'Tab' key: Toggle rectification
31. ...
32. }

33. video.release();
34. return 0;
35. }
53
General 2D-3D Geometry

▪ Camera Calibration
– Unknown: Intrinsic + extrinsic parameters (5* + 6 DoF)
• The number of intrinsic parameters* can be varied w.r.t. user preference.
– Given: 3D points 𝐗1 , 𝐗 2 , …, 𝐗 𝑛 and their projected points 𝐱1 , 𝐱 2 , …, 𝐱 𝑛
– Constraints: 𝑛 x projection 𝐱 𝑖 = K R | 𝐭 𝐗 𝑖

9 x 6 chessboard

54
General 2D-3D Geometry

▪ Camera Calibration
– Unknown: Intrinsic + 𝑚 x extrinsic parameters (5* + 𝑚 x 6 DoF)
Given: 3D points 𝐗1 , 𝐗 2 , …, 𝐗 𝑛 and their projected points from 𝑗th camera 𝐱 𝑖
𝑗
–
Constraints: 𝑛 x 𝑚 x projection 𝐱 𝑖 = K R𝑗 | 𝐭𝑗 𝐗 𝑖
𝑗
–
– Solutions
• OpenCV cv::calibrateCamera() and cv::initCameraMatrix2D()
• Camera Calibration Toolbox for MATLAB, http://www.vision.caltech.edu/bouguetj/calib_doc/
• GML C++ Camera Calibration Toolbox, http://graphics.cs.msu.ru/en/node/909
• DarkCamCalibrator, http://darkpgmr.tistory.com/139
9 x 6 chessboard

(𝑚: the number of images)

55
 General 2D-3D Geometry

▪ Example: Camera Calibration [camera_calibration.cpp]

960 x 540

## Camera Calibration Results

* The number of applied images = 22
* RMS error = 0.473353
field-of-view (FOV) = focal length
𝑓
* Camera matrix (K) =
- Horizontal: 2 × tan−1 𝑥 = 84° [432.7390364738057, 0, 476.0614994349778] ~ close to the center of the image, (480, 270)
𝑊/2
𝑓𝑦
- Vertical: 2 × tan−1 ℎ/2 = 116° [0, 431.2395555913084, 288.7602152621297]
[0, 0, 1]
* Distortion coefficient (k1, k2, p1, p2, k3, ...) =
[-0.2852754904152874, 0.1016466459919075,
-0.0004420196146339175, 0.0001149909868437517,
-0.01803978785585194] ~ close to zero (usually negligible)

56
1. #include "opencv2/opencv.hpp"

2. int main()
3. {
4. const char* input = "data/chessboard.avi";
5. cv::Size board_pattern(10, 7);
6. float board_cellsize = 0.025f;

7. // Select images
8. std::vector<cv::Mat> images;
9. ...

10. // Find 2D corner points from the given images

11. std::vector<std::vector<cv::Point2f>> img_points;
12. for (size_t i = 0; i < images.size(); i++)
13. {
14. std::vector<cv::Point2f> pts;
15. if (cv::findChessboardCorners(images[i], board_pattern, pts))
16. img_points.push_back(pts);
17. }
18. if (img_points.empty()) return -1;

19. // Prepare 3D points of the chess board

20. std::vector<std::vector<cv::Point3f>> obj_points(1);
21. for (int r = 0; r < board_pattern.height; r++)
22. for (int c = 0; c < board_pattern.width; c++)
23. obj_points[0].push_back(cv::Point3f(board_cellsize * c, board_cellsize * r, 0));
24. obj_points.resize(img_points.size(), obj_points[0]); // Copy

25. // Calibrate the camera

26. cv::Mat K = cv::Mat::eye(3, 3, CV_64F);
27. cv::Mat dist_coeff = cv::Mat::zeros(4, 1, CV_64F);
28. std::vector<cv::Mat> rvecs, tvecs;
29. double rms = cv::calibrateCamera(obj_points, img_points, images[0].size(), K, dist_coeff, rvecs, tvecs);

30. // Report calibration results

31. ...
32. return 0;
33. }

57
General 2D-3D Geometry

▪ Absolute Camera Pose Estimation (Perspective-n-Point; PnP)

– Unknown: Camera pose (6 DoF)
– Given: 3D points 𝐗1 , 𝐗 2 , …, 𝐗 𝑛 , their projected points 𝐱1 , 𝐱 2 , …, 𝐱 𝑛 , and camera matrix K
– Constraints: 𝑛 x projection 𝐱 𝑖 = K R | 𝐭 𝐗 𝑖
– Solutions (𝑛 ≥ 3) → 3-point algorithm
• OpenCV cv::solvePnP() and cv::solvePnPRansac()
• Efficient PnP (EPnP), http://cvlab.epfl.ch/EPnP/

landmark map

58
General 2D-3D Geometry

▪ Example: Pose Estimation (Chessboard) [pose_estimation_chessboard.cpp]

- Prepare a 3D box (AR model)

- Prepare 3D points (chessboard)

- Grab an image
- Find a chessboard on the image

[Camera Pose Estimation]

- Estimate camera pose using
cv::solvePnP()

[AR Rendering]
- Draw the 3D box on the image
using cv::projectPoints()

59
1. #include "opencv2/opencv.hpp"

2. int main()
3. {
4. const char* input = "data/chessboard.avi";
5. cv::Matx33d K(432.7390364738057, 0, 476.0614994349778, 0, 431.2395555913084, 288.7602152621297, 0, 0, 1);
6. std::vector<double> dist_coeff = { -0.2852754904152874, 0.1016466459919075, ... };
7. cv::Size board_pattern(10, 7);
8. double board_cellsize = 0.025;

9. // Open a video
10. cv::VideoCapture video;
11. if (!video.open(input)) return -1;

12. // Prepare a 3D box for simple AR

13. std::vector<cv::Point3d> box_lower = { cv::Point3d(4 * board_cellsize, 2 * board_cellsize, 0), ... };
14. std::vector<cv::Point3d> box_upper = { cv::Point3d(4 * board_cellsize, 2 * board_cellsize, -board_cellsize), ...) };

15. // Prepare 3D points on a chessboard

16. std::vector<cv::Point3d> obj_points;
17. for (int r = 0; r < board_pattern.height; r++)
18. for (int c = 0; c < board_pattern.width; c++)
19. obj_points.push_back(cv::Point3d(board_cellsize * c, board_cellsize * r, 0));

60
31. // Run pose estimation
32. while (true)
33. {
34. // Grab an image from the video
35. cv::Mat image;
36. video >> image;
37. if (image.empty()) break;

38. // Estimate camera pose

39. std::vector<cv::Point2d> img_points;
40. bool success = cv::findChessboardCorners(image, board_pattern, img_points, ...);
41. if (success)
42. {
43. cv::Mat rvec, tvec;
44. cv::solvePnP(obj_points, img_points, K, dist_coeff, rvec, tvec);

45. // Draw the box on the image

46. cv::Mat line_lower, line_upper;
47. cv::projectPoints(box_lower, rvec, tvec, K, dist_coeff, line_lower);
48. cv::projectPoints(box_upper, rvec, tvec, K, dist_coeff, line_upper);
49. ...

50. // Print camera position

51. cv::Mat R;
52. cv::Rodrigues(rvec, R);
53. cv::Mat p = -R.t() * tvec;
54. cv::String info = cv::format("XYZ: [%.3f, %.3f, %.3f]", cv::Point3d(p));
55. cv::putText(image, info, cv::Point(5, 15), cv::FONT_HERSHEY_PLAIN, 1, cv::Vec3b(0, 255, 0));
56. }

57. // Show the image

58. cv::imshow("3DV Tutorial: Pose Estimation", image);
59. int key = cv::waitKey(1);
60. ...
61. }

62. video.release();
63. return 0;
64. }
61
 General 2D-3D Geometry

▪ Example: Pose Estimation (Book) [pose_estimation_book1.cpp]

- Load the cover image

- Extract ORB features
- Build 3D object points 𝐗1 , 𝐗 2 , …

Pass tentative matches Pass correct matches

- Grab an input image [Outlier Rejection] [Camera Pose Estimation]

- Extract ORB features 𝐱1 , 𝐱 2 , … - Remove wrong matches using - Estimate camera pose using
- Match them to the object features cv::solvePnPRansac() cv::solvePnP()

[Camera Pose Update]

- Intrinsic: K (given and fixed)
Draw the box using cv::projectPoints()
- Extrinsic: R, 𝐭

62
 General 2D-3D Geometry

▪ Example: Pose Estimation (Book) + Camera Calibration [pose_estimation_book2.cpp]

(∵ unknown, autofocus)
- Load the cover image
- Extract ORB features
- Build 3D object points 𝐗1 , 𝐗 2 , …

Pass tentative matches Pass correct matches

- Grab an input image [Outlier Rejection] [Camera Pose Estimation]

- Extract ORB features 𝐱1 , 𝐱 2 , … - Remove wrong matches using - Estimate camera parameters using
- Match them to the object features cv::solvePnPRansac() cv::calibrateCamera()

[Camera Pose Update]

- Intrinsic: K (initially known)
Draw the box using cv::projectPoints()
- Extrinsic: R, 𝐭

63
 General 2D-3D Geometry
[pose_estimation_book3.cpp]
▪ Example: Pose Estimation (Book) + Camera Calibration – Initially Given K
(∵ unknown)
- Load the cover image
- Extract ORB features
- Build 3D object points 𝐗1 , 𝐗 2 , …

Pass tentative matches Pass correct matches

- Grab an input image [Outlier Rejection] [Camera Pose Estimation]

- Extract ORB features 𝐱1 , 𝐱 2 , … - Remove wrong matches using - Estimate camera parameters using
- Match them to the object features cv::findHomography() cv::calibrateCamera()
Free from K

[Camera Pose Update]

- Intrinsic: K (initially known)
Draw the box using cv::projectPoints()
- Extrinsic: R, 𝐭

64
Two-view Geometry
Planar 2D-2D Geometry (Projective Geometry)
- Planar Homography Estimation
General 2D-2D Geometry (Epipolar Geometry)
- Relative Camera Pose Estimation
- Triangulation

윤두서(1668-1715) 자화상, 국보 제240호

Korean National Treasure No. 240
Planar 2D-2D Geometry (Projective Geometry)

plane

transformation

shear
composition
rotation

translation

perspective
scaling / aspect ratio projection

66
 Overview of Projective Geometry
Euclidean Transform Similarity Transform Affine Transform Projective Transform
(a.k.a. Rigid Transform) (a.k.a. Planar Homography)

Matrix Forms

DoF 3 4 6 8

Transformations
- rotation O O O O
- translation O O O O
- scaling X O O O
- aspect ratio X X O O
- shear X X O O
- perspective projection X X X O
Invariants
- length O X X X
- angle O O X X
- ratio of lengths O O X X
- parallelism O O O X
- incidence O O O O
- cross ratio O O O O
cv::getAffineTransform() cv::getPerspectiveTransform()
cv::estimateRigidTransform() -
OpenCV Functions
- cv::findHomography()
cv::warpAffine() cv::warpPerspective()

cf. Similarly 3D transformations (3D-3D geometry) are represented as 4x4 matrices. 67

Planar 2D-2D Geometry (Projective Geometry)

▪ Planar Homography Estimation

– Unknown: Planar homography (8 DoF)
– Given: Point correspondence (𝐱1 , 𝐱1′ ), …, (𝐱 𝑛 , 𝐱 𝑛′ )
– Constraints: 𝑛 x projective transformation 𝐱 𝑖′ = H𝐱 𝑖
– Solutions (𝑛 ≥ 4) → 4-point algorithm
• OpenCV cv::getPerspectiveTransform() and cv::findHomography()
• cf. More simplified transformations need less number of minimal correspondence.
– Affine (𝑛 ≥ 3), similarity (𝑛 ≥ 2), Euclidean (𝑛 ≥ 2)

– [Note] Planar homography can be decomposed as relative camera pose.

• OpenCV cv::decomposeHomographyMat()
• cf. However, the decomposition needs to know camera matrices.

relative pose R, 𝐭

68
 Planar 2D-2D Geometry (Projective Geometry)

▪ Example: Perspective Distortion Correction [perspective_correction.cpp]

point_dst

Click
point_src

69
1. #include "opencv2/opencv.hpp"

2. void MouseEventHandler(int event, int x, int y, int flags, void* param)

3. {
4. ...
5. std::vector<cv::Point> *points_src = (std::vector<cv::Point> *)param;
6. points_src->push_back(cv::Point(x, y));
7. ...
8. }

9. int main()
10. {
11. const char* input = "data/sunglok_desk.jpg";
12. cv::Size card_size(450, 250);

13. // Prepare the rectified points

14. std::vector<cv::Point> points_dst;
15. points_dst.push_back(cv::Point(0, 0));
16. points_dst.push_back(cv::Point(card_size.width, 0));
17. points_dst.push_back(cv::Point(0, card_size.height));
18. points_dst.push_back(cv::Point(card_size.width, card_size.height));

19. // Load an image

20. cv::Mat original = cv::imread(input);
21. if (original.empty()) return -1;

22. // Get the matched points from a user's mouse

23. std::vector<cv::Point> points_src;
24. cv::namedWindow("3DV Tutorial: Perspective Correction");
25. cv::setMouseCallback("3DV Tutorial: Perspective Correction", MouseEventHandler, &points_src);
26. while (points_src.size() < 4)
27. {
28. ...
29. }
30. if (points_src.size() < 4) return -1;

31. // Calculate planar homography and rectify perspective distortion

32. cv::Mat H = cv::findHomography(points_src, points_dst);
33. cv::Mat rectify;
34. cv::warpPerspective(original, rectify, H, card_size);

35. // Show the rectified image

36. ... 70
37. }
Planar 2D-2D Geometry (Projective Geometry)

▪ Example: Planar Image Stitching [image_stitching.cpp]

1. #include "opencv2/opencv.hpp"

2. int main()
3. {
4. // Load two images (cf. We assume that two images have the same size and type)
5. cv::Mat image1 = cv::imread("data/hill01.jpg");
6. cv::Mat image2 = cv::imread("data/hill02.jpg");
7. if (image1.empty() || image2.empty()) return -1;

8. // Retrieve matching points

9. cv::Ptr<cv::FeatureDetector> fdetector = cv::ORB::create();
10. std::vector<cv::KeyPoint> keypoint1, keypoint2;
11. cv::Mat descriptor1, descriptor2;
12. fdetector->detectAndCompute(image1, cv::Mat(), keypoint1, descriptor1);
13. fdetector->detectAndCompute(image2, cv::Mat(), keypoint2, descriptor2);
14.
15.
cv::Ptr<cv::DescriptorMatcher> fmatcher = cv::DescriptorMatcher::create(“BruteForce-Hamming”);
std::vector<cv::DMatch> match;
+
16. fmatcher->match(descriptor1, descriptor2, match);

17. // Calculate planar homography and merge them

18. std::vector<cv::Point2f> points1, points2;
19. for (size_t i = 0; i < match.size(); i++)
20. {
21. points1.push_back(keypoint1.at(match.at(i).queryIdx).pt);
22. points2.push_back(keypoint2.at(match.at(i).trainIdx).pt);

=
23. }
24. cv::Mat H = cv::findHomography(points2, points1, cv::RANSAC);
25. cv::Mat merge;
26. cv::warpPerspective(image2, merge, H, cv::Size(image1.cols * 2, image1.rows));
27. merge.colRange(0, image1.cols) = image1 * 1; // Copy

28. // Show the merged image

29. ...
30. } 71
Planar 2D-2D Geometry (Projective Geometry)

▪ Example: 2D Video Stabilization [video_stabilization.cpp]

1. #include "opencv2/opencv.hpp"

2. int main()
3. {
4. // Open a video and get the reference image and feature points
5. cv::VideoCapture video;
6. if (!video.open("data/traffic.avi")) return -1;

7. cv::Mat gray_ref;
8. video >> gray_ref;
9. if (gray_ref.empty())
10. {
11. video.release();
12. return -1;
13. }
14. if (gray_ref.channels() > 1) cv::cvtColor(gray_ref, gray_ref, cv::COLOR_RGB2GRAY);

15. std::vector<cv::Point2f> point_ref; A shaking CCTV video: “data/traffic.avi”

16. cv::goodFeaturesToTrack(gray_ref, point_ref, 2000, 0.01, 10);
17. if (point_ref.size() < 4)
18. {
19. video.release();
20. return -1;
21. }

72
22. // Run and show video stabilization
23. while (true)
24. {
25. // Grab an image from the video
26. cv::Mat image, gray;
27. video >> image;
28. if (image.empty()) break;
29. if (image.channels() > 1) cv::cvtColor(image, gray, cv::COLOR_RGB2GRAY);
30. else gray = image.clone();

31. // Extract optical flow and calculate planar homography

32. std::vector<cv::Point2f> point;
33. std::vector<uchar> m_status;
34. cv::Mat err, inlier_mask;
35. cv::calcOpticalFlowPyrLK(gray_ref, gray, point_ref, point, m_status, err);
36. cv::Mat H = cv::findHomography(point, point_ref, inlier_mask, cv::RANSAC);

37. // Synthesize a stabilized image

38. cv::Mat warp;
39. cv::warpPerspective(image, warp, H, cv::Size(image.cols, image.rows));

40. // Show the original and rectified images together

41. for (int i = 0; i < point_ref.size(); i++)
42. {
43. if (inlier_mask.at<uchar>(i) > 0) cv::line(image, point_ref[i], point[i], cv::Vec3b(0, 0, 255));
44. else cv::line(image, point_ref[i], point[i], cv::Vec3b(0, 127, 0));
45. }
46. cv::hconcat(image, warp, image);
47. cv::imshow("3DVT Tutorial: Video Stabilization", image);
48. if (cv::waitKey(1) == 27) break; // 'ESC' key
49. }

50. video.release();
51. return 0;
52. }

Assumption: A plane observed by two views

- Image stitching: Distance ≫ depth variation
- Video stabilization: Small motion
73
General 2D-2D Geometry (Epipolar Geometry)

▪ Fundamental Matrix

image plane

camera #1
camera #2

relative pose R, 𝐭

Epipolar Constraint: on the image plane

on the normalized image plane

74
General 2D-2D Geometry (Epipolar Geometry)

▪ Essential Matrix

normalized image plane

camera #1
camera #2

relative pose R, 𝐭

Epipolar Constraint:
(Derivation)

75
General 2D-2D Geometry (Epipolar Geometry)

▪ Epipolar Geometry

image plane epipolar plane

camera #1
baseline camera #2

epipole

76
General 2D-2D Geometry (Epipolar Geometry)
unknown
▪ Epipolar Geometry

epipolar line
image plane

camera #1
baseline camera #2

epipole

Special case) Stereo Cameras

epipolar line

no epipole
relative pose

77
General 2D-2D Geometry (Epipolar Geometry)

▪ Relative Camera Pose Estimation (~ Fundamental/Essential Matrix Estimation)

– Unknown: Rotation and translation R, 𝐭 (5 DoF; up-to scale “scale ambiguity”)
– Given: Point correspondence (𝐱1 , 𝐱1′ ), …, (𝐱 𝑛 , 𝐱 𝑛′ ) and camera matrices K, K’
– Constraints: 𝑛 x epipolar constraint (𝐱′⊺ F𝐱 = 0 or 𝐱ො ′⊺ Eො𝐱 = 0)
– Solutions (OpenCV)
• Fundamental matrix: 7/8-point algorithm (7 DoF)
– Estimation: cv::findFundamentalMat() → 1 solution
– Conversion to E: E = K′⊺ FK
• Essential matrix: 5-point algorithm (5 DoF)
– Estimation: cv::findEssentialMat() → 𝑘 solutions
– Decomposition: cv::decomposeEssentialMat() → 4 solutions “relative pose ambiguity”
– Decomposition with positive-depth check: cv::recoverPose() → 1 solution

image plane

camera #1
camera #2

relative pose R, 𝐭

78
 Epipolar Geometry: Scale Ambiguity

(a) 2-meter-wide tunnel (b) 1-meter-wide tunnel

Images are from Resolving Scale Ambiguity in Monocular Visual Odometry (by Choi et al., 2013). 79
 Epipolar Geometry: Scale Ambiguity

The Sandcrawler @ Star Wars IV: A New Hope (1977)

Images are from http://ricardodelgadoart.blogspot.com/, https://www.galangpratama.com/, and https://www.pinterest.com/ 80

 Epipolar Geometry: Resolving Scale Ambiguity

(a) The reconstructed world (b) The real world

How to resolve scale ambiguity

▪ Additional sensors: speedometers, IMUs, GPSs, depth/distance (stereo, RGB-D, LiDAR, …)
𝐿
▪ More constraints: known size of objects, known and fixed height of camera ( 𝜌 = )
𝑙

Images are from Resolving Scale Ambiguity in Monocular Visual Odometry (by Choi et al., 2013). 81
 Epipolar Geometry: Relative Pose Ambiguity

How to resolve pose ambiguity

▪ Positive depth constraint

Images are from Multiple View Geometry in Computer Vision (by Hartley and Zisserman, 2nd Edition, 2004). 82
 General 2D-2D Geometry (Epipolar Geometry)

▪ Relative Camera Pose Estimation

– Unknown: Rotation and translation R, 𝐭 (5 DoF; up-to scale)
– Given: Point correspondence (𝐱1 , 𝐱1′ ), …, (𝐱 𝑛 , 𝐱 𝑛′ ) and camera matrices K, K’
– Constraints: 𝑛 x epipolar constraint (𝐱′⊺ F𝐱 = 0 or 𝐱ො ′⊺ Eො𝐱 = 0)
– Solutions (OpenCV)
intrinsic & extrinsic
• Fundamental matrix: 7/8-point algorithm (7 DoF) &
camera parameters
– Estimation: cv::findFundamentalMat() → 1 solution
– Conversion to E: E = K′⊺ FK
– Degenerate cases: No translation, correspondence from a single plane
extrinsic
• Essential matrix: 5-point algorithm (5 DoF)
camera parameters
– Estimation: cv::findEssentialMat() → 𝑘 solutions
– Decomposition: cv::decomposeEssentialMat() → 4 solutions
– Decomposition with positive-depth check: cv::recoverPose() → 1 solution
– Degenerate cases: No translation (∵ E = 𝐭 × R)

83
 General 2D-2D Geometry (Epipolar Geometry)

▪ Relative Camera Pose Estimation

– Unknown: Rotation and translation R, 𝐭 (5 DoF; up-to scale)
– Given: Point correspondence (𝐱1 , 𝐱1′ ), …, (𝐱 𝑛 , 𝐱 𝑛′ ) and camera matrices K, K’
– Constraints: 𝑛 x epipolar constraint (𝐱′⊺ F𝐱 = 0 or 𝐱ො ′⊺ Eො𝐱 = 0)
– Solutions (OpenCV)
intrinsic & extrinsic
• Fundamental matrix: 7/8-point algorithm (7 DoF) &
camera parameters
– Estimation: cv::findFundamentalMat() → 1 solution
– Conversion to E: E = K′⊺ FK
– Degenerate cases: No translation, correspondence from a single plane
extrinsic
• Essential matrix: 5-point algorithm (5 DoF)
camera parameters
– Estimation: cv::findEssentialMat() → 𝑘 solutions
– Decomposition: cv::decomposeEssentialMat() → 4 solutions
– Decomposition with positive-depth check: cv::recoverPose() → 1 solution
– Degenerate cases: No translation (∵ E = 𝐭 × R)
intrinsic & extrinsic
• Planar homography: 4-point algorithm (8 DoF)
camera parameters
– Estimation: cv::findHomography() → 1 solutions
+ plane normal
– Conversion to calibrated H: H
෡ = K′−1 HK
– Decomposition: cv::decomposeHomographyMat() → 4 solutions
– Degenerate cases: Correspondence not from a single plane

84
 Overview of Epipolar Geometry

Items General 2D-2D Geometry Planar 2D-2D Geometry

Model Fundamental Matrix (7 DoF) Planar Homography (8 DoF)
Formulation ෡ = K′−1 HK
F = K′−⊺ EK−1 E = K′⊺ FK ෡ K −1
H = K′H H
On Image Planes

Estimation - 7-point algorithm (𝑛 ≥ 7) → 𝑘 solution - 4-point algorithm (𝑛 ≥ 4) → 1 solution

- (normalized) 8-point algorithm → 1 solution - cv::findHomography()
- cv::findFundamentalMat()
Input - (𝐱𝑖 , 𝐱𝑖′ ) [px] on the image plane - (𝐱𝑖 , 𝐱𝑖′ ) [px] on a plane in the image plane
Degenerate - no translational motion - correspondence not from a single plane
Cases - correspondence from a single plane
Decomposition - convert to an essential matrix and decompose it - cv::decomposeHomographyMat()
to R and 𝐭
Model Essentials Matrix (5 DoF) (Calibrated) Planar Homgraphy (8 DoF)
On Normalized Image Planes

Formulation 1 ⊺
E = 𝐭 ×R ෡ =R+
H 𝐭𝐧
𝑑
Estimation - 5-point algorithm (𝑛 ≥ 5) → k solution - 4-point algorithm (𝑛 ≥ 4) → 1 solution
- cv::findEssentialMat() - cv::findHomography()
Input - (ො𝐱𝑖 , 𝐱ො 𝑖′ ) [m] on the normalized image plane - (ො𝐱𝑖 , 𝐱ො 𝑖′ ) [m] on a plane in the normalized image plane
Degenerate - no translational motion - correspondence not from a single plane
Cases
Decomposition - cv::decomposeEssentialMat() - cv::decomposeHomographyMat() with K = I3×3
to R and 𝐭 - cv::recoverPose()

86
 General 2D-2D Geometry (Epipolar Geometry)

▪ Example: Monocular Visual Odometry (Epipolar Version) [vo_epipolar.cpp]

KITTI Odometry Dataset (#: 00, Left): “data/KITTI_07_L/%06d.png”

relative pose R, 𝐭

Given K Use_5pt = true

[Tentative Matching] [Outlier Rejection] [Relative Pose Estimation]

- Extract optical flow - Remove wrong matches using - Decompose camera pose using
between adjacent images cv::findEssentialMat() cv::recoverPose()
Pass tentative matches Pass the essential matrix E

[Relative Pose Accumulation]

Pass the relative pose R, 𝐭 (cf. ∥ 𝐭 ∥ = 1)

87
Result: “visual_odometry_epipolar.xyz”

88
1. #include "opencv2/opencv.hpp"

2. int main()
3. {
4. const char* input = "data/KITTI_07_L/%06d.png";
5. double f = 707.0912;
6. cv::Point2d c(601.8873, 183.1104);
7. bool use_5pt = true;
8. int min_inlier_num = 100;

9. ...

10. // Open a video and get the initial image

11. cv::VideoCapture video;
12. if (!video.open(input)) return -1;

13. cv::Mat gray_prev;

14. video >> gray_prev;
15. ...

16. // Run and record monocular visual odometry

17. cv::Mat camera_pose = cv::Mat::eye(4, 4, CV_64F);
18. while (true)
19. {
20. // Grab an image from the video
21. cv::Mat image, gray;
22. video >> image;
23. if (image.empty()) break;
24. if (image.channels() > 1) cv::cvtColor(image, gray, cv::COLOR_RGB2GRAY);
25. else gray = image.clone();

26. // Extract optical flow

27. std::vector<cv::Point2f> point_prev, point;
28. cv::goodFeaturesToTrack(gray_prev, point_prev, 2000, 0.01, 10);
29. std::vector<uchar> m_status;
30. cv::Mat err;
31. cv::calcOpticalFlowPyrLK(gray_prev, gray, point_prev, point, m_status, err);
32. gray_prev = gray;

89
33. // Calculate relative pose
34. cv::Mat E, inlier_mask;
35. if (use_5pt)
36. {
37. E = cv::findEssentialMat(point_prev, point, f, c, cv::RANSAC, 0.99, 1, inlier_mask);
38. }
39. else
40. {
41. cv::Mat F = cv::findFundamentalMat(point_prev, point, cv::FM_RANSAC, 1, 0.99, inlier_mask);
42. cv::Mat K = (cv::Mat_<double>(3, 3) << f, 0, c.x, 0, f, c.y, 0, 0, 1);
43. E = K.t() * F * K;
44. }
45. cv::Mat R, t;
46. int inlier_num = cv::recoverPose(E, point_prev, point, R, t, f, c, inlier_mask);

47. // Accumulate relative pose if result is reliable

48. if (inlier_num > min_inlier_num)
49. {
50. cv::Mat T = cv::Mat::eye(4, 4, R.type());
51. T(cv::Rect(0, 0, 3, 3)) = R * 1.0;
52. T.col(3).rowRange(0, 3) = t * 1.0;
53. camera_pose = camera_pose * T.inv();
54. }

55. // Show the image and write camera pose

56. ...
57. }

58. video.release();
59. fclose(camera_traj);
60. return 0;
61. }

90
General 2D-2D Geometry (Epipolar Geometry)

▪ Triangulation (Point Localization)

– Unknown: Position of a 3D point 𝐗 (3 DoF)
– Given: Point correspondence (𝐱, 𝐱’), camera matrices (K, K’), and relative pose (R, 𝐭)
– Constraints: 2 x projection 𝐱 = K I | 𝟎 𝐗, 𝐱′ = K′ R | 𝐭 𝐗
– Solution (OpenCV): cv::triangulatePoints()
Special case) Stereo cameras

image plane

camera #1
camera #2

relative pose R, 𝐭

91
▪ Example: Triangulation (Two-view Reconstruction) [triangulation.cpp]

1. #include "opencv2/opencv.hpp"

2. int main()
3. {
4. // cf. You need to run 'image_formation.cpp' to generate point observation.
5. const char *input0 = "image_formation0.xyz", *input1 = "image_formation1.xyz";
6. double f = 1000, cx = 320, cy = 240;

7. // Load 2D points observed from two views

8. std::vector<cv::Point2d> points0, points1;
9. ...

10. // Estimate relative pose of two views

Result: “triangulation.xyz”
11. cv::Mat F = cv::findFundamentalMat(points0, points1, cv::FM_8POINT);
12. cv::Mat K = (cv::Mat_<double>(3, 3) << f, 0, cx, 0, f, cy, 0, 0, 1);
13. cv::Mat E = K.t() * F * K;
14. cv::Mat R, t;
15. cv::recoverPose(E, points0, points1, K, R, t);

16. // Reconstruct 3D points (triangulation)

17. cv::Mat P0 = K * cv::Mat::eye(3, 4, CV_64F);
18. cv::Mat Rt, X;
19. cv::hconcat(R, t, Rt);
20. cv::Mat P1 = K * Rt;
21. cv::triangulatePoints(P0, P1, points0, points1, X);
22. X.row(0) = X.row(0) / X.row(3);
23. X.row(1) = X.row(1) / X.row(3);
24. X.row(2) = X.row(2) / X.row(3);
25. X.row(3) = 1;

26. // Store the 3D points

27. FILE* fout = fopen("triangulation.xyz", "wt");
28. if (fout == NULL) return -1;
29. for (int c = 0; c < X.cols; c++)
30. fprintf(fout, "%f %f %f\n", X.at<double>(0, c), X.at<double>(1, c), X.at<double>(2, c));
31. fclose(fout);
32. return 0;
33. } 92
Correspondence Problem
Feature Correspondence
Robust Parameter Estimation
- RANSAC
- M-estimator

Images are from .

Quiz

What is it?

600 pixels

94
Quiz

What is it?

600 pixels

95
 Quiz

▪ Why corners ( junction)?

– a.k.a. keypoints, interest points, salient points, and feature points
– cf. ⊂ local invariant features (e.g. corner, edge, region, …)

“Duck”

600 pixels 1095 pixels 600 pixels

▪ Requirements of local invariance features

– repeatability (invariance, robustness), distinctiveness, locality (due to occlusion)
– quantity, accuracy, efficiency

[Reference] Tuytelaars et al., Local Invariant Feature Detectors: A Survey, Foundations and Trends in Computer Graphics and Vision, 2008 96
 Harris Corner (1988)

▪ Key idea: Sliding window

Small 𝜆1 and 𝜆2 Large 𝜆1 and small 𝜆2 Large 𝜆1 and 𝜆2

“flat” region: “edge”: “corner”:

no change no change significant change
in all directions along the edge direction in all directions

Harris corner response:

𝑀 cf. Good-Feature-to-Track (Shi-Tomasi; 1994):

Images are from Matching with Invariant Features (by Frolova and Simakov, Lecture Slides, 2004). 97
 Harris Corner (1988)

▪ Properties
– Invariant to translation, rotation, and intensity shift (𝐼 → 𝐼 + 𝑏) intensity scaling (𝐼 → 𝑎𝐼)
– But variant to image scaling

Edges

Corner!

Images are from Matching with Invariant Features (by Frolova and Simakov, Lecture Slides, 2004). 98
SIFT (Scale-Invariant Feature Transform; 1999)

▪ Key idea: Scale-space (~ image pyramid)

Gaussian Scale-space 𝐿(𝑥, 𝑦, 𝜎) = 𝐺(𝑥, 𝑦, 𝜎) ∗ 𝐼(𝑥, 𝑦)

DOG Scale-space 𝐷 𝑥, 𝑦, 𝜎 = 𝐺 𝑥, 𝑦, 𝑘𝜎 − 𝐺(𝑥, 𝑦, 𝜎)

▪ Part #1) Feature point detection

1. Find local extrema (minima and maxima) in DOG scale-space
2. Localize its position accurately (sub-pixel level) using 3D quadratic function
3. Eliminate low contrast candidates, |𝐷(𝐱)| < 𝜏
4. Eliminate candidates on edges, trace 𝐻 2 𝑟+1 2 𝐷𝑥𝑥 𝐷𝑥𝑦
< where 𝐻 =
det(𝐻) 𝑟 𝐷𝑥𝑦 𝐷𝑦𝑦 99
 SIFT (Scale-Invariant Feature Transform; 1999)

local extrema (N: 11479) feature points (N: 971) feature scales and orientations

cf. SURF (Speeded Up Robust Features; 2006):

- Key idea: Approximation of SIFT using integral image and ...

𝐷𝑦𝑦 𝐷𝑥𝑦

𝐷′𝑦𝑦 𝐷′𝑥𝑦

[Reference] Bay et. al., SURF: Speeded Up Robust Features, CVIU, 2008 100
 FAST (Features from Accelerated Segment Test; 2006)

▪ Key idea: Continuous arc of 𝑁 or more pixels

– Is this patch a corner?
• Is the segment brighter than 𝑝 + 𝑡? Is the segment darker than 𝑝 − 𝑡?
• 𝑡: The threshold of similar intensity
– Too many corners! it needs non-maximum suppression.

▪ Versions
– FAST-9 (𝑁: 9), FAST-12 (𝑁: 12), …
– FAST-ER: Training a decision tree to enhance repeatability with more pixels

[Reference] Rosten et al., FASTER and better: A machine learning approach to corner detection, T-PAMI, 2010 101
 LIFT (Learned Invariant Feature Transform; 2016)

▪ Key idea: Deep neural network

– DET (feature detector) + ORI (orientation estimator) + DESC (feature descriptor)

SIFT LIFT

[Reference] Yi et. al., LIFT: Learned Invariant Feature Transform, ECCV, 2016 102
Overview of Feature Correspondence
▪ Features
– Corners: Harris corner, GFTT (Shi-Tomasi corner), SIFT, SURF, FAST, LIFT, …
– Edges, line segments, regions, …
▪ Feature Descriptors and Matching
– Patch: Raw intensity
• Measures: SSD (sum of squared difference), ZNCC (zero normalized cross correlation), …
– Floating-point descriptors: SIFT, SURF, (DAISY), LIFT, … → e.g. A 128-dim. vector (a histogram of gradients)
• Measures: Euclidean distance, cosine distance, (the ratio of first and second bests)
• Matching: Brute-force matching (𝑂(𝑁 2 )), ANN (approximated nearest neighborhood) search (𝑂(log𝑁))
• Pros (+): High discrimination power
• Cons (–): Heavy computation
– Binary descriptors: BRIEF, ORB, (BRISK), (FREAK), … → e.g. A 128-bit string (a series of intensity comparison)
• Measures: Hamming distance
• Matching: Brute-force matching (𝑂(𝑁 2 ))
• Pros (+): Less storage and faster extraction/matching
• Cons (–): Less performance
▪ Feature Tracking (a.k.a. Optical Flow)
– Optical flow: (Horn-Schunck method), Lukas-Kanade method
• Measures: SSD (sum of squared difference)
• Tracking: Finding displacement of a similar patch
• Pros (+): No descriptor and matching (faster and compact)
• Cons (–): Not working in wide baseline
103
SIFT (Scale-Invariant Feature Transform; 1999)

▪ Part #2) Orientation assignment

1. Derive magnitude and orientation of gradient of each patch
2
𝑚(𝑥, 𝑦) = 𝐿 𝑥 + 1, 𝑦 − 𝐿 𝑥 − 1, 𝑦 + (𝐿 𝑥, 𝑦 + 1 − 𝐿 𝑥, 𝑦 − 1 )2

𝐿 𝑥, 𝑦 + 1 − 𝐿(𝑥, 𝑦 − 1)
𝜃 𝑥, 𝑦 = tan−1
𝐿 𝑥 + 1, 𝑦 − 𝐿(𝑥 − 1, 𝑦)

2. Find the strongest orientation

• Histogram voting (36 bins) with Gaussian-weighted magnitude
feature orientation ≈ 15 [deg]

feature patch gradient magnitude gradient orientation histogram voting

104
 SIFT (Scale-Invariant Feature Transform; 1999)

▪ Part #3) Feature descriptor extraction

– Build a 4x4 gradient histogram (8 bins) from each patch (16x16 pixels)
• Use Gaussian-weighted magnitude again
• Use relative angles w.r.t. the assigned feature orientation
– Encode the histogram into a 128-dimensional vector

feature scales and orientations

2 2 2 1 1 4 2 1 2 …

Keypoint descriptor (dim: 128)

Images are from Jonas Hurrelmann (https://www.eecs.tu-berlin.de/fileadmin/fg144/Courses/06WS/scanning/Jonas/html/index.html). 105

 BRIEF (Binary Robust Independent Elementary Features; 2010)

▪ Key idea: A sequence of intensity comparison of random pairs

– Applying smoothing for stability and repeatability
– Path size: 31 x 31 pixels

▪ Versions: The number of tests

– BRIEF-32, BRIEF-64, BRIEF-128, BRIEF-256 …
▪ Examples of combinations
– CenSurE detector (a.k.a. Star detector) + BRIEF descriptor
– SURF detector + BRIEF descriptor

[Reference] Calonder et. al., BRIEF: Computing a Local Binary Descriptor Very Fast, T-PAMI, 2012 106
 ORB (Oriented FAST and rotated BRIEF, 2011)

▪ Key idea: Adding rotation invariance to BRIEF

– Oriented FAST
• Generate scale pyramid for scale invariance
• Detect FAST-9 points (filtering with Harris corner response)
• Calculate feature orientation by intensity centroid
𝑚
𝜃 = tan−1 𝑚01 where 𝑚𝑝𝑞 = σ𝑥,𝑦 𝑥 𝑝 𝑦 𝑞 𝐼(𝑥, 𝑦)
10

– Rotation-aware BRIEF
• Extract BRIEF descriptors w.r.t. the known orientation
• Use better comparison pairs trained by greedy search

▪ Combination: ORB
– FAST-9 detector (with orientation) + BRIEF-256 descriptor (with trained pairs)

▪ Computing time
– ORB: 15.3 [msec] / SURF: 217.3 [msec] / SIFT: 5228.7 [msec] @ 24 images (640x480) in Pascal dataset

[Reference] Rublee et. al., ORB: An Efficient Alternative to SIFT or SURF, ICCV, 2011 107
Lukas-Kanade Optical Flow

▪ Key idea: Finding movement of a patch

Brightness constancy constraint: 𝐼 𝑥, 𝑦, 𝑡 = 𝐼 𝑥 + ∆𝑥 , 𝑦 + ∆𝑦 , 𝑡 + ∆𝑡
(if same patch) ∆ ∆𝑦
𝐼𝑥 ∆𝑥 + 𝐼𝑦 ∆ + 𝐼𝑡 = 0 because 𝐼 𝑥 + ∆𝑥 , 𝑦 + ∆𝑦 , 𝑡 + ∆𝑡 ≈ 𝐼 𝑥, 𝑦, 𝑡 + 𝐼𝑥 ∆𝑥 + 𝐼𝑦 ∆𝑦 +𝐼𝑡 ∆𝑡
𝑡 𝑡

𝐼𝑥 (𝑝1 ) 𝐼𝑦 (𝑝1 ) −𝐼𝑡 (𝑝1 )

𝑉𝑥
A𝒗 = 𝒃 where A = ⋮ ⋮ ,𝒗= 𝑉 ,𝒃= ⋮ , and 𝑝𝑖 ∈ 𝑊
𝑦
𝐼𝑥 (𝑝𝑛 ) 𝐼𝑦 (𝑝𝑛 ) −𝐼𝑡 (𝑝𝑛 )

∴ 𝒗 = A† 𝒃 = A⊺ A −1 ⊺
A 𝒃
▪ Combination: KLT tracker
– Shi-Tomasi detector (a.k.a. GFTT) + Lukas-Kanade optical flow

109
Why Outliers?
Putative matches (inliers + outliers)

After applying RANSAC (inliers)

Model: Homography H (Parameters: 9 elements of H)

110
RANSAC: Random Sample Consensus

▪ Example: Line Fitting with RANSAC

– Model: 2D line (𝑎𝑥 + 𝑏𝑦 + 𝑐 = 0; 𝑎2 + 𝑏2 = 1)
• Parameters: 𝑎, 𝑏, 𝑐
• 2 points can lead to a line.
– Error function: Geometric distance

𝐱 𝑖 = (𝑥𝑖 , 𝑦𝑖 )

111
RANSAC: Random Sample Consensus

Hypothesis
Generation

Hypothesis
Evaluation

112
RANSAC: Random Sample Consensus

Hypothesis
Generation

Hypothesis
Evaluation

# of Inlier Candidates: 3

113
RANSAC: Random Sample Consensus

Hypothesis
Generation

Hypothesis 7
Evaluation

114
RANSAC: Random Sample Consensus

Parameters:
Hypothesis
▪ The inlier threshold Generation
▪ The number of iterations
- 𝑠: Confidence level
Hypothesis 7
- 𝛾: Inlier ratio
Evaluation
- 𝑑: The number of samples

115
▪ Example: Line Fitting with RANSAC [line_fitting_ransac.cpp]

1. #include "opencv2/opencv.hpp"

2. // Convert a line format, [n_x, n_y, x_0, y_0] to [a, b, c]

3. // cf. A line model in OpenCV: n_x * (x - x_0) = n_y * (y - y_0)
4. #define CONVERT_LINE(line) (cv::Vec3d(line[0], -line[1], -line[0] * line[2] + line[1] * line[3]))

5. int main()
6. {
7. cv::Vec3d truth(1.0 / sqrt(2.0), 1.0 / sqrt(2.0), -240.0); // The line model: a*x + b*y + c = 0 (a^2 + b^2 = 1)
8. int ransac_trial = 50, ransac_n_sample = 2;
9. double ransac_thresh = 3.0; // 3 x 'data_inlier_noise'
10. int data_num = 1000;
11. double data_inlier_ratio = 0.5, data_inlier_noise = 1.0;

12. // Generate data

13. std::vector<cv::Point2d> data;
14. cv::RNG rng;
15. for (int i = 0; i < data_num; i++)
16. {
17. if (rng.uniform(0.0, 1.0) < data_inlier_ratio)
18. {
19. double x = rng.uniform(0.0, 480.0);
20. double y = (truth(0) * x + truth(2)) / -truth(1);
21. x += rng.gaussian(data_inlier_noise);
22. y += rng.gaussian(data_inlier_noise);
23. data.push_back(cv::Point2d(x, y)); // Inlier
24. }
25. else data.push_back(cv::Point2d(rng.uniform(0.0, 640.0), rng.uniform(0.0, 480.0))); // Outlier
26. }

27. // Estimate a line using RANSAC ...

55. // Estimate a line using least-squares method (for reference) ...

59. // Display estimates

60. printf("* The Truth: %.3f, %.3f, %.3f\n", truth[0], truth[1], truth[2]);
61. printf("* Estimate (RANSAC): %.3f, %.3f, %.3f (Score: %d)\n", best_line[0], best_line[1], ..., best_score);
62. printf("* Estimate (LSM): %.3f, %.3f, %.3f\n", lsm_line[0], lsm_line[1], lsm_line[2]);
63. return 0;
64. } 116
27. // Estimate a line using RANSAC
28. int best_score = -1;
29. cv::Vec3d best_line;
30. for (int i = 0; i < ransac_trial; i++)
31. {
32. // Step 1: Hypothesis generation
33. std::vector<cv::Point2d> sample;
34. for (int j = 1; j < ransac_n_sample; j++)
35. {
36. int index = rng.uniform(0, int(data.size()));
37. sample.push_back(data[index]); Hypothesis
38. } Generation
39. cv::Vec4d nnxy;
40. cv::fitLine(sample, nnxy, CV_DIST_L2, 0, 0.01, 0.01);
41. cv::Vec3d line = CONVERT_LINE(nnxy); Hypothesis
Evaluation
42. // Step 2: Hypothesis evaluation
43. int score = 0;
44. for (size_t j = 0; j < data.size(); j++)
45. {
46. double error = fabs(line(0) * data[j].x + line(1) * data[j].y + line(2));
47. if (error < ransac_thresh) score++;
48. }

49. if (score > best_score) Line Fitting Result

50. {
* The Truth: 0.707, 0.707, -240.000
51. best_score = score;
52. best_line = line; * Estimate (RANSAC): 0.712, 0.702, -242.170 (Score: 434)
53. } * Estimate (LSM): 0.748, 0.664, -314.997
54. }

55. // Estimate a line using least squares method (for reference)

56. cv::Vec4d nnxy;
57. cv::fitLine(data, nnxy, CV_DIST_L2, 0, 0.01, 0.01);
58. cv::Vec3d lsm_line = CONVERT_LINE(nnxy);

117
 Least Squares Method, RANSAC, and M-estimator

▪ Least Squares Method

– Find a model while minimizing sum of squared errors,

▪ RANSAC
– Find a model while maximizing the number of supports (~ inlier candidates)
~ minimizing the number of outlier candidates

▪ Why RANSAC was robust to outliers?

In the view of loss functions 𝜌,

– Least squares method:

– RANSAC:

▪ M-estimator (~ weighted least squares) | MSAC (in OpenCV)

– Find a model while minimizing sum of (squared) errors with a truncated loss function

[Reference] Choi et al., Performance Evaluation of RANSAC Family, BMVC, 2009 118
 Ceres (an asteroid)

One-page Tutorial for Ceres Solver

▪ Ceres Solver?
– An open source C++ library for modelling and solving large and complicated optimization problems.
• Since 2010 by Google (BSD license)
– Problem types: 1) Non-linear least squares (with bounds), 2) General unconstrained minimization
– Homepage: http://ceres-solver.org/

▪ Solving Non-linear Least Squares

1. Define residual functions (or cost function or error function)
2. Instantiate ceres::Problem and add residuals using its member function, AddResidualBlock()
• Instantiate each residual 𝑟𝑖 in the form of ceres::CostFunction and add it
– Select how to calculate its derivative (Jacobian)
(ceres::AutoDiffCostFunction or ceres::NumericDiffCostFunction or ceres::SizedCostFunction)
– cf. Automatic derivation (using the chain rule) is recommended for convenience and performance.
• Instantiate its ceres::LossFunction 𝜌𝑖 and add it (if the problem needs robustness against outliers)
3. Instantiate ceres::Solver::Options (and also ceres::Solver::Summary) and configure the option
4. Run ceres::Solve()
loss functions
▪ Solving General Minimization
– ceres::CostFunction → ceres::FirstOrderFunction, ceres::GradientFunction
– ceres::Problem → ceres::GradientProblem
– ceres::Solver → ceres::GradientProblemSolver

Images are from Wikipedia (https://en.wikipedia.org/wiki/Ceres_(dwarf_planet)) and Ceres Solver 119

▪ Example: Line Fitting with M-estimator [line_fitting_m_est.cpp]
1. #include "opencv2/opencv.hpp"
2. #include "ceres/ceres.h"
3. ...
4. struct GeometricError
5. {
6. GeometricError(const cv::Point2d& pt) : datum(pt) { }
7. template<typename T> 1) Define a residual as C++ generic function (T ~ double)
8. bool operator()(const T* const line, T* residual) const cf. The generic is necessary for automatic differentiation.
9. {
10. residual[0] = (line[0] * T(datum.x) + line[1] * T(datum.y) + line[2]) / sqrt(line[0] * line[0] + line[1] * line[1]);
11. return true;
12. }
13. private:
14. const cv::Point2d datum;
15. };

16. int main()

17. {
18. ...
19. // Estimate a line using M-estimator
20. cv::Vec3d opt_line(1, 0, 0);
21. ceres::Problem problem;
22. for (size_t i = 0; i < data.size(); i++) 2) Instantiate a problem and add a residual for each datum
23. {
24. ceres::CostFunction* cost_func = new ceres::AutoDiffCostFunction<GeometricError, 1, 3>(new GeometricError(data[i]));
25. ceres::LossFunction* loss_func = NULL;
26. if (loss_width > 0) loss_func = new ceres::CauchyLoss(loss_width); The dimension of a residual
27. problem.AddResidualBlock(cost_func, loss_func, opt_line.val); The dimension of the first model parameter
28. }
29. ceres::Solver::Options options; 3) Instantiate options and configure it
30. options.linear_solver_type = ceres::ITERATIVE_SCHUR;
31. options.num_threads = 8;
32. options.minimizer_progress_to_stdout = true;
33. ceres::Solver::Summary summary;
34. ceres::Solve(options, &problem, &summary); 4) Solve the minimization problem
35. std::cout << summary.FullReport() << std::endl;
36. opt_line /= sqrt(opt_line[0] * opt_line[0] + opt_line[1] * opt_line[1]); // Normalize
37. ...
38. return 0;
39. } 120
Overview of Robust Parameter Estimation

▪ Bottom-up Approaches (~ Voting) e.g. line fitting, relative pose estimation

– Hough transform
• A datum votes multiple parameter candidates.
– cf. The parameter space is maintained as a multi-dimensional histogram (discretization).
• Score: The number of hits by data
• Selection: Finding a peak on the histogram after voting
– RANSAC family
• A sample of data votes a single parameter candidate.
• Score: The number of inlier candidates (whose error is within threshold)
• Selection: Keeping the best model during RANSAC’s iterations
– cf. RANSAC involves many iterations of parameter estimation and error calculation.

▪ Top-down Approaches e.g. graph SLAM, multi-view reconstruction

– M-estimator
• All data aims to find the best parameter (from its initial guess).
• Score: A cost function
– The cost function includes a truncated loss function.
• Selection: Minimizing the cost function (following its gradient)
– cf. Nonlinear optimization is computationally heavy and leads to a local minima.

121
Multi-view Geometry

The Stanford Multi-Camera Array,

Stanford Computer Graphics Lab.
Bundle Adjustment

▪ Bundle Adjustment
– Unknown: Position of 3D points and each camera’s relative pose (6𝑛 + 3𝑚 DoF)
– Given: Point correspondence, camera matrices, position of 3D points, and each camera’s relative pose
cf. initial values
Constraints: 𝑛 x 𝑚 x projection 𝐱 𝑖 = P𝑗 𝐗 𝑖 = K 𝑗 R𝑗 | 𝐭𝑗 𝐗 𝑖
𝑗
–
– Solution: Non-linear least-square optimization
• Cost Function: Reprojection error

• Optimization: Levenberg–Marquardt method, Gauss-Newton method

camera #m

image plane

relative pose R 𝑚 , 𝐭 𝑚

camera #1
camera #2

relative pose R 2 , 𝐭 2

123
Bundle Adjustment

▪ Bundle Adjustment
– Unknown: Position of 3D points and each camera’s relative pose (6𝑛 + 3𝑚 DoF)
– Given: Point correspondence, camera matrices, position of 3D points, and each camera’s relative pose
cf. initial values
Constraints: 𝑛 x 𝑚 x projection 𝐱 𝑖 = P𝑗 𝐗 𝑖 = K 𝑗 R𝑗 | 𝐭𝑗 𝐗 𝑖
𝑗
–
– Solution: Non-linear least-square optimization
• Cost Function: Reprojection error

• Optimization: Levenberg–Marquardt method, Gauss-Newton method

▪ Bundle Adjustment and Optimization Tools

– OpenCV: No function (but cvsba is available as a sba wrapper.)
– Bundle adjustment: sba (Sparse Bundle Adjustment), SSBA (Simple Sparse Bundle Adjustment), pba
(Multicore Bundle Adjustment)
– (Graph) optimization: g2o (General Graph Optimization), iSAM (Incremental Smoothing and Mapping),
GTSAM, Ceres Solver (Google)

124
Bundle Adjustment

▪ Example: Bundle Adjustment (Global) [bundle_adjustment_global.cpp, bundle_adjustment.hpp]

Result: “triangulation.xyz” Result: “bundle_adjustment_global.xyz”

125
[Given]
- Intrinsic parameters: camera matrices K𝑗 (all cameras has same and fixed camera matrix.)

- 2D point correspondence 𝐱 𝑖 (all points are visible on all camera views.)

𝑗

[Unknown]
- Extrinsic parameters: camera poses

- 3D points 𝐗 𝑖 = 0, 0, 5.5 ⊺
initial values

Ceres Solver (non-linear least-square optimization)

126
1. #include "bundle_adjustment.hpp"

2. int main()
3. {
4. // cf. You need to run 'image_formation.cpp' to generate point observation.
5. const char* input = "image_formation%d.xyz";
6. int input_num = 5;
7. double f = 1000, cx = 320, cy= 240;

8. // Load 2D points observed from multiple views

9. std::vector<std::vector<cv::Point2d>> xs;
10. ... 𝐭

11. // Initialize cameras and 3D points

12. std::vector<cv::Vec6d> cameras(xs.size());
cameras[j]: 0 1 2 3 4 5
13. std::vector<cv::Point3d> Xs(xs.front().size(), cv::Point3d(0, 0, 5.5));

14. // Run bundle adjustment

Axis-angle representation of R
15. ceres::Problem ba; (a.k.a. Rodrigues’ rotation formula)
16. for (size_t j = 0; j < xs.size(); j++)
17. {
18. for (size_t i = 0; i < xs[j].size(); i++)
19. {
20. ceres::CostFunction* cost_func = ReprojectionError::create(xs[j][i], f, cv::Point2d(cx, cy));
21. double* view = (double*)(&(cameras[j]));
22. double* X = (double*)(&(Xs[i]));
23. ba.AddResidualBlock(cost_func, NULL, view, X);
24. }
25. }
26. ceres::Solver::Options options;
27. options.linear_solver_type = ceres::ITERATIVE_SCHUR;
28. options.num_threads = 8;
29. options.minimizer_progress_to_stdout = true;
30. ceres::Solver::Summary summary;
31. ceres::Solve(options, &ba, &summary);

32. // Store the 3D points and camera pose to an XYZ file

33. ...
34. cv::Vec3d rvec(cameras[j][0], cameras[j][1], cameras[j][2]), t(cameras[j][3], cameras[j][4], cameras[j][5]);
35. cv::Matx33d R;
36. cv::Rodrigues(rvec, R);
37. ...
38. return 0; Images are from Wikipedia (https://en.wikipedia.org/wiki/Axis–angle_representation). 127
39. }
1. #include "opencv2/opencv.hpp"
2. #include "ceres/ceres.h"
3. #include "ceres/rotation.h"

4. // Reprojection error for bundle adjustment

5. struct ReprojectionError
6. {
7. ReprojectionError(const cv::Point2d& _x, double _f, cv::Point2d& _c) : x(_x), f(_f), c(_c) { }

8. template <typename T>

9. bool operator()(const T* const camera, const T* const point, T* residuals) const
10. {
11. // X' = R*X + t
12. T X[3];
13. ceres::AngleAxisRotatePoint(camera, point, X);
14. X[0] += camera[3];
15. X[1] += camera[4];
16. X[2] += camera[5];

17. // x' = K*X'

18. T x_p = f * X[0] / X[2] + cx;
19. T y_p = f * X[1] / X[2] + cy;

20. // residual = x – x'

21. residuals[0] = T(x.x) - x_p;
22. residuals[1] = T(x.y) - y_p;
23. return true;
24. }

25. static ceres::CostFunction* create(const cv::Point2d& _x, double _f, cv::Point2d& _c)
26. {
27. return (new ceres::AutoDiffCostFunction<ReprojectionError, 2, 6, 3>(new ReprojectionError(_x, _f, _c)));
28. }

29. private:
30. const cv::Point2d x;
31. const double f;
32. const cv::Point2d c;
33. };
[bundle_adjustment.hpp] 128
Bundle Adjustment

▪ Example: Bundle Adjustment (Incremental) [bundle_adjustment_inc.cpp, bundle_adjustment.hpp]

Result: “triangulation.xyz” Result: “bundle_adjustment_inc.xyz”

[bundle_adjusetment_global.xyz]
- # of iterations: 20
- Reprojection error: 0.113 ← 2637.626

[bundle_adjusetment_inc.xyz]
- # of iterations: 17 + 4 + 6
- Reprojection error: 0.113 ← 722.350

129
[Given]
- Intrinsic parameters: camera matrices K𝑗 (all cameras has same and fixed camera matrix.)

- 2D point correspondence 𝐱 𝑖 (all points are visible on all camera views.)

𝑗

1) Best-pair selection

[Unknown]
- Extrinsic parameters: camera poses

2) Epipolar geometry

3) Triangulation
- 3D points 𝐗 𝑖

130
[Given]
- Intrinsic parameters: camera matrices K𝑗 (all cameras has same and fixed camera matrix.)

- 2D point correspondence 𝐱 𝑖 (all points are visible on all camera views.)

𝑗

[Unknown] 4) Next-view selection

- Extrinsic parameters: camera poses

5) Perspective-n-point (PnP)

- 3D points 𝐗 𝑖
6) Triangulation

131
[Given]
- Intrinsic parameters: camera matrices K𝑗 (all cameras has same and fixed camera matrix.)

- 2D point correspondence 𝐱 𝑖 (all points are visible on all camera views.)

𝑗

[Unknown] 4) Next-view selection Repeat 4 – 7 until the last image

- Extrinsic parameters: camera poses

5) Perspective-n-point (PnP)

- 3D points 𝐗 𝑖
6) Triangulation
7) Bundle Adjustment

132
Applications

▪ Structure-from-Motion (SfM) → 3D Reconstruction, Photo Browsing

– Bundler, COLMAP, MVE, Theia, openMVG, OpenSfM, Toy SfM / VisualSFM (GUI, binary only)
▪ Visual SLAM → Augmented Reality, Navigation (Mapping and Localization)
– PTAM (Parallel Tracking and Mapping), DTAM (Dense Tracking and Mapping), ORB-SLAM2, LSD-SLAM
– cf. Visual loop closure (a.k.a. visual place recognition): DBoW2, FBoW, PoseNet, NetVLAD
▪ Visual Odometry → Navigation (Localization)
– LIBVISO2 (C++ Library for Visual Odometry 2), SVO (Semi-direct Monocular Visual Odometry), DVO
(Direct Sparse Odometry), DeepVO, UnDeepVO

Structure-from-Motion
image set
Visual SLAM
(unordered)
image sequences
(ordered) Visual Odometry

w/o loop closing

w/o global mapping

133
 Structure-from-Motion

▪ Example: Structure-from-Motion (Global) [sfm_global.cpp]

0) Load images and extract features
0) Match features and find good matches (which has enough inliers)

1855 / 5615 205 / 5615 77 / 5615 72 / 5615

1569 / 5821 139 / 5821 82 / 5821

cf. # of inliers / # of matches

1324 / 5763 112 / 5763

1834 / 5767

134
 Structure-from-Motion

▪ Example: Structure-from-Motion (Global) [sfm_global.cpp]

0) Load images and extract features
0) Match features and find good matches (which has enough inliers)

1855 / 5615 1569 / 5615 1324 / 5615 1834 / 5615

205 / 5615

1) Initialize cameras (R, 𝐭, 𝑓, …)

2) Initialize 3D points and build a visibility graph

3) Optimize camera pose and 3D points together (BA)

Ceres Solver (non-linear least-square optimization)

135
Structure-from-Motion

▪ Example: Structure-from-Motion (Global) [sfm_global.cpp]

136
 Structure-from-Motion

▪ Example: Structure-from-Motion (Incremental) [sfm_inc.cpp]

0) Build a viewing graph (well-matched pairs) while finding inliers

1855 / 5615 1569 / 5615 1324 / 5615 1834 / 5615

1) Select the best pair

2) Estimate relative pose from the best two views (epipolar geometry)

3) Reconstruct 3D points of the best two views (triangulation)

4) Select the next image to add

Separate points into known and unknown for PnP (known) and triangulation (unknown)

5) Estimate relative pose of the next view (PnP)

6) Reconstruct newly observed 3D points (triangulation)

7) Optimize camera pose and 3D points together (BA)

Incrementally add more views

137
Structure-from-Motion
Result: “sfm_global(point).xyz” / ”sfm_global(camera).xyz”

- # of points: 2503
- # of iterations: 150
- Reprojection error: 50.962 ← 17385.5

Result: “sfm_inc(point).xyz” / ”sfm_inc(camera).xyz”

- # of points: 2133
- # of iterations: 26 + 23 + 25
- Reprojection error: 0.380 ← 1.261

138
Structure-from-Motion using VisualSFM

Sparse 3D Reconsturction by VisualSFM (SiftGPU and pba)

139
Structure-from-Motion using VisualSFM

Dense 3D Reconsturction by VisualSFM (CMVS and PMVS2)

140
Structure-from-Motion using VisualSFM

Surface Reconstruction by VisualSFM (Poisson Surface Reconstruction) [URL]

141
 Visual Odometry and SLAM: History
SLAM in Robotics
Monocular Camera

MonoSLAM (2003): features, Bayesian filtering

v.s.
SfM in Computer Vision

PTAM (2007): more features, bundle adjustment RGB-D Camera

(keyframes)

DTAM (2011): dense volumetric KineticFusion (2011): dense volumetric

ORB-SLAM (2014) v.s. LSD-SLAM (2014)

better feature management direct, semi-dense
ElasticFusion (2015) DynamicFusion (2015)
dense surfels dynamic dense

142
Paradigm #1: Bayesian Filtering v.s. Bundle Adjustment

Robotics Computer Vision

Localization Mapping Motion Estimation Reconstruction

SLAM SfM
(Simultaneous Localization and Mapping) (Structure from Motion)
Bayesian Filtering
VS Bundle Adjustment

e.g. MonoSLAM e.g. PTAM

[Davison et al., T-PAMI, 2007] [Klein and Murray, ISMAR, 2007]

“Real-time Monocular SLAM: Why Filter?”

[Strasdat et al., ICRA, 2010]

≤
1. Global optimization
2. # of features (100 vs 4000)

143
Paradigm #2: Feature-based Method v.s. Direct Method

▪ e.g. Image stitching

+ =

Feature-based Method VS Direct Method

feature extraction and matching

144
Paradigm #2: Feature-based Method v.s. Direct Method

145
Why Visual Odometry?

Visual odometry
VS Wheel odometry

+ direct motion measure − indirect motion measure (e.g. slippage)

+ six degree-of-freedoms − two degree-of-freedoms
+ easy to install − necessary to be on rotor/shaft
− heavy computation + simple calculation
− visual disturbance (e.g. moving objects)

146
Why Visual Odometry?

Visual odometry
VS Wheel odometry

+ direct motion measure − indirect motion measure (e.g. slippage)

+ six degree-of-freedoms − two degree-of-freedoms
+ easy to install − necessary to be on rotor/shaft
− heavy computation + simple calculation
− visual disturbance (e.g. moving objects)
no wheels / rough terrains, in-the-sky, under-the-water

147
Why Visual Odometry?

Visual odometry
VS Wheel odometry

+ direct motion measure − indirect motion measure (e.g. slippage)

+ six degree-of-freedoms − two degree-of-freedoms
+ easy to install − necessary to be on rotor/shaft
− heavy computation + simple calculation
− visual disturbance (e.g. moving objects)
no wheels / rough terrains, in-the-sky, under-the-water

Visual Odometry
VS Visual SLAM

no assumption on trajectories closed-loop is preferred for convergence

→ navigation / large space (outdoor) → mapping / small space (indoor, campus)

148
Why Visual Odometry?

Visual odometry
VS Wheel odometry

+ direct motion measure − indirect motion measure (e.g. slippage)

+ six degree-of-freedoms − two degree-of-freedoms
+ easy to install − necessary to be on rotor/shaft
− heavy computation + simple calculation
− visual disturbance (e.g. moving objects)
no wheels / rough terrains, in-the-sky, under-the-water

Visual Odometry
VS Visual SLAM

no assumption on trajectories closed-loop is preferred for convergence

→ navigation / large space (outdoor) → mapping / small space (indoor, campus)

real navigation situations

149
Feature-based Monocular Visual Odometry

Two-view Motion Estimation

- Feature: Good-Feature-to-Track [Shi94_CVPR]

Find 2D-2D feature with bucketing to distribute features
correspondence - Correspondence: Lucas-Kanade optical flow [Lucas81_IJCAI]

Reject outlier - Adaptive MSAC [Choi09_IROS]

correspondence - Iterative 5-point algorithm [Choi15_IJCAS]
- Error measure: Sampson distance
Estimate
(scaled) relative pose - Normalized 8-point algorithm
- Scale-from-ground with asymmetric kernels [Choi13_URAI]
Accumulate
relative pose

150
Feature-based Monocular Visual Odometry

PnP Pose Estimation

Find 2D-2D feature - Feature: Good-Feature-to-Track [Shi94_CVPR]

correspondence with bucketing to distribute features
- Correspondence: Lucas-Kanade optical flow
If few points [Lucas81_IJCAI]
Find 2D-3D point
correspondence
- Adaptive MSAC
If few inliers Reject outlier - Iterative PnP algorithm (3-point algorithm)
correspondence
- Error measure: Projection error

Try two-view - Iterative PnP algorithm

Estimate pose
motion estimation - Scale-from-ground with asymmetric kernels
[Choi13_URAI]
Add Update - Bundle adjustment over last 𝐾 keyframes
3D points map 3D points map Reprojection error with Cauchy loss function

151
Feature-based Monocular Visual Odometry

152
 Slides and example codes are available:
Summary https://github.com/sunglok/3dv_tutorial

▪ What is 3D Vision?
▪ Single-view Geometry
– Camera Projection Model
• Pinhole Camera Model
• Geometric Distortion Models
– General 2D-3D Geometry
• Camera Calibration
• Absolute Camera Pose Estimation (PnP Problem)
▪ Two-view Geometry
– Planar 2D-2D Geometry (Projective Geometry)
• Planar Homography
– General 2D-2D Geometry (Epipolar Geometry)
• Fundamental/Essential Matrix
• Relative Camera Pose Estimation
• Triangulation (Point Localization)
▪ Multi-view Geometry
– Bundle Adjustment (Non-linear Optimization)
– Applications: Structure-from-motion, Visual SLAM, and Visual Odometry
▪ Correspondence Problem
– Feature Correspondence: Feature Matching and Tracking
– Robust Parameter Estimation: (Hough Transform), RANSAC, M-estimator
153
Applications in Deep Learning Era

▪ There are still many researches and applications.

– 3D reconstruction
– Real-time visual odometry/SLAM
– Augmented reality (mixed reality), virtual reality

model-based problem solving unknown models and procedures

(e.g. calculating camera pose; minimizing a cost function) (e.g. recognizing objects, finding correspondence)

▪ 3D understanding of results from deep learning

– e.g. Bounding boxes from object recognition → metric position and size

▪ Designing cost functions for deep learning

– e.g. Left-right consistency in MonoDepth [CVPR 2017]
– e.g. Eigendecomposition-free training of deep learning [ECCV 2018]

▪ Building datasets for deep learning

– e.g. LIFT [ECCV 2016]
Appendix: How to Solve Problems

▪ Linear Equations
– Inhomogeneous cases: Multiplying a pseudo-inverse

– Homogenous cases: Finding a null vector (or a vector which has the smallest singular value)

▪ Non-linear Equations
– Non-linear optimization
– General cases: Gradient-descent method, Newton method
– Least squares cases: Gauss-Newton method, Levenberg–Marquardt method

155
Appendix: Further Information
▪ Beyond Point Features
– Other features: OPVO, Kimera
– Direct methods (w/o features): Already mentioned (including deep learning)
▪ Real-time / Large-scale SfM
▪ (Spatially / Temporally) Non-static SfM
– Deformable (or moving) objects: Non-rigid SfM
▪ Depth, Object, and Semantic Recognition

▪ Sensor Fusion and New Sensors

– + Depth (ORB-SLAM2, LSD-SLAM, DVO have their variants with depth.)
• RGB-D: ElasticFusion, RGB-D SLAM, BaMVO, RTAB-Map
• Stereo cameras: S-PTAM, ProSLAM
• LiDAR: LIMO / cf. Cartographer
– + IMU: OKVIS, ROVIO, VINS-Mono
• Visual-inertia Calibration: Kalib
– + GPS
– Omni-directional cameras: Multi-FoV datasets
– Light-field cameras
– Event camera: ETAM*
▪ Minimal Solvers / Self-calibration
– OpenGV, Minimal Problems in Computer Vision
▪ Public Datasets and Evaluations
– Datasets: Awesome Robotics Datasets
– Evaluations: KITTI Odometry/SLAM Evaluation 2012, GSLAM, evo
157