Real Time pose estimation of a textured object (solvePnP)
Real Time pose estimation of a textured object
Nowadays, augmented reality is one of the top research topic in computer vision and robotics fields. The most elemental problem in augmented reality is the estimation of the camera pose respect of an object in the case of computer vision area to do later some 3D rendering or in the case of robotics obtain an object pose in order to grasp it and do some manipulation. However, this is not a trivial problem to solve due to the fact that the most common issue in image processing is the computational cost of applying a lot of algorithms or mathematical operations for solving a problem which is basic and immediateley for humans.
Goal
In this tutorial is explained how to build a real time application to estimate the camera pose in order to track a textured object with six degrees of freedom given a 2D image and its 3D textured model.
The application will have the followings parts:
- Read 3D textured object model and object mesh.
- Take input from Camera or Video.
- Extract ORB features and descriptors from the scene.
- Match scene descriptors with model descriptors using Flann matcher.
- Pose estimation using PnP + Ransac.
- Linear Kalman Filter for bad poses rejection.
Theory
In computer vision estimate the camera pose from n 3D-to-2D point correspondences is a fundamental and well understood problem. The most general version of the problem requires estimating the six degrees of freedom of the pose and five calibration parameters: focal length, principal point, aspect ratio and skew. It could be established with a minimum of 6 correspondences, using the well known Direct Linear Transform (DLT) algorithm. There are, though, several simplifications to the problem which turn into an extensive list of different algorithms that improve the accuracy of the DLT.
The most common simplification is to assume known calibration parameters which is the so-called Perspective-n-Point problem:
Problem Formulation: Given a set of correspondences between 3D points expressed in a world reference frame, and their 2D projections onto the image, we seek to retrieve the pose ( and ) of the camera w.r.t. the world and the focal length .
OpenCV provides four different approaches to solve the Perspective-n-Point problem which return and . Then, using the following formula it’s possible to project 3D points into the image plane:
The complete documentation of how to manage with this equations is in Camera Calibration and 3D Reconstruction.
Source code
You can find the source code of this tutorial in the samples/cpp/tutorial_code/calib3d/real_time_pose_estimation/ folder of the OpenCV source library.
阅读更多- Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields ∗
- Robust Real-Time Extreme Head Pose Estimation
- YOLO9000( YOLO v2 ), a state of the art, real time object detection system
- [OpenPose翻译] Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields ∗原文翻译(注释版)
- 论文阅读:《Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields》CVPR 2017
- caffe openpose/Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields配置(转)
- KeyPoint of 《Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks》
- 姿态估计论文思路整理 -- Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
- 论文阅读-Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
- Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
- 姿态论文整理--02-Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
- Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields ∗ 实时多人人体姿态估计论文原理讲解
- key point of 《You Only Look Once: Unified, Real-Time Object Detection》
- 拜读曹哲的Realtime Multi-Person 2D Pose Estimation using Part Affinity Fileds
- 论文阅读笔记: 2017 cvpr Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
- 行人姿态估计--Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
- READING NOTE: PVANET: Deep but Lightweight Neural Networks for Real-time Object Detection
- 目标检测--PVANET: Deep but Lightweight Neural Networks for Real-time Object Detection
- 论文阅读--PVANET: Deep but Lightweight Neural Networks for Real-time Object Detection
- Real time example of using BADI in SAP