Augmented Reality SDK with OpenCV

I am developing an Augmented Reality SDK on OpenCV. I had some problems to find tutorials on the topic, which steps to follow, possible algorithms, fast and efficient coding for real-time performance etc.

So far I have gathered the next information and useful links.

OpenCV installation

Download latest release version.

You can find installation guides here (platforms: linux, mac, windows, java, android, iOS).

Online documentation.

Augmented Reality

For begginers here is a simple augmented reality code in OpenCV. It is a good start.

For anyone searching for a well designed state-of-the-art SDK I found some general steps that every augmented-reality based on marker tracking should have, considering OpenCV functions.

1. Main program: creates all classes, initialization, capture frames from video.

2. AR_Engine class: Controls the parts of an augmented reality application. There should be 2 main states:

   • detection: tries to detect the marker in the scene
   • tracking: once it is detected, uses lower computational techniques for traking the marker in upcoming frames.

Also there should be some algorithms for finding the position and orientation of the camera in every frame. This is achieve by detecting the homography transformation between the marker detected in the scene, and a 2D image of the marker we have processed offline. The explanation of this method here (page 18). The main steps for Pose Estimations are:

1. Load camera Intrinsic Parameters. Previously extracted offline through calibration.

2. Load the pattern (marker) to track: It is an image of the planar marker we are going to track. It is necessary to extract features and generate descriptors (keypoints) for this pattern so later we can compare with features from the scene. Algorithms for this task:

   • SIFT
   • FAST
   • SURF

3. For every frame update, run a detection algorithm for extracting features from the scene and generate descriptors. Again we have several options.

   • SIFT
   • FAST
   • SURF
   • FREAK: A new method (2012) supossed to be the fastest.
   • ORB

4. Find matches between pattern and the scene descriptors.

   • FLANN matcher

5. Find Homography matrix from those matches. RANSAC can be used before to find inliers/outliers in the set of matches.

6. Extract Camera Pose from homography.

   • Sample code on Pose from Homography.
   • Sample code on Homography from Pose.

Complete examples:

• aruco

• Mastering OpenCV samples

Since AR applications often run on mobile devices, you could consider also other features detector/descriptor:

• FREAK
• ORB

Generally if you can chose the markers you first detect a square target using an edge detector and then either Hough or simply contours - then identify the particular marker from the internal design. Rather than using a general point matcher.

Take a look at Aruco for well written example code.