Offline Registration for the Kinect
I am releasing source code for a program that supports off-line calibration and registration of raw Kinect data collected by the libfreenect record program. The interesting parts of this code were all developed by the OpenKinect community, I simply patched parts of their code together in a way that was useful for my Kinect work. I have argued in the past for collecting the raw data so that different (and better) calibrations can be applied in the future, and this code maintains this philosophy.
This code uses the internal calibration parameters that are shipped with each Kinect, as opposed to extrinsically determining the calibration (the method used by most checker-board image calibration methods). Offline means that the calibration parameters can be captured once when the Kinect is plugged in, and then used in the processing phase without the Kinect on data that has already been acquired. Calibration is defined as the act of mapping from the raw sensor digital numbers (DN) in (pixel, pixel, DN) coordinates to world (x,y,z) coordinates. Registration is defined as mapping the RGB pixels onto the depth data.
Importantly, this registration maps RGB to depth, whereas all other registration algorithms I know of map depth to RGB. Since the data of interest with the Kinect is the depth data, it does not make sense to perform any unnecessary error-increasing mathematical operations on the depth data.
The code can be downloaded from https://github.com/mankoff/libfreenect/tree/offline_register and built with the following commands, assuming all dependencies are already installed. The easiest way to verify your are able to build this code is to install libfreenect through an existing package manager system, then over-write that with this custom build.
git clone https://github.com/mankoff/libfreenect/
cd libfreenect
git checkout offline_register
mkdir build
cd build
cmake ..
make
make install
Optionally, skip the make install line and access the new program in build/bin/. The new program is called “kinect_register” and when run without any arguments it prints usage instructions:
$ kinect_register
Kinect Offline Registration
Usage:
kinect_register [-h] -s <regfile> | -a <regfile> <PGM> [<PPM>]
-h: Display this help message
-s: Save the registration parameters from a connected Kinect to <regfile>
-a: Apply the registration from <regfile> to <PGM> (from 'record')
Optionally align a PPM file with the PGM file
Data Formats (units: mm):
file.x: 640x480 double of x values
file.y: 640x480 double of y values
file.z: 640x480 integer of z values
file.xyz: ASCII file of x y z r g b. RGB only if PPM argument provided
file.reg.ppm: PPM shifted so pixels align with file.{x,y,z} data
April 5th, 2013 at 06:55
Hi Mr. Ken Mankoff,
I am looking into the contribution you had towards OpenKinect for Offline Registration and I was wondering if you can suggest me a paper or a document which explains how the registration is actually done? I am going through the code but it looks like an implementation of some mathematical equations.
I am interested in understanding those equations for better clarity of the registration code.
Can you please kindly help me in providing such information?
Thanks a lot in advance
Sriram
April 5th, 2013 at 07:02
Hi Sriram,
That code came from this conversation https://groups.google.com/d/msg/openkinect/pgo6bNwkoRg/czL-DePeP7EJ and was originally implemented here: https://github.com/OpenKinect/libfreenect/pull/284/files
I do not know of a paper or document that describes this reverse registration. The original registration method (depth -> RGB) is documented but only in the conversations that took place on the OpenKinect Google Group.