KenMankoff

Kinect for Earth Scientists

We have successfully used a Kinect outdoors to study ablation on a glacier, map a subglacial cave in 3D, and tested it in a variety of hydrological situations (imaging roughness on the base of a stream, calibrating the Kinect data through water, and imaging surface waves). Results will be presented at the 2011 AGU conference.

There are a variety software interfaces to the Kinect. One high-level tool that is easy to use (binaries provided, no need to compile source, supports ‘scene painting’) is RGB-Demo. It is a good tool to start with if you want to work with the Kinect.

However, most Kinect software and calibrations so far have been developed by the robotics and computer vision communities. I am grateful for the work they have done, but those communities have different data needs than earth scientists. For example, quadrotor obstacle avoidance (link (PDF), link) has distance measurement errors that appear to be on order cm, but it still works fine as the helicopter avoids obstacles by an amount larger than the error.

Earth scientists should aim for a better model of the world than the one currently provided by the Kinect and its primary users. I suggest recording and storing the raw digital numbers (DN) from the Kinect rather than higher-level auto-calibrated real-world coordinates. It will require more post-processing, but storing the DNs will allow the data to be re-processed as better calibrations are developed. In addition, the low level recorder operates at 30 Hz and the higher level point-cloud products currently do not record data at that rate.

The best supported low-level interface is the LibFreenect Fakenect record program. It dumps the uncalibrated RGB and depth images to a folder at 30 Hz until you kill the process. Uncalibrated means both that the depth data is in sensor units, and that the depth and RGB images are not aligned. You can easily convert the depth data to real world x,y,z coordinates using existing published algorithms (link, link, link, and many others exist on the web), but importantly the raw data is stored and can be used with better calibrations in the future.

After processing the raw ‘record’ data, you can work with the point cloud data or DEMs using a variety of standard software for pointclouds, LiDAR, etc. I have had great success with CloudCompare and Poinst2Grid, in addition to custom code in MATLAB, IDL, and Python. A good list of software is available at the NSF OpenTopography site.

To work with the depth data to we initially use the following algorithms found on the various sites dedicated to Kinect hacking. The data provided by these algorithms is sufficient for certain uses, and for testing algorithms and visualizations, while better calibrations are performed.

DN to distance (source):

k1 = 1.1863d
k2 = 2842.5d
k3 = 0.1236d
Z = k3 * tan( double( DN ) / k2 + k1 )

XYZ to world (source):

Xres = 640
Yres = 480
FovH = 1.0144686707507438 (rad)
FovV = 0.78980943449644714 (rad)
XtoZ = tan( FovH / 2 ) * 2
YtoZ = tan( FovV / 2 ) * 2
X = ( X_pixel / Xres – 0.5 ) * Z * XtoZ
Y = ( 0.5 – Y_pixel / Yres ) * Z * YtoZ

Question or comments? Post below…