| Follow with RSS
The Kinect is a new sensor and is actively being used and researched, and new uses and information appears frequently. I cited what I could in my paper, but some of that information is already out-dated. In the comments below I attempt to catalog new projects, studies, publications, and information as it becomes available.
Tags: Code, Computers, HowTo, Kinect, ResearchEmacs org-mode has a companion MobileOrg app that lets you view and edit your org files while mobile. But, since this is emacs, unfortunately it doesn’t work ‘out-of-the-box’ as smoothly as some other applications. The documented method is to manually execute org-mobile-push to push local edits into the cloud (to your mobile device), and then org-mobile-pull to pull mobile edits onto the desktop.
The code below sets up MobileOrg, and introduces some additional functions that auto-pushes whenever a change is made, and polls the incoming file and auto-pulls whenever the mobile apps make a change. The functions work asynchronously in the background so they don’t tie up your emacs process.
First, set up MobileOrg in emacs:
;; Mobile Org (require 'org-mobile) (setq org-mobile-inbox-for-pull "~/Dropbox/org/mobile.org") (setq org-mobile-directory "~/Dropbox/MobileOrg") (define-key org-mode-map "\C-cp" 'org-mobile-pull) (define-key org-agenda-mode-map "\C-cp" 'org-mobile-pull)
The following is a modified version of this gist. Each time you save an org buffer it will wait 10 seconds and then execute org-mobile-push in the background.
;; Fork the work (async) of pushing to mobile ;; https://gist.github.com/3111823 ASYNC org mobile push... (require 'gnus-async) ;; Define a timer variable (defvar org-mobile-push-timer nil "Timer that `org-mobile-push-timer' used to reschedule itself, or nil.") ;; Push to mobile when the idle timer runs out (defun org-mobile-push-with-delay (secs) (when org-mobile-push-timer (cancel-timer org-mobile-push-timer)) (setq org-mobile-push-timer (run-with-idle-timer (* 1 secs) nil 'org-mobile-push))) ;; After saving files, start an idle timer after which we are going to push (add-hook 'after-save-hook (lambda () (if (or (eq major-mode 'org-mode) (eq major-mode 'org-agenda-mode)) (dolist (file (org-mobile-files-alist)) (if (string= (expand-file-name (car file)) (buffer-file-name)) (org-mobile-push-with-delay 10))) ))) ;; Run after midnight each day (or each morning upon wakeup?). (run-at-time "00:01" 86400 '(lambda () (org-mobile-push-with-delay 1))) ;; Run 1 minute after launch, and once a day after that. (run-at-time "1 min" 86400 '(lambda () (org-mobile-push-with-delay 1)))
Finally, poll the incoming mobileorg.org file every 30 seconds, and execute org-mobile-pull if it has been updated.
;; watch mobileorg.org for changes, and then call org-mobile-pull ;; http://stackoverflow.com/questions/3456782/emacs-lisp-how-to-monitor-changes-of-a-file-directory (defun install-monitor (file secs) (run-with-timer 0 secs (lambda (f p) (unless (< p (second (time-since (elt (file-attributes f) 5)))) (org-mobile-pull))) file secs)) (defvar monitor-timer (install-monitor (concat org-mobile-directory "/mobileorg.org") 30) "Check if file changed every 30 s.")
The only remaining improvement would be if the MobileOrg app had an option to auto-sync on launch. Perhaps the next version…
Tags: Code, Computers, HowTo
This post introduces yet another utility that might be useful for Kinect users. My previous posts have suggested using the libfreenect record utility to capture data. That is the workflow that works best with my kinect_register code. However, record does not show you what it is recording, which can be problematic.
I have patched together the OpenKinect record and glview utilities into a program called kinect_record. This program has output identical to record, but also shows the data as it captures it.
You can download the source from GitHub and compile and run it by following the text in the README.txt file.
Tags: Code, Computers, Kinect, VisualizationI am releasing source code for a program that supports off-line calibration and registration of raw Kinect data collected by the libfreenect record program. The interesting parts of this code were all developed by the OpenKinect community, I simply patched parts of their code together in a way that was useful for my Kinect work. I have argued in the past for collecting the raw data so that different (and better) calibrations can be applied in the future, and this code maintains this philosophy.
This code uses the internal calibration parameters that are shipped with each Kinect, as opposed to extrinsically determining the calibration (the method used by most checker-board image calibration methods). Offline means that the calibration parameters can be captured once when the Kinect is plugged in, and then used in the processing phase without the Kinect on data that has already been acquired. Calibration is defined as the act of mapping from the raw sensor digital numbers (DN) in (pixel, pixel, DN) coordinates to world (x,y,z) coordinates. Registration is defined as mapping the RGB pixels onto the depth data.
Importantly, this registration maps RGB to depth, whereas all other registration algorithms I know of map depth to RGB. Since the data of interest with the Kinect is the depth data, it does not make sense to perform any unnecessary error-increasing mathematical operations on the depth data.
The code can be downloaded from https://github.com/mankoff/libfreenect/tree/offline_register and built with the following commands, assuming all dependencies are already installed. The easiest way to verify your are able to build this code is to install libfreenect through an existing package manager system, then over-write that with this custom build.
git clone https://github.com/mankoff/libfreenect/
cd libfreenect
git checkout offline_register
mkdir build
cd build
cmake ..
make
make install
Optionally, skip the make install line and access the new program in build/bin/. The new program is called “kinect_register” and when run without any arguments it prints usage instructions:
$ kinect_register
Kinect Offline Registration
Usage:
kinect_register [-h] -s <regfile> | -a <regfile> <PGM> [<PPM>]
-h: Display this help message
-s: Save the registration parameters from a connected Kinect to <regfile>
-a: Apply the registration from <regfile> to <PGM> (from 'record')
Optionally align a PPM file with the PGM file
Data Formats (units: mm):
file.x: 640x480 double of x values
file.y: 640x480 double of y values
file.z: 640x480 integer of z values
file.xyz: ASCII file of x y z r g b. RGB only if PPM argument provided
file.reg.ppm: PPM shifted so pixels align with file.{x,y,z} data
The file format produced by the Kinect libfreenect record program is not well documented and suffers from endian complication issues on most computers. Below are four snippets of code that can be used to read the big-endian PGM (depth data) files produced by record. The PPM (RGB data) are a more popular image format and should be easier to work with.
// C
fp = fopen("file.pgm", "r");
while (getc(fp) != '\n'); // skip header line
uint16_t data[640*480];
fread(data, sizeof(uint16_t), 640*480, fp); // read the data
fclose(fp);
# Python
import numpy as np
infile = open('file.pgm','r')
header = next(infile)
infile.seek(len(header))
data = np.fromfile(infile, dtype=np.uint16).reshape((480, 640))
;; IDL
openr, lun, "file.pgm", /get_lun
header = {P5:BYTARR(2),width:BYTARR(4),height:BYTARR(4),maxV:BYTARR(7)}
readu, lun, header
data = intarr( string(header.width), string(header.height) )
readu, lun, data
free_lun, lun
% MATLAB
data = imread('file.pgm');
data = swapbytes(data);
If using my kinect_register program then reading a PGM is not absolutely necessary, as that program converts a single frame of depth data to an ASCII x,y,z file format. However, a single frame is noisy, and I suggest using the above PGM reader snippets to load several (or several hundred) PGMs and average all the good values of each pixel. Then, write out a new (averaged) PGM to be used as input to kinect_register. The snippets above can be used as templates for the writer function. The averaging will produce fractional numbers, but they will need to be rounded to whole numbers before writing the PGM.
If you have code to load 16-bit big-endian PGM files in another language, or a PGM writer function, feel free to share.
Tags: Code, Computers, HowTo, KinectFollowing up on my previous post about using the Kinect for earth science applications, I’m documenting the Kinect Go Kit I built for fieldwork.
Kinect Go Kit: Top level
Kinect Go Kit Pelican Case
I travel with two Kinects, two power supplies, and two computers in case one gets damaged or destroyed in the field, although only one computer is in the kit.
The Netbook is a cheap $240 computer running Ubuntu and the libfreenect software stack at a minimum. I have also found it useful to have more advanced data collection software (ROS, RGBDemo, RGBDSLAM), and some analysis software (CloudCompare, points2grid, Viewpoints, etc.). This netbook works fine for raw data dumps from the libfreenect ‘record’ program. It can run the more computationally expensive scene stitching algorithms such as RGBDSLAM, but it takes about 10 seconds per stitch, while a more powerful laptop (but still a few years old) can do it at 0.5 to 1 Hz. Since ‘record’ collects about 1.5 GB of data per minute, it is good to have a lot of free space on the hard drive.
The plugs and cables are shown laid out below, and in addition, some velcro straps are stored in that compartment, used to attach the Kinect to the tripod arm.
The Pelican 1510 case supports two levels, and the lower level looks like this:
Kinect Go Kit: Bottom level
Battery #1 is a 12 V 5 Ah sealed led acid battery. It provides >5 hours of Kinect runtime, about equal to the runtime of the netbook.
Battery #2 is 8 AA batteries (12 V), and underneath is an 8 AA battery holder and a battery charger. If I need to turn the Kinect on for a short amount of time and want to travel lightly, these will do.
Kinect cable layouts
As shown above the cord to the Kinect can be cut and alligator clips or some other electrical termination can be attached. I often have wall power and have attached clips to the detached plug so I can use it as originally intended. However, when in the field, the clips can connect directly to the 12 V battery or the AA battery pack.
Kinect mounted on tripod
The Kinect Go Kit above is close to the minimum necessary for fieldwork. Things that I would like in it, but are not yet, include:
Additional tools I have found handy to have with me in the field include, but are not limited to, the following:
Tags: Computers, HowTo, Kinect, PhD, Research
We have successfully used a Kinect outdoors to study ablation on a glacier, map a subglacial cave in 3D, and tested it in a variety of hydrological situations (imaging roughness on the base of a stream, calibrating the Kinect data through water, and imaging surface waves). Results will be presented at the 2011 AGU conference.
There are a variety software interfaces to the Kinect. One high-level tool that is easy to use (binaries provided, no need to compile source, supports ‘scene painting’) is RGB-Demo. It is a good tool to start with if you want to work with the Kinect.
However, most Kinect software and calibrations so far have been developed by the robotics and computer vision communities. I am grateful for the work they have done, but those communities have different data needs than earth scientists. For example, quadrotor obstacle avoidance (link (PDF), link) has distance measurement errors that appear to be on order cm, but it still works fine as the helicopter avoids obstacles by an amount larger than the error.
Earth scientists should aim for a better model of the world than the one currently provided by the Kinect and its primary users. I suggest recording and storing the raw digital numbers (DN) from the Kinect rather than higher-level auto-calibrated real-world coordinates. It will require more post-processing, but storing the DNs will allow the data to be re-processed as better calibrations are developed. In addition, the low level recorder operates at 30 Hz and the higher level point-cloud products currently do not record data at that rate.
The best supported low-level interface is the LibFreenect Fakenect record program. It dumps the uncalibrated RGB and depth images to a folder at 30 Hz until you kill the process. Uncalibrated means both that the depth data is in sensor units, and that the depth and RGB images are not aligned. You can easily convert the depth data to real world x,y,z coordinates using existing published algorithms (link, link, link, and many others exist on the web), but importantly the raw data is stored and can be used with better calibrations in the future.
After processing the raw ‘record’ data, you can work with the point cloud data or DEMs using a variety of standard software for pointclouds, LiDAR, etc. I have had great success with CloudCompare and Poinst2Grid, in addition to custom code in MATLAB, IDL, and Python. A good list of software is available at the NSF OpenTopography site.
To work with the depth data to we initially use the following algorithms found on the various sites dedicated to Kinect hacking. The data provided by these algorithms is sufficient for certain uses, and for testing algorithms and visualizations, while better calibrations are performed.
DN to distance (source):
k1 = 1.1863d
k2 = 2842.5d
k3 = 0.1236d
Z = k3 * tan( double( DN ) / k2 + k1 )
XYZ to world (source):
Xres = 640
Yres = 480
FovH = 1.0144686707507438 (rad)
FovV = 0.78980943449644714 (rad)
XtoZ = tan( FovH / 2 ) * 2
YtoZ = tan( FovV / 2 ) * 2
X = ( X_pixel / Xres – 0.5 ) * Z * XtoZ
Y = ( 0.5 – Y_pixel / Yres ) * Z * YtoZ
Question or comments? Post below…
Tags: Code, Computers, HowTo, Kinect, ResearchIt has been almost one year since the Kinect was released, and there have been some amazing projects that use it. Microsoft appears to be embracing the hackers and highlights some of the non-video-game related uses in a new video:
Tags: Computers, Kinect, Research, VideoGoogle Earth does not officially support vertical curtains of data. However, it does support buildings with images on the side. If you are willing to stretch the definition of a building, you can put any vertical data you like into KML. This implementation was inspired by the Goddard Earth Science (GES) Data and Information Service Center (DISC) sample files for CloudSat, CALIPSO, and Aqua.
If you use IDL and would like to image your data in Google Earth, be it points, lines, regions, surface images, or vertical data, you should be using my IDL interface to the KML API.
Vertical Data in Google Earth
A new beta version of the Community Ice Sheet Model, Glimmer-CISM, has been released. Below are instructions to compile and run it on OS X.
# build NetCDF
export CFLAGS=-m32
export FFLAGS=-m32
./configure --prefix=/Users/mankoff/local/netcdf-4.1.1 \
--disable-cxx --disable-curl --disable-dap
make && make install
say netCDF done
# build GLIMMER
cd ~/local/src/
wget http://download.berlios.de/glimmer-cism/glimmer-1.7.0.tar.gz
tar zxvf glimmer-1.7.0.tar.gz
cd glimmer-1.7.0/
# OS X has issues with 32 and 64 bit libraries.
# The -m32 flag forces 32 bit compilation.
# The following should be one long line:
./configure --prefix=/Users/mankoff/local/glimmer-1.7.0 \
--with-netcdf=/Users/mankoff/local/netcdf-4.0.1 \
FC=gfortran F77=gfortran CFLAGS=-m32
make
make install
say GLIMMER done
There are a few ways to test the installation. The source folder provides a test folder:
export PATH=/Users/mankoff/local/glimmer-1.7.0/bin:$PATH cd ~/local/src/glimmer-1.7.0/tests/shelf python circular-shelf.py circular-shelf.PP.config python confined-shelf.py confined-shelf.PP.config say GLIMMER Test Done # Takes a while. Turn up your volume
Running the above command will result in NetCDF files being created in the output/ subdirectory. You can view the contents of example.nc with most any generic NetCDF viewer. While theses tests run over a given period of time, the output only has one time stored. If you want to see an evolution of the ice shelf, older test suites available from the previous code repository site should be used:
cd ~/tmp/ wget http://forge.nesc.ac.uk/download.php/200/glimmer-example-0.6.tar.gz tar zxvf glimmer-example-0.6.tar.gz cd glimmer-example-0.6/ ~/local/glimmer-1.7.0/bin/glide_launch.py ./example.config say done
Examine the output file example.nc to see ice sheet evolution over time. Basal melt is shown below:
@article{Rutt:2009,
Author = {Ian C. Rutt and Nicholas R. J. Hulton and Antony J. Payne},
Title = {{The Glimmer community ice sheet model}},
Year = {2009}}
Journal = {J. Geophys. Res.},
Volume = {114},
Number = {F2},
Tags: Antarctica, Code, Computers, Greenland, HowTo, ice, Research 