Difference between revisions of "OpenNI2"

From Wiki for iCub and Friends
Jump to: navigation, search
(yarp openni2 driver manual fixes)
m (→‎Troubleshooting for OS X: -> macOS)
 
(65 intermediate revisions by 2 users not shown)
Line 1: Line 1:
−
The OpenNI2 device driver for YARP works with PrimeSense compatible RGB-D sensors, including Asus Xtion Pro. Tested on Debian Jessie (testing), Ubuntu 12.04+ and Mac OS X 10.8.4+.
+
The OpenNI2 device driver for YARP exposes streams of data (color and depth videos, skeleton user tracking) from RGB-D sensors, including: Asus Xtion Pro, Kinect v1 and Kinect v2 (soon).
  
−
Debian Wheezy (stable) is not supported by OpenNI2, due to glibc version being < 2.14.
+
Note: other projects with similar purposes exist:
 +
# [http://www.yarp.it/classyarp_1_1dev_1_1depthCameraDriver.html YARP depthCamera plugin], based on OpenNI 2;
 +
# https://github.com/robotology/kinect-wrapper, based on OpenNI 1 or Microsoft Kinect SDK;
 +
# [http://wiki.icub.org/yarpdoc/group__kinect__openni.html Kinect YARP device driver], also based on OpenNI 1.
  
−
Note: Microsoft Kinect sensors are not supported by this driver. However, they are supported by the old [http://wiki.icub.org/yarpdoc/group__kinect__openni.html Kinect YARP driver], which was based on OpenNI version 1.
+
= Installation =
 +
 
 +
This driver was tested on Debian Jessie (stable), Ubuntu 12.04+ and macOS 10.8.4+.
 +
 
 +
== Dependencies ==
 +
 
 +
For Debian and Ubuntu, install the following dependencies:
 +
sudo apt-get install g++ python libusb-1.0-0-dev libudev-dev freeglut3-dev doxygen graphviz libxmu-dev libxi-dev
 +
Also, you must have installed a Java SDK (Oracle recommended).
 +
 
 +
Debian Wheezy (oldstable) is not supported by OpenNI2, due to glibc version being < 2.14.
  
−
= Installation =
+
For macOS, the above packages are either preinstalled or available through Homebrew.
  
 
== OpenNI2 ==
 
== OpenNI2 ==
  
−
Download and install:
+
Install OpenNI2 (from zip packages or from GitHub source - see http://structure.io/openni) and NiTE2.
−
  git clone https://github.com/OpenNI/OpenNI2.git
+
 
 +
For macOS, OpenNI2 is also available through Homebrew.
 +
 
 +
== Environment variables ==
 +
 
 +
Follow the installation instructions for OpenNI2 and NiTE2 by running the respective install.sh scripts. These scripts will create two files called OpenNIDevEnvironment and NiTEDevEnvironment. To copy their content permanently into your .bashrc, do:
 +
  cat OpenNIDevEnvironment >> ~/.bashrc
 +
cat NiTEDevEnvironment >> ~/.bashrc
 +
 
 +
At the end of this process, you will have these environment variables:
 +
OPENNI2_INCLUDE: full path to OpenNI2/Include
 +
OPENNI2_REDIST: full path to OpenNI2/''arch''-Release
 +
NITE2_INCLUDE: full path to NiTE-''OS''-''arch''-2.x
 +
NITE2_REDIST64 or NITE2_REDIST: full path to NiTE-''OS''-''arch''-2.x/Redist
 +
where ''OS'' = Linux, ''arch'' = x64 or x86.
 +
 
 +
== Enabling OpenNI2 YARP driver ==
  
−
If you are using Debian testing (jessie), you have to make small changes:
+
In the CMake configuration for YARP, set the following:
−
1) pthread
+
CREATE_DEVICE_LIBRARY_MODULES ON
−
2) comment suppress errors
+
ENABLE_yarpmod_OpenNI2DeviceServer ON
 +
ENABLE_yarpmod_OpenNI2DeviceClient ON
  
−
cd OpenNI2
+
== Finalizing the installation ==
−
make
 
−
cd Packaging
 
−
./Release x64  // or: ./Release x86
 
−
cd Final
 
  
−
== NiTE2 ==
+
These final steps are necessary to not incur in runtime errors like ''Couldn't create user tracker''.
  
−
Download and install NiTE2:
+
The OpenNI2 YARP driver must always be called from a directory that contains a copy (or symbolic link) of the NiTE.ini file from the stock NiTE2 distribution, properly configured. To prepare such a setup, do something along these lines:
 +
mkdir ~/yarp_openni2_root
 +
cd ~/yarp_openni2_root
 +
cp $NITE2_REDIST64/NiTE.ini .
 +
// or if you have the 32-bit version:
 +
// cp $NITE2_REDIST/NiTE.ini .
  
−
tbc
+
Edit NiTE.ini, setting DataDir to be the full path to $NITE2_REDIST64/NiTE2 (or $NITE2_REDIST/NiTE2).
−
./install.sh
 
  
−
add source ..../OpenNidev
+
Now you're ready to call the yarpdev command from the yarp_openni2_root directory.
−
ad source ..../NiTEDev
 
  
−
source ~/.bashrc  // or: logout and login
+
= Support for different RGB-D cameras =
  
−
== YARP driver ==
+
== PrimeSense-compatible sensors ==
  
−
Enable devices
+
These sensors (Asus Xtion, Carmine) work out of the box with the driver.
−
SET OpenNI2DeviceServer to ON
 
−
set OpenNI2DeviceClient to ON
 
  
−
set INSTALL_WITH_RPATH to ON
+
== Kinect v1 ==
 +
 
 +
Install [https://github.com/OpenKinect/libfreenect libfreenect], including the OpenNI2-freenect bridge. Then copy (or create a symbolic link of) libFreenectDriver.so inside yarp_openni2_root/OpenNI2/Drivers.
 +
 
 +
== Kinect v2 ==
 +
 
 +
Support coming soon.
  
 
= Usage =
 
= Usage =
  
−
Print a short driver manual:
+
To print a list of all available options (help):
 
  yarpdev --device OpenNI2DeviceServer --verbose
 
  yarpdev --device OpenNI2DeviceServer --verbose
  
−
Available options:
+
General options:
 +
{| class="wikitable" border="1"
 +
|-
 +
!  option
 +
!  description
 +
!  default
 +
|-
 +
|  --name
 +
|  Set port prefix.
 +
|  /OpenNI2
 +
|-
 +
|  --printVideoModes
 +
|  Print supported video modes.
 +
| 
 +
|-
 +
|  --depthVideoMode
 +
|  Set depth camera video mode.
 +
|  0 (i.e., 320x240 @ 30fps)
 +
|-
 +
|  --colorVideoMode
 +
|  Set color (RGB) camera video mode.
 +
|  0 (i.e., 320x240 @ 30fps)
 +
|-
 +
|  --noRGB
 +
|  Disable color (RGB) camera, while keeping depth sensor and user skeleton enabled. Useful for RGB-less sensors and to reduce CPU load.
 +
|  off (i.e., RGB on)
 +
|-
 +
|  --noRGBMirror
 +
|  Disable mirroring for the color (RGB) stream. Mirroring causes the stream to appear as if seen in a mirror, i.e., the image is transformed by reflecting all pixels across the vertical axis.
 +
|  off (i.e., mirroring on)
 +
|-
 +
|  --noDepthMirror
 +
|  Disable mirroring for the depth stream. Mirroring causes the depth to appear as if seen in a mirror, i.e., the image is transformed by reflecting all pixels across the vertical axis.
 +
|  off (i.e., mirroring on)
 +
|-
 +
|  --noUserTracking
 +
|  Disable skeleton user tracking. Useful to reduce CPU load.
 +
|  off (i.e., user tracking on)
 +
|-
 +
|  --minConfidence x (double)
 +
|  Set minimum confidence required by skeleton user tracking.
 +
|  0.6
 +
|-
 +
|  --syncFrames
 +
|  Keep color and depth frames always in temporal sync.
 +
|  off
 +
|-
 +
|  --imageRegistration
 +
|  Align color and depth video streams. This computation is performed on the sensor hardware.
 +
|  off
 +
|}
 +
 
 +
Other options related to recording and playback:
 +
{| class="wikitable" border="1"
 +
|-
 +
!  option
 +
!  description
 +
!  default
 +
|-
 +
|  --record filename.oni
 +
|  Record (save) to the specified oni file.
 +
| 
 +
|-
 +
|  --playback filename.oni
 +
|  Playback the specified oni file.
 +
| 
 +
|-
 +
|  --loop
 +
|  Force the playback to loop.
 +
|  off (i.e., playback stops after one run)
 +
|}
 +
 
 +
Note: in playback, options like --depthVideoMode and --colorVideoMode are ignored.
  
−
--device
+
= Output and data format =
  
−
--playback filename.oni
+
The driver opens these ports:
−
    Playbacks the specified oni file.
+
* [portPrefix]/userSkeleton:o - userSkeleton detection port (only opened if user detection is on)
 +
* [portPrefix]/depthFrame:o - depth frame port
 +
* [portPrefix]/imageFrame:o - rgb camera frame port
  
−
--loop
+
imageFrame:o and depthFrame:o stream YARP images (PixelRGB and PixelMono16, respectively), and if connected to a yarpview, the video is available.
−
    Forces the playback to loop. If not used, the playback will stop after one run.
 
  
−
--record filename.oni
+
During tracking, userTracker:o contains YARP Bottles in the following format:
−
    records to the specified oni file
 
  
−
--name
+
'''([USER] userID)   [POS] (x y z) positionConfidence [ORI] (w x y z) orientationConfidence    ...'''
−
    chenge default port prefix (default=/OpenNI2)
 
−
--noCameras
 
−
    use only user tracker
 
  
−
--noMirror
+
where
−
    disable mirroring
+
* the part between "[POS]" and "orientationConfidence" is repeated as many times as the number of skeleton joints (15): HEAD NECK LEFT_SHOULDER RIGHT_SHOULDER LEFT_ELBOW RIGHT_ELBOW LEFT_HAND RIGHT_HAND TORSO LEFT_HIP RIGHT_HIP LEFT_KNEE RIGHT_KNEE LEFT_FOOT RIGHT_FOOT
−
--noUserTracking
+
* the position is expressed in 3D Cartesian coordinates
 +
* the orientation is expressed in four-dimensional quaternions
  
−
--minConfidence x (double)
+
= Using the driver from an application =
−
    set minimum confidence required. Default is 0.6
 
−
Doesn't start the user tracker.
 
  
−
--
+
Refer to http://wiki.icub.org/yarpdoc/IOpenNI2DeviceDriver_8h_source.html
  
−
The driver opens three ports:
+
tbc
−
imageFrame:o
 
−
depthFrame:o
 
−
userTracker:o
 
  
−
imageFrame and depthFrame stream YARP images, and if connectd to a yarpview, the video is available.
+
= Advanced options =
−
UserTracker outputs bottles in the format:
 
  
−
To use the driver in an application:
+
In case you do not want to use RPATH, toggle CMAKE_SKIP_INSTALL_RPATH to ON in YARP. With this setting, the YARP OpenNI2 driver must always be called from a directory that contains a copy (or symbolic link) of the stock OpenNI2 and NiTE2 'Drivers' directories. To prepare such a setup, do something along these lines:
 +
mkdir ~/yarp_openni2_root
 +
cd ~/yarp_openni2_root
 +
cp -R $OPENNI2_REDIST/OpenNI2 .
 +
cp -R $NITE2_REDIST64/NiTE2 .
 +
// or if you have the 32-bit version:
 +
// cp -R $NITE2_REDIST/NiTE2 .
 +
and call the yarpdev command from the yarp_openni2_root directory.
  
−
tbc
+
= Troubleshooting for macOS =
 +
 
 +
Due to a combination of changes in the handling of RPATHs in macOS 10.10 Yosemite and CMake 3, the following is needed (once) before running the device:
 +
 
 +
cd $NITE2_REDIST64
 +
install_name_tool -id $PWD/libNiTE2.dylib libNiTE2.dylib
 +
install_name_tool -change libOpenNI2.dylib /usr/local/lib/ni2/libOpenNI2.dylib libNiTE2.dylib

Latest revision as of 16:23, 7 April 2017

The OpenNI2 device driver for YARP exposes streams of data (color and depth videos, skeleton user tracking) from RGB-D sensors, including: Asus Xtion Pro, Kinect v1 and Kinect v2 (soon).

Note: other projects with similar purposes exist:

  1. YARP depthCamera plugin, based on OpenNI 2;
  2. https://github.com/robotology/kinect-wrapper, based on OpenNI 1 or Microsoft Kinect SDK;
  3. Kinect YARP device driver, also based on OpenNI 1.

Installation

This driver was tested on Debian Jessie (stable), Ubuntu 12.04+ and macOS 10.8.4+.

Dependencies

For Debian and Ubuntu, install the following dependencies: 

sudo apt-get install g++ python libusb-1.0-0-dev libudev-dev freeglut3-dev doxygen graphviz libxmu-dev libxi-dev

Also, you must have installed a Java SDK (Oracle recommended).

Debian Wheezy (oldstable) is not supported by OpenNI2, due to glibc version being < 2.14.

For macOS, the above packages are either preinstalled or available through Homebrew.

OpenNI2

Install OpenNI2 (from zip packages or from GitHub source - see http://structure.io/openni) and NiTE2.

For macOS, OpenNI2 is also available through Homebrew.

Environment variables

Follow the installation instructions for OpenNI2 and NiTE2 by running the respective install.sh scripts. These scripts will create two files called OpenNIDevEnvironment and NiTEDevEnvironment. To copy their content permanently into your .bashrc, do: 

cat OpenNIDevEnvironment >> ~/.bashrc
cat NiTEDevEnvironment >> ~/.bashrc

At the end of this process, you will have these environment variables: 

OPENNI2_INCLUDE: full path to OpenNI2/Include
OPENNI2_REDIST: full path to OpenNI2/arch-Release 
NITE2_INCLUDE: full path to NiTE-OS-arch-2.x
NITE2_REDIST64 or NITE2_REDIST: full path to NiTE-OS-arch-2.x/Redist

where OS = Linux, arch = x64 or x86.

Enabling OpenNI2 YARP driver

In the CMake configuration for YARP, set the following: 

CREATE_DEVICE_LIBRARY_MODULES ON
ENABLE_yarpmod_OpenNI2DeviceServer ON
ENABLE_yarpmod_OpenNI2DeviceClient ON

Finalizing the installation

These final steps are necessary to not incur in runtime errors like Couldn't create user tracker.

The OpenNI2 YARP driver must always be called from a directory that contains a copy (or symbolic link) of the NiTE.ini file from the stock NiTE2 distribution, properly configured. To prepare such a setup, do something along these lines: 

mkdir ~/yarp_openni2_root
cd ~/yarp_openni2_root
cp $NITE2_REDIST64/NiTE.ini .
// or if you have the 32-bit version:
// cp $NITE2_REDIST/NiTE.ini .

Edit NiTE.ini, setting DataDir to be the full path to $NITE2_REDIST64/NiTE2 (or $NITE2_REDIST/NiTE2).

Now you're ready to call the yarpdev command from the yarp_openni2_root directory.

Support for different RGB-D cameras

PrimeSense-compatible sensors

These sensors (Asus Xtion, Carmine) work out of the box with the driver.

Kinect v1

Install libfreenect, including the OpenNI2-freenect bridge. Then copy (or create a symbolic link of) libFreenectDriver.so inside yarp_openni2_root/OpenNI2/Drivers.

Kinect v2

Support coming soon.

Usage

To print a list of all available options (help): 

yarpdev --device OpenNI2DeviceServer --verbose

General options:

option description default
--name Set port prefix. /OpenNI2
--printVideoModes Print supported video modes.
--depthVideoMode Set depth camera video mode. 0 (i.e., 320x240 @ 30fps)
--colorVideoMode Set color (RGB) camera video mode. 0 (i.e., 320x240 @ 30fps)
--noRGB Disable color (RGB) camera, while keeping depth sensor and user skeleton enabled. Useful for RGB-less sensors and to reduce CPU load. off (i.e., RGB on)
--noRGBMirror Disable mirroring for the color (RGB) stream. Mirroring causes the stream to appear as if seen in a mirror, i.e., the image is transformed by reflecting all pixels across the vertical axis. off (i.e., mirroring on)
--noDepthMirror Disable mirroring for the depth stream. Mirroring causes the depth to appear as if seen in a mirror, i.e., the image is transformed by reflecting all pixels across the vertical axis. off (i.e., mirroring on)
--noUserTracking Disable skeleton user tracking. Useful to reduce CPU load. off (i.e., user tracking on)
--minConfidence x (double) Set minimum confidence required by skeleton user tracking. 0.6
--syncFrames Keep color and depth frames always in temporal sync. off
--imageRegistration Align color and depth video streams. This computation is performed on the sensor hardware. off

Other options related to recording and playback:

option description default
--record filename.oni Record (save) to the specified oni file.
--playback filename.oni Playback the specified oni file.
--loop Force the playback to loop. off (i.e., playback stops after one run)

Note: in playback, options like --depthVideoMode and --colorVideoMode are ignored.

Output and data format

The driver opens these ports:

  • [portPrefix]/userSkeleton:o - userSkeleton detection port (only opened if user detection is on)
  • [portPrefix]/depthFrame:o - depth frame port
  • [portPrefix]/imageFrame:o - rgb camera frame port

imageFrame:o and depthFrame:o stream YARP images (PixelRGB and PixelMono16, respectively), and if connected to a yarpview, the video is available.

During tracking, userTracker:o contains YARP Bottles in the following format:

([USER] userID) [POS] (x y z) positionConfidence [ORI] (w x y z) orientationConfidence ...

where

  • the part between "[POS]" and "orientationConfidence" is repeated as many times as the number of skeleton joints (15): HEAD NECK LEFT_SHOULDER RIGHT_SHOULDER LEFT_ELBOW RIGHT_ELBOW LEFT_HAND RIGHT_HAND TORSO LEFT_HIP RIGHT_HIP LEFT_KNEE RIGHT_KNEE LEFT_FOOT RIGHT_FOOT
  • the position is expressed in 3D Cartesian coordinates
  • the orientation is expressed in four-dimensional quaternions

Using the driver from an application

Refer to http://wiki.icub.org/yarpdoc/IOpenNI2DeviceDriver_8h_source.html

tbc

Advanced options

In case you do not want to use RPATH, toggle CMAKE_SKIP_INSTALL_RPATH to ON in YARP. With this setting, the YARP OpenNI2 driver must always be called from a directory that contains a copy (or symbolic link) of the stock OpenNI2 and NiTE2 'Drivers' directories. To prepare such a setup, do something along these lines: 

mkdir ~/yarp_openni2_root
cd ~/yarp_openni2_root
cp -R $OPENNI2_REDIST/OpenNI2 .
cp -R $NITE2_REDIST64/NiTE2 .
// or if you have the 32-bit version:
// cp -R $NITE2_REDIST/NiTE2 .

and call the yarpdev command from the yarp_openni2_root directory.

Troubleshooting for macOS

Due to a combination of changes in the handling of RPATHs in macOS 10.10 Yosemite and CMake 3, the following is needed (once) before running the device: 

cd $NITE2_REDIST64
install_name_tool -id $PWD/libNiTE2.dylib libNiTE2.dylib
install_name_tool -change libOpenNI2.dylib /usr/local/lib/ni2/libOpenNI2.dylib libNiTE2.dylib
Retrieved from "http://wiki.icub.org/index.php?title=OpenNI2&oldid=22084"