From Wiki for iCub and Friends
Jump to navigation Jump to search
The correct title of this article is stereoVisualTracker. The initial letter is shown capitalized due to technical restrictions.

Author: Micha Hersch

This module performs a color-based 3D tracker. It assumes a constant vergence. Up to three objects can be tracked. The colors to be tracked are selected by the user with the mouse. Kalman filtering can also be applied on the tracking. Potentially, it can also track things in 2D but it is not activated. If there is a need for it, please let me know.

Tracking is made with a combination of camshift and GMM-based tracking. Camera calibration is made with the standard chessboard procedure. (Hold a chessboard of given size in front of the camera).

Back to iCub YARP module specifications

Compiler & Linker Dependencies

Include files




Run-time Dependencies

Module arguments

--name <portBaseName> --file <configFile>
<portBaseName> is the port prefix. Default is /stereoVisualTracker
<configFile> is the configuration file. 

The configuration file is used for calibration. It is described below

Ports accessed

Ports that are assumed to exist prior to instantiation of the module (i.e. some other module must create them)


Those are the ports where the camera image are available. They can have other names, as connection is made externally (manually or with a script).

Ports created

Ports that are instantiated by the module and are then available for other modules to use (using yarp connect)

  • <portBaseName>/image0;i the input port where receiving the images of camera 1
  • <portBaseName>/image1:i the input port where receiving the images of camera 2
  • <portBaseName>/position:o the output port where object positions are sent. The output is sent in the following format:

<obj_1> <x1> <y1> <z1> [<obj_2> <x2> <y2> <z2>][<obj_3> <x3> <y3> <z3>] where integer <obj_i> is the object index, and doubles <xi>,<yi> and ,<zi> the corresponding position in a frame of refererence given by the 3d calibration process (command: '3').

Input data files

camera calibration parameter files

Output data files

camera calibration parameter files (if modified)

Configuration files

The configuration file is used for calibration. It should contain the following:

directory <directory_name>
height <height>
width  <width>
size   <size>

<directory_name> specifies the name of a directory within params where to read and write the calibration parameters. If there is no 'directory' entry in the <configFile> parameters are written directly in params. If there is an entry, the directory ./params/<directory_name> is assumed to exist.

The integers <height> and <width> specifiy the number squares of the calibration chessboard, which is <heigth+1>*<width+1>. <size> is the size of the square, in the units in which we want the localization to be.

User interface mechanism

Run-time modification of module parameters

Color selection is made with the opencv windows.

The user can give commands through the keyboard. Accepted commands are

  * 't' enable/disable 2d tracking
  * 'l' enable/disable 3d localization
  * 'a' add an object to be tracked
  * 'u' start/stop sending the 3d location through yarp
  * 'k' enable/disable kalman filtering for all objects
  * 'n' switch to next object
  * 'M' raise measurement noise of the kalman filter for the current object
  * 'm' lower measurement noise of the kalman filter for the current object
  * 'P' raise process noise of the kalman filter for the current object
  * 'p' lower process noise of the kalman filter for the current object
  * 'r' stop/resume tracking the current object (only one of them can be stopped)
  * 'c' calibrate the camera intrinsics parameters (distortion coeffs)
  * '3' calibrate the camera extrinsics parameters (for position and orientation)
  * 'w' switch color mode (YCbCr vs normalized)
  * 'd' show effect of undistortion
  * 'q' exit application

OS on which the module was developed


OS on which the module was tested

Linux, partly tested on Windows

Operating system dependencies

The file paths are using '/' (linux) and not '\' (windows). This is likely to cause a problem when running the module on windows. I'll try to update this soon.

Example instantiation of the module

Run & port connection commands

Make sure you have directory called params in the working directory. Otherwise make one.

$stereoVisualTracking --name /tulip --file filename
$yarp connect /cam/right /tulip/image0:i
$sleep(1) # to let the time to create the second port
$yarp connect /cam/left /tulip/image1:i

Two windows will appear showing the image stream of each camera. On linux, the red and blue channels are inverted, which means you will see red things blue and blue things red.

Press 't' (for tracking).

In each window, select with the mouse a region of the color you want to track. You can adjust as many time as you want. You can tune the sensitivity of color filter using the trackbar at the top of the window. The result of the color filter is shown with transparency.

Press 'l' (for 3d localization)

This may require extrinsic parameter calibration. In that case, bring the chessboard in the field of view of both cameras.

Press 'u' for output the result of the tracking.
Press 'a' to add a color to track, and select the color with the mouse.
Press 'n' to shift from one object to the other for adjusting the color filter.

You need to have one of the opencv window active in order to send commands. The tracked positions can be see using

$yarp read /reader /tulip/position:o

iCub Capabilities

iCub capability code

Other Yarp modules required to effect this capability

Example instantiation


Back to iCub YARP module specifications