Tracker
- The correct title of this article is tracker. The initial letter is shown capitalized due to technical restrictions.
Author: Paul Fitzpatrick
Visual tracker. Tracks a single visual target using input from a single camera. If mapping from visual coordinates to motor commands is not known, an adaptation mode can be selected, which will make small exploratory movements.
tracker implements tracking in a low-level sense, not any of the many
possible high-level senses. Specifically: if the robot is looking at
something in this instant, "tracker" tries to figure out how to keep the
robot looking at the same thing in the next instant, despite movements
of the object and/or head.
tracker is meant to work with other modules.
It does not decide what to look at, when to look at it, and when to look
away. That's the job of other modules (e.g. salience + attentionSelection + ...).
By implementing low-level tracking in a distinct module, the
control-loop can be kept tight, and the rest of the system can react to
changes in the environment at a slower or more variable rate without
making the robot's gaze seem jerky and unpredictable. Basically, we
transform the default low-level behavior of the robot's visual system
(in the absence of commands from higher-up) from being "look in the same
direction you are currently looking" to being "keep looking at what
you're looking". This is almost always what you want, and corresponds
well to the default reaction in human vision too.
Finally, tracker does not need to directly control the eyes, it
is configured to pass on its suggestions to another module. Basically,
it provides a stream that says "if you want to keep the eyes looking at
the same object, move them like this". So it could be an input to
something like controlGaze or simpleHeadcontrol or whatever actually
gets to talk to the physical hardware. However, it shouldn't be too
far removed from the real controller or latencies will cause ugly
behavior. This allows for switching to other modes of operation (e.g.
ballistic saccades, if desired) without contaminating the tracker with
new functionality, thereby keeping each module cohesive.
In summary,
Tracker is cross-correlation based image patch matching module that produces
image frame-of-reference positional offsets that can be mapped by another module to joint control offsets or control point
or whatever. These offsets can (optionally) be adaptively weighted if they don't inially match well with the controller
constants/gains. The initial seed position from which the initial patch to match (track) is provided by, e.g.,
salience+attentionSelection, but probably not egosphere (because it's frame of reference is neck based in units of degrees,
rather than image based in units of pixels). The output of tracker can be used selectively by any intermediary module to
control the gaze (eyes, head, ..), or ignored should a higher priority control strategy kick in, typically re-initializing
tracker with a new position/patch to track.
Back to iCub YARP module specifications
Compiler & Linker Dependencies
Include files
YARP // miscellaneous STL // for <deque>
Libraries
YARP
Run-time Dependencies
Module arguments
--file <filename> // see #Input data files
Port name control:
--name NAME: changes prefix of port names to NAME --image_name NAME: changes just the image port name --pos_name NAME: changes just the position port name --enc_name NAME: changes just the encoder port name
Default port names are:
/tracker/img (for incoming and outgoing images) /tracker/pos (for streaming position vector output)
Input format control:
--vvv: for "summer school" input format
Output format control:
--rel: for "[set] [rels] ($x $y)" input format
Adaptivity control:
--monitor: turns on adaptation behavior, where tracker monitors system response to its commands
and modifies commands to get desired response.
Ports created
Ports that are instantiated by the module and are then available for other modules to use (using yarp connect)
Under normal use, the following ports are created. The port names are just examples (controlled by options):
Input ports
/icub/tracker/img: This port expects to receive a stream of
wide-field-of-view images from a camera.
/icub/tracker/pos: This port expects to receive occasional commands
resetting the tracker or changing the area to be tracked.
/icub/tracker/enc: This (optional) port expects to receive a stream of encoder readings
from the motors. This is required if adaptation is enabled.
Output ports
(overlap with input ports)
/icub/tracker/img: This port generates an annotated copy of the stream of images received,
showing the location of the tracked area.
This is intended for human monitoring purposes only.
/icub/tracker/pos: This port generates a stream of locations, corresponding to the
currently tracked area's position. Locations can be expressed as
simple coordinates or mapped onto motor commands.
Ports accessed
Ports that are assumed to exist prior to instantiation of the module (i.e. some other module must create them)
There needs to be a wide-field-of-view-camera input available. On iCub, the tradition seems to be to call this:
/icub/cam/left
There needs to be something to control the head movement; ideally there is also a way to measure how much the head actually moved (as opposed to how it was commanded).
On iCub, control and state for the head [1] is available at:
/icub/head/rpc:i
/icub/head/state:o
Typically, the tracker is not connected directly to control the head; there is usually an intermediary module called "simple_headcontrol" [2]. This is usually
configured to create a port called:
/icub/target
That port, for historical reasons, accepts messages in "summer school format" (generated by --vvv flag to tracker).
So, to summarize, the external ports and typical connections are:
/icub/cam/left --> /icub/tracker/img
/icub/head/state:o --> /icub/tracker/enc
/icub/tracker/pos --> /icub/target
The tracker will work without an external position command input, and is frequently used this way for demos. If an attention system is available, it can be connected
as an input to:
/icub/tracker/pos.
References:
[1] http://eris.liralab.it/wiki/ICub_joints [2] http://eris.liralab.it/wiki/YARP_Tutorial
Input data files
None
Output data files
None
Configuration files
If desired, command line arguments can be moved to a configuration file by specifying --file NAME.ini.
User interface mechanism
Run-time modification of module parameters
The module is intended to run autonomously. For testing, new targets to track can be sent to the module by hooking up the click output of a yarpview process to the
modules' motor input.
OS on which the module was developed
Linux
OS on which the module was tested
Linux, Windows
Operating system dependencies
None known.
Example instantiation of the module
Run & port connection commands
(run commands in separate terminals when necessary)
(syntax needs to be a little different on Windows vs Linux)
(ICUB_ROOT = location of iCub code, YARP_ROOT = location of yarp)
cd $ICUB_ROOT/src/tracker ./tracker --name /icub/tracker --rel yarpview --name /viewer --out /clicker yarp connect /icub/cam/left /icub/tracker/img yarp connect /icub/tracker/img /viewer yarp connect /clicker /icub/tracker/pos
New standard avaiable
now tracker has a new standard for /tracker/pos output port it is possible to use it using the --pmONE option, like:
./tracker --name /icub/tracker --pmONE
iCub Capabilities
iCub capability code
C25 Gaze control: smooth pursuit (with prediction); possibly tuned by learning (but no prediction; some crude adaptation)
Other Yarp modules required to effect this capability
simple_headcontrol yarpdev (for image source, motor output)
Example instantiation
For testing, it is easiest to instantiate the module and connect it to the "fakebot" (http://eris.liralab.it/wiki/Fakebot)simulation:
cd $YARP_ROOT/examples/tutorial ./run_fakebot --file admin.ini yarp connect /fakebot/camera /icub/tracker/img yarp connect /icub/tracker/pos /fakebot/rpc:i
EXAMPLE 1
The tracker for fakebot boat example, in 8 steps:
0-(remember to run "yarp server" somewhere)
1-launch fakebot: ./run_fakebot --file admin.ini
2-launch yarpview: yarpview --name /viewer --out /clicker
3-launch the tracker: ./tracker --name /tracker --rel
4-connect yarpview and the camera of fakebot: yarp connect /fakebot/camera /viewer
5-connect tracker and fakebot's camera: yarp connect /fakebot/camera /tracker/img
6-connect yarpview and the tracker: yarp connect /tracker/img /viewer
7-connect the clicker and the tracker: yarp connect /clicker /tracker/pos
8-connect fakebot motor input and tracker position output: yarp connect /tracker/pos /fakebot/motor/rpc:i
EXAMPLE 2
The tracker for iCub's head, in 9 steps:
(I suppose you are connected with the head's dragonfly camera :o) )
0-(remember to run "yarp server" somewhere)
1-launch iCubInterface: ./iCubInterface --file icubHeadOnly.ini
2-launch yarpview: yarpview --name /viewer --out /clicker
3-launch the tracker: ./tracker --name /tracker --vvv
4-launch the controller: ./simpleHeadControl --robot icub --part head
5-launch the grabber: yarpdev --device dragonfly --d 0 --width 320 --height 240 --white_balance 20 50 --gain 600 --name /icub/cam/left --port_number 1 --shutter 0.023
6-connect yarpview and iCub's camera: yarp connect /icub/cam/left /tracker/img
7-connect yarpview and tracker: yarp connect /tracker/img /viewer
8-connect clicker and tracker: yarp connect /clicker /tracker/pos
9-connect motor input and tracker position output: yarp connect /tracker/pos /icub/target
some information about what you are trying to do:
Remember that: if you are working inside a net where other people can use the same hardware you are going to use, probably, is better to call the ports with your USERNAME inside the nameport,
for example: instead of "yarpview --name /viewer --out /clicker" write "yarpview --name /MATTIA/viewer --out /MATTIA/clicker" r
0-usually you have not to run "yarp server" because usually, each lab which uses yarp, has a server running. Yarp is able to check if someone has launched "yarp server" before you. To be sure about it you can write "yarp where" on your shell to check it.
1-ICubInterface, in this case, is going to open 3 ports:
* /icub/head/rpc:i * /icub/head/command:i * /icub/head/state:o
3-tracker, in this case, is going to open:
* /tracker/img * /tracker/pos * /tracker/enc
4-simpleHeadControl, in this case, is going to open 5 ports:
* /icub/head/control/rpc:o * /icub/head/control/command:o * /icub/head/control/state:i * /icub/target * /icub/inertial_data
5-some options have been added to yarpdev, for example, you can select the firewirecamera card you are going to use, in case your machine has more than one. the command is "port_name NumberOfThePort". Usually number 0 is used if you have just 1 firewire card and number 1 can be useful if you are using 2 cards.
Remember that:
- yarpdev has to be launched from the machine where the dragonfly card is connected
p.s. you can use the option "--shutter" to manage the brightness of your image, or you can run the program frameGrabberGui which helps you to manage Brightness, Gain, Shutter, Red and Blue online.
Back to iCub YARP module specifications