This website uses browsing/session and functional cookies to ensure you get the best experience. Learn More

YARP ResourceFinder

From Wiki for iCub and Friends
Jump to: navigation, search

NOTE: THIS PAGE IS A DRAFT --Daniele.Domenichelli@iit.it 18:17, 27 February 2013 (CET)



Contents

Directories used by ResourceFinder

Yarp resource finder uses the following environment variables to locate files

$YARP_DATA_HOME
Location where user data files are stored <path to a directory>

Unless the user wants to test some specific setting, this environment variable should be unset

$YARP_CONFIG_HOME
Location where user config files are stored <path to a directory>

Unless the user wants to test some specific setting, this environment variable should be unset

$YARP_DATA_DIRS
Locations where installed data and config files are stored <list of path separated by columns>.

These are usually the directories where the installers put the data files.

If a user wants to use uninstalled packages, he should modify this environment variable

$YARP_CONFIG_DIRS
Locations where system administrator data and config files are stored <list of path separated by columns>

These directories can be used by the system administrator to modify some specific setting for the system


The default value for these variables is dependent on the operating system


path.d
This is a special folder used by yarp to provide a portable way to install 3rd party applications in separate folders.

Files in this folder contain paths that will be inserted in the search path when looking for resources.

This allows for example to install iCub in a different folder, and still use all the plugins and resources, just by installing a single file in this directory

See later for a complete description of how it works.

robot
This is the name of the robot currently used. This is set by the $YARP_ROBOT_NAME or $ICUB_ROBOTNAME (deprecated, and perhaps we shouldn't use it in yarp) environment variables.

If the $YARP_ROBOT_NAME variable is not set, it uses by default the value "robot"

This corresponds to one extra folder that will be used to search for resources

See later for a complete description of how it works.

context
This is the current context. Modules have a default one, that can be changed by adding --context xxx to the command line.

This corresponds to one extra folder that will be used to search for resources

See later for a complete description of how it works.

current directory
This is the directory where a module is executed.

This is the first place where files are searched

See later for a complete description of how it works.

Linux

XDG Base Directory Specification

See http://standards.freedesktop.org/basedir-spec/basedir-spec-0.8.html

$XDG_DATA_HOME
-> $HOME/.local/share/
$XDG_CONFIG_HOME
-> $HOME/.config/
$XDG_DATA_DIRS
-> /usr/local/share/:/usr/share/
$XDG_CONFIG_DIRS
-> /etc/xdg/
I don't like this at all, and I think it is not used almost by any program. I suggest that we use /etc/ as default instead
if $XDG_CONFIG_DIRS is not set. Anyway in this document we assume to use the correct default --Daniele.Domenichelli@iit.it 18:28, 27 February 2013 (CET)

Not used?

$XDG_CACHE_HOME
-> $HOME/.cache/
$XDG_RUNTIME_DIR
-> /tmp/something (? see specs)

Default values

$YARP_DATA_HOME
-> $XDG_DATA_HOME + /yarp/
$YARP_CONFIG_HOME
-> $XDG_CONFIG_HOME + /yarp/
$YARP_DATA_DIRS
-> $XDG_DATA_DIRS + /yarp/ (appended to each dir)
$YARP_CONFIG_DIRS
-> $XDG_CONFIG_DIRS + /yarp/ (appended to each dir)

Windows

Environment Variables

 %YARP_DIR%
-> C:\Program Files\robotology\YARP_2.X.XX or C:\Program Files (x86)\robotology\YARP_2.X.XX (Set by the installer or by the user)
 %HOMEPATH%
-> C:\Users\{username} or C:\Documents and Settings\{username} (User home)
 %APPDATA%
-> %HOMEPATH%\AppData\Roaming (Windows 7) or %HOMEPATH%\Application Data (Windows XP) (Location where applications should store their data by default)
 %ALLUSERSPROFILE%
-> C:\ProgramData (Windows Vista) or C:\Documents and Settings\All Users (Windows XP) (Location of the "All Users" or "Common" profile folder)

Not used?

 %LOCALAPPDATA%
Location of temporary files of applications. (Since Windows Vista)


Default values

$YARP_DATA_HOME
-> %APPDATA%\yarp
$YARP_CONFIG_HOME
-> %APPDATA%\yarp\config\
$YARP_DATA_DIRS
-> %YARP_DIR%\share\yarp\
$YARP_CONFIG_DIRS
-> %ALLUSERSPROFILE%\yarp\

Mac OsX

System Directories

$HOME
-> /Users/<username>

Default values

$YARP_DATA_HOME
-> ?? $HOME/Library/Application Support/yarp
$YARP_CONFIG_HOME
-> ?? $HOME/Library/Preferences/yarp
$YARP_DATA_DIRS
-> ?? /usr/local/share/yarp
$YARP_CONFIG_DIRS
-> ?? /Library/Preferences/yarp

Files

Configuration Files

Config files are INI files.

a) User $YARP_CONFIG_HOME

       # LINUX     -> $HOME/.config/yarp/
       # WINDOWS   -> %APPDATA%\yarp\config
       # MAC       -> ?

b) Sysadmin $YARP_CONFIG_DIRS + /config/

       # LINUX      -> /etc/xdg/yarp/config/
       # WINDOWS    -> %ALLUSERSPROFILE%\yarp\config\
       # MAC        -> ?

c) Installed ($YARP_DATA_DIRS && path.d) + /config/

       # LINUX      -> /usr/local/share/yarp/config/:/usr/share/yarp/config/
       # WINDOWS    -> %YARP_DIR%\share\yarp\config\
       # MAC        -> ?

A config files are searched in a), b), c), always in this order. All of the files found should be considered, but for a specific setting, only the first value is considered. This allows to have all settings installed in c), the sysadmin can overwrite some of them in b), and the user can tweak the behaviour in a) Command line parameters should always take precedence over config files,

CMake will export a ${YARP_CONFIG_INSTALL_DIR} variable, that should be used in 3rd party software to install config files in the right directory

path.d

path.d is a special folder used by yarp to provide a portable way to install 3rd party applications in separate folders. It is searched by the resource finder in the same location as configuration files, but only for the "Installed" one (i.e. /usr/local/share/yarp/config/path.d) before searching for a resource. If more than one path.d is found, all the files with unique name are used. In case of multiple files with the same name, only the first found is used.

Paths found are appended to the list of search directories after "Installed", sorted by name.

The reason why only the "Installed" folder is allowed is that this method is specific for packaging stuff, so only the packagers should use it. Users and Sysadmins should use the environment variables

Data Files

Data files are all the architecture independent files (text files, images, audio, video, etc) that are installed by yarp or by some 3rd party software)

Generic data files are located as follow

a) User $YARP_DATA_HOME

       # LINUX     -> $HOME/.local/share/yarp/
       # WINDOWS   -> %APPDATA%\yarp\
       # MAC       -> ?

b) Sysadmin $YARP_CONFIG_DIRS

       # LINUX     -> /etc/xdg/yarp/
       # WINDOWS   -> %ALLUSERSPROFILE%\yarp\
       # MAC       -> ?

c) Installed ($YARP_DATA_DIRS && path.d)

       # LINUX     -> /usr/local/share/yarp/:/usr/share/yarp/
       # WINDOWS   -> %YARP_DIR%\share\yarp\
       # MAC       -> ?

d) Some resource types can be searched also in the directory where the module is executed, and in the "Robot" and "Context" folders.

Searches are not recursive, but they search for a filename in the exact directory as specified.

Depending on the type of data file, the resource finder should be able to have different behaviour (in pseudo-code a possible implementation of the methods) The ResourceManager should have methods for these different behaviour.

Class ResourceManager {
  enum SearchLocations {
    NoLocation  = 0x0000,
    Directory   = 0x0001, // Search current directory
    Context     = 0x0002, // Search current context directory
    Robot       = 0x0004, // Search current robot directory
    User        = 0x0008, // Search user directory
    Sysadmin    = 0x0010, // Search system config directories
    Installed   = 0x0020, // Search system data directories + path.d
    Default = User | Sysadmin | Installed,
    ModuleDefault = Default | Robot | Context | Directory
  };
  enum DuplicateFilesPolicy {
    First,      // Keep only the first file found
    All         // Keep all the files
  };
};
  • FIND ALL BY TYPE
    • Find all the resources of a specified type, and show them to the user as different files, perhaps in a treeview (for example template files).

      We should also be able to choose only certain paths (i.e. only "Installed", "Sysadmin", "User", "Robot", "Context", or "Directory", and any combination).

      If more than one file with the specified name was found we should be able to choose if show the first found, or all of them. If in one level that can have several paths ("Installed", "Sysadmin"), more than one file with the same name was found, and we are searching only for the first, then the one that comes first in the list of paths (the first found from left to right) is returned.

      • list<string> RF::findAll(string resourceType, {RF::Installed|RF::Sysadmin|RF::User|RF::Robot|RF::Context|RF::Directory}, {RF::First,RF::All});


  • FIND ALL BY TYPE AND NAME
    • Find all the resources of a specified type and name and return all of them.

      We should also be able to choose only certain paths (i.e. only "Installed", "Sysadmin", "User", "Robot", "Context", or "Directory" and any combination).

      If more than one file with the specified name was found we should be able to choose if show the first found, or all of them. If in one level that can have several paths, more than one file with the same name was found, and we are searching only for the first, then the one that comes first in the list of paths (the first found from left to right) is returned.

      • list<string> RF::findAll(string resourceType, string name, {RF::Installed|RF::Sysadmin|RF::User|RF::Robot|RF::Context|RF::Directory}, {RF::First,RF::All});
  • FIND ONE BY TYPE AND NAME
    • Find all the resources of a specified type and name and return only the top one.

      Some resource types are searched also in the folders.<p>We should also be able to choose only certain paths (i.e. only "Installed", "Sysadmin", "User", "Robot", "Context", or "Directory" and any combination)

      The priority in the search path order is always User, Sysadmin, Installed, Robot, Context, Directory) FIXME Correct order<p>If in one level that can have several paths ("Installed", "Sysadmin"), more than one file with the same name was found, the one that comes first in the list of paths (the first found from left to right) is returned;

      • string RF::find(string resourceType, string name, {RF::Installed|RF::Sysadmin|RF::User|RF::Robot|RF::Context|RF::Directory});
  • READ CONFIG
    • Read one config files and return a Searchable containing the values.

      It should have 2 different behaviours

      1. Read the first found.

        If in one level that can have several paths ("Installed", "Sysadmin"), more than one file with the same name was found, the one that comes first in the list of paths (the first found from left to right) is read;

      2. Merge the configurations files found (priority is given in this order: User, Sysadmin, Installed, Robot, Context, Directory) FIXME Correct order

        If in one level that can have several paths ("Installed", "Sysadmin"), more than one file with the same name was found, all of them are read, and priority is given to the one that comes first in the list of paths (the first found from left to right).<!p>

      • Searchable RF::readConfig(string resourceType, string name, {RF::Installed|RF::Sysadmin|RF::User|RF::Robot|RF::Context|RF::Directory}, {RF::First,RF::All});


CMake will export a ${YARP_DATA_INSTALL_DIR} variable, that should be used in 3rd party software to install config files in the right directory

Plugins Manifest Files

Plugin manifest files are INI files.

a) User $YARP_DATA_HOME + /plugins/

       # LINUX     -> $HOME/.local/share/yarp/plugins/
       # WINDOWS   -> %APPDATA%\yarp\plugins\
       # MAC       -> ?

b) Sysadmin $YARP_CONFIG_DIRS + /plugins/

       # LINUX     -> /etc/xdg/yarp/plugins/
       # WINDOWS   -> %ALLUSERSPROFILE%\yarp\plugins\
       # MAC       -> ?

c) Installed ($YARP_DATA_DIRS && path.d) + /plugins/

       # LINUX     -> /usr/local/share/yarp/plugins/:/usr/share/yarp/plugins/
       # WINDOWS   -> %YARP_DIR%\share\yarp\plugins\
       # MAC       -> ?

Plugins should include the absolute path to the library to open that includes the class implementing the plugin.

While this is not a problem for installed plugins, it might be a problem for
uninstalled stuff. Therefore we might need to modify the CMakeLists.txt to generate
2 different plugin manifest files, one that stays in the build directory, and one
that is installed (like we have for YARPConfig.cmake). We should have a CMake macro
to do this quickly.
Another option is to use the relative path, but I'm not sure if this is a good
idea, it probably complicates things anyway during installation.


CMake will export a ${YARP_PLUGIN_MANIFESTS_INSTALL_DIR} variable, that should be used in 3rd party software to install config files in the right directory


Module Files

Module files are XML files. They are usually associated to one executable

a) User $YARP_DATA_HOME + /modules/

       # LINUX     -> $HOME/.local/share/yarp/modules/
       # WINDOWS   -> %APPDATA%\yarp\modules\
       # MAC       -> ?

b) Sysadmin $YARP_CONFIG_DIRS + /modules/

       # LINUX     -> /etc/xdg/yarp/modules/
       # WINDOWS   -> %ALLUSERSPROFILE%\yarp\modules\
       # MAC       -> ?

c) Installed ($YARP_DATA_DIRS && path.d) + /modules/

       # LINUX     -> /usr/local/share/yarp/modules/:/usr/share/yarp/modules/
       # WINDOWS   -> %YARP_DIR%\share\yarp\modules\
       # MAC       -> ?


User data is writable, the other directories no, so when a user creates a new module, opens a template or a module in a read only folder, it will be saved in the User folder.

Configuration for modules is set with parameters passed in the application, and with configuration files. Configuration files are handled in the same way, as described before.

CMake will export a ${YARP_MODULES_INSTALL_DIR} variable, that should be used in 3rd party software to install config files in the right directory

Robots

Robots are directories that contain files of different type (configuration for a specific robot, etc.)

a) User: $YARP_DATA_HOME + /robots/

       # LINUX     -> $HOME/.local/share/yarp/robots/
       # WINDOWS   -> %APPDATA%\yarp\robot\
       # MAC       -> ?

b) Sysadmin: $YARP_CONFIG_DIRS + /robots/

       # LINUX     -> /etc/xdg/yarp/robots/
       # WINDOWS   -> %ALLUSERSPROFILE%\yarp\robots\
       # MAC       -> ?

c) Installed: ($YARP_DATA_DIRS && path.d) + /robots/ + path.d

       # LINUX     -> /usr/local/share/yarp/robots/:/usr/share/yarp/robots/
       # WINDOWS   -> %YARP_DIR%\share\yarp\robots\
       # MAC       -> ?

The variable $YARP_ROBOT_NAME (defaulting to "default") corresponds to the robot to use. The default value allows to set the robot just by making a symbolic link to the robot folder.

The first directory found with the same name as the environmment variable is added to the search path when looking for resources, if RF::Robot is used.

Applications

Applications are XML files, used by Yarpmanager and Yarpbuilder to run a collection of modules and establish connections between them.


a) User $YARP_DATA_HOME + /applications/

       # LINUX     -> $HOME/.local/share/yarp/applications/
       # WINDOWS   -> %APPDATA%\yarp\applications\
       # MAC       -> ?

b) Sysadmin $YARP_CONFIG_DIRS + /applications/

       # LINUX     -> /etc/xdg/yarp/applications/
       # WINDOWS   -> %ALLUSERSPROFILE%\yarp\applications\
       # MAC       -> ?

c) Installed ($YARP_DATA_DIRS && path.d) + /applications/

       # LINUX     -> /usr/local/share/yarp/applications/:/usr/share/yarp/applications/
       # WINDOWS   -> %YARP_DIR%\share\yarp\applications\
       # MAC       -> ?


User data is writable, the other directories no, so when a user creates a new application, opens a template or a application in a read only folder, it will be saved in the User folder


Even though applications are usually shipped as templates, CMake will export a ${YARP_APPLICATIONS_INSTALL_DIR} variable, that should be used in 3rd party software to install application files in the right directory.

Context

The "context directory" is the first folder that has the exact same name of the current context. This directory is added to the search path when looking for data and config files, if RF::Context is used.

a) User $YARP_DATA_HOME + /contexts/

       # LINUX     -> $HOME/.local/share/yarp/contexts/
       # WINDOWS   -> %APPDATA%\yarp\contexts\
       # MAC       -> ?

b) Sysadmin $YARP_CONFIG_DIRS + /contexts/

       # LINUX     -> /etc/xdg/yarp/contexts/
       # WINDOWS   -> %ALLUSERSPROFILE%\yarp\contexts\
       # MAC       -> ?

c) Installed ($YARP_DATA_DIRS && path.d) + /contexts/

       # LINUX     -> /usr/local/share/yarp/contexts/:/usr/share/yarp/contexts/
       # WINDOWS   -> %YARP_DIR%\share\yarp\contexts\
       # MAC       -> ?

The current context can usually be set for modules, using the --context command line argument. If --context is not used, the module uses the default context defined in the executable.


CMake will export a ${YARP_CONTEXTS_INSTALL_DIR} variable, that should be used in 3rd party software to install context-related config files in the right directory

Resource Files

Resource files are XML files. They are not installed with yarp or with 3rd party software, but are configured by the user or by the system administrator.

The name "Resource" is in my opinion ambiguous, so I strongly recommend to try to find a better name for these files


a) User $YARP_DATA_HOME + /resources/

       # LINUX     -> $HOME/.local/share/yarp/resources/
       # WINDOWS   -> %APPDATA%\yarp\resources\
       # MAC       -> ?

b) Sysadmin $YARP_CONFIG_DIRS + /resources/

       # LINUX     -> /etc/xdg/yarp/resources/
       # WINDOWS   -> %ALLUSERSPROFILE%\yarp\resources\
       # MAC       -> ?


Installed directories is not needed, because this kind of files is not supposed to be installed by external packages.

For the same reason also CMake variable is not needed.

Template Files

Template files are usually XML files. They are not supposed to be modified by the user, but both the user and the sysadmin can add their own templates.

Template base dir:

a) User $YARP_DATA_HOME + /templates/

       # LINUX     -> $HOME/.local/share/yarp/templates
       # WINDOWS   -> %APPDATA%\yarp\templates
       # MAC       -> ?

b) Sysadmin $YARP_CONFIG_DIRS + /templates/

       # LINUX     -> /etc/xdg/yarp/templates/
       # WINDOWS   -> %ALLUSERSPROFILE%\yarp\templates\
       # MAC       -> ?

c) Installed $YARP_DATA_DIRS + /templates/

       # LINUX     -> /usr/local/share/yarp/templates/:/usr/share/yarp/templates/
       # WINDOWS   -> %YARP_DIR%\share\yarp\templates
       # MAC       -> ?

Each template is installed in a subdirectory (applications, modules, resources, etc.), depending on its type.

Template files with the same name are not exclusive, that means that if an application template file "foo.xml" is found in a), b), and c), the user should see all of them when creating a new application from template


CMake will export the ${YARP_TEMPLATE_INSTALL_DIR} (generic), ${YARP_APPLICATION_TEMPLATES_INSTALL_DIR} and ${YARP_MODULE_TEMPLATES_INSTALL_DIR} (specific for application and modules) variables, that should be used in 3rd party software to install template files in the right directory. Resource templates should be installed only by YARP, therefore there is no need for a CMake variable


Generation of the search path

RF::User

  1. $YARP_DATA_HOME + /type/ (data) or $YARP_CONFIG_HOME (config)
  2. path.d for $YARP_CONFIG_HOME + /type/


RF::Sysadmin

  1. $YARP_CONFIG_DIRS + /type/
  2. path.d for $YARP_CONFIG_DIRS + /type/


RF::Installed

  1. $YARP_DATA_DIRS + /type/
  2. path.d for $YARP_DATA_DIRS + /type/


RF::Robot

  1. RF::find(robots, $YARP_ROBOT_NAME, RF::Installed|RF::Sysadmin|RF::User);


RF::Context

  1. RF::find(contexts, <current_context>, RF::Installed|RF::Sysadmin|RF::User);

Advanced Uses

Environment customization

(Note that in the following examples, the XDG_ variables are used on linux only)


  • User wants to use a separate folder to test something, maintaining multiple configurations of the same application
 # For ALL applications  -> Change $XDG_CONFIG_HOME
 # For YARP only         -> Change $YARP_CONFIG_HOME


  • User wants to avoid using sysadmin settings
 # For ALL applications  -> Change $XDG_CONFIG_DIRS (config files are searched
                            in the yarp subdirectories)
 # For YARP only         -> Change $YARP_CONFIG_DIRS


  • User wants to add settings from a package not installed
 # For ALL applications  -> Append values to $XDG_CONFIG_DIRS (i.e.
                            "/etc/xdg:/new/folder") (config files are searched
                            in the yarp subdirectories)
 # For YARP only         -> Append values to $YARP_CONFIG_DIRS (i.e.
                            "/etc/xdg/yarp:/new/folder/yarp")


  • User wants to add data from a package not installed
 # For ALL applications  -> Append values to $XDG_DATA_DIRS (i.e.
                            "/etc/xdg:/new/folder") (config files are searched
                            in the yarp subdirectories)
 # For YARP only         -> Append values to $YARP_DATA_DIRS (i.e.
                            "/etc/xdg/yarp:/new/folder/yarp")


  • User wants to keep 2 separate installations of yarp
 # For ALL applications  -> Change $XDG_DATA_DIRS (config files are searched in
                            the yarp/config subdirectories)
 # For YARP only         -> Change $YARP_DATA_DIRS (config files are searched
                            in the config subdirectories)

3rd party packages

Installed packages

Using the correct CMake variables when installing files will ensure that those files will be in the correct place on every platform.

Also non-root installation is supported, but both yarp and 3rd party software will require to be configured with the same $PREFIX, and the environment variables will require to be tweaked accordingly.

Uninstalled packages can be appended to the $YARP_DATA_DIRS environment variable

LINUX
User can set the variable in his .bashrc, sysadmin can set the variable system wide by adding a .sh file in /etc/profile.d/
     -> $YARP_DATA_DIRS -> $XDG_DATA_DIRS/yarp/:$ICUB_ROOT
WINDOWS
Edit system variables (user and system)
     -> $YARP_DATA_DIRS -> %YARP_DIR%\share\yarp\:%ICUB_DIR%\share\yarp\
MAC
 ?
     -> $YARP_DATA_DIRS -> ?


Packages that want to keep their stuff in their own data directory

If we want to allow packages to install their files in their own directory, you should install a .ini file in $YARP_PREFIX/yarp/config/path.d (or in /etc/xdg/yarp/config/path.d).

Personal tools
Namespaces

Variants
Actions
Navigation
Print/export
Toolbox