Difference between revisions of "YARP ResourceFinder"

From Wiki for iCub and Friends
Jump to: navigation, search
(Mac OsX)
 
(51 intermediate revisions by 4 users not shown)
Line 7: Line 7:
 
= Directories used by ResourceFinder =
 
= Directories used by ResourceFinder =
  
Yarp resource finder uses the following environment variables locate files
+
Yarp resource finder uses the following environment variables to locate files
  
 
; $YARP_DATA_HOME:    Location where user data files are stored <path to a directory><!--
 
; $YARP_DATA_HOME:    Location where user data files are stored <path to a directory><!--
Line 17: Line 17:
 
                 --><p>If a user wants to use uninstalled packages, he should modify this environment variable</p>
 
                 --><p>If a user wants to use uninstalled packages, he should modify this environment variable</p>
 
; $YARP_CONFIG_DIRS:  Locations where system administrator data and config files are stored <list of path separated by columns><!--
 
; $YARP_CONFIG_DIRS:  Locations where system administrator data and config files are stored <list of path separated by columns><!--
                 --><p>These directories can be used by the system administrator to modify some specific setting for the system
+
                 --><p>These directories can be used by the system administrator to modify some specific setting for the system</p>
  
  
Line 24: Line 24:
  
 
; path.d: This is a special folder used by yarp to provide a portable way to install 3rd party applications in separate folders.<!--
 
; path.d: This is a special folder used by yarp to provide a portable way to install 3rd party applications in separate folders.<!--
--><p>Files in this folder contain paths that will be appended to '''$YARP_DATA_DIRS'''</p><!--
+
--><p>Files in this folder contain paths that will be inserted in the search path when looking for resources.</p><!--
 
--><p>This allows for example to install iCub in a different folder, and still use all the plugins and resources, just by installing a single file in this directory</p><!--
 
--><p>This allows for example to install iCub in a different folder, and still use all the plugins and resources, just by installing a single file in this directory</p><!--
--><p>Since this is</p>
+
--><p>See later for a complete description of how it works.</p>
  
; robot: This is the name of the robot currently used. This is set by the $YARP_ROBOT_NAME variable
+
; robot: This is the name of the robot currently used. This is set by the '''$YARP_ROBOT_NAME''' or '''$ICUB_ROBOTNAME''' (deprecated, and perhaps we shouldn't use it in yarp) environment variables.<!--
 +
--><p>If the '''$YARP_ROBOT_NAME''' variable is not set, it uses by default the value "robot"</p><!--
 +
--><p>This corresponds to one extra folder that will be used to search for resources</p><!--
 +
--><p>See later for a complete description of how it works.</p>
  
; context: Modules have a default context, that can be changed by adding <code>--context xxx</code> to the command line.
+
; context: This is the current context. Modules have a default one, that can be changed by adding <code>--context xxx</code> to the command line.<!--
 +
--><p>This corresponds to one extra folder that will be used to search for resources</p><!--
 +
--><p>See later for a complete description of how it works.</p>
 +
 
 +
; current directory: This is the directory where a module is executed.<!--
 +
--><p>This is the first place where files are searched</p><!--
 +
--><p>See later for a complete description of how it works.</p>
  
 
== Linux ==
 
== Linux ==
Line 81: Line 90:
 
=== System Directories ===
 
=== System Directories ===
  
?
+
; $HOME:              -> /Users/<username>
  
 
=== Default values ===
 
=== Default values ===
  
; $YARP_DATA_HOME:    -> ?
+
; $YARP_DATA_HOME:    -> ?? $HOME/Library/Application Support/yarp
; $YARP_CONFIG_HOME:  -> ?
+
; $YARP_CONFIG_HOME:  -> ?? $HOME/Library/Preferences/yarp
; $YARP_DATA_DIRS:    -> ?
+
; $YARP_DATA_DIRS:    -> ?? /usr/local/share/yarp
; $YARP_CONFIG_DIRS:  -> ?
+
; $YARP_CONFIG_DIRS:  -> ?? /Library/Preferences/yarp
 
 
 
 
  
 
= Files =
 
= Files =
Line 111: Line 118:
 
         # MAC        -> ?
 
         # MAC        -> ?
  
c) Installed    $YARP_DATA_DIRS  +  /config/
+
c) Installed    ($YARP_DATA_DIRS && path.d) +  /config/
 
         # LINUX      -> /usr/local/share/yarp/config/:/usr/share/yarp/config/
 
         # LINUX      -> /usr/local/share/yarp/config/:/usr/share/yarp/config/
 
         # WINDOWS    -> %YARP_DIR%\share\yarp\config\
 
         # WINDOWS    -> %YARP_DIR%\share\yarp\config\
 
         # MAC        -> ?
 
         # MAC        -> ?
  
 
+
A config files are searched in a), b), c), always in this order.
 
 
A config files are searched in a), b), and c), in this order.
 
 
All of the files found should be considered, but for a specific setting, only
 
All of the files found should be considered, but for a specific setting, only
 
the first value is considered. This allows to have all settings installed in c),
 
the first value is considered. This allows to have all settings installed in c),
Line 127: Line 132:
 
CMake will export a '''${YARP_CONFIG_INSTALL_DIR}''' variable, that should be used in
 
CMake will export a '''${YARP_CONFIG_INSTALL_DIR}''' variable, that should be used in
 
3rd party software to install config files in the right directory
 
3rd party software to install config files in the right directory
 
 
  
 
== path.d ==
 
== path.d ==
  
  
'''path.d''' are special folders used by yarp to provide a portable way to
+
'''path.d''' is a special folder used by yarp to provide a portable way to install 3rd party applications in separate folders.
install 3rd party applications in separate folders.
+
It is searched by the resource finder in the same location as configuration files, but only for the "Installed" one (i.e. <code>/usr/local/share/yarp/config/path.d</code>) before searching for a resource.
They are searched by the resource finder in the same location as configuration
+
If more than one path.d is found, all the files with unique name are used. In case of multiple files with the same name, only the first found is used.
files (User, Sysadmin and Installed) (i.e.
 
<code>/usr/local/share/yarp/config/path.d</code>) before searching for a
 
resource. Paths found are inserted to the list of directories as if they were in
 
$YARP_DATA_DIRS, after the item, sorted by name.
 
Paths found are immediately scanned for path.d directories.
 
To avoid recursive paths, the  
 
 
 
 
 
For example if
 
* $YARP_DATA_DIRS=/usr/local/share/yarp/:/usr/share/yarp/
 
* $YARP_CONFIG_DIRS=/etc/xdg/yarp/
 
* $YARP_CONFIG_HOME=$HOME/.config/yarp
 
and in the path.d folders there are the these files
 
* /usr/share/yarp/config/path.d/icub.ini -> /usr/share/icub/
 
* /usr/local/share/yarp/config/path.d/foo.ini -> /usr/local/share/foo/
 
* /usr/local/share/yarp/config/path.d/bar.ini -> /usr/local/share/bar/
 
* /usr/local/share/bar/config/path.d/bar.ini -> /usr/local/share/goo/
 
* $HOME/.config/path.d/user.ini -> $HOME/user/
 
* /etc/xdg/yarp/config/path.d/sysadmin.ini -> /opt/sysadmin/
 
 
 
The full search path for resource files is
 
 
 
* $HOME/.config/yarp/
 
* $HOME/user/
 
* /etc/xdg/yarp/
 
* /opt/sysadmin/
 
* /usr/local/share/yarp/
 
* /usr/local/share/bar/
 
* /usr/local/share/goo/
 
* /usr/local/share/foo/
 
* /usr/share/yarp/
 
* /usr/share/icub/
 
  
'''NOTE''': this example is intentionally complicated, usually the full search
+
Paths found are appended to the list of search directories after "Installed", sorted by name.
path will be a lot easier, for example
 
  
* $HOME/.config/yarp/
+
The reason why only the "Installed" folder is allowed is that this method is specific for packaging stuff, so only the packagers should use it. Users and Sysadmins should use the environment variables
* /etc/xdg/yarp/
 
* /usr/share/yarp/
 
* /usr/share/icub/
 
  
 
== Data Files ==
 
== Data Files ==
Line 196: Line 163:
 
         # MAC      -> ?
 
         # MAC      -> ?
  
c) Installed    $YARP_DATA_DIRS
+
c) Installed    ($YARP_DATA_DIRS && path.d)
 
         # LINUX    -> /usr/local/share/yarp/:/usr/share/yarp/
 
         # LINUX    -> /usr/local/share/yarp/:/usr/share/yarp/
 
         # WINDOWS  -> %YARP_DIR%\share\yarp\
 
         # WINDOWS  -> %YARP_DIR%\share\yarp\
 
         # MAC      -> ?
 
         # MAC      -> ?
  
 +
d) Some resource types can be searched also in the directory where the module is executed, and in the "Robot" and "Context" folders.
 +
 +
Searches are not recursive, but they search for a filename in the exact directory as specified.
  
 
Depending on the type of data file, the resource finder should be able to have
 
Depending on the type of data file, the resource finder should be able to have
Line 206: Line 176:
 
The ResourceManager should have methods for these different behaviour.
 
The ResourceManager should have methods for these different behaviour.
  
 +
<pre>Class ResourceManager {
 +
  enum SearchLocations {
 +
    NoLocation  = 0x0000,
 +
    Directory  = 0x0001, // Search current directory
 +
    Context    = 0x0002, // Search current context directory
 +
    Robot      = 0x0004, // Search current robot directory
 +
    User        = 0x0008, // Search user directory
 +
    Sysadmin    = 0x0010, // Search system config directories
 +
    Installed  = 0x0020, // Search system data directories + path.d
 +
    Default = User | Sysadmin | Installed,
 +
    ModuleDefault = Default | Robot | Context | Directory
 +
  };
 +
  enum DuplicateFilesPolicy {
 +
    First,      // Keep only the first file found
 +
    All        // Keep all the files
 +
  };
 +
};
 +
</pre>
  
 
*'''FIND ALL BY TYPE'''
 
*'''FIND ALL BY TYPE'''
 
** Find all the resources of a specified type, and show them to the user as different files, perhaps in a treeview (for example template files).<!--
 
** Find all the resources of a specified type, and show them to the user as different files, perhaps in a treeview (for example template files).<!--
--><p>We should also be able to choose only certain paths (i.e. only "Installed", "Sysadmin", or "User", and any combination).</p><!--
+
--><p>We should also be able to choose only certain paths (i.e. only "Installed", "Sysadmin", "User", "Robot", "Context", or "Directory", and any combination).</p><!--
 
--><p>If more than one file with the specified name was found we should be able to choose if show the first found, or all of them. If in one level that can have several paths ("Installed", "Sysadmin"), more than one file with the same name was found, and we are searching only for the first, then the one that comes first in the list of paths (the first found from left to right) is returned.</p>
 
--><p>If more than one file with the specified name was found we should be able to choose if show the first found, or all of them. If in one level that can have several paths ("Installed", "Sysadmin"), more than one file with the same name was found, and we are searching only for the first, then the one that comes first in the list of paths (the first found from left to right) is returned.</p>
***<code>list<string> RF::findAll(string resourceType, {RF::Installed|RF::Sysadmin|RF::User}, {RF::First,RF::All});</code>
+
***<code>list<string> RF::findAll(string resourceType, {RF::Installed|RF::Sysadmin|RF::User|RF::Robot|RF::Context|RF::Directory}, {RF::First,RF::All});</code>
 +
 
  
 
*'''FIND ALL BY TYPE AND NAME'''
 
*'''FIND ALL BY TYPE AND NAME'''
 
** Find all the resources of a specified type and name and return all of them.<!--
 
** Find all the resources of a specified type and name and return all of them.<!--
--><p>We should also be able to choose only certain paths (i.e. only "Installed", "Sysadmin", or "User", and any combination).</p><!--
+
--><p>We should also be able to choose only certain paths (i.e. only "Installed", "Sysadmin", "User", "Robot", "Context", or "Directory" and any combination).</p><!--
 
--><p>If more than one file with the specified name was found we should be able to choose if show the first found, or all of them. If in one level that can have several paths, more than one file with the same name was found, and we are searching only for the first, then the one that comes first in the list of paths (the first found from left to right) is returned.</p>
 
--><p>If more than one file with the specified name was found we should be able to choose if show the first found, or all of them. If in one level that can have several paths, more than one file with the same name was found, and we are searching only for the first, then the one that comes first in the list of paths (the first found from left to right) is returned.</p>
***<code>list<string> RF::findAll(string resourceType, string name, {RF::Installed|RF::Sysadmin|RF::User}, {RF::First,RF::All});</code>
+
***<code>list<string> RF::findAll(string resourceType, string name, {RF::Installed|RF::Sysadmin|RF::User|RF::Robot|RF::Context|RF::Directory}, {RF::First,RF::All});</code>
  
 
*'''FIND ONE BY TYPE AND NAME'''
 
*'''FIND ONE BY TYPE AND NAME'''
** Find all the resources of a specified type and name and return only the top one (in this order: User, Sysadmin, and finally Installed).<!--
+
** Find all the resources of a specified type and name and return only the top one.<!--
--><p>We should also be able to choose only certain paths (i.e. only "Installed", "Sysadmin", or "User", and any combination)</p><!--
+
--><p>Some resource types are searched also in the  folders.<!--
 +
--><p>We should also be able to choose only certain paths (i.e. only "Installed", "Sysadmin", "User", "Robot", "Context", or "Directory" and any combination)</p><!--
 +
--><p>The priority in the search path order is always User, Sysadmin, Installed, Robot, Context, Directory) '''FIXME''' Correct order<!--
 
--><p>If in one level that can have several paths ("Installed", "Sysadmin"), more than one file with the same name was found, the one that comes first in the list of paths (the first found from left to right) is returned;</p>
 
--><p>If in one level that can have several paths ("Installed", "Sysadmin"), more than one file with the same name was found, the one that comes first in the list of paths (the first found from left to right) is returned;</p>
***<code>string RF::find(string resourceType, string name, {RF::Installed|RF::Sysadmin|RF::User});</code>
+
***<code>string RF::find(string resourceType, string name, {RF::Installed|RF::Sysadmin|RF::User|RF::Robot|RF::Context|RF::Directory});</code>
  
 
* '''READ CONFIG'''
 
* '''READ CONFIG'''
Line 230: Line 221:
 
**# Read the first found.<!--
 
**# Read the first found.<!--
 
--><p>If in one level that can have several paths ("Installed", "Sysadmin"), more than one file with the same name was found, the one that comes first in the list of paths (the first found from left to right) is read;</p>
 
--><p>If in one level that can have several paths ("Installed", "Sysadmin"), more than one file with the same name was found, the one that comes first in the list of paths (the first found from left to right) is read;</p>
**# Merge the configurations files found (priority is given first to the User file, then to the Sysadmin file, and finally to the Installed file)<!--
+
**# Merge the configurations files found (priority is given in this order: User, Sysadmin, Installed, Robot, Context, Directory) '''FIXME''' Correct order<!--
 
--><p>If in one level that can have several paths ("Installed", "Sysadmin"), more than one file with the same name was found, all of them are read, and priority is given to the one that comes first in the list of paths (the first found from left to right).<!p>
 
--><p>If in one level that can have several paths ("Installed", "Sysadmin"), more than one file with the same name was found, all of them are read, and priority is given to the one that comes first in the list of paths (the first found from left to right).<!p>
***<code>Searchable RF::readConfig(string resourceType, string name, {RF::Installed|RF::Sysadmin|RF::User}, {RF::First,RF::All});</code>
+
***<code>Searchable RF::readConfig(string resourceType, string name, {RF::Installed|RF::Sysadmin|RF::User|RF::Robot|RF::Context|RF::Directory}, {RF::First,RF::All});</code>
 
 
 
 
 
 
I strongly recommend that we avoid allowing recursive search, or it will make a lot harder everything. It doesn't mean that we don't allow to have
 
subdirectories, but each subdirectory will contain one resource type. For example the templates folder will have an application directory: application
 
template types will be "templates/applications" --[[User:Daniele.Domenichelli@iit.it|Daniele.Domenichelli@iit.it]] 09:58, 28 February 2013 (CET)
 
 
 
 
 
  
  
 
CMake will export a '''${YARP_DATA_INSTALL_DIR}''' variable, that should be used in
 
CMake will export a '''${YARP_DATA_INSTALL_DIR}''' variable, that should be used in
 
3rd party software to install config files in the right directory
 
3rd party software to install config files in the right directory
 
 
 
  
 
== Plugins Manifest Files ==
 
== Plugins Manifest Files ==
 
  
  
Line 265: Line 244:
 
         # MAC      -> ?
 
         # MAC      -> ?
  
c) Installed    $YARP_DATA_DIRS  +  /plugins/
+
c) Installed    ($YARP_DATA_DIRS && path.d) +  /plugins/
 
         # LINUX    -> /usr/local/share/yarp/plugins/:/usr/share/yarp/plugins/
 
         # LINUX    -> /usr/local/share/yarp/plugins/:/usr/share/yarp/plugins/
 
         # WINDOWS  -> %YARP_DIR%\share\yarp\plugins\
 
         # WINDOWS  -> %YARP_DIR%\share\yarp\plugins\
 
         # MAC      -> ?
 
         # MAC      -> ?
  
Plugins should include the absolute path to the library to open that includes
+
Plugins should include the absolute path to the library to open that
the class implementing the plugin.
+
includes the class implementing the plugin.
While this is not a problem for installed plugins, it might be a problem for
+
While this is not a problem for installed plugins, it might be a problem for
uninstalled stuff. Therefore we might need something
+
uninstalled stuff. Therefore we might need to modify the CMakeLists.txt to generate
 +
2 different plugin manifest files, one that stays in the build directory, and one
 +
that is installed (like we have for YARPConfig.cmake). We should have a CMake macro
 +
to do this quickly.
 +
Another option is to use the ''relative'' path, but I'm not sure if this is a good
 +
idea, it probably complicates things anyway during installation.
  
  
 
CMake will export a '''${YARP_PLUGIN_MANIFESTS_INSTALL_DIR}''' variable, that should
 
CMake will export a '''${YARP_PLUGIN_MANIFESTS_INSTALL_DIR}''' variable, that should
 
be used in 3rd party software to install config files in the right directory
 
be used in 3rd party software to install config files in the right directory
 
  
  
 
== Module Files ==
 
== Module Files ==
 
  
  
Line 298: Line 280:
 
         # MAC      -> ?
 
         # MAC      -> ?
  
c) Installed    $YARP_DATA_DIRS  +  /modules/
+
c) Installed    ($YARP_DATA_DIRS && path.d) +  /modules/
 
         # LINUX    -> /usr/local/share/yarp/modules/:/usr/share/yarp/modules/
 
         # LINUX    -> /usr/local/share/yarp/modules/:/usr/share/yarp/modules/
 
         # WINDOWS  -> %YARP_DIR%\share\yarp\modules\
 
         # WINDOWS  -> %YARP_DIR%\share\yarp\modules\
Line 306: Line 288:
 
User data is writable, the other directories no, so when a user creates a new
 
User data is writable, the other directories no, so when a user creates a new
 
module, opens a template or a module in a read only folder, it will be saved
 
module, opens a template or a module in a read only folder, it will be saved
in the User folder
+
in the User folder.
 
 
  
 
Configuration for modules is set with parameters passed in the application, and
 
Configuration for modules is set with parameters passed in the application, and
 
with configuration files. Configuration files are handled in the same way, as
 
with configuration files. Configuration files are handled in the same way, as
 
described before.
 
described before.
 
  
 
CMake will export a '''${YARP_MODULES_INSTALL_DIR}''' variable, that should be used in
 
CMake will export a '''${YARP_MODULES_INSTALL_DIR}''' variable, that should be used in
 
3rd party software to install config files in the right directory
 
3rd party software to install config files in the right directory
 
 
  
 
== Robots ==
 
== Robots ==
Line 323: Line 301:
  
 
Robots are directories that contain files of different type (configuration for
 
Robots are directories that contain files of different type (configuration for
a specific robot, etc
+
a specific robot, etc.)
  
a) User        $YARP_DATA_HOME  +  /robot/
+
a) User:         $YARP_DATA_HOME  +  /robots/
         # LINUX    -> $HOME/.local/share/yarp/robot/
+
         # LINUX    -> $HOME/.local/share/yarp/robots/
 
         # WINDOWS  -> %APPDATA%\yarp\robot\
 
         # WINDOWS  -> %APPDATA%\yarp\robot\
 
         # MAC      -> ?
 
         # MAC      -> ?
  
b) Sysadmin    $YARP_CONFIG_DIRS  +  /robot/
+
b) Sysadmin:     $YARP_CONFIG_DIRS  +  /robots/
         # LINUX    -> /etc/xdg/yarp/robot/
+
         # LINUX    -> /etc/xdg/yarp/robots/
         # WINDOWS  -> %ALLUSERSPROFILE%\yarp\robot\
+
         # WINDOWS  -> %ALLUSERSPROFILE%\yarp\robots\
 
         # MAC      -> ?
 
         # MAC      -> ?
  
c) Installed    $YARP_DATA_DIRS  +  /robot/
+
c) Installed:   ($YARP_DATA_DIRS && path.d) +  /robots/ +  path.d
         # LINUX    -> /usr/local/share/yarp/robot/:/usr/share/yarp/robot/
+
         # LINUX    -> /usr/local/share/yarp/robots/:/usr/share/yarp/robots/
         # WINDOWS  -> %YARP_DIR%\share\yarp\robot\
+
         # WINDOWS  -> %YARP_DIR%\share\yarp\robots\
 
         # MAC      -> ?
 
         # MAC      -> ?
  
 +
The variable $YARP_ROBOT_NAME (defaulting to "default") corresponds to the robot to use.
 +
The default value allows to set the robot just by making a symbolic link to the robot folder.
  
User data is writable, the other directories no, so when a user creates a new
+
The first directory found with the same name as the environmment variable is added to the search path when looking for resources, if RF::Robot is used.
application, opens a template or a application in a read only folder, it will
 
be saved in the User folder
 
  
 +
== Applications ==
  
Even though applications are usually shipped as templates, CMake will export a
 
'''${YARP_APPLICATIONS_INSTALL_DIR}''' variable, that should be used in 3rd party
 
software to install config files in the right directory.
 
 
 
 
== Applications ==
 
  
 +
Applications are XML files, used by Yarpmanager and Yarpbuilder to run a collection of modules and establish connections between them.
  
Applications are directories that contain files of different type (demo,
 
examples, tests, etc)
 
  
a) User        $YARP_DATA_HOME  +  /app/
+
a) User        $YARP_DATA_HOME  +  /applications/
         # LINUX    -> $HOME/.local/share/yarp/app/
+
         # LINUX    -> $HOME/.local/share/yarp/applications/
         # WINDOWS  -> %APPDATA%\yarp\app\
+
         # WINDOWS  -> %APPDATA%\yarp\applications\
 
         # MAC      -> ?
 
         # MAC      -> ?
  
b) Sysadmin    $YARP_CONFIG_DIRS  +  /app/
+
b) Sysadmin    $YARP_CONFIG_DIRS  +  /applications/
         # LINUX    -> /etc/xdg/yarp/app/
+
         # LINUX    -> /etc/xdg/yarp/applications/
         # WINDOWS  -> %ALLUSERSPROFILE%\yarp\app\
+
         # WINDOWS  -> %ALLUSERSPROFILE%\yarp\applications\
 
         # MAC      -> ?
 
         # MAC      -> ?
  
c) Installed    $YARP_DATA_DIRS  +  /app/
+
c) Installed    ($YARP_DATA_DIRS && path.d) +  /applications/
         # LINUX    -> /usr/local/share/yarp/app/:/usr/share/yarp/app/
+
         # LINUX    -> /usr/local/share/yarp/applications/:/usr/share/yarp/applications/
         # WINDOWS  -> %YARP_DIR%\share\yarp\app\
+
         # WINDOWS  -> %YARP_DIR%\share\yarp\applications\
 
         # MAC      -> ?
 
         # MAC      -> ?
  
Line 381: Line 352:
 
Even though applications are usually shipped as templates, CMake will export a
 
Even though applications are usually shipped as templates, CMake will export a
 
'''${YARP_APPLICATIONS_INSTALL_DIR}''' variable, that should be used in 3rd party
 
'''${YARP_APPLICATIONS_INSTALL_DIR}''' variable, that should be used in 3rd party
software to install config files in the right directory.
+
software to install application files in the right directory.
  
 
== Context ==
 
== Context ==
  
The "context directory" is the first application folder that has the exact same
+
The "context directory" is the first folder that has the exact same
name of the current context. This directory is added to the search path for
+
name of the current context. This directory is added to the search path when
resource and config files if RF::Context is used
+
looking for data and config files, if RF::Context is used.
 +
 
 +
a) User        $YARP_DATA_HOME  +  /contexts/
 +
        # LINUX    -> $HOME/.local/share/yarp/contexts/
 +
        # WINDOWS  -> %APPDATA%\yarp\contexts\
 +
        # MAC      -> ?
 +
 
 +
b) Sysadmin    $YARP_CONFIG_DIRS  +  /contexts/
 +
        # LINUX    -> /etc/xdg/yarp/contexts/
 +
        # WINDOWS  -> %ALLUSERSPROFILE%\yarp\contexts\
 +
        # MAC      -> ?
 +
 
 +
c) Installed    ($YARP_DATA_DIRS && path.d)  +  /contexts/
 +
        # LINUX    -> /usr/local/share/yarp/contexts/:/usr/share/yarp/contexts/
 +
        # WINDOWS  -> %YARP_DIR%\share\yarp\contexts\
 +
        # MAC      -> ?
 +
 
 +
The current context can usually be set for modules, using the --context command line argument.
 +
If --context is not used, the module uses the default context defined in the executable.
 +
 
  
 +
CMake will export a '''${YARP_CONTEXTS_INSTALL_DIR}''' variable, that should be used in
 +
3rd party software to install context-related config files in the right directory
  
 
== Resource Files ==
 
== Resource Files ==
Line 394: Line 386:
  
  
Resource files are INI files. They are not installed with yarp or with 3rd party
+
Resource files are XML files. They are not installed with yarp or with 3rd party
 
software, but are configured by the user or by the system administrator.
 
software, but are configured by the user or by the system administrator.
  
Line 416: Line 408:
  
 
For the same reason also CMake variable is not needed.
 
For the same reason also CMake variable is not needed.
 
 
  
 
== Template Files ==
 
== Template Files ==
Line 460: Line 450:
  
  
 +
 +
= Generation of the search path =
 +
 +
== RF::User ==
 +
 +
# '''$YARP_DATA_HOME''' + /type/ (data) or '''$YARP_CONFIG_HOME''' (config)
 +
# path.d for '''$YARP_CONFIG_HOME''' + /type/
 +
 +
 +
== RF::Sysadmin ==
 +
 +
# '''$YARP_CONFIG_DIRS''' + /type/
 +
# path.d for '''$YARP_CONFIG_DIRS''' + /type/
 +
 +
 +
== RF::Installed ==
 +
 +
# '''$YARP_DATA_DIRS''' + /type/
 +
# path.d for '''$YARP_DATA_DIRS ''' + /type/
 +
 +
 +
== RF::Robot ==
 +
 +
# <code>RF::find(robots, $YARP_ROBOT_NAME, RF::Installed|RF::Sysadmin|RF::User);</code>
 +
 +
 +
== RF::Context ==
 +
 +
# <code>RF::find(contexts, <current_context>, RF::Installed|RF::Sysadmin|RF::User);</code>
  
 
= Advanced Uses =
 
= Advanced Uses =

Latest revision as of 14:09, 12 June 2013

NOTE: THIS PAGE IS A DRAFT --Daniele.Domenichelli@iit.it 18:17, 27 February 2013 (CET)



Directories used by ResourceFinder

Yarp resource finder uses the following environment variables to locate files

$YARP_DATA_HOME
Location where user data files are stored <path to a directory>

Unless the user wants to test some specific setting, this environment variable should be unset

$YARP_CONFIG_HOME
Location where user config files are stored <path to a directory>

Unless the user wants to test some specific setting, this environment variable should be unset

$YARP_DATA_DIRS
Locations where installed data and config files are stored <list of path separated by columns>.

These are usually the directories where the installers put the data files.

If a user wants to use uninstalled packages, he should modify this environment variable

$YARP_CONFIG_DIRS
Locations where system administrator data and config files are stored <list of path separated by columns>

These directories can be used by the system administrator to modify some specific setting for the system


The default value for these variables is dependent on the operating system


path.d
This is a special folder used by yarp to provide a portable way to install 3rd party applications in separate folders.

Files in this folder contain paths that will be inserted in the search path when looking for resources.

This allows for example to install iCub in a different folder, and still use all the plugins and resources, just by installing a single file in this directory

See later for a complete description of how it works.

robot
This is the name of the robot currently used. This is set by the $YARP_ROBOT_NAME or $ICUB_ROBOTNAME (deprecated, and perhaps we shouldn't use it in yarp) environment variables.

If the $YARP_ROBOT_NAME variable is not set, it uses by default the value "robot"

This corresponds to one extra folder that will be used to search for resources

See later for a complete description of how it works.

context
This is the current context. Modules have a default one, that can be changed by adding --context xxx to the command line.

This corresponds to one extra folder that will be used to search for resources

See later for a complete description of how it works.

current directory
This is the directory where a module is executed.

This is the first place where files are searched

See later for a complete description of how it works.

Linux

XDG Base Directory Specification

See http://standards.freedesktop.org/basedir-spec/basedir-spec-0.8.html

$XDG_DATA_HOME
-> $HOME/.local/share/
$XDG_CONFIG_HOME
-> $HOME/.config/
$XDG_DATA_DIRS
-> /usr/local/share/:/usr/share/
$XDG_CONFIG_DIRS
-> /etc/xdg/
I don't like this at all, and I think it is not used almost by any program. I suggest that we use /etc/ as default instead
if $XDG_CONFIG_DIRS is not set. Anyway in this document we assume to use the correct default --Daniele.Domenichelli@iit.it 18:28, 27 February 2013 (CET)

Not used?

$XDG_CACHE_HOME
-> $HOME/.cache/
$XDG_RUNTIME_DIR
-> /tmp/something (? see specs)

Default values

$YARP_DATA_HOME
-> $XDG_DATA_HOME + /yarp/
$YARP_CONFIG_HOME
-> $XDG_CONFIG_HOME + /yarp/
$YARP_DATA_DIRS
-> $XDG_DATA_DIRS + /yarp/ (appended to each dir)
$YARP_CONFIG_DIRS
-> $XDG_CONFIG_DIRS + /yarp/ (appended to each dir)

Windows

Environment Variables

 %YARP_DIR%
-> C:\Program Files\robotology\YARP_2.X.XX or C:\Program Files (x86)\robotology\YARP_2.X.XX (Set by the installer or by the user)
 %HOMEPATH%
-> C:\Users\{username} or C:\Documents and Settings\{username} (User home)
 %APPDATA%
-> %HOMEPATH%\AppData\Roaming (Windows 7) or %HOMEPATH%\Application Data (Windows XP) (Location where applications should store their data by default)
 %ALLUSERSPROFILE%
-> C:\ProgramData (Windows Vista) or C:\Documents and Settings\All Users (Windows XP) (Location of the "All Users" or "Common" profile folder)

Not used?

 %LOCALAPPDATA%
Location of temporary files of applications. (Since Windows Vista)


Default values

$YARP_DATA_HOME
-> %APPDATA%\yarp
$YARP_CONFIG_HOME
-> %APPDATA%\yarp\config\
$YARP_DATA_DIRS
-> %YARP_DIR%\share\yarp\
$YARP_CONFIG_DIRS
-> %ALLUSERSPROFILE%\yarp\

Mac OsX

System Directories

$HOME
-> /Users/<username>

Default values

$YARP_DATA_HOME
-> ?? $HOME/Library/Application Support/yarp
$YARP_CONFIG_HOME
-> ?? $HOME/Library/Preferences/yarp
$YARP_DATA_DIRS
-> ?? /usr/local/share/yarp
$YARP_CONFIG_DIRS
-> ?? /Library/Preferences/yarp

Files

Configuration Files

Config files are INI files.

a) User $YARP_CONFIG_HOME

       # LINUX     -> $HOME/.config/yarp/
       # WINDOWS   -> %APPDATA%\yarp\config
       # MAC       -> ?

b) Sysadmin $YARP_CONFIG_DIRS + /config/

       # LINUX      -> /etc/xdg/yarp/config/
       # WINDOWS    -> %ALLUSERSPROFILE%\yarp\config\
       # MAC        -> ?

c) Installed ($YARP_DATA_DIRS && path.d) + /config/

       # LINUX      -> /usr/local/share/yarp/config/:/usr/share/yarp/config/
       # WINDOWS    -> %YARP_DIR%\share\yarp\config\
       # MAC        -> ?

A config files are searched in a), b), c), always in this order. All of the files found should be considered, but for a specific setting, only the first value is considered. This allows to have all settings installed in c), the sysadmin can overwrite some of them in b), and the user can tweak the behaviour in a) Command line parameters should always take precedence over config files,

CMake will export a ${YARP_CONFIG_INSTALL_DIR} variable, that should be used in 3rd party software to install config files in the right directory

path.d

path.d is a special folder used by yarp to provide a portable way to install 3rd party applications in separate folders. It is searched by the resource finder in the same location as configuration files, but only for the "Installed" one (i.e. /usr/local/share/yarp/config/path.d) before searching for a resource. If more than one path.d is found, all the files with unique name are used. In case of multiple files with the same name, only the first found is used.

Paths found are appended to the list of search directories after "Installed", sorted by name.

The reason why only the "Installed" folder is allowed is that this method is specific for packaging stuff, so only the packagers should use it. Users and Sysadmins should use the environment variables

Data Files

Data files are all the architecture independent files (text files, images, audio, video, etc) that are installed by yarp or by some 3rd party software)

Generic data files are located as follow

a) User $YARP_DATA_HOME

       # LINUX     -> $HOME/.local/share/yarp/
       # WINDOWS   -> %APPDATA%\yarp\
       # MAC       -> ?

b) Sysadmin $YARP_CONFIG_DIRS

       # LINUX     -> /etc/xdg/yarp/
       # WINDOWS   -> %ALLUSERSPROFILE%\yarp\
       # MAC       -> ?

c) Installed ($YARP_DATA_DIRS && path.d)

       # LINUX     -> /usr/local/share/yarp/:/usr/share/yarp/
       # WINDOWS   -> %YARP_DIR%\share\yarp\
       # MAC       -> ?

d) Some resource types can be searched also in the directory where the module is executed, and in the "Robot" and "Context" folders.

Searches are not recursive, but they search for a filename in the exact directory as specified.

Depending on the type of data file, the resource finder should be able to have different behaviour (in pseudo-code a possible implementation of the methods) The ResourceManager should have methods for these different behaviour.

Class ResourceManager {
  enum SearchLocations {
    NoLocation  = 0x0000,
    Directory   = 0x0001, // Search current directory
    Context     = 0x0002, // Search current context directory
    Robot       = 0x0004, // Search current robot directory
    User        = 0x0008, // Search user directory
    Sysadmin    = 0x0010, // Search system config directories
    Installed   = 0x0020, // Search system data directories + path.d
    Default = User | Sysadmin | Installed,
    ModuleDefault = Default | Robot | Context | Directory
  };
  enum DuplicateFilesPolicy {
    First,      // Keep only the first file found
    All         // Keep all the files
  };
};
  • FIND ALL BY TYPE
    • Find all the resources of a specified type, and show them to the user as different files, perhaps in a treeview (for example template files).

      We should also be able to choose only certain paths (i.e. only "Installed", "Sysadmin", "User", "Robot", "Context", or "Directory", and any combination).

      If more than one file with the specified name was found we should be able to choose if show the first found, or all of them. If in one level that can have several paths ("Installed", "Sysadmin"), more than one file with the same name was found, and we are searching only for the first, then the one that comes first in the list of paths (the first found from left to right) is returned.

      • list<string> RF::findAll(string resourceType, {RF::Installed|RF::Sysadmin|RF::User|RF::Robot|RF::Context|RF::Directory}, {RF::First,RF::All});


  • FIND ALL BY TYPE AND NAME
    • Find all the resources of a specified type and name and return all of them.

      We should also be able to choose only certain paths (i.e. only "Installed", "Sysadmin", "User", "Robot", "Context", or "Directory" and any combination).

      If more than one file with the specified name was found we should be able to choose if show the first found, or all of them. If in one level that can have several paths, more than one file with the same name was found, and we are searching only for the first, then the one that comes first in the list of paths (the first found from left to right) is returned.

      • list<string> RF::findAll(string resourceType, string name, {RF::Installed|RF::Sysadmin|RF::User|RF::Robot|RF::Context|RF::Directory}, {RF::First,RF::All});
  • FIND ONE BY TYPE AND NAME
    • Find all the resources of a specified type and name and return only the top one.

      Some resource types are searched also in the folders.<p>We should also be able to choose only certain paths (i.e. only "Installed", "Sysadmin", "User", "Robot", "Context", or "Directory" and any combination)

      The priority in the search path order is always User, Sysadmin, Installed, Robot, Context, Directory) FIXME Correct order<p>If in one level that can have several paths ("Installed", "Sysadmin"), more than one file with the same name was found, the one that comes first in the list of paths (the first found from left to right) is returned;

      • string RF::find(string resourceType, string name, {RF::Installed|RF::Sysadmin|RF::User|RF::Robot|RF::Context|RF::Directory});
  • READ CONFIG
    • Read one config files and return a Searchable containing the values.

      It should have 2 different behaviours

      1. Read the first found.

        If in one level that can have several paths ("Installed", "Sysadmin"), more than one file with the same name was found, the one that comes first in the list of paths (the first found from left to right) is read;

      2. Merge the configurations files found (priority is given in this order: User, Sysadmin, Installed, Robot, Context, Directory) FIXME Correct order

        If in one level that can have several paths ("Installed", "Sysadmin"), more than one file with the same name was found, all of them are read, and priority is given to the one that comes first in the list of paths (the first found from left to right).<!p>

      • Searchable RF::readConfig(string resourceType, string name, {RF::Installed|RF::Sysadmin|RF::User|RF::Robot|RF::Context|RF::Directory}, {RF::First,RF::All});


CMake will export a ${YARP_DATA_INSTALL_DIR} variable, that should be used in 3rd party software to install config files in the right directory

Plugins Manifest Files

Plugin manifest files are INI files.

a) User $YARP_DATA_HOME + /plugins/

       # LINUX     -> $HOME/.local/share/yarp/plugins/
       # WINDOWS   -> %APPDATA%\yarp\plugins\
       # MAC       -> ?

b) Sysadmin $YARP_CONFIG_DIRS + /plugins/

       # LINUX     -> /etc/xdg/yarp/plugins/
       # WINDOWS   -> %ALLUSERSPROFILE%\yarp\plugins\
       # MAC       -> ?

c) Installed ($YARP_DATA_DIRS && path.d) + /plugins/

       # LINUX     -> /usr/local/share/yarp/plugins/:/usr/share/yarp/plugins/
       # WINDOWS   -> %YARP_DIR%\share\yarp\plugins\
       # MAC       -> ?

Plugins should include the absolute path to the library to open that includes the class implementing the plugin.

While this is not a problem for installed plugins, it might be a problem for
uninstalled stuff. Therefore we might need to modify the CMakeLists.txt to generate
2 different plugin manifest files, one that stays in the build directory, and one
that is installed (like we have for YARPConfig.cmake). We should have a CMake macro
to do this quickly.
Another option is to use the relative path, but I'm not sure if this is a good
idea, it probably complicates things anyway during installation.


CMake will export a ${YARP_PLUGIN_MANIFESTS_INSTALL_DIR} variable, that should be used in 3rd party software to install config files in the right directory


Module Files

Module files are XML files. They are usually associated to one executable

a) User $YARP_DATA_HOME + /modules/

       # LINUX     -> $HOME/.local/share/yarp/modules/
       # WINDOWS   -> %APPDATA%\yarp\modules\
       # MAC       -> ?

b) Sysadmin $YARP_CONFIG_DIRS + /modules/

       # LINUX     -> /etc/xdg/yarp/modules/
       # WINDOWS   -> %ALLUSERSPROFILE%\yarp\modules\
       # MAC       -> ?

c) Installed ($YARP_DATA_DIRS && path.d) + /modules/

       # LINUX     -> /usr/local/share/yarp/modules/:/usr/share/yarp/modules/
       # WINDOWS   -> %YARP_DIR%\share\yarp\modules\
       # MAC       -> ?


User data is writable, the other directories no, so when a user creates a new module, opens a template or a module in a read only folder, it will be saved in the User folder.

Configuration for modules is set with parameters passed in the application, and with configuration files. Configuration files are handled in the same way, as described before.

CMake will export a ${YARP_MODULES_INSTALL_DIR} variable, that should be used in 3rd party software to install config files in the right directory

Robots

Robots are directories that contain files of different type (configuration for a specific robot, etc.)

a) User: $YARP_DATA_HOME + /robots/

       # LINUX     -> $HOME/.local/share/yarp/robots/
       # WINDOWS   -> %APPDATA%\yarp\robot\
       # MAC       -> ?

b) Sysadmin: $YARP_CONFIG_DIRS + /robots/

       # LINUX     -> /etc/xdg/yarp/robots/
       # WINDOWS   -> %ALLUSERSPROFILE%\yarp\robots\
       # MAC       -> ?

c) Installed: ($YARP_DATA_DIRS && path.d) + /robots/ + path.d

       # LINUX     -> /usr/local/share/yarp/robots/:/usr/share/yarp/robots/
       # WINDOWS   -> %YARP_DIR%\share\yarp\robots\
       # MAC       -> ?

The variable $YARP_ROBOT_NAME (defaulting to "default") corresponds to the robot to use. The default value allows to set the robot just by making a symbolic link to the robot folder.

The first directory found with the same name as the environmment variable is added to the search path when looking for resources, if RF::Robot is used.

Applications

Applications are XML files, used by Yarpmanager and Yarpbuilder to run a collection of modules and establish connections between them.


a) User $YARP_DATA_HOME + /applications/

       # LINUX     -> $HOME/.local/share/yarp/applications/
       # WINDOWS   -> %APPDATA%\yarp\applications\
       # MAC       -> ?

b) Sysadmin $YARP_CONFIG_DIRS + /applications/

       # LINUX     -> /etc/xdg/yarp/applications/
       # WINDOWS   -> %ALLUSERSPROFILE%\yarp\applications\
       # MAC       -> ?

c) Installed ($YARP_DATA_DIRS && path.d) + /applications/

       # LINUX     -> /usr/local/share/yarp/applications/:/usr/share/yarp/applications/
       # WINDOWS   -> %YARP_DIR%\share\yarp\applications\
       # MAC       -> ?


User data is writable, the other directories no, so when a user creates a new application, opens a template or a application in a read only folder, it will be saved in the User folder


Even though applications are usually shipped as templates, CMake will export a ${YARP_APPLICATIONS_INSTALL_DIR} variable, that should be used in 3rd party software to install application files in the right directory.

Context

The "context directory" is the first folder that has the exact same name of the current context. This directory is added to the search path when looking for data and config files, if RF::Context is used.

a) User $YARP_DATA_HOME + /contexts/

       # LINUX     -> $HOME/.local/share/yarp/contexts/
       # WINDOWS   -> %APPDATA%\yarp\contexts\
       # MAC       -> ?

b) Sysadmin $YARP_CONFIG_DIRS + /contexts/

       # LINUX     -> /etc/xdg/yarp/contexts/
       # WINDOWS   -> %ALLUSERSPROFILE%\yarp\contexts\
       # MAC       -> ?

c) Installed ($YARP_DATA_DIRS && path.d) + /contexts/

       # LINUX     -> /usr/local/share/yarp/contexts/:/usr/share/yarp/contexts/
       # WINDOWS   -> %YARP_DIR%\share\yarp\contexts\
       # MAC       -> ?

The current context can usually be set for modules, using the --context command line argument. If --context is not used, the module uses the default context defined in the executable.


CMake will export a ${YARP_CONTEXTS_INSTALL_DIR} variable, that should be used in 3rd party software to install context-related config files in the right directory

Resource Files

Resource files are XML files. They are not installed with yarp or with 3rd party software, but are configured by the user or by the system administrator.

The name "Resource" is in my opinion ambiguous, so I strongly recommend to try to find a better name for these files


a) User $YARP_DATA_HOME + /resources/

       # LINUX     -> $HOME/.local/share/yarp/resources/
       # WINDOWS   -> %APPDATA%\yarp\resources\
       # MAC       -> ?

b) Sysadmin $YARP_CONFIG_DIRS + /resources/

       # LINUX     -> /etc/xdg/yarp/resources/
       # WINDOWS   -> %ALLUSERSPROFILE%\yarp\resources\
       # MAC       -> ?


Installed directories is not needed, because this kind of files is not supposed to be installed by external packages.

For the same reason also CMake variable is not needed.

Template Files

Template files are usually XML files. They are not supposed to be modified by the user, but both the user and the sysadmin can add their own templates.

Template base dir:

a) User $YARP_DATA_HOME + /templates/

       # LINUX     -> $HOME/.local/share/yarp/templates
       # WINDOWS   -> %APPDATA%\yarp\templates
       # MAC       -> ?

b) Sysadmin $YARP_CONFIG_DIRS + /templates/

       # LINUX     -> /etc/xdg/yarp/templates/
       # WINDOWS   -> %ALLUSERSPROFILE%\yarp\templates\
       # MAC       -> ?

c) Installed $YARP_DATA_DIRS + /templates/

       # LINUX     -> /usr/local/share/yarp/templates/:/usr/share/yarp/templates/
       # WINDOWS   -> %YARP_DIR%\share\yarp\templates
       # MAC       -> ?

Each template is installed in a subdirectory (applications, modules, resources, etc.), depending on its type.

Template files with the same name are not exclusive, that means that if an application template file "foo.xml" is found in a), b), and c), the user should see all of them when creating a new application from template


CMake will export the ${YARP_TEMPLATE_INSTALL_DIR} (generic), ${YARP_APPLICATION_TEMPLATES_INSTALL_DIR} and ${YARP_MODULE_TEMPLATES_INSTALL_DIR} (specific for application and modules) variables, that should be used in 3rd party software to install template files in the right directory. Resource templates should be installed only by YARP, therefore there is no need for a CMake variable


Generation of the search path

RF::User

  1. $YARP_DATA_HOME + /type/ (data) or $YARP_CONFIG_HOME (config)
  2. path.d for $YARP_CONFIG_HOME + /type/


RF::Sysadmin

  1. $YARP_CONFIG_DIRS + /type/
  2. path.d for $YARP_CONFIG_DIRS + /type/


RF::Installed

  1. $YARP_DATA_DIRS + /type/
  2. path.d for $YARP_DATA_DIRS + /type/


RF::Robot

  1. RF::find(robots, $YARP_ROBOT_NAME, RF::Installed|RF::Sysadmin|RF::User);


RF::Context

  1. RF::find(contexts, <current_context>, RF::Installed|RF::Sysadmin|RF::User);

Advanced Uses

Environment customization

(Note that in the following examples, the XDG_ variables are used on linux only)


  • User wants to use a separate folder to test something, maintaining multiple configurations of the same application
 # For ALL applications  -> Change $XDG_CONFIG_HOME
 # For YARP only         -> Change $YARP_CONFIG_HOME


  • User wants to avoid using sysadmin settings
 # For ALL applications  -> Change $XDG_CONFIG_DIRS (config files are searched
                            in the yarp subdirectories)
 # For YARP only         -> Change $YARP_CONFIG_DIRS


  • User wants to add settings from a package not installed
 # For ALL applications  -> Append values to $XDG_CONFIG_DIRS (i.e.
                            "/etc/xdg:/new/folder") (config files are searched
                            in the yarp subdirectories)
 # For YARP only         -> Append values to $YARP_CONFIG_DIRS (i.e.
                            "/etc/xdg/yarp:/new/folder/yarp")


  • User wants to add data from a package not installed
 # For ALL applications  -> Append values to $XDG_DATA_DIRS (i.e.
                            "/etc/xdg:/new/folder") (config files are searched
                            in the yarp subdirectories)
 # For YARP only         -> Append values to $YARP_DATA_DIRS (i.e.
                            "/etc/xdg/yarp:/new/folder/yarp")


  • User wants to keep 2 separate installations of yarp
 # For ALL applications  -> Change $XDG_DATA_DIRS (config files are searched in
                            the yarp/config subdirectories)
 # For YARP only         -> Change $YARP_DATA_DIRS (config files are searched
                            in the config subdirectories)

3rd party packages

Installed packages

Using the correct CMake variables when installing files will ensure that those files will be in the correct place on every platform.

Also non-root installation is supported, but both yarp and 3rd party software will require to be configured with the same $PREFIX, and the environment variables will require to be tweaked accordingly.

Uninstalled packages can be appended to the $YARP_DATA_DIRS environment variable

LINUX
User can set the variable in his .bashrc, sysadmin can set the variable system wide by adding a .sh file in /etc/profile.d/
     -> $YARP_DATA_DIRS -> $XDG_DATA_DIRS/yarp/:$ICUB_ROOT
WINDOWS
Edit system variables (user and system)
     -> $YARP_DATA_DIRS -> %YARP_DIR%\share\yarp\:%ICUB_DIR%\share\yarp\
MAC
 ?
     -> $YARP_DATA_DIRS -> ?


Packages that want to keep their stuff in their own data directory

If we want to allow packages to install their files in their own directory, you should install a .ini file in $YARP_PREFIX/yarp/config/path.d (or in /etc/xdg/yarp/config/path.d).