Difference between revisions of "YARP ResourceFinder"

From Wiki for iCub and Friends
Jump to: navigation, search
(3rd party packages)
(Installed packages)
Line 429: Line 429:
 
=== Installed packages ===
 
=== Installed packages ===
  
  Using the correct CMake variables when installing files will ensure that those
+
Using the correct CMake variables when installing files will ensure that those files will be in the correct place on every platform.
  files will be in the correct place on every platform.
 
 
 
  Also non-root installation is supported, but both yarp and 3rd party software
 
  will require to be configured with the same $PREFIX, and the environment
 
  variables will require to be tweaked accordingly.
 
  
 +
Also non-root installation is supported, but both yarp and 3rd party software will require to be configured with the same $PREFIX, and the environment variables will require to be tweaked accordingly.
  
 
=== Uninstalled packages can be appended to the $YARP_DATA_DIRS environment variable ===
 
=== Uninstalled packages can be appended to the $YARP_DATA_DIRS environment variable ===

Revision as of 10:02, 28 February 2013

NOTE: THIS PAGE IS A DRAFT --Daniele.Domenichelli@iit.it 18:17, 27 February 2013 (CET)



Directories used by ResourceFinder

Yarp resource finder uses the following environment variables locate files

$YARP_DATA_HOME
Location where user data files are stored <path to a directory>
$YARP_CONFIG_HOME
Location where user config files are stored <path to a directory>
$YARP_DATA_DIRS
Locations where system data files are stored <list of path separated by columns>
$YARP_CONFIG_DIRS
Locations where system config files are stored <list of path separated by columns>

The default value for these variables is dependent on the operating system

Linux

XDG Base Directory Specification

See http://standards.freedesktop.org/basedir-spec/basedir-spec-0.8.html

$XDG_DATA_HOME
-> $HOME/.local/share/
$XDG_CONFIG_HOME
-> $HOME/.config/
$XDG_DATA_DIRS
-> /usr/local/share/:/usr/share/
$XDG_CONFIG_DIRS
-> /etc/xdg/
I don't like this at all, and I think it is not used almost by any program. I suggest that we use /etc/ as default instead
if $XDG_CONFIG_DIRS is not set. Anyway in this document we assume to use the correct default --Daniele.Domenichelli@iit.it 18:28, 27 February 2013 (CET)

Not used?

$XDG_CACHE_HOME
-> $HOME/.cache/
$XDG_RUNTIME_DIR
-> /tmp/something (? see specs)

Default values

$YARP_DATA_HOME
-> $XDG_DATA_HOME + /yarp/
$YARP_CONFIG_HOME
-> $XDG_CONFIG_HOME + /yarp/
$YARP_DATA_DIRS
-> $XDG_DATA_DIRS + /yarp/ (appended to each dir)
$YARP_CONFIG_DIRS
-> $XDG_CONFIG_DIRS + /yarp/ (appended to each dir)

Windows

Environment Variables

 %YARP_DIR%
-> C:\Program Files\robotology\YARP_2.X.XX or C:\Program Files (x86)\robotology\YARP_2.X.XX (Set by the installer or by the user)
 %HOMEPATH%
-> C:\Users\{username} or C:\Documents and Settings\{username} (User home)
 %APPDATA%
-> %HOMEPATH%\AppData\Roaming (Windows 7) or %HOMEPATH%\Application Data (Windows XP) (Location where applications should store their data by default)
 %ALLUSERSPROFILE%
-> C:\ProgramData (Windows Vista) or C:\Documents and Settings\All Users (Windows XP) (Location of the "All Users" or "Common" profile folder)

Not used?

 %LOCALAPPDATA%
Location of temporary files of applications. (Since Windows Vista)


Default values

$YARP_DATA_HOME
-> %APPDATA%\yarp
$YARP_CONFIG_HOME
-> %APPDATA%\yarp\config\
$YARP_DATA_DIRS
-> %YARP_DIR%\share\yarp\
$YARP_CONFIG_DIRS
-> %ALLUSERSPROFILE%\yarp\

Mac OsX

System Directories

?

Default values

$YARP_DATA_HOME
-> ?
$YARP_CONFIG_HOME
-> ?
$YARP_DATA_DIRS
-> ?
$YARP_CONFIG_DIRS
-> ?


Files

Configuration Files

Config files are INI files.

a) User $YARP_CONFIG_HOME

       # LINUX     -> $HOME/.config/yarp/
       # WINDOWS   -> %APPDATA%\yarp\config
       # MAC       -> ?

b) Sysadmin $YARP_CONFIG_DIRS + /config/

       # LINUX      -> /etc/xdg/yarp/config/
       # WINDOWS    -> %ALLUSERSPROFILE%\yarp\config\
       # MAC        -> ?

c) Installed $YARP_DATA_DIRS + /config/

       # LINUX      -> /usr/local/share/yarp/config/:/usr/share/yarp/config/
       # WINDOWS    -> %YARP_DIR%\share\yarp\config\
       # MAC        -> ?


A config files are searched in a), b), and c), in this order. All of the files found should be considered, but for a specific setting, only the first value is considered. This allows to have all settings installed in c), the sysadmin can overwrite some of them in b), and the user can tweak the behaviour in a) Command line parameters should always take precedence over config files,

CMake will export a ${YARP_CONFIG_INSTALL_DIR} variable, that should be used in 3rd party software to install config files in the right directory


Data Files

Data files are all the architecture independent files (text files, images, audio, video, etc) that are installed by yarp or by some 3rd party software)

Generic data files are located as follow

a) User $YARP_DATA_HOME

       # LINUX     -> $HOME/.local/share/yarp/
       # WINDOWS   -> %APPDATA%\yarp\
       # MAC       -> ?

b) Sysadmin $YARP_CONFIG_DIRS

       # LINUX     -> /etc/xdg/yarp/
       # WINDOWS   -> %ALLUSERSPROFILE%\yarp\
       # MAC       -> ?

c) Installed $YARP_DATA_DIRS

       # LINUX     -> /usr/local/share/yarp/:/usr/share/yarp/
       # WINDOWS   -> %YARP_DIR%\share\yarp\
       # MAC       -> ?


Depending on the type of data file, the resource finder should be able to have different behaviour (in pseudo-code a possible implementation of the methods) The ResourceManager should have methods for these different behaviour.


  • FIND ALL BY TYPE
    • Find all the resources of a specified type, and show them to the user as different files, perhaps in a treeview (for example template files)
      • list<string> RF::findAll(string resourceType, RF::Installed|RF::Sysadmin|RF::User);
    • Find all the resources of a specified type and for each unique name, return only one choosing in order User, Sysadmin and finally Installed. If more than one file with the specified name was found in the same level (i.e. Installed) we should be able to choose if show the first, the last, or all of them.
      • list<string> RF::findAll(string resourceType, RF::Installed|RF::Sysadmin|RF::User|RF::One|RF::First);
      • list<string> RF::findAll(string resourceType, RF::Installed|RF::Sysadmin|RF::User|RF::One|RF::Last);
      • list<string> RF::findAll(string resourceType, RF::Installed|RF::Sysadmin|RF::User|RF::One|RF::All);
    • Find all the resources of a specified type in the "Installed" directories. If more than one file with the specified name was found we should be able to choose if show the first, the last, or all of them
      • list<string> RF::findAll(string resourceType, RF::Installed|RF::First);
      • list<string> RF::findAll(string resourceType, RF::Installed|RF::Last);
      • list<string> RF::findAll(string resourceType, RF::Installed|RF::All);
    • Find all the resources of a specified type in the "Sysadmin" directories
      • list<string> RF::findAll(string resourceType, RF::Sysadmin|RF::First);
      • list<string> RF::findAll(string resourceType, RF::Sysadmin|RF::Last);
      • list<string> RF::findAll(string resourceType, RF::Sysadmin|RF::All);
    • Find all the resources of a specified type in the "User" directory
      • list<string> RF::findAll(string resourceType, RF::User);


  • FIND ALL BY TYPE AND NAME
    • Find all the resources of a specified type and name and return all of them
      • list<string> RF::findAll(string resourceType, string name, RF::Installed|RF::Sysadmin|RF::User);
    • Find all the resources of a specified type and name in the "Installed" directories and return all of them.
      • list<string> RF::findAll(string resourceType, string name, RF::Installed);
    • Find all the resources of a specified type and name in the "Installed" directories and return all of them.
      • list<string> RF::findAll(string resourceType, string name, RF::Sysadmin);


  • FIND ONE BY TYPE AND NAME
    • Find all the resources of a specified type and name and return only the top one (in this order: User, Sysadmin, and finally Installed). If more than one file with the specified name was found in the same level (i.e. Installed) we should be able to choose if show the first or the last.
      • string RF::find(string resourceType, string name, RF::Installed|RF::Sysadmin|RF::User|RF::First);
      • string RF::find(string resourceType, string name, RF::Installed|RF::Sysadmin|RF::User|RF::Last);
    • Find all the resources of a specified type and name in the "Installed" directories and return only one of them. If more than one file with the specified name was found in the same level (i.e. Installed) we should be able to choose if show the first or the last.
      • string RF::find(string resourceType, string name, RF::Installed|RF::First);
      • string RF::find(string resourceType, string name, RF::Installed|RF::Last);
    • Find all the resources of a specified type and name in the "Sysadmin" directories and return only one of them. If more than one file with the specified name was found in the same level (i.e. Installed) we should be able to choose if show the first or the last.
      • string RF::find(string resourceType, string name, RF::Installed|RF::First);
      • string RF::find(string resourceType, string name, RF::Installed|RF::Last);
    • Find the resource of a specified type and name in the "User" directory
      • string RF::find(string resourceType, string name, RF::User);


  • MERGE ALL FILES WITH GIVEN TYPE AND NAME
    • Find all of them and merge them (for example for config files, a setting should be searched first in the User file, if not found in the Sysadmin file, and finally if not found also in the Installed file). If more than one file with the specified name was found in the same level (i.e. Installed) we should be able to choose whether priority should be given to the first found or to the last found
      • Searchable RF::readConfig(string resourceType, string name, RF::First);
      • Searchable RF::readConfig(string resourceType, string name, RF::Last);


I strongly recommend that we avoid allowing recursive search, or it will make a lot harder everything. It doesn't mean that we don't allow to have
subdirectories, but each subdirectory will contain one resource type. For example the templates folder will have an application directory: application
template types will be "templates/applications" --Daniele.Domenichelli@iit.it 09:58, 28 February 2013 (CET)



CMake will export a ${YARP_DATA_INSTALL_DIR} variable, that should be used in 3rd party software to install config files in the right directory



Plugins Manifest Files

Plugin manifest files are INI files.

a) User $YARP_DATA_HOME + /plugins/

       # LINUX     -> $HOME/.local/share/yarp/plugins/
       # WINDOWS   -> %APPDATA%\yarp\plugins\
       # MAC       -> ?

b) Sysadmin $YARP_CONFIG_DIRS + /plugins/

       # LINUX     -> /etc/xdg/yarp/plugins/
       # WINDOWS   -> %ALLUSERSPROFILE%\yarp\plugins\
       # MAC       -> ?

c) Installed $YARP_DATA_DIRS + /plugins/

       # LINUX     -> /usr/local/share/yarp/plugins/:/usr/share/yarp/plugins/
       # WINDOWS   -> %YARP_DIR%\share\yarp\plugins\
       # MAC       -> ?

Plugins should include the absolute path to the library to open that includes the class implementing the plugin. While this is not a problem for installed plugins, it might be a problem for uninstalled stuff. Therefore we might need something


CMake will export a ${YARP_PLUGIN_MANIFESTS_INSTALL_DIR} variable, that should be used in 3rd party software to install config files in the right directory


Module Files

Module files are XML files. They are usually associated to one executable

a) User $YARP_DATA_HOME + /modules/

       # LINUX     -> $HOME/.local/share/yarp/modules/
       # WINDOWS   -> %APPDATA%\yarp\modules\
       # MAC       -> ?

b) Sysadmin $YARP_CONFIG_DIRS + /modules/

       # LINUX     -> /etc/xdg/yarp/modules/
       # WINDOWS   -> %ALLUSERSPROFILE%\yarp\modules\
       # MAC       -> ?

c) Installed $YARP_DATA_DIRS + /modules/

       # LINUX     -> /usr/local/share/yarp/modules/:/usr/share/yarp/modules/
       # WINDOWS   -> %YARP_DIR%\share\yarp\modules\
       # MAC       -> ?


User data is writable, the other directories no, so when a user creates a new module, opens a template or a module in a read only folder, it will be saved in the User folder


Configuration for modules is set with parameters passed in the application, and with configuration files. Configuration files are handled in the same way, as described before.


CMake will export a ${YARP_MODULES_INSTALL_DIR} variable, that should be used in 3rd party software to install config files in the right directory


Application Files

Application files are XML files.

a) User $YARP_DATA_HOME + /applications/

       # LINUX     -> $HOME/.local/share/yarp/applications/
       # WINDOWS   -> %APPDATA%\yarp\applications\
       # MAC       -> ?

b) Sysadmin $YARP_CONFIG_DIRS + /applications/

       # LINUX     -> /etc/xdg/yarp/applications/
       # WINDOWS   -> %ALLUSERSPROFILE%\yarp\applications\
       # MAC       -> ?

c) Installed $YARP_DATA_DIRS + /applications/

       # LINUX     -> /usr/local/share/yarp/applications/:/usr/share/yarp/applications/
       # WINDOWS   -> %YARP_DIR%\share\yarp\applications\
       # MAC       -> ?


User data is writable, the other directories no, so when a user creates a new application, opens a template or a application in a read only folder, it will be saved in the User folder


Even though applications are usually shipped as templates, CMake will export a ${YARP_APPLICATIONS_INSTALL_DIR} variable, that should be used in 3rd party software to install config files in the right directory.


Resource Files

Resource files are INI files. They are not installed with yarp or with 3rd party software, but are configured by the user or by the system administrator.

The name "Resource" is in my opinion ambiguous, so I strongly recommend to try to find a better name for these files


a) User $YARP_DATA_HOME + /resources/

       # LINUX     -> $HOME/.local/share/yarp/resources/
       # WINDOWS   -> %APPDATA%\yarp\resources\
       # MAC       -> ?

b) Sysadmin $YARP_CONFIG_DIRS + /resources/

       # LINUX     -> /etc/xdg/yarp/resources/
       # WINDOWS   -> %ALLUSERSPROFILE%\yarp\resources\
       # MAC       -> ?


Installed directories is not needed, because this kind of files is not supposed to be installed by external packages.

For the same reason also CMake variable is not needed.


Template Files

Template files are usually XML files. They are not supposed to be modified by the user, but both the user and the sysadmin can add their own templates.

Template base dir:

a) User $YARP_DATA_HOME + /templates/

       # LINUX     -> $HOME/.local/share/yarp/templates
       # WINDOWS   -> %APPDATA%\yarp\templates
       # MAC       -> ?

b) Sysadmin $YARP_CONFIG_DIRS + /templates/

       # LINUX     -> /etc/xdg/yarp/templates/
       # WINDOWS   -> %ALLUSERSPROFILE%\yarp\templates\
       # MAC       -> ?

c) Installed $YARP_DATA_DIRS + /templates/

       # LINUX     -> /usr/local/share/yarp/templates/:/usr/share/yarp/templates/
       # WINDOWS   -> %YARP_DIR%\share\yarp\templates
       # MAC       -> ?

Each template is installed in a subdirectory (applications, modules, resources, etc.), depending on its type.

Template files with the same name are not exclusive, that means that if an application template file "foo.xml" is found in a), b), and c), the user should see all of them when creating a new application from template


CMake will export the ${YARP_TEMPLATE_INSTALL_DIR} (generic), ${YARP_APPLICATION_TEMPLATES_INSTALL_DIR} and ${YARP_MODULE_TEMPLATES_INSTALL_DIR} (specific for application and modules) variables, that should be used in 3rd party software to install template files in the right directory. Resource templates should be installed only by YARP, therefore there is no need for a CMake variable

Advanced Uses

(Note that in the following examples, the XDG_ variables are used on linux only)


  • User wants to use a separate folder to test something, maintaining multiple configurations of the same application
 # For ALL applications  -> Change $XDG_CONFIG_HOME
 # For YARP only         -> Change $YARP_CONFIG_HOME


  • User wants to avoid using sysadmin settings
 # For ALL applications  -> Change $XDG_CONFIG_DIRS (config files are searched
                            in the yarp subdirectories)
 # For YARP only         -> Change $YARP_CONFIG_DIRS


  • User wants to add settings from a package not installed
 # For ALL applications  -> Append values to $XDG_CONFIG_DIRS (i.e.
                            "/etc/xdg:/new/folder") (config files are searched
                            in the yarp subdirectories)
 # For YARP only         -> Append values to $YARP_CONFIG_DIRS (i.e.
                            "/etc/xdg/yarp:/new/folder/yarp")


  • User wants to add data from a package not installed
 # For ALL applications  -> Append values to $XDG_DATA_DIRS (i.e.
                            "/etc/xdg:/new/folder") (config files are searched
                            in the yarp subdirectories)
 # For YARP only         -> Append values to $YARP_DATA_DIRS (i.e.
                            "/etc/xdg/yarp:/new/folder/yarp")


  • User wants to keep 2 separate installations of yarp
 # For ALL applications  -> Change $XDG_DATA_DIRS (config files are searched in
                            the yarp/config subdirectories)
 # For YARP only         -> Change $YARP_DATA_DIRS (config files are searched
                            in the config subdirectories)

3rd party packages

Installed packages

Using the correct CMake variables when installing files will ensure that those files will be in the correct place on every platform.

Also non-root installation is supported, but both yarp and 3rd party software will require to be configured with the same $PREFIX, and the environment variables will require to be tweaked accordingly.

Uninstalled packages can be appended to the $YARP_DATA_DIRS environment variable

LINUX
User can set the variable in his .bashrc, sysadmin can set the variable system wide by adding a .sh file in /etc/profile.d/
     -> $YARP_DATA_DIRS -> $XDG_DATA_DIRS/yarp/:$ICUB_ROOT
   
WINDOWS
Edit system variables (user and system)
     -> $YARP_DATA_DIRS -> %YARP_DIR%\share\yarp\:%ICUB_DIR%\share\yarp\
MAC
 ?
     -> $YARP_DATA_DIRS -> ?


Packages that want to keep their stuff in their own data directory

If we want to allow packages to install their files in their own directory, we could use the /etc/yarp/path.d directory as we discussed about. Nonetheless, instead of using the /etc directory, since we are dealing with files installed by another package, I suggest to use /usr/share/yarp/path.d instead.