Difference between revisions of "YARP ResourceFinder"

From Wiki for iCub and Friends
Jump to: navigation, search
(Data Files)
(Data Files)
Line 147: Line 147:
 
** Find all the resources of a specified type, and show them to the user as different files, perhaps in a treeview (for example template files).<!--
 
** Find all the resources of a specified type, and show them to the user as different files, perhaps in a treeview (for example template files).<!--
 
--><p>We should also be able to choose only certain paths (i.e. only "Installed", "Sysadmin", or "User", and any combination).</p><!--
 
--><p>We should also be able to choose only certain paths (i.e. only "Installed", "Sysadmin", or "User", and any combination).</p><!--
--><p>If more than one file with the specified name was found we should be able to choose if show the first found, or all of them. If in one level that can have several paths ("Installed", "Sysadmin"), more than one file with the same name was found, and we are searching only for the first, then the one that comes first in the list of paths (the first found from left to right) is returned.
+
--><p>If more than one file with the specified name was found we should be able to choose if show the first found, or all of them. If in one level that can have several paths ("Installed", "Sysadmin"), more than one file with the same name was found, and we are searching only for the first, then the one that comes first in the list of paths (the first found from left to right) is returned.</p>
 
***<code>list<string> RF::findAll(string resourceType, {RF::Installed|RF::Sysadmin|RF::User}, {RF::First,RF::All});</code>
 
***<code>list<string> RF::findAll(string resourceType, {RF::Installed|RF::Sysadmin|RF::User}, {RF::First,RF::All});</code>
  
 
*'''FIND ALL BY TYPE AND NAME'''
 
*'''FIND ALL BY TYPE AND NAME'''
** Find all the resources of a specified type and name and return all of them<!--
+
** Find all the resources of a specified type and name and return all of them.<!--
 
--><p>We should also be able to choose only certain paths (i.e. only "Installed", "Sysadmin", or "User", and any combination).</p><!--
 
--><p>We should also be able to choose only certain paths (i.e. only "Installed", "Sysadmin", or "User", and any combination).</p><!--
--><p>If more than one file with the specified name was found we should be able to choose if show the first found, or all of them. If in one level that can have several paths, more than one file with the same name was found, and we are searching only for the first, then the one that comes first in the list of paths (the first found from left to right) is returned..
+
--><p>If more than one file with the specified name was found we should be able to choose if show the first found, or all of them. If in one level that can have several paths, more than one file with the same name was found, and we are searching only for the first, then the one that comes first in the list of paths (the first found from left to right) is returned.</p>
 
***<code>list<string> RF::findAll(string resourceType, string name, {RF::Installed|RF::Sysadmin|RF::User}, {RF::First,RF::All});</code>
 
***<code>list<string> RF::findAll(string resourceType, string name, {RF::Installed|RF::Sysadmin|RF::User}, {RF::First,RF::All});</code>
  

Revision as of 19:39, 28 February 2013

NOTE: THIS PAGE IS A DRAFT --Daniele.Domenichelli@iit.it 18:17, 27 February 2013 (CET)



Directories used by ResourceFinder

Yarp resource finder uses the following environment variables locate files

$YARP_DATA_HOME
Location where user data files are stored <path to a directory>
$YARP_CONFIG_HOME
Location where user config files are stored <path to a directory>
$YARP_DATA_DIRS
Locations where system data files are stored <list of path separated by columns>
$YARP_CONFIG_DIRS
Locations where system config files are stored <list of path separated by columns>

The default value for these variables is dependent on the operating system

Linux

XDG Base Directory Specification

See http://standards.freedesktop.org/basedir-spec/basedir-spec-0.8.html

$XDG_DATA_HOME
-> $HOME/.local/share/
$XDG_CONFIG_HOME
-> $HOME/.config/
$XDG_DATA_DIRS
-> /usr/local/share/:/usr/share/
$XDG_CONFIG_DIRS
-> /etc/xdg/
I don't like this at all, and I think it is not used almost by any program. I suggest that we use /etc/ as default instead
if $XDG_CONFIG_DIRS is not set. Anyway in this document we assume to use the correct default --Daniele.Domenichelli@iit.it 18:28, 27 February 2013 (CET)

Not used?

$XDG_CACHE_HOME
-> $HOME/.cache/
$XDG_RUNTIME_DIR
-> /tmp/something (? see specs)

Default values

$YARP_DATA_HOME
-> $XDG_DATA_HOME + /yarp/
$YARP_CONFIG_HOME
-> $XDG_CONFIG_HOME + /yarp/
$YARP_DATA_DIRS
-> $XDG_DATA_DIRS + /yarp/ (appended to each dir)
$YARP_CONFIG_DIRS
-> $XDG_CONFIG_DIRS + /yarp/ (appended to each dir)

Windows

Environment Variables

 %YARP_DIR%
-> C:\Program Files\robotology\YARP_2.X.XX or C:\Program Files (x86)\robotology\YARP_2.X.XX (Set by the installer or by the user)
 %HOMEPATH%
-> C:\Users\{username} or C:\Documents and Settings\{username} (User home)
 %APPDATA%
-> %HOMEPATH%\AppData\Roaming (Windows 7) or %HOMEPATH%\Application Data (Windows XP) (Location where applications should store their data by default)
 %ALLUSERSPROFILE%
-> C:\ProgramData (Windows Vista) or C:\Documents and Settings\All Users (Windows XP) (Location of the "All Users" or "Common" profile folder)

Not used?

 %LOCALAPPDATA%
Location of temporary files of applications. (Since Windows Vista)


Default values

$YARP_DATA_HOME
-> %APPDATA%\yarp
$YARP_CONFIG_HOME
-> %APPDATA%\yarp\config\
$YARP_DATA_DIRS
-> %YARP_DIR%\share\yarp\
$YARP_CONFIG_DIRS
-> %ALLUSERSPROFILE%\yarp\

Mac OsX

System Directories

?

Default values

$YARP_DATA_HOME
-> ?
$YARP_CONFIG_HOME
-> ?
$YARP_DATA_DIRS
-> ?
$YARP_CONFIG_DIRS
-> ?


Files

Configuration Files

Config files are INI files.

a) User $YARP_CONFIG_HOME

       # LINUX     -> $HOME/.config/yarp/
       # WINDOWS   -> %APPDATA%\yarp\config
       # MAC       -> ?

b) Sysadmin $YARP_CONFIG_DIRS + /config/

       # LINUX      -> /etc/xdg/yarp/config/
       # WINDOWS    -> %ALLUSERSPROFILE%\yarp\config\
       # MAC        -> ?

c) Installed $YARP_DATA_DIRS + /config/

       # LINUX      -> /usr/local/share/yarp/config/:/usr/share/yarp/config/
       # WINDOWS    -> %YARP_DIR%\share\yarp\config\
       # MAC        -> ?


A config files are searched in a), b), and c), in this order. All of the files found should be considered, but for a specific setting, only the first value is considered. This allows to have all settings installed in c), the sysadmin can overwrite some of them in b), and the user can tweak the behaviour in a) Command line parameters should always take precedence over config files,

CMake will export a ${YARP_CONFIG_INSTALL_DIR} variable, that should be used in 3rd party software to install config files in the right directory


Data Files

Data files are all the architecture independent files (text files, images, audio, video, etc) that are installed by yarp or by some 3rd party software)

Generic data files are located as follow

a) User $YARP_DATA_HOME

       # LINUX     -> $HOME/.local/share/yarp/
       # WINDOWS   -> %APPDATA%\yarp\
       # MAC       -> ?

b) Sysadmin $YARP_CONFIG_DIRS

       # LINUX     -> /etc/xdg/yarp/
       # WINDOWS   -> %ALLUSERSPROFILE%\yarp\
       # MAC       -> ?

c) Installed $YARP_DATA_DIRS

       # LINUX     -> /usr/local/share/yarp/:/usr/share/yarp/
       # WINDOWS   -> %YARP_DIR%\share\yarp\
       # MAC       -> ?


Depending on the type of data file, the resource finder should be able to have different behaviour (in pseudo-code a possible implementation of the methods) The ResourceManager should have methods for these different behaviour.


  • FIND ALL BY TYPE
    • Find all the resources of a specified type, and show them to the user as different files, perhaps in a treeview (for example template files).

      We should also be able to choose only certain paths (i.e. only "Installed", "Sysadmin", or "User", and any combination).

      If more than one file with the specified name was found we should be able to choose if show the first found, or all of them. If in one level that can have several paths ("Installed", "Sysadmin"), more than one file with the same name was found, and we are searching only for the first, then the one that comes first in the list of paths (the first found from left to right) is returned.

      • list<string> RF::findAll(string resourceType, {RF::Installed|RF::Sysadmin|RF::User}, {RF::First,RF::All});
  • FIND ALL BY TYPE AND NAME
    • Find all the resources of a specified type and name and return all of them.

      We should also be able to choose only certain paths (i.e. only "Installed", "Sysadmin", or "User", and any combination).

      If more than one file with the specified name was found we should be able to choose if show the first found, or all of them. If in one level that can have several paths, more than one file with the same name was found, and we are searching only for the first, then the one that comes first in the list of paths (the first found from left to right) is returned.

      • list<string> RF::findAll(string resourceType, string name, {RF::Installed|RF::Sysadmin|RF::User}, {RF::First,RF::All});
  • FIND ONE BY TYPE AND NAME
    • Find all the resources of a specified type and name and return only the top one (in this order: User, Sysadmin, and finally Installed).

      We should also be able to choose only certain paths (i.e. only "Installed", "Sysadmin", or "User", and any combination)

      If in one level that can have several paths ("Installed", "Sysadmin"), more than one file with the same name was found, the one that comes first in the list of paths (the first found from left to right) is returned;

      • string RF::find(string resourceType, string name, {RF::Installed|RF::Sysadmin|RF::User});
  • READ CONFIG
    • Read one config files and return a Searchable containing the values.

      It should have 2 different behaviours

      1. Read the first found.

        If in one level that can have several paths ("Installed", "Sysadmin"), more than one file with the same name was found, the one that comes first in the list of paths (the first found from left to right) is read;

      2. Merge the configurations files found (priority is given first to the User file, then to the Sysadmin file, and finally to the Installed file)

        If in one level that can have several paths ("Installed", "Sysadmin"), more than one file with the same name was found, all of them are read, and priority is given to the one that comes first in the list of paths (the first found from left to right).<!p>

      • Searchable RF::readConfig(string resourceType, string name, {RF::Installed|RF::Sysadmin|RF::User}, {RF::First,RF::All});


I strongly recommend that we avoid allowing recursive search, or it will make a lot harder everything. It doesn't mean that we don't allow to have
subdirectories, but each subdirectory will contain one resource type. For example the templates folder will have an application directory: application
template types will be "templates/applications" --Daniele.Domenichelli@iit.it 09:58, 28 February 2013 (CET)



CMake will export a ${YARP_DATA_INSTALL_DIR} variable, that should be used in 3rd party software to install config files in the right directory



Plugins Manifest Files

Plugin manifest files are INI files.

a) User $YARP_DATA_HOME + /plugins/

       # LINUX     -> $HOME/.local/share/yarp/plugins/
       # WINDOWS   -> %APPDATA%\yarp\plugins\
       # MAC       -> ?

b) Sysadmin $YARP_CONFIG_DIRS + /plugins/

       # LINUX     -> /etc/xdg/yarp/plugins/
       # WINDOWS   -> %ALLUSERSPROFILE%\yarp\plugins\
       # MAC       -> ?

c) Installed $YARP_DATA_DIRS + /plugins/

       # LINUX     -> /usr/local/share/yarp/plugins/:/usr/share/yarp/plugins/
       # WINDOWS   -> %YARP_DIR%\share\yarp\plugins\
       # MAC       -> ?

Plugins should include the absolute path to the library to open that includes the class implementing the plugin. While this is not a problem for installed plugins, it might be a problem for uninstalled stuff. Therefore we might need something


CMake will export a ${YARP_PLUGIN_MANIFESTS_INSTALL_DIR} variable, that should be used in 3rd party software to install config files in the right directory


Module Files

Module files are XML files. They are usually associated to one executable

a) User $YARP_DATA_HOME + /modules/

       # LINUX     -> $HOME/.local/share/yarp/modules/
       # WINDOWS   -> %APPDATA%\yarp\modules\
       # MAC       -> ?

b) Sysadmin $YARP_CONFIG_DIRS + /modules/

       # LINUX     -> /etc/xdg/yarp/modules/
       # WINDOWS   -> %ALLUSERSPROFILE%\yarp\modules\
       # MAC       -> ?

c) Installed $YARP_DATA_DIRS + /modules/

       # LINUX     -> /usr/local/share/yarp/modules/:/usr/share/yarp/modules/
       # WINDOWS   -> %YARP_DIR%\share\yarp\modules\
       # MAC       -> ?


User data is writable, the other directories no, so when a user creates a new module, opens a template or a module in a read only folder, it will be saved in the User folder


Configuration for modules is set with parameters passed in the application, and with configuration files. Configuration files are handled in the same way, as described before.


CMake will export a ${YARP_MODULES_INSTALL_DIR} variable, that should be used in 3rd party software to install config files in the right directory


Application Files

Application files are directories that contain files of different type

a) User $YARP_DATA_HOME + /app/

       # LINUX     -> $HOME/.local/share/yarp/app/
       # WINDOWS   -> %APPDATA%\yarp\app\
       # MAC       -> ?

b) Sysadmin $YARP_CONFIG_DIRS + /app/

       # LINUX     -> /etc/xdg/yarp/app/
       # WINDOWS   -> %ALLUSERSPROFILE%\yarp\app\
       # MAC       -> ?

c) Installed $YARP_DATA_DIRS + /app/

       # LINUX     -> /usr/local/share/yarp/app/:/usr/share/yarp/app/
       # WINDOWS   -> %YARP_DIR%\share\yarp\app\
       # MAC       -> ?


User data is writable, the other directories no, so when a user creates a new application, opens a template or a application in a read only folder, it will be saved in the User folder


Even though applications are usually shipped as templates, CMake will export a ${YARP_APPLICATIONS_INSTALL_DIR} variable, that should be used in 3rd party software to install config files in the right directory.

Resource Files

Resource files are INI files. They are not installed with yarp or with 3rd party software, but are configured by the user or by the system administrator.

The name "Resource" is in my opinion ambiguous, so I strongly recommend to try to find a better name for these files


a) User $YARP_DATA_HOME + /resources/

       # LINUX     -> $HOME/.local/share/yarp/resources/
       # WINDOWS   -> %APPDATA%\yarp\resources\
       # MAC       -> ?

b) Sysadmin $YARP_CONFIG_DIRS + /resources/

       # LINUX     -> /etc/xdg/yarp/resources/
       # WINDOWS   -> %ALLUSERSPROFILE%\yarp\resources\
       # MAC       -> ?


Installed directories is not needed, because this kind of files is not supposed to be installed by external packages.

For the same reason also CMake variable is not needed.


Template Files

Template files are usually XML files. They are not supposed to be modified by the user, but both the user and the sysadmin can add their own templates.

Template base dir:

a) User $YARP_DATA_HOME + /templates/

       # LINUX     -> $HOME/.local/share/yarp/templates
       # WINDOWS   -> %APPDATA%\yarp\templates
       # MAC       -> ?

b) Sysadmin $YARP_CONFIG_DIRS + /templates/

       # LINUX     -> /etc/xdg/yarp/templates/
       # WINDOWS   -> %ALLUSERSPROFILE%\yarp\templates\
       # MAC       -> ?

c) Installed $YARP_DATA_DIRS + /templates/

       # LINUX     -> /usr/local/share/yarp/templates/:/usr/share/yarp/templates/
       # WINDOWS   -> %YARP_DIR%\share\yarp\templates
       # MAC       -> ?

Each template is installed in a subdirectory (applications, modules, resources, etc.), depending on its type.

Template files with the same name are not exclusive, that means that if an application template file "foo.xml" is found in a), b), and c), the user should see all of them when creating a new application from template


CMake will export the ${YARP_TEMPLATE_INSTALL_DIR} (generic), ${YARP_APPLICATION_TEMPLATES_INSTALL_DIR} and ${YARP_MODULE_TEMPLATES_INSTALL_DIR} (specific for application and modules) variables, that should be used in 3rd party software to install template files in the right directory. Resource templates should be installed only by YARP, therefore there is no need for a CMake variable

Advanced Uses

(Note that in the following examples, the XDG_ variables are used on linux only)


  • User wants to use a separate folder to test something, maintaining multiple configurations of the same application
 # For ALL applications  -> Change $XDG_CONFIG_HOME
 # For YARP only         -> Change $YARP_CONFIG_HOME


  • User wants to avoid using sysadmin settings
 # For ALL applications  -> Change $XDG_CONFIG_DIRS (config files are searched
                            in the yarp subdirectories)
 # For YARP only         -> Change $YARP_CONFIG_DIRS


  • User wants to add settings from a package not installed
 # For ALL applications  -> Append values to $XDG_CONFIG_DIRS (i.e.
                            "/etc/xdg:/new/folder") (config files are searched
                            in the yarp subdirectories)
 # For YARP only         -> Append values to $YARP_CONFIG_DIRS (i.e.
                            "/etc/xdg/yarp:/new/folder/yarp")


  • User wants to add data from a package not installed
 # For ALL applications  -> Append values to $XDG_DATA_DIRS (i.e.
                            "/etc/xdg:/new/folder") (config files are searched
                            in the yarp subdirectories)
 # For YARP only         -> Append values to $YARP_DATA_DIRS (i.e.
                            "/etc/xdg/yarp:/new/folder/yarp")


  • User wants to keep 2 separate installations of yarp
 # For ALL applications  -> Change $XDG_DATA_DIRS (config files are searched in
                            the yarp/config subdirectories)
 # For YARP only         -> Change $YARP_DATA_DIRS (config files are searched
                            in the config subdirectories)

3rd party packages

Installed packages

Using the correct CMake variables when installing files will ensure that those files will be in the correct place on every platform.

Also non-root installation is supported, but both yarp and 3rd party software will require to be configured with the same $PREFIX, and the environment variables will require to be tweaked accordingly.

Uninstalled packages can be appended to the $YARP_DATA_DIRS environment variable

LINUX
User can set the variable in his .bashrc, sysadmin can set the variable system wide by adding a .sh file in /etc/profile.d/
     -> $YARP_DATA_DIRS -> $XDG_DATA_DIRS/yarp/:$ICUB_ROOT
   
WINDOWS
Edit system variables (user and system)
     -> $YARP_DATA_DIRS -> %YARP_DIR%\share\yarp\:%ICUB_DIR%\share\yarp\
MAC
 ?
     -> $YARP_DATA_DIRS -> ?


Packages that want to keep their stuff in their own data directory

If we want to allow packages to install their files in their own directory, we could use the /etc/yarp/path.d directory as we discussed about.

Nonetheless, instead of using the /etc directory, since we are dealing with files installed by another package, I suggest to use /usr/share/yarp/path.d instead.