Plugins manifests and config files

From Wiki for iCub and Friends
Jump to: navigation, search

Discussion page to better understand how to organize files like plugins ini files, manifests for modules and applications and configuration files.

The idea is to come up wiith a solution that works for the following situations:

  1. YARP as independent library
  2. YARP and iCub coexisting together

In addition we support the usual view in which software can be installed or compiled locally.

Configuration files that need to be found one way or another

There are a few low-level config files used by YARP:

  • yarp_namespace.conf -- current yarp namespace
  • yarp.conf -- this stores the network location of a name server, and what type of name server it is
    • alternate .conf files are used if yarp is in a namespace other than the default one.

These config files predate the existence of ResourceFinder.

YARP plugins that are loaded at runtime rather than compiled in may come with config files, currently named like this:

  • etc/yarp/plugins/FOO.ini

This is new stuff, unstable.

  • yarpmanager.ini for gyarpbuilder
  • iCub applications have lots of config files, bootstrapped from an ICUB_ROOT.ini file
  • iCub plugins will maybe have config files? (not vital for device plugins; vital for carrier plugins)
  • The name server may have a yarpserver.conf file
  • yarpview may have one/many .conf files?
  • ... what else? ...

Possible layout

(work in progress)

For plugins, app modules and resources files

 $prefix
    /share
      /yarp
        /plugins (?)
        /app
        /modules
        /resources
          
 /etc/yarp/paths.d 

a list of paths that contains a structure similar to the one in $prefix/share/yarp

In addition these directories take precedence:

$XDG_CONFIG_HOME/yarp
$XDG_CONFIG_HOME/yarp/paths.d

See below (order of precedence in RF specs).

These defaults directories are all overridden if the YARP_DATA_PATH (or YARP_SOMETHING, name TBD) is set; it should be a list of paths that contain the same directory structure as above.

For yarp and gyarpmanager gyarpbuidler ini files

ymanager.ini can be searched in:

1 $YARP_CONF
2 ~/.yarp
  or $XDG_CONFIG_HOME/yarp
3 /etc/yarp

Files like yarp.conf yarpserver.conf yarp_namespace.conf are generated automatically and we don't have good defaults to install in system directories so they should stay in the user home:

1 $YARP_CONF
2 ~/.yarp
  or $XDG_CONFIG_HOME/yarp

How the RF should change

The RF at the moment works by searching a list of locations and return the first match of the file or directory requested by the user.

In the current proposal this should be extended so that when looking for directories the RF returns a list of locations. This is useful to aggregate plugins configuration files, templates, and modules from different sources. For files the current mechanism seems to be appropriate.


The ResourceFinder should be modified to return all paths, in order of precedence, that contain a required folder, i.e., "app". For example, in src/ymanager/gybuilder/src/gybuilder.cpp, lines 100 and following could become:

   if(!config.check("apppath"))
   {
      yarp::os::Bottle appPaths= rf.findPaths("app");
      config.put("apppath", appPaths.toString()); //maybe toString is not consistent with expected apppath though
   }
       //config.put("apppath", "./"); 


Although yarpmanager and yarpbuilder do not need to have explicitly specified whether the folders contain modules or applications or resources, enforcing this directory structure helps in file localization (both for RF and users...).

More detailed info on the Rf:

YARP_ResourceFinder


Order of precedence

Please comment. In particular:

  • Should it be ~/yarp or ~/.yarp or ~/.config/yarp ?
    • ~/.config/yarp [PF]
  • Equivalently, should it be ${XDG_CONFIG_HOME}/.yarp o /yarp?
      • ${XDG_CONFIG_HOME}/yarp [PF]
  • Should we pick Option 1) or Option 2) below?
  • Should we give a specific name to this policy?
 1 if (exists YARP_SOMETHING) goto 2
   else goto 3
 2 Search for all entries in $YARP_SOMETHING (this can be a list)
   2.1 append all matched to return value
   2.2 Option 1) return list of paths and stops OR
       Option 2) search continues, goto 3
 3 if XDG_CONFIG_HOME exists, scan ${XDG_CONFIG_HOME}/yarp
   else scan ~/yarp/
 4 Scan /usr/share/yarp
 5 Scan /etc/yarp/path.d/ for additional directories
   5.1 Scan all the additional directories

How the RF has changed

  • ResourceFinder::getPaths(...) has been added. It returns a list of *all* instances of the requested file (or directory) that were found in ResourceFinder's search path.
 yarp::os::Bottle bot = finder.findPaths("app");
 printf("Paths: %s\n", bot.toString().c_str());
 --> Paths: app "/usr/share/yarp/app" "/usr/share/iCub/app"
  • ResourceFinder now checks /etc/yarp/yarp.d/*.ini for paths to search. The ini files should look like this:
 yarp.ini:
   [search yarp]
   path /usr/share/yarp
 icub.ini:
   [search icub]
   path /usr/share/iCub
 funky.ini:
   [search funky]
   path /usr/share/funky /opt/funky /tmp/funky /var/spool/funky 
  • The yarp.ini file is supposed to go away / not be needed. That's not done yet.
  • Nothing done for windows operations yet.
  • Path priority issues not looked at.

Writable files

There are some files that the user should be able to modify. Examples are config files for modules and application files.

The idea is that installers provide these files as read-only in system directories and they are then modified in a user's private workspace which should be somewhere in the home of the user, e.g. XDG_DATA_HOME/app.

This happens in two ways:

  1. After installation a script moves configuration files from the origin (e.g. /usr/share/yarp) to the user's workspace.
  2. The user through gyarpbuilder modifies a template or builds it own application and saves it to the workspace.

These files should be then picked up by gyarpbuilder and gyarpmanager. So maybe XDG_DATA_HOME/app should end up in the RF search order?

For iCub modules the current ICUB_ROOT policy can be configured to point to XDG_DATA_HOME from which configuration files will be read at runtime.

Old text

Online meeting of 5/02

Discussion focused on plugins and module configuration files since they are those that are extended by users. yarp.conf, yarp_namespace.conf, yarpserver.conf can be handled as they are now with minor modifications.

Plugins

There seems to be agreement on the following:

  • When YARP and other framework are installed they all install plugins in the same place:
 /etc/yarp/plugins
   I would avoid /etc for plugins manifest files, their place is /usr/share/yarp, /etc is for system config files editable by the administrator
   (manifest files are not supposed to be edited) --Daniele.Domenichelli@iit.it 15:09, 6 February 2013 (CET)
  • When YARP and other framework are compiled locally plugins are in the binary tree.

YARP will search for plugins config files in a certain order:

  • Default directories:
 /etc/yarp/plugins
 ~/.yarp/plugins
 ...
   Same here, my suggestion is:
   * ${INSTALL_PREFIX}/share/yarp/plugins
   * ${XDG_CONFIG_HOME}/plugins
   --Daniele.Domenichelli@iit.it 15:09, 6 February 2013 (CET)

In addition the env variable YARP_PLUGINS can specify a set of directories that will be searched for plugins. Examples:

 YARP_PLUGINS=$YARP_DIR/plugins:$ICUB_ROOT/plugins
   Just to clarify, the manifest file should contain the absolute path to the library that contains the plugin,
   these paths are where the manifest files are located --Daniele.Domenichelli@iit.it 15:09, 6 February 2013 (CET)

Note: search uses the ResourceFinder, but no policy files are required. Requires modifications to ResourceFinder.

   For debugging purposes and to help when building software depending on yarp we should also have an executable
       yarp-config (or a "yarp config" argument)
   that prints the list of directories searched, and other useful stuff
   
   For example kde has kde4-config
   
       $ kde4-config --help
       Usage: kde4-config [Qt-options] [KDE-options] [options] 
       
       A little program to output installation paths
       
       Generic options:
         --help                    Show help about options
         --help-qt                 Show Qt specific options
         --help-kde                Show KDE specific options
         --help-all                Show all options
         --author                  Show author information
         -v, --version             Show version information
         --license                 Show licence information
         --                        End of options
       
       Options:
         --expandvars              Left for legacy support
         --prefix                  Compiled in prefix for KDE libraries
         --exec-prefix             Compiled in exec_prefix for KDE libraries
         --libsuffix               Compiled in library path suffix
         --localprefix             Prefix in $HOME used to write files
         --kde-version             Compiled in version string for KDE libraries
         --types                   Available KDE resource types
         --path type               Search path for resource type
         --locate filename         Find filename inside the resource type given to --path
         --userpath type           User path: desktop|autostart|document
         --install type            Prefix to install resource files to
         --qt-prefix               Installation prefix for Qt
         --qt-binaries             Location of installed Qt binaries
         --qt-libraries            Location of installed Qt libraries
         --qt-plugins              Location of installed Qt plugins
       
       $ kde4-config --version
       Qt: 4.8.2
       KDE Development Platform: 4.8.4 (4.8.4)
       kde4-config: 1.0
   Useful for example to get information about version used by people reporting bugs
   
       $ kde4-config --prefix
       /usr
   Useful if kde is not installed in $prefix so that you can run "cmake .. -DCMAKE_INSTALL_PREFIX=`kde4-config --prefix`" in order to install plugins in the right directory
   
       $ kde4-config --localprefix
       /home/drdanz/.kde/
   Useful to locate the user directory
   
       $ kde4-config --types
       apps - Applications menu (.desktop files)
       cache - Cached information (e.g. favicons, web-pages)
       cgi - CGIs to run from kdehelp
       config - Configuration files
       data - Where applications store data
       emoticons - Emoticons
       exe - Executables in $prefix/bin
       html - HTML documentation
       icon - Icons
       include - Includes/Headers
       kcfg - Configuration description files
       kdeinit - kdeinit shared executables (resource added by Debian)
       lib - Libraries
       locale - Translation files for KLocale
       mime - Mime types
       module - Loadable modules
       pixmap - Legacy pixmaps
       qtplugins - Qt plugins
       services - Services
       servicetypes - Service types
       socket - UNIX Sockets (specific for both current host and current user)
       sound - Application sounds
       templates - Templates
       tmp - Temporary files (specific for both current host and current user)
       wallpaper - Wallpapers
       xdgconf-autostart - XDG autostart directory
       xdgconf-menu - XDG Menu layout (.menu files)
       xdgdata-apps - XDG Application menu (.desktop files)
       xdgdata-dirs - XDG Menu descriptions (.directory files)
       xdgdata-icon - XDG Icons
       xdgdata-mime - XDG Mime Types
       xdgdata-pixmap - Legacy pixmaps
   List all the resource types
   
       $ kde4-config --path config
       /home/drdanz/.kde/share/config/:/etc/kde4/:/usr/share/kde4/config/:/usr/share/desktop-base/profiles/kde-profile/share/config/
       $ kde4-config --install config
       /usr/share/kde4/config/
   Equivalent to our configuration files
   
       $ kde4-config --path services
       /home/drdanz/.kde/share/kde4/services/:/usr/share/kde4/services/
       $ kde4-config --install services
       /usr/share/kde4/services/
   Equivalent to our plugins manifests
   
       $ kde4-config --path servicetypes
       /home/drdanz/.kde/share/kde4/servicetypes/:/usr/share/kde4/servicetypes/
       $ kde4-config --install servicetypes
       /usr/share/kde4/servicetypes/
   Descriptions of the types of plugins to make it extendable (in our case we could have this for devices and for resource types)
   
       $ kde4-config --path servicetypes --locate kfileplugin.desktop
       /usr/share/kde4/servicetypes/kfileplugin.desktop
   Useful to locate a file, or if you have several version of a file with the same name in different locations, to know which one is currently used
   
   --Daniele.Domenichelli@iit.it 15:09, 6 February 2013 (CET)

gyarpbuilder

This is Lorenzo's proposal. This module needs to be able to find manifest files which I guess will be:

  • installed in a single directory for installations
/etc/yarp/modules
   Also for those files that are not supposed to be modified by the administrator, the path should be ${PREFIX}/share/yarp/modules
   --Daniele.Domenichelli@iit.it 15:24, 6 February 2013 (CET)
  • or kept in $YARP_DIR/modules

Other frameworks like iCub can contribute modules in the first case by installing them in /etc/yarp/modules in the second case by keeping them in $ICUB_ROOT/modules (or $ICUB_DIR/modules).

gyarpbuilder need a way similar to plugins to search for the manifest files in multiple directories. So an option could be to set the YARP_BUILDER env variable and configure gyarpbuilder accordingly (using the RF as for the plugins).

Note by Elena: gyarpbuilder can already search the files in multiple directories, as listed in the ymanager.ini configuration file (which contains other options as well, so it is needed anyway, right Ali?). If the directories in which YARP may install module files are known, they can be added in the default ymanager.ini file we provide with yarp and iCub (will the string "$YARP_DIR/modules" be expanded correctly in Windows? I need to check). We "just" need to define how the ymanager.ini file will be located, so what policy the RF uses. Right now I am using YARP_POLICY=ICUB_ROOT to locate the ymanager.ini file installed in $ICUB_ROOT/app, but of course we can provide one with yarp as well, that can be found with the default policy.

Lorenzo: yes, this works but it forces to have multiple ymanager.ini which is a bit messy. My idea would be to have a single ymanager.ini with general configurations (i.e. editor other things) and let the search path for module be specified in the environment (which will be needed only when modules are not installed). I don't have a strong feeling against the current situation, I'm just trying to see if we can find a better solution.

Elena: I completely agree that multiple .ini files are messy, however they give you the flexibility to change the "context" where you work at runtime (for example, one may decide to work only with a subset of modules instead of the whole, potentially large, collection). In this case, changing the environment variables every time can be time consuming. So, it all depends on how we expect people will use it (or how we want them to). Furthermore, the .ini file also contains the path to applications and resources, not only modules. Maybe we should first discuss some future features we want to have on the builder, like the possibility to organize modules/apps by category, and then decide what we need?

Alternative solution

This is Lorenzo's proposal. Instead of YARP_BUILDER and YARP_POLICY we could use a single environment variable e.g. YARP_RESOURCES. Plugins and manifest files could be searched in $YARP_RESOURCES but in default directories (plugins inside plugins, manifests in modules).

Less flexibility but less environment variables.

Do we need YARP_POLICY ?

This variable seems to be ok for redirecting the RF in modules like yarpscope. In this case I don't see a good reason for YARP to have configuration file for it so redirecting could be ok.

Plugins

The plugin system need to fins *.ini files. These files could be in a single or multiple directories.

When YARP is installed, plugins ini files will be installed in a known directory.

When YARP is compiled locally plugins ini files will be in the binary tree.

Other packages i.e. iCub can contribute other plugins. They may be installed in a known location or compiled locally.

Note: not talking about dll or .so, these follow the rules of the os.

From the current yarp tutorial:

YARP is configured to aggregate plugin.ini files in a directory 
called etc/yarp/plugins. The ini   files are collected in this 
directory irrespective of whether the corresponding plugins are 
compiled or not. To add plugins not included with YARP, an ini file 
may be placed in etc/yarp/plugins.

Question: do you need env variables to get this behavior? I would say no but sentence below seems to contradict.

YARP looks for etc/yarp/plugins using a yarp::os::ResourceFinder. 
If you don't already use configuration files with YARP, one way 
to get going would be to create a file in your home directory 
called .yarp/plugins.ini (or yarp\plugins.ini on Windows) containing:
style capability
default_capability /path/to/your/yarp/build

and then set a YARP_POLICY environment variable to plugins (matching the name of the ini file you placed in the .yarp/yarp directory).

  • No, env variables not needed, that was just an example --pf

Question: wouldn't it be better if plugins could use a specific policy (i.e. YARP_PLUGINS); in this way there would be no need for setting YARP_POLICY when yarp is installed (assuming this is the case). Other possibilities:

  • put plugins ini files in the same directories in which we put configuration files (for YARP and for iCub); each "framework" will have its own set of configuration files and plugins (and maybe manifest), following rules according to their own policies (YARP_POLICY and ICUB_ROOT).
  • similar to the above but no need to have separate policy for plugins, if we add to the policy file (i.e. YARP_POLICY.ini/ICUB_ROOT.ini) a specific entry for plugins

Comment from pf: Constraint on solution: YARP will need to be able to find all plugins, since it is responsible for loading them when needed (am I correct in this assumption? --pf). How should YARP find iCub plugins?

Comment from lorenzo: Assumption correct. I thought doing

 export YARP_POLICY=ICUB_ROOT yarpdev --device ...

was the idea. Not optimal but sort of ok.

Configuration files

We use the ResourceFinder and a policy file. iCub checks:

  1. /etc/ICUB_ROOT.ini --> ICUB_ROOT.ini points to /usr/share/iCub or something like that.
  2. $ICUB_ROOT/ICUB_ROOT.ini --> ICUB_ROOT.ini points to "app" locally in $ICUB_ROOT.

So far this has been OK to support installations 1) and local compilation 2).

In YARP yarpscope and gyarpmanager/gyarpbuilder need also config files. YARP_POLICY is checked. This allows for YARP users to have their own configuration files. I guess as specified in /etc/YARP_POLICY.ini or $YARP_POLICY/YARP_POLICY.ini.

YARP and iCub can coexist. If YARP_POLICY is equal to an existing environment variable, i.e. ICUB_ROOT then iCub and YARP can share the same config files.

Manifest files

gyarpbuilder needs manifest files for modules. At the moment it uses YARP_POLICY to find ymanager.ini; this contains a list of directories to be parsed looking for xml manifest files. Normally iCub users use $ICUB_ROOT/app/yarpmanager.ini.

This works for local compilation. However in the case in which we install YARP and iCub, no environment variables are set and gyarpmanager goes to search /etc/YARP_POLICY.ini.

General considerations

  1. Policy files are in /etc; however there was previous suggestion from Daniele that they should go to /etc/yarp or /etc/iCub. This is a good idea especially if we add more files to /etc/yarp.
  2. It looks like using YARP_POLICY and ICUB_ROOT allows handling almost all situations. This does not seem to work well for gyarpbuilder on the other hand, since this module needs to access multiple directories at the same time. Since this problem seems only relevant for gyarpbuilder we could also find a specific solution.

At the moment gyarpbuilder has it own yarpmanager.ini configuration file. This could go in ~/.yarp, /etc/yarp or be located with YARP_POLICY.

To make iCub and YARP coexist:

  1. Users modify manually yarpmanager.ini to add $ICUB_ROOT/app
  2. iCub provides its own yarpmanager.ini; users modify it to add $YARP_ROOT/YARP_DIR/path/to/modules and point YARP_POLICY to ICUB_ROOT.

I prefer 1 which seems less work. The only problem is that fixing yarpmanager.ini automatically at install time is not something easy to do (need to know where either ICUB_ROOT or YARP_ROOT are).