Plugins manifests and config files
Discussion page to better understand how to organize files like plugins ini files, manifests for modules and applications and configuration files.
The idea is to come up wiith a solution that works for the following situations:
- YARP as independent library
- YARP and iCub coexisting together
In addition we support the usual view in which software can be installed or compiled locally.
Configuration files that need to be found one way or another
There are a few low-level config files used by YARP:
- yarp_namespace.conf -- current yarp namespace
- yarp.conf -- this stores the network location of a name server, and what type of name server it is
- alternate .conf files are used if yarp is in a namespace other than the default one.
These config files predate the existence of ResourceFinder.
YARP plugins that are loaded at runtime rather than compiled in may come with config files, currently named like this:
This is new stuff, unstable.
- yarpmanager.ini for gyarpbuilder
- iCub applications have lots of config files, bootstrapped from an ICUB_ROOT.ini file
- iCub plugins will maybe have config files? (not vital for device plugins; vital for carrier plugins)
- The name server may have a yarpserver.conf file
- yarpview may have one/many .conf files?
- ... what else? ...
Online meeting of 5/02
Discussion focused on plugins and module configuration files since they are those that are extended by users. yarp.conf, yarp_namespace.conf, yarpserver.conf can be handled as they are now with minor modifications.
There seems to be agreement on the following:
- When YARP and other framework are installed they all install plugins in the same place:
- When YARP and other framework are compiled locally plugins are in the binary tree.
YARP will search for plugins config files in a certain order:
- Default directories:
/etc/yarp/plugins ~/.yarp/plugins ...
In addition the env variable YARP_PLUGINS can specify a set of directories that will be searched for plugins. Examples:
Note: search uses the ResourceFinder, but no policy files are required. Requires modifications to ResourceFinder.
This is Lorenzo's proposal. This module needs to be able to find manifest files which I guess will be:
- installed in a single directory for installations
- or kept in $YARP_DIR/modules
Other frameworks like iCub can contribute modules in the first case by installing them in /etc/yarp/modules in the second case by keeping them in $ICUB_ROOT/modules (or $ICUB_DIR/modules).
gyarpbuilder need a way similar to plugins to search for the manifest files in multiple directories. So an option could be to set the YARP_BUILDER env variable and configure gyarpbuilder accordingly (using the RF as for the plugins).
Note by Elena: gyarpbuilder can already search the files in multiple directories, as listed in the ymanager.ini configuration file (which contains other options as well, so it is needed anyway, right Ali?). If the directories in which YARP may install module files are known, they can be added in the default ymanager.ini file we provide with yarp and iCub (will the string "$YARP_DIR/modules" be expanded correctly in Windows? I need to check). We "just" need to define how the ymanager.ini file will be located, so what policy the RF uses. Right now I am using YARP_POLICY=ICUB_ROOT to locate the ymanager.ini file installed in $ICUB_ROOT/app, but of course we can provide one with yarp as well, that can be found with the default policy.
Lorenzo: yes, this works but it forces to have multiple ymanager.ini which is a bit messy. My idea would be to have a single ymanager.ini with general configurations (i.e. editor other things) and let the search path for module be specified in the environment (which will be needed only when modules are not installed). I don't have a strong feeling against the current situation, I'm just trying to see if we can find a better solution.
Elena: I completely agree that multiple .ini files are messy, however they give you the flexibility to change the "context" where you work at runtime (for example, one may decide to work only with a subset of modules instead of the whole, potentially large, collection). In this case, changing the environment variables every time can be time consuming. So, it all depends on how we expect people will use it (or how we want them to). Furthermore, the .ini file also contains the path to applications and resources, not only modules. Maybe we should first discuss some future features we want to have on the builder, like the possibility to organize modules/apps by category, and then decide what we need?
This is Lorenzo's proposal. Instead of YARP_BUILDER and YARP_POLICY we could use a single environment variable e.g. YARP_RESOURCES. Plugins and manifest files could be searched in $YARP_RESOURCES but in default directories (plugins inside plugins, manifests in modules).
Less flexibility but less environment variables.
Do we need YARP_POLICY ?
This variable seems to be ok for redirecting the RF in modules like yarpscope. In this case I don't see a good reason for YARP to have configuration file for it so redirecting could be ok.
The plugin system need to fins *.ini files. These files could be in a single or multiple directories.
When YARP is installed, plugins ini files will be installed in a known directory.
When YARP is compiled locally plugins ini files will be in the binary tree.
Other packages i.e. iCub can contribute other plugins. They may be installed in a known location or compiled locally.
Note: not talking about dll or .so, these follow the rules of the os.
From the current yarp tutorial:
YARP is configured to aggregate plugin.ini files in a directory called etc/yarp/plugins. The ini files are collected in this directory irrespective of whether the corresponding plugins are compiled or not. To add plugins not included with YARP, an ini file may be placed in etc/yarp/plugins.
Question: do you need env variables to get this behavior? I would say no but sentence below seems to contradict.
YARP looks for etc/yarp/plugins using a yarp::os::ResourceFinder. If you don't already use configuration files with YARP, one way to get going would be to create a file in your home directory called .yarp/plugins.ini (or yarp\plugins.ini on Windows) containing:
style capability default_capability /path/to/your/yarp/build
and then set a YARP_POLICY environment variable to plugins (matching the name of the ini file you placed in the .yarp/yarp directory).
- No, env variables not needed, that was just an example --pf
Question: wouldn't it be better if plugins could use a specific policy (i.e. YARP_PLUGINS); in this way there would be no need for setting YARP_POLICY when yarp is installed (assuming this is the case). Other possibilities:
- put plugins ini files in the same directories in which we put configuration files (for YARP and for iCub); each "framework" will have its own set of configuration files and plugins (and maybe manifest), following rules according to their own policies (YARP_POLICY and ICUB_ROOT).
- similar to the above but no need to have separate policy for plugins, if we add to the policy file (i.e. YARP_POLICY.ini/ICUB_ROOT.ini) a specific entry for plugins
Comment from pf: Constraint on solution: YARP will need to be able to find all plugins, since it is responsible for loading them when needed (am I correct in this assumption? --pf). How should YARP find iCub plugins?
Comment from lorenzo: Assumption correct. I thought doing
export YARP_POLICY=ICUB_ROOT yarpdev --device ...
was the idea. Not optimal but sort of ok.
We use the ResourceFinder and a policy file. iCub checks:
- /etc/ICUB_ROOT.ini --> ICUB_ROOT.ini points to /usr/share/iCub or something like that.
- $ICUB_ROOT/ICUB_ROOT.ini --> ICUB_ROOT.ini points to "app" locally in $ICUB_ROOT.
So far this has been OK to support installations 1) and local compilation 2).
In YARP yarpscope and gyarpmanager/gyarpbuilder need also config files. YARP_POLICY is checked. This allows for YARP users to have their own configuration files. I guess as specified in /etc/YARP_POLICY.ini or $YARP_POLICY/YARP_POLICY.ini.
YARP and iCub can coexist. If YARP_POLICY is equal to an existing environment variable, i.e. ICUB_ROOT then iCub and YARP can share the same config files.
gyarpbuilder needs manifest files for modules. At the moment it uses YARP_POLICY to find ymanager.ini; this contains a list of directories to be parsed looking for xml manifest files. Normally iCub users use $ICUB_ROOT/app/yarpmanager.ini.
This works for local compilation. However in the case in which we install YARP and iCub, no environment variables are set and gyarpmanager goes to search /etc/YARP_POLICY.ini.
- Policy files are in /etc; however there was previous suggestion from Daniele that they should go to /etc/yarp or /etc/iCub. This is a good idea especially if we add more files to /etc/yarp.
- It looks like using YARP_POLICY and ICUB_ROOT allows handling almost all situations. This does not seem to work well for gyarpbuilder on the other hand, since this module needs to access multiple directories at the same time. Since this problem seems only relevant for gyarpbuilder we could also find a specific solution.
At the moment gyarpbuilder has it own yarpmanager.ini configuration file. This could go in ~/.yarp, /etc/yarp or be located with YARP_POLICY.
To make iCub and YARP coexist:
- Users modify manually yarpmanager.ini to add $ICUB_ROOT/app
- iCub provides its own yarpmanager.ini; users modify it to add $YARP_ROOT/YARP_DIR/path/to/modules and point YARP_POLICY to ICUB_ROOT.
I prefer 1 which seems less work. The only problem is that fixing yarpmanager.ini automatically at install time is not something easy to do (need to know where either ICUB_ROOT or YARP_ROOT are).