This website uses browsing/session and functional cookies to ensure you get the best experience. Learn More

YARP Cleanup

From Wiki for iCub and Friends
Jump to: navigation, search

This is a list of issues that are required and/or desirable in order to cleanup YARP, and to make it easier to package, and appealing for other projects.


Contents

General

  • Create a public and public archived "yarp-devel" mailing list
    • Mail to the rc-hackers mailing list about the new mailing list

CMake 2.6.0

  • At the moment YARP is not compiling with cmake 2.6.0, but officially we are supporting it. Moreover internally it searches for cmake 2.6.4. My suggestion is to procede in this way:
    • Fix the actual issue with cmake 2.6.0.
    • Release 2.3.20.
    • Write a mail to the mailing list about the new policies for supported debian/ubuntu releases. YARP_Supported_Distributions
    • Add a wiki page for supported versions.
    • Check if the build machines for the new policy are available and eventually add them.
    • Create 2.4 branch and continue 2.3 development in master.
    • Bump cmake version to 2.6.4.
    • Cleanup and remove all workarounds for cmake 2.6.0.
    • Ensure that everywhere 2.6.4 is required in all code.
    • Ensure that everywhere 2.6.4 is required in documentation.
    • Ensure that after those changes, YARP actually builds with cmake 2.6.4 (check the dashboard)
Looks like there was agreement on this, so let's do it (Lorenzo, 14/11)
  • Rename all macros to YARP_XXXX (by keeping compatibility)

FindXXX

  • [Recommended] Fix all the FindXXX.cmake files dropping compatibility, according to CMake_Modules_Guidelines
  • Remove ACE4YARP
  Ok, removed - updated conf/FindACE.cmake --paulfitz
  • [Recommended] Use FeatureSummary and replace the YARP_USE_XXX and YARP_SKIP_ACE with HAVE_XXX
  • [Optional] Investigate if FindGtk shipped with cmake can be used instead of our homemade version, and eventually fix it upstream.
  • [Required] Mail to the mailing lists about those changes.
ok, but checks the guidelines (Lorenzo, 14/11)

Dependency Issues

If you link libYARP_os, you don't need to link directly to ACE, unless you are using it directly in your code. So the aim here is to fix wherever libraries not needed are linked, in order to reduce the dependencies needed for debian packages (in this case you don't need the libace-dev package containing the libACE.so file)

  • Create and export variables YARP_OS_LIBRARY, and use them instead of YARP_LIBRARIES
  • Use LINK_PRIVATE in order not to link unneeded packages.
 This requires CMake 2.8.7, we can fix it later --Daniele.Domenichelli@iit.it 15:43, 22 January 2013 (CET)
  • Ensure that all packages are linked internally using only the required packages and not using YARP_LIBRARIES
 Same --Daniele.Domenichelli@iit.it 15:43, 22 January 2013 (CET)
  • [Required] Mail to the mailing lists about those changes.
ok (Lorenzo 14/11)

Plugin framework

Current plugin framework has several issues with dependencies. The solution would be to have plugins that are loaded dynamically and not linked statically (RUNTIME option is still not a solution because if one .so file is deleted, the library crashes, therefore the dependencies are still the same).

  • [Recommended] Rewrite plugin framework that does not require to link everything from every carrier and device. libYARP_init can be dropped if the plugin framework is implemented properly.
  This was already done, although only tested on Linux, and not tested by users --paulfitz
    For drivers: http://bazaar.launchpad.net/~vcs-imports/yarp/main/revision/4727
  • [Suggested] Remove all the non-free plugin stuff and put it in another repository. (This would be necessary to have yarp in the free section of debian repository, and the nvidia/cuda plugins in the contrib section, otherwise the whole yarp should be in contrib). When plugins will be loaded at runtime, there will be no problem in keeping them separate.
 This would be ok.  There were more non-free plugins, which were kicked out in a previous purge.  Where to put them?  Maybe move to git first? :-) --paulfitz
  • [Required] Mail to the mailing lists about those changes.
 I'm in favor of the new plug-in system (Lorenzo 14/11).
What are the reasons in favor of keeping the old "static" system once the new one is in place? (Lorenzo 14/11).
 * There are a few cases in which static libraries might be better than shared; with static linking you link only the methods that you need, so the final executable should be a little bit faster.
 * It should be just a couple of lines of code, setting the default value (STATIC or SHARED) for add_library
so I think it's safe to leave it --Daniele.Domenichelli@iit.it 09:44, 15 November 2012 (CET)
 ** Why not just stick with CMake's global BUILD_SHARED_LIBS flag, as currently? -- paulfitz
 Reminder: we agreed that we should have just 2 methods.
 * If yarp is built STATIC, the plugins are linked statically into yarp and there is no plugin system.
 * If yarp is built SHARED, all the plugins are built with the RUNTIME option = ON (we should actually get rid of that option)

RobotInterface

When we will have a dynamic plugin framework, there wonì't be any reason to keep the robotInterface in iCub, since the only reason why it is there now is because we need to init the icub drivers library.

  • [Recommended] Move the robotInterface in YARP

Extern libraries

Debian/Ubuntu

There are a few issues with extern libraries in Debian/Ubuntu:

  1. TinyXML is required both by YARP and iCub, and the version installed with squeeze is too old.
  2. OpenCV requires several flags, and not all of them are supported in all debian/ubuntu versions.
  3. GtkDataboxMM is packaged only for Debian Experimental.
 At the moment I don't know how to solve these issues. --Daniele.Domenichelli@iit.it 16:14, 14 November 2012 (CET)

The ideal solution would be not to have them in YARP/iCub and have only the FindXXX.cmake for them. To arrive to this goal,

  • [Recommended] We should have apt repositories with dependencies with the debian/ubuntu versions that we support. In those repositories we should have the required versions of the external libraries.
ok, it looks reasonable for OpenCV. For the other dependencies can we handle them so that we use the system library when available and known to work and the internal one otherwise. It is messier but I would like to reduce the effort/time required.
  • [Recommended] We should work a lot together with packagers for Debian/Ubuntu to get the versions and the build flags that we need in the upstream packages.

Windows

Windows is a problem by itself, but on this issue, it is relatively safe, since we are already shipping binaries for most of the external libraries.

  • [Suggested] We should ship external packages together with GtkMM and the other dependencies.
  • [Suggested] We should investigate if it is possible to use CoApp for building YARP packages and its dependencies (website seems to be dead at the moment)
  I've been listening in on their mailing list.  Making progress, but still far from useable by us today.  Hope for the future... --paulfitz

If one day we manage to do so, we could remove all the extern stuff and keep that in some external repository. But for now I don't know what's the best way to deal with it.

General

  • [Proposal] Move external dependencies in a "robocub-support" repository (to be installed before YARP and ICUB).

Most of this stuff is not required for packaging for linux distro, because it's already there. Moreover Debian policies require to remove all non-free stuff and all binary blobs, therefore the tarball released might have to be repackaged downstream. Handling this in an external repository might make things easier. From this repository we can check if a package is not installed, and eventually build and install a private version of that package.

I would defer this (Lorenzo 14/11)

API/ABI and Versioning

Application Programming Interface

A "stable" API guarantees that code written for a version of Yarp should compile with a newer version. A library that guarantees API stability is usually preferable when writing some software, because this implies that you won't have to update your software when the library is updated and/or you won't have to keep different branches working with different versions of the library.

Application Binary Interface

A "stable" ABI guarantees that code compiled for a version of Yarp should run with a newer version of the library installed. The advantage is that when packaging newer version of the library, if there are programs built using the previous version, they don't need to be recompiled.

SONAME

From Debian Policy Manual

Every time the shared library ABI changes in a way that may break binaries linked against older versions of the shared library, the SONAME of the library and the corresponding name for the binary package containing the runtime shared library should change. Normally, this means the SONAME should change any time an interface is removed from the shared library or the signature of an interface (the number of parameters or the types of parameters that it takes, for example) is changed. This practice is vital to allowing clean upgrades from older versions of the package and clean transitions between the old ABI and new ABI without having to upgrade every affected package simultaneously.
The SONAME and binary package name need not, and indeed normally should not, change if new interfaces are added but none are removed or changed, since this will not break binaries linked against the old shared library.

To check the SONAME you can run objdump -p /opt/iit/lib/libYARP_OS.so | grep SONAME

References

YARP Versioning

Running API/ABI checkers tools on YARP, unfortunatly the results (for the 2.3 series) are not positive, there are several API and ABI breaks in almost every patch version.

    • Yep. There is SOVERSION code lying around in Cmake only due to efforts of Anne van Rossum, who took a stab at one point on Debian packaging. Maybe once there is a yarp-devel mailing list, the first thing to do is to raise awareness of this, and then get serious about it. I personally never bothered maintaining a soversion since there are too many committers modifying header files who don't know about this stuff, and I didn't want to take on the role of nagging people without a clear benefit to show them. --paulfitz

Currently in YARP we these version numbers + some commented code to handle svn version:

set(YARP_VERSION_MAJOR "2")
set(YARP_VERSION_MINOR "3")
set(YARP_VERSION_PATCH "19")
set(YARP_VERSION_MODIFIER "")
set(YARP_VERSION_TWEAK "9424")
#"${YARP_WC_REVISION}")
set(YARP_VERSION_ABI "1")

set(CPACK_PACKAGE_VERSION_MAJOR "${YARP_VERSION_MAJOR}")
set(CPACK_PACKAGE_VERSION_MINOR "${YARP_VERSION_MINOR}")
set(CPACK_PACKAGE_VERSION_PATCH "${YARP_VERSION_PATCH}${YARP_VERSION_MODIFIER}")

set(YARP_GENERIC_VERSION "${YARP_VERSION_MAJOR}.${YARP_VERSION_MINOR}.${YARP_VERSION_PATCH}")
set(YARP_GENERIC_SOVERSION "${YARP_VERSION_ABI}")

Current version is 2.3.19, and all the libraries always use SOVERSION 1, even if the ABI was broken with every new release

 I'm not sure if YARP_VERSION_MODIFIER is used. --Daniele.Domenichelli@iit.it 16:14, 14 November 2012 (CET)

Versioning should reflect the changes in the code I suggest to adopt a strict policy in numbering, and to use:

  • YARP_VERSION_MAJOR
    Fixed (2) until there is some very big change
    YARP_VERSION_MINOR
    Increase if API is changed
    YARP_VERSION_PATCH
    Increase if API is not changed, reset to 0 when YARP_VERSION_MINOR is increased.
    SONUMBER
    rename the libraries to libYARP2_xxx.so and restart counting from 0 and increase it properly. This number should not be "global", but depending on the library.

Ideally, when increasing YARP_VERSION_MINOR number, the API should be stable (no methods removed, but marked as deprecated instead) and all those deprecated methods should be removed when increasing the YARP_VERSION_MAJOR number

Let's try this (Lorenzo 14/11)
To do this, all YARP committers need to be on board (Ugo, Marco, Ali, etc) and know the consequences of touching interfaces --paulfitz

When doing a minor release, a branch should be started as well, and bugfix should be applied both to master and stable branches.

A policy often adopted is to use the MINOR number to distinguish between stable and unstable releases. Another policy adopted is to set in the master branch the MINOR number very big (i.e. 60) and increase it when there is an incompatible change, or a new feature (i.e. the role of YARP_VERSION_TWEAK)

TODO

  • [Suggested] Constify methods and pass const references whenever possible before the 2.4.0 release. For example
    • virtual bool open(Searchable& config) should probably be virtual bool open(const Searchable& config)
    • virtual Bottle& findGroup(const char *key) should probably be virtual const Bottle& findGroup(const char *key) const
  • [Suggested] Starting with the 2.4 series we should run API and ABI checker tools before every release to ensure that API and ABI are stable during the release cycle.
    • Methods should be marked as deprecated instead of modifying/removing them, and eventually removed in a major release, after a reasonable amount of time (Example).
 Yay!  Let's do this. --paulfitz
  • [Suggested] Do not install private headers (impl)
    • People should not rely on those headers that should contain pure implementative details.
    • If someone requires parts of them, we should consider moving that part to the public headers.
  • [Check] -fvisibility=hidden and -fvisibility-inlines-hidden should be used by default and only the public methods should be exported.
 Checked, the YARP_CLEAN_API does -fvisibility=hidden but there are some issues with tests (that require the impl stuff) --Daniele.Domenichelli@iit.it 16:48, 4 January 2013 (CET)
  • [Check] Exported classes should not contain private members and methods, since those are implementation details (see 1 and 2).
  • [Check] inline methods are dangerous for ABI stability, check if we have and eventually implement them as normal method.
  • [Recommended] Adopt a strict versioning policy, including branching stable releases when a doing a minor release
 I'm not sure if this is how it is supposed to be handled with SVN, but this is how it is usually handled using GIT --Daniele.Domenichelli@iit.it 16:37, 14 November 2012 (CET)

Other Optional Cleanup

  • [Recommended] Remove debian-related CPack stuff.
  Why? --paulfitz
  Is it used? when we will have a proper debian package I think that it will be useless --Daniele.Domenichelli@iit.it 10:20, 19 November 2012 (CET)
  Ah, you mean debian-related CPack stuff, yes that's fine to remove if it is still lying around.  CPack is in use for generating basic zips/tarballs --paulfitz
  Yes, sorry that stuff :) Fixing. --Daniele.Domenichelli@iit.it 16:22, 22 November 2012 (CET)
  • [Recommended] Add versioning to online documentation.
  • [If we have time] Write coding style policy and use them through all the code (replace all tabs with spaces and remove trailing spaces, indenting, brackets position, etc.)
  • [Is it possible?] Add modelines that works on emacs, vi and windows. (extabsettings addon for windows, Editor Modeline Generator)
  • [If we have time] Headers should should not require other headers before in order to compile
  • [If we have time] Headers should contain the minimum amount of other headers, using forward declarations where possible.
  • [If we have time] Remove packaging stuff from the main repository.
  • [If we have time] Not all the classes/features in YARP are covered by unit tests, we should add them and use coverage tools.
  • Enable -Wextra and more errors by default, to help finding possible errors, and add an option for -Werror
  • Try to build yarp using clang.
  • Migrate YARP to git.
  • Move policies from /etc to /etc/yarp/policies
  • Do not use environment variables in CMake, but use cmake options instead
  • [Recommended] Require a single version for Gtk/GtkMM
  • [Recommended] Devices should not need to be declared in yarp::dev namespace
  • [Recommended] Fix tests so that they can be built with CLEAN_API
  • [Recommended] Check for compiler instead of UNIX (or WINDOWS) when setting COMPILE_FLAGS
  • [Recommended] Use POSITION_INDEPENDENT_CODE property from CMake 2.8.9 if available instead of -fPIC

TODO iCub

  • Ensure that everywhere 2.6.4 is required in iCub.
  • Update to the new CMake Macros.
  • Update to the new const methods.
  • Use YARP_OS_LIBRARY, etc. variables instead of YARP_OS
Personal tools
Namespaces

Variants
Actions
Navigation
Print/export
Toolbox