Better Repository old

From Wiki for iCub and Friends
Jump to navigation Jump to search

Author: Lorenzo Natale

For historical reasons the iCub repository and the build system have been designed in a way that make it quite difficult to provide the functionalities of a mature package.

In parallel cmake has evolved quite a lot since version 2.4 and offers much more robust and useful features than the ones we relayed upon when we started the project.

Repository organization

The root of the repository contains various directories.

  • admin: contains scripts for maintainance, of no interest for most developers
  • tutorials: some tutorials on the software
  • pc104: pc104 startup scripts
  • license: keep information about licensing
  • app: stores applications scripts and configuration files

Code is now contained in two different directories main and contrib.

  • contrib stores modules that can be compiled individually. This is a good place to upload new modules that are still under test. No automatic tests are executed on these directories.
  • main this is the directory that contains the main build. This structure contains code that respects iCub standards (dependencies, supported operating systems and compilers, etc.). To avoid causing troubles to other users place here only modules you are reasonably confident will compile correctly. To help you check this automatic tests run periodically on this build.
  • main/conf contains cmake helper code, including FIND* scripts
  • main/src contains source code. This is further organized in examples, libraries, simulators and tools with obvious meaning. In addition the modules directory contains iCub modules (notice that this is different from the old modules directory which is now called libraries/icubmod). The tools directory contains programs of general use including graphic interfaces. Finally inside core we place modules that are required to operate the robot (like iCubInterface).

The philosophy

We still would like to keep at the minimum the amount of work required to maintain the repository. Users should be allowed to add new modules to the repository or the main build without much effort. However we also would like to give the possibility to use the software from other packages or libraries, in particular we would like this to be possible in two configurations:

  • out-of-source or in-source builds
  • from installation

We do a first distinction between modules that are "visible" only within the iCub repository and modules that are "exported", or in other words, can be used externally with respect to the big iCub build. Another distinction is between executables and libraries. Executables are easy to export: they do not provide header files or other resources. It is enough to put them in the system path. This can be done at install time, or by appending the binary directory where executables are built to the system path (something like ICUB_DIR/bin). Libraries are a bit more complicated because they define symbols, have header files and dependencies.

It is also important to stress that the iCub build is not just a normal package with a set of libraries and executables. We like to see the build more like a collection of modules, made of executables and libraries. This is how external packages view the iCub software.

Executables

Adding an executable to the build is simple, it does not take more than a few lines of cmake code:

 project(mymodule)
 set(folder_source main.cpp myclass.cpp)
 set(folder_header myclass.h)
 source_group("Source Files" FILES ${folder_source})
 source_group("Header Files" FILES ${folder_header})
 add_executable(mymodule ${folder_source} ${folder_header})

To cope with intallation this is enough:

 install(TARGETS mymodule DESTINATION bin)

Using YARP or other packages

If you are using YARP or other packages you do not have to call the macro find_pacakge(). This was done already for you at the beginning of the build.

You have, however, to add the include directory and link the library. For YARP before calling add_executable:

 include_directories(${YARP_INCLUDE_DIRS})

and afterwards:

 target_link_libraries(mymodule ${YARP_LIBRARIES})

Important with respect to the previous repository this has changed, in the old style build system calling find_package(YARP) was enough.

Libraries

You should be aware that a library has a public and a private interface. The public interface is a set of header files that users of the library will be offered to include; the private interface is a set of header files that are included only by code inside the library. This distinction is important. Header files that are the public interface of the library should be clean; they should define only the minimum set of symbols required by the user, and avoid polluting the main scope with unnecessary symbols (ideally all symbols should be hidden in a separate namespace, but for now let's not be too picky about this). Private header files are only required when you compile the library and, in principle, could be destroyed afterwards (ok don't do that).

It is a good idea to maintain the distinction between public and private interface at the file system level (public header files in one directory, private header files in another one).

This is because at install time only public header files are copied to the target destination (maybe with the exception of inline files/templates which should also be visible).

Files structure

Everyone is free to organize files in any way they like, clearly there is not a single solution that works.

How you organize header files is the combined result of how you include them in your code and how you configure the compiler.

The only restriction in this case is that public files are shared between external code and the code of the library, so, whatever you do, make sure you include them in the same way. This is the point where you need to be careful and follow normal practices.

When external users include a public file of a library they expect to do it like:

 #include <library-name/header-file.h>

For many reasons this is preferable than just:

 #include <header-file.h>

In our case this becomes:

 #include <iCub/header-file.h>

Even better, in case you define a namespace:

 #include <iCub/namespace/header-file.h>

Usually you instruct your compiler to know where to start looking for header files, so you do not have to wonder how the compiler will be able to fine the iCub directory. However, this means that the local directory structure of the files in your library should match the one above.

In short, all public header files should be placed in a directory called iCub. Since you want to be clean, this is also inside a directory called include (to keep it separate from the src directory where you put source files).

If you define a namespace, make a new directory there (in include/iCub).

Now private header files can go in a separate directory of your choice, e.g. include/private and included as:

#include "include/private/local.h"

We really do not care because it will be your job, as a maintainer of the library, to make sure everything compiles.

Using a library

We make a difference between using the library within the repository and importing it as an external package.

Both cases involve (obviously) linking against the library, and, sometimes, adding the appropriate path to header files.

Within the repository

CMake knows what the targets in the build are. Linking a library is as easy as linking against the corresponding target/project.

If lib1 is a library compiled in the main build, then it is enough to do something like:

 include_directories(${lib1_INCLUDE_DIRS})
 add_executable(example main.cpp)
 target_link_libraries(example lib1)

where example is the new project/executable and lib1 is the name of the project that compiles lib1 (often the two are coincident).

Adding the path to header files is a bit more complicated. Since you are within the iCub main build you can rely on a set of cmake variables that are already been filled for you. We enforce the standard by which each library should define the variable libname_INCLUDE_DIRS.

This involves obviously linking against the library, and, in some cases, including the appropriate header files.

We view the iCub repository as a collection of libraries, so we give the possibility to link against each individual library. This might turn our useful in the future to avoid conflicts. As far as header files are concerned it is enough to instruct the compiler to point to the root installation or build directory.

Externally to the repository

In cmake:

 find_package(ICUB)
 
 include_directories(${ICUB_INCLUDE_DIRS})
 add_executable(example main.cpp)
 target_link_libraries(example lib1)

This creates a project called example and links the library lib1 -- supposedly from the iCub package. CMake gives the possiblity to append a namespace to libraries to avoid conflicts it is an option we have to consider beforehand.

We organized the iCub software so that the code above can work in cases the iCub software is installed or exported from the build.

As anticipated above, path to header files, on the other hand, follows this convention:

All libraries header files are located in a single root directory. In case you export the build tree this is $ICUB_DIR/include, otherwise it is where you installed the library (this can be one of the system directories like C:/Program Files/iCub/include or /usr/lib/include).

Inside the root include directory each namespace or library will create its own directory.

For example:

 $ICUB_DIR/include/iCub/iKin
 $ICUB_DIR/include/iCub/contrib
 ...

Since header files are included with the prefix iCub/namespace (e.g. #include<iCub/iKin/iKin.h>) it is enough to instruct the compiler to know the root of the header files (i.e. where the "/include/iCub" directory is) to work. This is taken care of, by the ICUB_INCLUDE_DIRS cmake variable.

Writing a library

We distinguish two types of libraries.

  1. libraries that are used locally by a few modules, we call these internal libraries
  2. libraries that are exported to be used by users of the iCub package, we call these exported libraries

Internal libraries

Nothing special has to be done in this case. You add the library to the build as you normally would in cmake.

For example:

 project(mylib)
 file(GLOB folder_source *.cpp)
 file(GLOB folder_public_headers include/iCub/*.h)
 file(GLOB folder_private_headers include/private/*.h)
 set(folder_header ${folder_public_headers} ${folder_private_headers})
 # Declare groups of source and header files -- makes things pretty in MSVC.
 source_group("Source Files" FILES ${folder_source})
 source_group("Header Files" FILES ${folder_header})
 # need to define a cache variable to keep track of include directories
 set(mylib_INCLUDE_DIRS ${PROJECT_SOURCE_DIR}/include/iCub CACHE STRING "mylib include directory")  
 include_directories(${PROJECT_SOURCE_DIR}/include/private)
 include_directories(${PROJECT_SOURCE_DIR}/include/iCub)
 add_library(mylib ${folder_source} ${folder_header})

You can now use this library in other projects within the repository. As pointed out above this can be done simply by:

 project(mymodule)
 include_directories(${mylib_INCLUDE_DIRS})
 add_executable(mymodule main.cpp)
 target_link_libraries(mymodule lib1)

External libraries

Things here become a bit more complicated because you want to export the library so that it can be used from external packages/users. Of course this is normally trivial, what is interesting is to have cmake do that.

The cmake code we have written above is still valid. In addition you have to instruct cmake to produce code that will be used afterwards to find and link the library, and add the correct directory that contain the public interface of the library.

Of course you can write your own cmake code to do this. We provide a macro that automate all this. In short it is enough you call:

 icub_export_library(libraryname PRIVATE_HEADERS_LIST DESTINATION PUBLIC_HEADER_FILES) 

e.g.

 icub_export_library(libraryname "${PROJECT_SOURCE_DIR}/include" 
 include/iCub/libraryname ${folder_header})

at the end of the cmake file (after the call to add_library()). Here libraryname is the name of the target, and include/iCub/libraryname is the destination directory inside ${CMAKE_INSTALL_PREFIX} where files in folder_header will be copied. If your library does not define a namespace it is sufficient to specify here include/iCub; all header files specified in folder_header will be copied to ${CMAKE_INSTALL_PREFIX}/include/iCub. We can remove this behavior, it is here for back compatibility.

To see what is going on under the hood of the call icub_export_library() see below.

Making things nice in Windows

Note: we have found a way to set this option globally in the build (still to be tested accurately). In windows it is common habit to give different names to debug and release libraries. Often libraries compiled in debug mode have a d appended at the end of the name. We recommend you do this in your library, in cmake it is quite easy, just add, after the call to add_library():

 set_target_properties(libraryname PROPERTIES DEBUG_POSTFIX "d")

Adding dependencies to libraries

To work properly CMake needs to know the dependencies of all libraries. If lib1 depends on lib0 you have to tell CMake, like this:

 target_link_libraries(lib1 lib0) #target lib1 depends on target lib0

Notice that this is not useless as it might seem (you do not link static libraries when building a static library). This tells cmake that lib1 depends on lib0 (lib1 inherits dependencies of lib0, if any) and is actually fundamental to avoid strange linking errors (especially in Linux where the link order does matter).

Under the hood

What is going on under the hood.

Basic notion. CMake has to generate a file called icub-config.cmake, which includes the code to import all the libraries built (and exported) by the project. This files is used by CMake to locate and import the package when you call find_package(iCub).

There are two such files, one goes to the build directory and is used when you "export the build tree" (i.e. you link the libraries from the location where they have been built). The other one is copied at install time to ${CMAKE_INSTALL_PREFIX), and it is used when you important the library after it has been installed. The two files have the same name, CMake will use the one it finds (CMake uses certain rules, in short: $ICUB_DIR env variable, some system directories...).

What does icub-config.cmake do? Broadly speaking this file is responsible for creating cmake variables that allows using the project in users code, namely:

 ICUB_FOUND TRUE/FALSE, when the package is found
 ICUB_INCLUDE_DIRS a directory (or a list) that should be added to the compiler include path

It also imports all the targets (libraries) that have been compiled and exported by the package (we use the export mechanism available in CMake 2.6). The export command generates a file that contains all the code that needs to be executed to import the package.

This is implemented in different ways at install time and when exporting the build tree.

Installation

This situation is a bit easier because the main build knows where all the header files will go.

  • icub-config.cmake defines ICUB_FOUND and ICUB_INCLUDE_DIRS. It also includes icub-export-install.cmake.
  • icub-config.cmake is generated from the template in conf/template/icub-config-install.cmake
  • icub-export-install.cmake is generated automatically by cmake when calling export().

Main code is in conf/ExportForInstall.cmake.

icub-config.cmake is first generated in the build directory with the name icub-config-for-install.cmake and then copied and renamed at install time to ${CMAKE_INSTALL_PREFIX}; the installer also copies icub-export-install.cmake.

Exporting the build

In this case header files are scattered through the source tree, ICUB_INCLUDE_DIRS needs to be generated by scanning a list of targets and corresponding INCLUDE_DIRS property (both are properties that are created and populated in subsequent calls to icub_export_library(target)).

The code that set the ICUB_INCLUDE_DIRS variable is automatically generated (see conf/iCub/ExportBuildTree.cmake) and goes into icub-export-build-includes.cmake.

The list of targets that need to be imported is generated by calling export() and is called icub-export-build.cmake.

Both files are merged in icub-config.cmake, which creates ICUB_FOUND and includes icub-export-build-includes.cmake and icub-build-export.cmake.

Main code is in conf/ExportBuildTree.cmake