Managing dependencies with vcpkg

Previously

Working with dev containers in CLion

So many dependencies, so little time

One of the interesting side effects of using a completely custom toolchain to build bpf-iotrace is that every single one of the third-party libraries it uses must be built with the same toolchain, transitive dependencies included.

This drastically increases the complexity of bpf-iotrace’s build environment because every library’s source code must be checked out, configured, and built. Then, those libraries and their headers must be staged where the main project can find them. This explosion of complexity is especially challenging when a project’s direct dependencies have dependencies of their own (also known as transitive dependencies). Thankfully, tools like vcpkg and conan can generate complete dependency graphs for a project and provide APIs for fetching, building, and installing third-party libraries in consistent and maintainable ways. Both projects also have large collections of pre-written build scripts for most popular, open source C and C++ libraries.

This is especially helpful where libraries and frameworks like protobuf and gRPC are concerned because projects that use them frequently benefit from object libraries and up-to-date code generation utilities being available at configuration time.

bpf-iotrace uses vcpkg because vcpkg offers simpler integration with CMake and greater flexibility when integrating caching for source code and pre-built binary artifacts.

Integrating vcpkg with the bpf-iotrace build system

vcpkg’s README file recommends installing vcpkg as a submodule within the project that uses it. I like this advice because it makes it easier to automate bootstrapping vcpkg as part of the project configuration step.

The installation process is as simple as changing to the project directory where vcpkg should be located and running git submodule add https://github.com/microsoft/vcpkg.git

To integrate vcpkg with bpf-iotrace’s CMake build system, I added the following lines to the top-level CMakeLists.txt before the first project() command:

set(VCPKG_FIXUP_ELF_RPATH ON)

set(CMAKE_TOOLCHAIN_FILE "${CMAKE_CURRENT_LIST_DIR}/tools/vcpkg/scripts/buildsystems/vcpkg.cmake"
        CACHE STRING "Vcpkg toolchain file")

if(NOT EXISTS ${CMAKE_TOOLCHAIN_FILE})
    # Initialize / update the vcpkg submodule
    execute_process(COMMAND git submodule update --init --depth 1 -- tools/vcpkg
            WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}
            COMMAND_ERROR_IS_FATAL ANY)
endif()

This is where things get tricky because bpf-iotrace also relies on CMAKE_TOOLCHAIN_FILE to point CMake at the custom toolchain in its development container. Fortunately, vcpkg offers two mechanisms for resolving this conflict. I can include() vcpkg/scripts/buildsystems/vcpkg.cmake in my own toolchain file, or I can set VCPKG_CHAINLOAD_TOOLCHAIN_FILE to point at my compiler toolchain file, and vcpkg and CMake will use its contents for setting up the project’s compilers. I opted for the latter approach to be consistent with how vcpkg configures its package builds.

So the preamble of my top-level CMakeLists.txt becomes:

set(VCPKG_CHAINLOAD_TOOLCHAIN_FILE ${CMAKE_CURRENT_LIST_DIR}/.devcontainer/toolchain.cmake
        CACHE STRING "Dev container toolchain file")
set(VCPKG_FIXUP_ELF_RPATH ON)

set(CMAKE_TOOLCHAIN_FILE "${CMAKE_CURRENT_LIST_DIR}/tools/vcpkg/scripts/buildsystems/vcpkg.cmake"
        CACHE STRING "Vcpkg toolchain file")

if(NOT EXISTS ${CMAKE_TOOLCHAIN_FILE})
    # Initialize / update the vcpkg submodule
    execute_process(COMMAND git submodule update --init --depth 1 -- tools/vcpkg
            WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}
            COMMAND_ERROR_IS_FATAL ANY)
endif()

Now, even on a fresh git clone of bpf-iotrace, CMake will download vcpkg to the correct directory, and bootstrap it–and all the project’s other dependencies–at configuration time. It will also use the dev container’s custom toolchain to build the main project.

Manifest mode

vcpkg has two modes for managing dependencies: “classic,” and “manifest.” vcpkg documentation recommends manifest mode for new projects which–along with installing vcpkg as a submodule–allows a project to be completely self-contained.

The manifest they are talking about is vcpkg.json and bpf-iotrace uses it to declare its dependencies:

{
  "dependencies": [
    "libbpf"
  ],
  "overrides": [
    {
      "name": "libbpf",
      "version": "1.4.1#8"
    }
  ],
  "builtin-baseline": "943c5ef1c8f6b5e6ced092b242c8299caae2ff01"
}

In the spirit of “Include what you use” from Google’s excellent C++ style guide, bpf-iotrace declares its direct dependencies and only its direct dependencies in vcpkg.json. This means that if a directly included package depends on other software, its vcpkg.json needs to declare those dependencies, and so on. Conversely, if we find we start directly using a library that vcpkg already brought in as a transitive dependency, we need to declare it directly in bpf-iotrace/vcpkg.json.

Also note that bpf-iotrace uses the overrides key in vcpkg.json to pin its dependency versions. This is intended to make builds more deterministic by preventing vcpkg from automatically pulling in the latest release of a dependency.

In a future post, I will write about an automated workflow for updating explicitly versioned dependencies with review and approval gates to help developers avoid nasty surprises.

Overlays in vcpkg

As nifty as vcpkg is, some of its default’s don’t work for bpf-iotrace. Even with VCPKG_CHAINLOAD_TOOLCHAIN_FILE set for the main project, it insists on using the dev container’s default compiler rather than its custom toolchain. It also tries to statically link every third-party dependency. This is usually desirable because it would simplify the heck out of the bpf-iotrace installation process. But it is problematic when any project depends on LGPL-licensed libraries like elfutils libelf. Compliance with LGPL license terms is way easier when LGPL-licensed libraries are dynamically linked. (DISCLAIMER: the preceding sentence is engineering advice, NOT legal advice. Always make sure you and your company’s lawyer(s) are on the same page before incorporating any LGPL code into your products.)

Another bpf-iotrace-specific limitation for vcpkg is that its default registry does not include bpf-trace’s most important third-party dependency: libbpf.

Fortunately, vcpkg provides hooks called overlays that allow users to customize its behavior for their needs and add missing packages without having to create a full-blown registry.

Building vcpkg dependencies with a custom toolchain

In vcpkg, triplets encapsulate the definition of a library’s build environment, and the library’s build configuration. vcpkg ships with default triplets for several popular target platforms including Linux on x86_64 called x64-linux.cmake.

To tailor the x64-linux triplet for bpf-iotrace’s needs, we can create a directory to contain triplet overlays in tools/vcpkg-overlays/triplets, and reference that directory in vcpkg-configuration.json. Then we can create a modified copy of x64-linux.cmake in tools/vcpkg-overlays/triplets with the following contents:

set(VCPKG_CHAINLOAD_TOOLCHAIN_FILE "${CMAKE_CURRENT_LIST_DIR}/../../../.devcontainer/toolchain.cmake")

# List of LGPL-licensed dependencies AND transitive dependencies that must be dynamically linked to them.
set(LGPL_DEPENDENCIES elfutils bzip2)

set(VCPKG_TARGET_ARCHITECTURE x64)
set(VCPKG_CRT_LINKAGE dynamic)
set(VCPKG_LIBRARY_LINKAGE static)
# Force LGPL dependencies to be dynamically linked
if(PORT IN_LIST LGPL_DEPENDENCIES)
    set(VCPKG_LIBRARY_LINKAGE dynamic)
endif()

set(VCPKG_CMAKE_SYSTEM_NAME Linux)

Setting VCPKG_CHAINLOAD_TOOLCHAIN_FILE in a triplet file causes vcpkg libraries to build with the toolchain it specifies instead of using vcpkg’s default compiler detection logic.

We also selectively enable dynamic linking elfutils (libelf) which is LGPL-licensed, and bzip2 because shared libelf builds fail unless bzip2 is also built as a shared library.

New or customized port files

The process of publishing or updating a port published in a vcpkg registry is complex enough to make rapid iteration difficult. vcpkg’s overlay ports feature makes it possible to quickly test new ports, or changes to an existing port before publishing them to a registry.

Results

After configuring the bpf-iotrace project with CMake, we see vcpkg has installed headers and a statically linkable archive library for libbpf as well as headers and binary libraries for libbpf’s transitive dependencies.

mikep@b593cb81e890 /IdeaProjects/bpf-iotrace/cmake-build-debug/vcpkg_installed/x64-linux $ ls
debug  etc  include  lib  share  tools

mikep@b593cb81e890 /IdeaProjects/bpf-iotrace/cmake-build-debug/vcpkg_installed/x64-linux $ ls include
bpf      dwarf.h   gelf.h    lzma    nlist.h  zdict.h  zstd.h
bzlib.h  elfutils  libelf.h  lzma.h  zconf.h  zlib.h   zstd_errors.h

mikep@b593cb81e890 /IdeaProjects/bpf-iotrace/cmake-build-debug/vcpkg_installed/x64-linux $ ls include/bpf
bpf.h            bpf_endian.h       bpf_helpers.h  btf.h     libbpf_common.h  libbpf_version.h  usdt.bpf.h
bpf_core_read.h  bpf_helper_defs.h  bpf_tracing.h  libbpf.h  libbpf_legacy.h  skel_internal.h

mikep@b593cb81e890 /IdeaProjects/bpf-iotrace/cmake-build-debug/vcpkg_installed/x64-linux $ ls lib
libasm-0.186.so  libbz2.so               libdebuginfod.so    libdw.so.1       liblzma.a
libasm.so        libbz2.so.1.0           libdebuginfod.so.1  libelf-0.186.so  libz.a
libasm.so.1      libbz2.so.1.0.8         libdw-0.186.so      libelf.so        libzstd.a
libbpf.a         libdebuginfod-0.186.so  libdw.so            libelf.so.1      pkgconfig

Up next

Packaging libbpf with vcpkg