Release packaging and distribution

How dsdlc / dsdl-opt / dsdld get from a git tag to a user's machine. This documents the pipeline as it is built today; channels and targets that do not exist yet (Homebrew, an apt repository, other architectures) are backlog and live in distribution-channels.md.

Today: a tag pushes packages to a draft GitHub release — .deb for Ubuntu on amd64 and arm64, and a tarball for macOS on each of Apple silicon and Intel. Each embeds the LLVM it needs, so nothing has to be added to the user's machine first.

1. What ships

Component Contents Debian package macOS
bin dsdlc, dsdl-opt, dsdld, SBOM, licences llvm-dsdl bin/ in the tarball
dev 8 static libraries, include/llvmdsdl, generated dialect headers, Version.h llvm-dsdl-dev included in the tarball

llvm-dsdl-dev pins llvm-dsdl (= <version>): the static libraries and headers are only coherent against the exact build they shipped with, and dpkg refuses the pairing otherwise.

The generated output is self-sufficient. Each backend writes its own runtime support scaffold into the output tree, and those scaffolds are compiled into the binary (tools/runtime/generate_embedded_runtime.py), so a packaged dsdlc emits code that compiles on a machine which has never seen this source tree.

2. The LLVM toolchain

The tools link an LLVM/MLIR this project builds for itself, pinned by revision in packaging/toolchain/llvm.pin and published by .github/workflows/toolchain.yml. It rebuilds when its build configuration changes -- that line, the CMake flags in build_llvm.py, or a Dockerfile -- not per release.

Owning the build is what settles five things a distribution otherwise settles on our behalf: the glibc floor, the C++ standard library, RTTI, whether a shared libLLVM exists, and the exact patch level every lane links.

Distribution LLVM packages

Static linking against one is unreachable by build flag. Measured against the Homebrew llvm 22 keg:

Fact Location
add_library(LLVM SHARED IMPORTED) lib/cmake/llvm/LLVMExports.cmake:1742
450 references to that target lib/cmake/mlir/MLIRTargets.cmake
set(LLVM_LINK_LLVM_DYLIB ON) lib/cmake/llvm/LLVMConfig.cmake:30
set(LLVM_WITH_Z3 1) lib/cmake/llvm/LLVMConfig.cmake:316
llvm-config --link-static --system-libs → -lm /opt/homebrew/lib/libz3.dylib -lz -lzstd -lxml2 —

Component archives are present; a monolithic libLLVM.a is not. -DLLVM_LINK_LLVM_DYLIB=OFF does not dislodge the dylib, because MLIR's exported targets name a target that is declared shared. Even the fully static path hard-codes libz3.dylib by absolute path.

The same reasoning closes the mirror-image option of building against libc++ on Linux, which is the obvious-looking way to get std::flat_set without include/llvmdsdl/Support/FlatSet.h: a distribution's libLLVM fixes the standard library, and two C++ runtimes either side of MLIR's std::string boundary yield undefined symbols or a quietly ODR-broken binary.

Configuration

packaging/toolchain/build_llvm.py owns every CMake flag; the Dockerfiles supply a base image and a compiler. Independently maintained flag lists would drift into describing different toolchains, and the divergence would surface as different generated bytes — the property this project pins.

  • LLVM_TARGETS_TO_BUILD=X86;AArch64;ARM;RISCV;AVR;Mips;WebAssembly. dsdlc --target-language obj assembles inside the compiler, so a triple it is asked for has to be a backend the prefix carries. These reach a Cyphal node or a host that builds for one; LLVM's remaining thirteen are the bulk of a full build and none of them is a target this compiler is asked for.
  • LLVM_ENABLE_RTTI=ON. Mandatory; the other settings here switch features off, this one on: llvmdsdl derives from mlir::Dialect and mlir::Pass, and an -fno-rtti LLVM fails at link with undefined reference to typeinfo for mlir::Pass. Costs ~31 MB of the prefix.
  • Optional features off — Z3_SOLVER, LIBEDIT, LIBXML2, ZLIB, ZSTD, FFI, PLUGINS. These account for the whole dependency tail a distribution's LLVM drags in.
  • LLVM_BUILD_TOOLS=OFF, worth 1.5 GB: without a shared libLLVM every one of llc, opt, lli, mlir-opt and about a hundred others statically links the world, and mlir-opt alone reaches 200 MB. The four tools the build and lit suite need total under 9 MB.
  • intrinsics_gen and analysis_gen are built as their own pass first. MLIR declares MLIRExecutionEngineUtils with DEPENDS intrinsics_gen, but its source reaches TargetLibraryInfo.inc through PassBuilder.h, which analysis_gen produces. Nothing orders the two. Which of them finishes first moves with the target list and the core count, and a machine with few cores has lost the race and failed on a missing header.

Flavours

A prefix is ~490 static archives built against one libc and does not cross that boundary, so there are two, each on both architectures:

glibc (Ubuntu 22.04) musl (Alpine 3.24)
Serves the release, llvm-dsdl-dev, CI fully static binaries (§1 of the backlog)
Floor GLIBC_2.34 none
Prefix 897 MB 956 MB
Build ~72 min aarch64 / ~89 min x86-64 ~83 min aarch64 / ~134 min x86-64

Both carry the seven target backends above alongside the LLVM and MLIR component archives, and link nothing beyond the C and C++ runtimes.

The glibc flavour is built on 22.04 rather than the CI image's 26.04 because glibc is backward compatible: archives built against 2.35 link in 2.43, and the reverse does not hold. One build therefore serves both the release and CI.

macOS is absent from that table because a Darwin toolchain cannot be a container image. Each macOS leg builds its own natively and caches the prefix on the pin.

Consumption

  • The release pulls ghcr.io/opencyphal-garage/llvm-dsdl-toolchain:glibc-<ref> and copies the prefix into the build image. A missing image fails the job outright. Building it inline is the obvious fallback and is deliberately absent: it costs one to two hours on a 4-vCPU runner, which once exceeded the job timeout, and a silent fallback would reproduce that as a mystery.
  • CI restores the prefix from a tarball attached to a prerelease tagged toolchain-<ref>, via .github/actions/restore-llvm-toolchain. 172 MB compressed. The action asserts the unpacked LLVM_REF matches the pin, so a stale tarball fails there rather than as generated output that differs from the release.

packaging/toolchain/verify_toolchain.py asserts each property above against a built prefix before it is published, so a Dockerfile regression fails there rather than downstream of every consumer.

3. Build environments

Linux: Ubuntu 22.04 (jammy)

packaging/docker/Dockerfile.ubuntu-release layers the glibc toolchain prefix onto jammy. 22.04 puts the glibc floor at 2.35, so one package installs on 22.04, 24.04 and 26.04 alike.

The CI toolshed image is not reused for releases. It is Ubuntu 26.04, and a .deb built there links a glibc that refuses to install on anything older — which would silently cost every 24.04 user, the largest group.

Two third-party apt sources remain, for tools rather than libraries: apt.llvm.org for clang-22, because jammy's GCC 11 predates much of C++23 and the project requires it, and apt.kitware.com for CMake ≥ 3.24 (jammy has 3.22). Which compiler does the compiling is independent of which LLVM is linked; LLVM_DIR decides the latter and names the toolchain prefix explicitly, so clang-22's own runtime LLVM under /usr/lib/llvm-22 never enters the build.

macOS: macos-15 (Apple silicon) and macos-15-intel

The Darwin toolchain, built natively in each job and cached on the pin — ~9 minutes on Apple silicon, ~30 on the 4-core Intel runner. Two prefixes at ~800 MB each share a 10 GB repository cache, so an eviction turns a cold toolchain build into the cost of an ordinary run; that is the first thing to look at if the 90-minute job timeout starts biting.

Intel ships until GitHub retires macos-15-intel in August 2027, which is the last x86_64 image Actions will offer. D2 of the backlog records the reasoning and the macOS 15 floor that comes with it.

Dependencies are derived

CPACK_DEBIAN_PACKAGE_SHLIBDEPS is off: dpkg-shlibdeps resolves every linked library against the dpkg database, which is the wrong question for a package whose libraries are embedded.

The list is derived from the built binaries at package time by packaging/deb/derive_depends.py: read the ELF program headers to decide whether a binary is dynamic at all, walk what it links, map the rest through dpkg -S. It runs from a CPack project-config script, because the binaries do not exist at configure time.

It distinguishes statically linked from unreadable and fails loudly on the second. ldd gives the same answer for both, and conflating them would produce a package claiming to need nothing.

Architecture: is stated from the target rather than taken from dpkg --print-architecture, which reports the architecture of the machine's dpkg: Alpine's answers musl-linux-arm64, and a package carrying that installs nowhere.

The derived list for the shipped package is libc6 and libstdc++6.

4. Debian packaging

.deb generation is CPack's DEB generator driven by the existing install components (cmake/Packaging.cmake). CPACK_DEB_COMPONENT_INSTALL maps bin/dev onto the two packages. Compression is xz rather than CPack's default gzip, since this is a file people download directly.

The llvm-dsdl package is 21.4 MB on arm64, 22.0 MB on amd64.

Policy metadata

Debian keys these off the binary package name, so each package needs its own copy. CMAKE_INSTALL_DOCDIR only covers llvm-dsdl; the llvm-dsdl-dev destination is spelled out.

  • packaging/deb/copyright — DEP-5, covering the project's MIT terms and the Apache-2.0-with-LLVM-exception of the LLVM code linked into the tools. Static linking redistributes that code as surely as shipping a shared object would.
  • packaging/deb/changelog — CPack has no changelog support, so it is gzipped (-n, for a byte-identical result across builds) and installed by hand. Its top entry restates the version; a package whose changelog disagrees with its control file is malformed, so configure fails on the mismatch.
  • packaging/deb/lintian-overrides/llvm-dsdl — the tools ship unstripped, neither stripped nor split into -dbgsym. For a compiler of an avionics-adjacent wire format, the symbols needed to read a backtrace should already be on the machine that produced the core. The override carries no file hint: lintian releases disagree on how hints are rendered, and a hint that does not match the running version is silently inert.

Both packages are lintian-clean.

Man pages

Generated from each tool's own --help by tools/man/generate_manpage.py, so the two cannot drift. Two help dialects are parsed: the NAME/SYNOPSIS sections dsdlc and dsdld print, and LLVM's cl::opt format dsdl-opt inherits. Anything not structurally mappable is emitted verbatim in a preformatted block rather than guessed at. The page date comes from the changelog's release trailer, not the clock, so rebuilds are byte-identical.

This runs each tool to document it, so it needs host-executable binaries — fine natively, but a cross-build would have to generate the pages host-side.

5. macOS packaging

The tarball is bin/, the dev component, licences and a manifest. It is relocatable — extract anywhere, add bin/ to PATH — and carries no dylibs. The three tools link exactly one library:

/usr/lib/libSystem.B.dylib

libSystem cannot be linked statically on macOS. Everything above it can, including libc++, which takes two settings:

  • build_llvm.py --with-libcxx — builds a static libc++ into the toolchain prefix. Apple ships libc++ as a dylib only, so there is otherwise nothing to link.
  • LLVMDSDL_STATIC_LIBCXX=ON — links it.

⚠️ Do not drop the LIBCXX_HERMETIC_STATIC_LIBRARY flags in build_llvm.py. Without them the build succeeds and otool -L still shows libSystem alone, but the tools crash at runtime: libSystem already provides libc++abi, and a non-hermetic libc++ adds a second copy. Check this by running a tool, not by reading otool.

Gatekeeper

The binaries are ad-hoc signed rather than notarised, and spctl -a -t exec rejects them. That matters only for files carrying com.apple.quarantine, which depends on how the archive was fetched, not on how it was opened:

Path Quarantine on extracted files Result
curl / gh release download + tar -xzf no runs
Browser download + tar -xzf yes blocked
Browser download + Finder double-click yes blocked
brew install no — brew fetches over curl runs

tar does propagate quarantine. The attribute is attached to the downloaded archive by the browser, and extracting it — with tar -xzf, from a shell, no Finder involved — copies it onto every extracted file. Measured on macOS 26.5.2 (bsdtar 3.5.3 / libarchive 3.7.4) against a Chrome-downloaded llvm-dsdl-0.2.0-darwin-arm64.tar.gz: all 120 extracted files came out carrying the archive's own quarantine UUID, and bin/dsdlc --version was killed by Gatekeeper with "Apple could not verify ... is free of malware". Do not re-derive this claim from first principles or from a synthetic archive stamped with xattr -w — a hand-written quarantine value is not registered in the LaunchServices quarantine database and does not reproduce the propagation. Test with a real browser download or not at all.

What actually decides the outcome is whether the archive was quarantined in the first place, so the fetch is what the install instructions have to get right:

curl -fLO https://github.com/OpenCyphal-Garage/llvm-dsdl/releases/download/v0.2.0/llvm-dsdl-0.2.0-darwin-arm64.tar.gz

A reader who already downloaded through a browser clears the attribute after extracting, which restores the ad-hoc binaries to runnable — verified on the same archive above:

xattr -d -r com.apple.quarantine llvm-dsdl-0.2.0-darwin-arm64

The tarball needs no Developer ID and no notarisation: the two clean paths are curl and brew, and every browser path needs the xattr step. Notarisation is the usual answer for shipping macOS binaries and would cost a paid certificate, notarytool, and secrets; it buys the whole browser column, which is the path most first-time readers take — see D5 in distribution-channels.md.

⚠️ smoke_macos.py cannot catch a regression here, and no CI job can as things stand: the verifier extracts a freshly built, never-downloaded tarball, so no quarantine attribute exists anywhere in the pipeline to propagate. This row of the table is checked by hand or not at all.

6. The release workflow

.github/workflows/release.yml, triggered by a v* tag or by workflow_dispatch (which defaults to a dry run that builds and verifies without publishing). CONTRIBUTING.md §14 is the procedure.

tag v* ──▶ stage ──▶ build   (ubuntu-22.04-arm, ubuntu-22.04) ──▶ publish
             │       macos   (macos-15, macos-15-intel)            │
        version/tag      each: build → package → verify        draft release,
        consistency                                            checksums, attestation
  • stage — derives the version from VERSION, fails if a tag disagrees with it, and checks the changelog's top entry matches. A tag that disagreed would ship binaries misreporting themselves.
  • build / macos — each runs on a native runner for its architecture; CI never emulates. Emulation is only how a developer on one architecture cross-checks the other locally, which is what smoke.py --platform exists for. The legs are independent (fail-fast: false): one architecture's package is installable without the other.
  • publish — downloads each artifact into its own subdirectory (every build job writes a SHA256SUMS, so merging would leave one overwriting the others), rebuilds one checksum file from what is actually attached, attests provenance, and creates a draft release.

7. Verification

A package that only works where it was built is the failure mode all of this exists to catch, and with no repository behind a downloaded file, nothing else would notice.

Check What it proves Where
smoke.py The .deb installs on pristine Ubuntu, all three tools run, no LLVM resolves from outside the package, generated C compiles with stock cc, a non-C backend emits release workflow
smoke_macos.py Every non-system reference in every binary resolves inside the tarball, tools run, generated C compiles, and the binaries carry the architecture the tarball is named for at no higher a macOS floor than the release claims release workflow
verify_toolchain.py The toolchain is static-only, backend-free, and carries the tools the build and lit suite need toolchain workflow, before publishing
corpus_determinism.py dsdlc built on amd64 and arm64 emits byte-identical output for all six backends CI (cross-arch-determinism)
check_deb_config.py Control fields, component split, version pin, lintian verdict — against real packages built in a container ctest (llvmdsdl-packaging-deb-config)
test_check_deb_config.py The verdict logic itself, against recorded output — no Docker needed ctest (llvmdsdl-packaging-deb-config-selftest)

Three design points worth keeping:

The Linux check asserts what is absent. No LLVM may resolve from outside the package. A build that reverted to a system LLVM would run on the build machine and fail on a user's.

macOS asserts linkage, not just behaviour. There is no pristine container to install into, and the build runner is the worst possible judge — a tarball that only worked there would pass every behavioural test.

The verdict logic is tested without Docker. The containerised check registers only when a docker binary exists and exits 77 — which SKIP_RETURN_CODE turns into a ctest skip — when the daemon is unreachable. The CI Linux lane runs inside a container and has no Docker of its own, so without this it would either fail to configure or report a green test that never ran. Pass --require-docker to make an unusable daemon a hard failure; the release lane should, since a silent skip there would let an unverified package through.

8. Why the release builds natively

Building every target from one Linux runner is the obvious way to avoid a native runner per architecture, and zig cc is the obvious tool. This is a dependency problem, not a compiler problem: zig supplies a clang that emits aarch64 or Mach-O code, libc headers and glibc version stubs, but not libLLVM for the target or the MLIR archives we link.

Owning the toolchain answers most of that — there is a target-built LLVM/MLIR, and the C++ ABI is ours on both sides. TableGen is mechanical: build a host-native mlir-tblgen and point the cross build at it.

What remains is that verification requires execution. Every check in §7 runs the binary, so cross-compiling removes the build runner and leaves the test runner, which is the one that matters. GitHub provides native x86-64 and arm64 Linux runners free on a public repository, and emulation costs a measured 3.8×, so for the two architectures that ship there is nothing to buy.

macOS additionally requires Apple's SDK, licensed for use on Apple hardware. Signing and notarisation can be performed from Linux, so those are not the obstacle.

Architectures with no runner — armv7, riscv64 — are where cross-compilation earns its place, and they are §2 of the backlog along with how far emulation carries their verification.