Linking LLVM in-tree as a shared library fails on macOS but not on Linux

[icmccorm]

Summary

I am trying to build LLVM in-tree as a shared library and then link against it as if it were an external, prebuilt installation (e.g. setting llvm-config in the bootstrap config). When I try this, stage 1 builds fail to locate libLLVM on macOS. This does not occur on an x86_64-unknown-linux-gnu system.

Command used

Replace <host> with the current host.

./x.py build llvm
export LD_LIBRARY_PATH="absolute/path/to/build/<host>/llvm/build/lib"
./x.py build —stage 1 --set target.<host>.llvm-config="absolute/path/to/build/<host>/llvm/bin/llvm-config"

Expected behaviour

The build should succeed on macOS like it does on Linux.

Actual behaviour

The build fails at the very end with the following message:

dyld[7250]: Library not loaded: @rpath/libLLVM.dylib

Bootstrap configuration (config.toml)

profile = "dist"
change-id = 134650

[llvm]
download-ci-llvm = false
link-shared = true

Operating system

• Ubuntu 22.04.5 (x86_64-unknown-linux-gnu)
• macOS 15.3.2 (aarch64-apple-darwin)

HEAD

4d30011

Additional context

I haven't modified anything other than my bootstrap config.

I am trying to do this so that I can test changes to a custom LLVM instrumentation pass without having to rebuild the Rust frontend. This is purely for convenience during development—the project's CI pipeline builds and tests everything with rustc statically linked to LLVM in the usual way. The variable LD_LIBRARY_PATH is set by default to that directory in the development environment.

Build Log

log.txt

[TimNN]

I think what might be happening here is that during the first build, libLLVM gets copied into the library directory of the Rust sysroot, and rustc is linked against that.

Then, during the second build, because the llvm-config option was specified, libLLVM does not get copied. I assume that bootstrap/Cargo don't detect any other changes, so rustc isn't re-linked, which means it will try and find LLVM in the sysroot where it is no longer present instead of the LLVM build directory.

Random notes on llvm-config+macOS

There are some know issues with llvm-config+dynamically linked LLVM+macOS: llvm/llvm-project#39599

Bootstrap has some custom logic for copying libLLVM on macOS in various places, e.g.:

rust/src/bootstrap/src/core/build_steps/llvm.rs

Line 588 in 8b87fef

if builder.llvm_link_shared() && target.contains("apple-darwin") {

and https://github.com/rust-lang/rust/blob/8b87fefd76665236a304d3c0998e1021710ce1b0/src/bootstrap/src/core/build_steps/dist.rs#L2025C16-L2035

Some of this logic doesn't run when using "system LLVM", i.e. when setting the llvm-config option as you are doing.

[icmccorm]

Thanks for your help! That's similar to what I was thinking, too. However...

during the first build, libLLVM gets copied into the library directory of the Rust sysroot, and rustc is linked against that.

From what I can tell, both the original libLLVM.dylib and the versioned symlink (libLLVM-20-rust-1.87.0-nightly.dylib) are made available within ./build/host/llvm/lib and ./build/host/llvm/build/lib, but nothing has been copied out of there yet.

Two other additional things to add from my original post:

• I'm also setting target.<host>.llvm-has-rust-patches to true
• On macOS I set DYLD_LIBRARY_PATH, and not LD_LIBRARY_PATH

[icmccorm]

I think I've figured out the issue. On Unix systems, the link to the sysroot is inserted into the @rpath for the rustc binary. I've found conflicting information on this, but it seems like this setting overrides any system-wide search paths to shared libraries.

If you dynamically link against an external "system" LLVM, then libLLVM.dylib won't be copied into the sysroot, so it won't be found in @rpath. If I add the LLVM libdir to @rpath within Cargo::configure_linker, then it fixes this issue and everything links correctly.

Patch for configure_linker

// If we are dynamically linking against a prebuilt system LLVM,
// then libraries will not be copied into the sysroot, so they
// cannot be located via rpath. We fix this by adding the path to
// the LLVM libdir to rpath, in addition to the sysroot.
if builder.llvm_link_shared() && builder.is_system_llvm(target) {
    if let Some(llvm_config) = builder.llvm_config(target) {
        let llvm_libdir =
            command(llvm_config).arg("--libdir").run_capture_stdout(builder).stdout();
        self.rustflags.arg(&format!("-Clink-args=-Wl,-rpath,{llvm_libdir}"));
    }
}

I'l be using the patch above until there's a more permanent fix. I'd make a PR, but it seems like there's still more to investigate here. Strangely enough, disabling @rpath altogether by setting target.<host>.rpath = false and adding the LLVM libdir to DYLD_LIBRARY_PATH does not fix the issue. I still get the same message that @rpath/libLLVM.dylib cannot be found.