Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add one more "late metadata"/"digest" file to rlib archives #138243

Open
petrochenkov opened this issue Mar 8, 2025 · 4 comments
Open

Add one more "late metadata"/"digest" file to rlib archives #138243

petrochenkov opened this issue Mar 8, 2025 · 4 comments
Labels
A-linkage Area: linking into static, shared libraries and binaries C-enhancement Category: An issue proposing an enhancement or a PR with one. E-help-wanted Call for participation: Help is requested to fix this issue. E-mentor Call for participation: This issue has a mentor. Use #t-compiler/help on Zulip for discussion. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.

Comments

@petrochenkov
Copy link
Contributor

petrochenkov commented Mar 8, 2025

The file should contain data about:

  • Native libraries bundled into the rlib, to avoid looking up native libraries during the regular "early" metadata construction, when it's unnecessary and cannot be done fully correctly (native_libs: More consistent native library search between rustc and linker #138170)
  • Object files bundled into the rlib, to support link time cfg on native libraries bundled as object files
  • Rust object files in the rlib to avoid hacks like looks_like_rust_object_file during LTO

Unlike regular metadata which is constructed early, this additional digest will be constructed and added to the archive in the very end, when all other members of the rlib archive are already added.
It is also going to be read only when rustc is going to link the rlib into something, not earlier.

The data could potentially use some text format like json, but it would probably be simpler to just use the same format as the regular metadata uses.
The implementation can also mirror what is done for regular metadata, in a simplified form.

Me and @belovdv wanted to do this when implementing +bundle,+whole-archive (#113301) and packed_bundled_libs (#108081), as a more principle solution, but ended up with just some workarounds instead, and never returned to this.

@rustbot rustbot added the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Mar 8, 2025
@petrochenkov petrochenkov added T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. A-linkage Area: linking into static, shared libraries and binaries E-mentor Call for participation: This issue has a mentor. Use #t-compiler/help on Zulip for discussion. E-help-wanted Call for participation: Help is requested to fix this issue. labels Mar 8, 2025
@petrochenkov
Copy link
Contributor Author

I suggest first adding some data that is enough to avoid the file skipping hacks in fn link_staticlib.

@jieyouxu jieyouxu added C-enhancement Category: An issue proposing an enhancement or a PR with one. and removed needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. labels Mar 9, 2025
@bjorn3
Copy link
Member

bjorn3 commented Mar 12, 2025

Why put it at the end? That would mean we have to scan over each rlib twice during linking. Once to find the "digest" and a second time to copy the actual content. Ar archives don't allow random access. You need to iterate over them one entry at a time. Wouldn't it be possible to create this "digest" during rlib writing before adding object files, but after we already know which object files will be added?

@petrochenkov
Copy link
Contributor Author

fn link_rlib contains a number of archive_builder.add_file(...); steps.
I assumed each of them would also add something into the digest, then the digest would become ready in the end.

Of course it can be refactored to be a bit smarter as an optimization.

@davidtwco
Copy link
Member

You might get more visibility on this as an MCP - it sounds reasonable to me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-linkage Area: linking into static, shared libraries and binaries C-enhancement Category: An issue proposing an enhancement or a PR with one. E-help-wanted Call for participation: Help is requested to fix this issue. E-mentor Call for participation: This issue has a mentor. Use #t-compiler/help on Zulip for discussion. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Development

No branches or pull requests

5 participants