rustc-dev component is broken by missing literal-escaper

[RalfJung]

Trying to load the rustc-dev packages with cargo leads to an error:

    Updating crates.io index
error: failed to get `literal-escaper` as a dependency of package `rustc_ast v0.0.0 (/home/r/.rustup/toolchains/miri/lib/rustlib/rustc-src/rust/compiler/rustc_ast)`
    ... which satisfies path dependency `rustc_ast` of package `rustc_codegen_ssa v0.0.0 (/home/r/.rustup/toolchains/miri/lib/rustlib/rustc-src/rust/compiler/rustc_codegen_ssa)`
    ... which satisfies path dependency `rustc_codegen_ssa` of package `rustc-main v0.0.0 (/home/r/.rustup/toolchains/miri/lib/rustlib/rustc-src/rust/compiler/rustc)`

Caused by:
  failed to load source for dependency `literal-escaper`

Caused by:
  Unable to update /home/r/.rustup/toolchains/miri/lib/rustlib/rustc-src/rust/library/literal-escaper

Caused by:
  failed to read `/home/r/.rustup/toolchains/miri/lib/rustlib/rustc-src/rust/library/literal-escaper/Cargo.toml`

Caused by:
  No such file or directory (os error 2)

This is caused by #136355: compiler crates cannot have path dependencies outside the compiler folder.

Cc @GuillaumeGomez @Amanieu

[GuillaumeGomez]

cc @rust-lang/bootstrap

[Kobzol]

Hmm, I'm not sure if we want to ship library sources in the rustc-dev rustup component (which contains only the compiler sources currently).

[Kobzol]

But I guess why not? Anyone have any concerns about doing that?

[bjorn3]

It should also be possible to add just library/literal-escaper, right?

[Amanieu]

One possible solution:

• rustc_lexer should not depend on literal-escaper via a Cargo.toml dependency, it should instead import it directly with extern crate literal_escaper to get the one from the library sysroot.
• For cfg(bootstrap) where literal-escaper is not available in the library sysroot just use #[path] to include the code directly.

[Veykril]

rustc_lexer should not depend on literal-escaper via a Cargo.toml dependency, it should instead import it directly with extern crate literal_escaper to get the one from the library sysroot.

This will break rustc_lexer from compiling on stable I believe. Not that it matters I suppose as the change done here in general makes it impossible to publish the crate to crates-io now for rust-analyzer :/

[Kobzol]

It should also be possible to add just library/literal-escaper, right?

Sure, we could selectively add only the required crates.

[bjorn3]

Another option would be to publish literal-escaper to crates.io. That would also allow proc-macro2 to use it. proc-macro2 needs to support not using libproc_macro, so it can't unconditionally use literal escaping through libproc_macro.

[GuillaumeGomez]

Publishing on crates.io seems like the simplest solution. Should I do it?

[Kobzol]

That doesn't sound like the simplest solution to me, tbh 😅 Unless the code is done and will never ever change? Otherwise we should probably have some setup for deploying it, managing the crates.io token, figuring out who will own the crate etc. (it probably shouldn't be owned by your GH account).

[GuillaumeGomez]

Very good point. Then I'll keep quiet in the back and step in if my help is needed. 😆

[RalfJung]

the change done here in general makes it impossible to publish the crate to crates-io now for rust-analyzer :/

So this change broke not just Miri but also RA?

I've posted a revert to avoid blocking multiple tools while we figure out a proper solution: #138661.

[GuillaumeGomez]

Just a small thing if publishing to crates.io is a possibility: we have a crate in rustdoc (https://crates.io/crates/rustdoc-json) that is published with rights given to https://cargo-public-api. Isn't it possible to do the same here?

[Veykril]

So this change broke not just Miri but also RA?

The change prevents us from doing subtree syncs, so technically yes (though there is another issue right now that prevents us from syncing until 1.86 releases)

[Kobzol]

I guess that the token wouldn't be such an issue, but rather the deployment strategy, as we currently don't have workflows or secrets configured for deploying crates from the rust-lang/rust repository. rustdoc-json is in a separate repository, where it is simpler.

I suppose that we should enumerate the possible solutions here, as it seems that there were also other concerns raised above, which would not be solved even by crates.io upload.

[GuillaumeGomez]

So the listed solutions so far:

• Add library/literal-escaper to rustup components
• Import literal-escaper with extern crate in rustc_lexer (seems to create issues with r-a though)
• Publish to crates.io

Another option would be to publish literal-escaper to crates.io. That would also allow proc-macro2 to use it. proc-macro2 needs to support not using libproc_macro, so it can't unconditionally use literal escaping through libproc_macro.

The point was for proc-macro2 to be able to use the new methods from Lit, so no need for them to import literal-escaper directly. But until this new feature is stabilized, I suppose it would allow to make a smoother transition.

[bjorn3]

I guess that the token wouldn't be such an issue, but rather the deployment strategy, as we currently don't have workflows or secrets configured for deploying crates from the rust-lang/rust repository.

I think literal-escaper would have to be moved to another repo if we want to publish to crates.io either way. Otherwise any breaking change to literal-escaper would require first changing it for proc_macro, getting the PR merged, publishing to crates.io and finally opening a PR to change rustc_lexer to use the newly published version.

[Kobzol]

I think that we could solve that with https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#multiple-locations? But even then I would probably also prefer for it to be a separate repo.

[GuillaumeGomez]

Having it into a separate repository would make things a lot easier so I also prefer this solution.

[bjorn3]

I think that we could solve that with https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#multiple-locations?

That doesn't work for rustc-dev. That functionality works by cargo stripping path = "..." when publishing to a registry. For as long as you build a local version (like from rustc-dev) it will use path = "..." instead.

[GuillaumeGomez]

I created https://github.com/GuillaumeGomez/literal-escaper. So if we agree on publishing it to crates.io, we can move it to the rust-lang organization.

[RalfJung]

Add library/literal-escaper to rustup components

That also won't help for RA I think.

crates.io is probably the only option for sharing a dependency between library and compiler.

[GuillaumeGomez]

Seems like crates.io it is. @Kobzol is gonna open a thread on zulip t-infra channel for the discussion to continue there.