Replace evaluated cfg_attr in AST with a placeholder attribute for accurate span tracking

[estebank]

Previously, when evaluating a #[cfg_attr(..)] to false, the entire attribute was removed from the AST. Afterwards, we insert in its place a #[rustc-cfg-placeholder] attribute so that checks for attributes can still know about their placement. This is particularly relevant when we suggest removing items with cfg_attrs (fix #56328). We use rustc-cfg-placeholder as it is an ident that can't be written by the end user to begin with. We tweak the wording of the existing "unused extern crate" lint.

warning: unused `extern crate`
  --> $DIR/removing-extern-crate.rs:9:1
   |
LL | extern crate removing_extern_crate as foo;
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ unused
   |
note: the lint level is defined here
  --> $DIR/removing-extern-crate.rs:6:9
   |
LL | #![warn(rust_2018_idioms)]
   |         ^^^^^^^^^^^^^^^^
   = note: `#[warn(unused_extern_crates)]` implied by `#[warn(rust_2018_idioms)]`
help: remove the unused `extern crate`
   |
LL - #[cfg_attr(test, macro_use)]
LL - extern crate removing_extern_crate as foo;
   |

r? @petrochenkov

[estebank]

I thought this was a rendering bug, but it is because of the //~ removed comment.

[rustbot]

Some changes occurred in src/tools/clippy

cc @rust-lang/clippy

[petrochenkov]

Are the placeholder attributes visible to proc macros?
Or they somehow produce empty token streams (from the proc macro point of view) and exist only in AST?
I'm also interested whether these placeholders mess up the token collection process, because removing false #[cfg(_attr)]s from the token stream representation is probably the most involved part of the cfg expansion.

Something like

#[cfg_eval]
#[my_proc_macro]
#[cfg_attr(all(), my_attr)]
struct S {}

can be used for testing, the cfg_eval macro will eagerly expand all cfg(_attr) attributes so macros like my_proc_macro can see only their results.

If it's possible to introduce placeholders like this (on best effort basis) only in AST, without making them observable in token streams in any way, then they should be able to solve some other problems as well, like removing expanded #[cfg(TRUE)]s from token streams without damaging rustdoc and clippy.

Upd: if these placeholders are visible to proc macros then we cannot add them, because they are internal details that should not be public and stable.

[estebank]

With a test

//@ proc-macro: cfg-placeholder.rs
#![feature(cfg_eval)]
#[macro_use] extern crate cfg_placeholder;

#[cfg_eval]
#[my_proc_macro]
#[cfg_attr(FALSE, my_attr1)]
#[cfg_attr(all(), my_attr2)]
struct S {}

and a my_proc_macro and my_attrs that just print what it was passed in, the output was

my_proc_macro:
TokenStream []
TokenStream [
    Punct {
        ch: '#',
        spacing: Alone,
        span: #0 bytes(160..161),
    },
    Group {
        delimiter: Bracket,
        stream: TokenStream [
            Ident {
                ident: "my_attr2",
                span: #0 bytes(178..186),
            },
        ],
        span: #0 bytes(161..188),
    },
    Ident {
        ident: "struct",
        span: #0 bytes(189..195),
    },
    Ident {
        ident: "S",
        span: #0 bytes(196..197),
    },
    Group {
        delimiter: Brace,
        stream: TokenStream [],
        span: #0 bytes(198..200),
    },
]

my_attr2:
TokenStream []
TokenStream [
    Ident {
        ident: "struct",
        span: #0 bytes(189..195),
    },
    Ident {
        ident: "S",
        span: #0 bytes(196..197),
    },
    Group {
        delimiter: Brace,
        stream: TokenStream [],
        span: #0 bytes(198..200),
    },
]

I'm honestly surprised it doesn't seem like we need to do anything else to avoid polluting the TTS. I think it is because of the check for is_cfg_placeholder in Attribute::token_trees, which was needed to be able to build the stdlib (IIRC) in the first place (likely precisely because of this exact problem you anticipated) so I addressed this issue reactively purely trying to get things to a working state to begin with.

[petrochenkov]

Sorry for the long delay.
I think the direction here is right, but I need to do some experiments with this.

[rustbot]

Some changes occurred in compiler/rustc_passes/src/check_attr.rs

cc @jdonszelmann

[estebank]

No problem. Rebased (but kept the separated commits) to hopefully make it easier to review. The recent changes to support cfg attrs on where clauses required yet another tweak, but besides that everything is the same as it was back in December.

[petrochenkov]

I've reimplemented the trace attributes in #138515 in a slightly more principled way, but without including any diagnostic changes from this PR.
@rustbot blocked

[bors]

☔ The latest upstream changes (presumably #138515) made this pull request unmergeable. Please resolve the merge conflicts.

[petrochenkov]

#138515 has landed, unblocking.
@rustbot author