regression processing c-style doc comments in proc macros between 1.46 and 1.47

[ahl]

When a proc macro is applied to an item that has an associated c-style doc comment the contents of the TokenStream changed between 1.46 and 1.47. In particular the surrounding * and newlines were previously elided (mostly), but as of 1.47 they are not.

Code

Here's a little repo I put together to demonstrate and reproduce the issue: https://github.com/ahl/cdoc_regression. It contains a proc macro that looks like this:

#[proc_macro_attribute]
pub fn return_as_is(_attr: TokenStream, item: TokenStream) -> TokenStream {
    for token in item.clone() {
        match token {
            TokenTree::Group(group) if group.delimiter() == Delimiter::Bracket => {
                println!("{:}", group)
            }
            _ => {}
        }
    }
    item
}

Here's the output:

$ cargo +1.46.0 run --example test
   Compiling fun v0.1.0 (/Users/ahl/src/cdoc_regression)
[doc = " This is a multi-"]
[doc = " line comment."]
[doc = " This is another multi-\n line comment."]
    Finished dev [unoptimized + debuginfo] target(s) in 0.30s
     Running `target/debug/examples/test`
$ cargo +1.47.0 run --example test
   Compiling fun v0.1.0 (/Users/ahl/src/cdoc_regression)
[doc = " This is a multi-"]
[doc = " line comment."]
[doc = "\n * This is another multi-\n * line comment.\n "]
    Finished dev [unoptimized + debuginfo] target(s) in 0.27s
     Running `target/debug/examples/test`

In both cases, the first multiline comment is for a ///-style comment and the second is for a /**-style comment.

Version it worked on

rustc 1.46.0 (04488afe3 2020-08-24)
binary: rustc
commit-hash: 04488afe34512aa4c33566eb16d8c912a3ae04f9
commit-date: 2020-08-24
host: x86_64-apple-darwin
release: 1.46.0
LLVM version: 10.0

Version with regression

rustc 1.47.0 (18bf6b4f0 2020-10-07)
binary: rustc
commit-hash: 18bf6b4f01a6feaf7259ba7cdae58031af1b7b39
commit-date: 2020-10-07
host: x86_64-apple-darwin
release: 1.47.0
LLVM version: 11.0

[jyn514]

I tested and this doesn't affect rustdoc, which strips \n * in both versions.

[apiraino]

Assigning P-medium as discussed as part of the Prioritization Working Group procedure and removing I-prioritize.

[Aaron1011]

I think this was caused by #74627 . The new behavior may be intentional - @petrochenkov

[moxian]

Can confirm that it does indeed bisect to #74627 (or maybe #73842 ).

cargo-bisect-rustc output

********************************************************************************
Regression in nightly-2020-08-08
********************************************************************************

fetching https://static.rust-lang.org/dist/2020-08-07/channel-rust-nightly-git-commit-hash.txt
nightly manifest 2020-08-07: 40 B / 40 B [=======================================================] 100.00 % 62.56 KB/s converted 2020-08-07 to 71f8d0c8f1060bbe74100f29cc6f2da63d697c28
fetching https://static.rust-lang.org/dist/2020-08-08/channel-rust-nightly-git-commit-hash.txt
nightly manifest 2020-08-08: 40 B / 40 B [=======================================================] 100.00 % 56.66 KB/s converted 2020-08-08 to 09f4c9f5082f78b0adcee83d3ab4337e000cd28e
looking for regression commit between 2020-08-07 and 2020-08-08
fetching (via remote github) commits from max(71f8d0c8f1060bbe74100f29cc6f2da63d697c28, 2020-08-05) to 09f4c9f5082f78b0adcee83d3ab4337e000cd28e
ending github query because we found starting sha: 71f8d0c8f1060bbe74100f29cc6f2da63d697c28
get_commits_between returning commits, len: 10
  commit[0] 2020-08-06: Auto merge of #75228 - tmiasko:keep-stdout-open, r=ecstatic-morse
  commit[1] 2020-08-07: Auto merge of #75238 - JohnTitor:rollup-llbk0sq, r=JohnTitor
  commit[2] 2020-08-07: Auto merge of #75233 - RalfJung:miri, r=RalfJung
  commit[3] 2020-08-07: Auto merge of #75121 - tmiasko:str-slicing, r=Mark-Simulacrum
  commit[4] 2020-08-07: Auto merge of #75244 - Manishearth:rollup-dzfyjva, r=Manishearth
  commit[5] 2020-08-07: Auto merge of #70052 - Amanieu:hashbrown7, r=Mark-Simulacrum
  commit[6] 2020-08-07: Auto merge of #73842 - euclio:doctest-expn, r=GuillaumeGomez
  commit[7] 2020-08-07: Auto merge of #74627 - petrochenkov:docbeauty2, r=Aaron1011
  commit[8] 2020-08-07: Auto merge of #74821 - oli-obk:const_eval_read_uninit_fast_path, r=wesleywiser
  commit[9] 2020-08-07: Auto merge of #75255 - davidtwco:polymorphisation-symbol-mangling-v0-upvar-closures, r=lcnr
ERROR: no CI builds available between 71f8d0c8f1060bbe74100f29cc6f2da63d697c28 and 09f4c9f5082f78b0adcee83d3ab4337e000cd28e within last 167 days

Narrowing this to a specific commit is somewhat challenging today, 4 years later (specifically I'm having trouble building the old llvm).
Hopefully this info is not too useful anymore since lots of code has changed since anyway. So opportunistically removing the label.

@rustbot labels: -E-needs-bisection

[petrochenkov]

Closing as an intentional change.