Auto merge of rust-lang#93176 - danielhenrymantilla:stack-pinning-macro, r=m-ou-se

Add a stack-`pin!`-ning macro to `core::pin`.

  - rust-lang#93178

`pin!` allows pinning a value to the stack. Thanks to being implemented in the stdlib, which gives access to `macro` macros, and to the private `.pointer` field of the `Pin` wrapper, [it was recently discovered](https://rust-lang.zulipchat.com/#narrow/stream/187312-wg-async-foundations/topic/pin!.20.E2.80.94.20the.20.22definitive.22.20edition.20.28a.20rhs-compatible.20pin-nin.2E.2E.2E/near/268731241) ([archive link](https://zulip-archive.rust-lang.org/stream/187312-wg-async-foundations/topic/A.20rhs-compatible.20pin-ning.20macro.html#268731241)), contrary to popular belief, that it is actually possible to implement and feature such a macro:

```rust
let foo: Pin<&mut PhantomPinned> = pin!(PhantomPinned);
stuff(foo);
```
or, directly:

```rust
stuff(pin!(PhantomPinned));
```

  - For context, historically, this used to require one of the two following syntaxes:

      - ```rust
        let foo = PhantomPinned;
        pin!(foo);
        stuff(foo);
        ```

      -  ```rust
         pin! {
             let foo = PhantomPinned;
         }
         stuff(foo);
         ```

This macro thus allows, for instance, doing things like:

```diff
fn block_on<T>(fut: impl Future<Output = T>) -> T {
    // Pin the future so it can be polled.
-   let mut fut = Box::pin(fut);
+   let mut fut = pin!(fut);

    // Create a new context to be passed to the future.
    let t = thread::current();
    let waker = Arc::new(ThreadWaker(t)).into();
    let mut cx = Context::from_waker(&waker);

    // Run the future to completion.
    loop {
        match fut.as_mut().poll(&mut cx) {
            Poll::Ready(res) => return res,
            Poll::Pending => thread::park(),
        }
    }
}
```

  - _c.f._, https://doc.rust-lang.org/1.58.1/alloc/task/trait.Wake.html

And so on, and so forth.

I don't think such an API can get better than that, barring full featured language support (`&pin` references or something), so I see no reason not to start experimenting with featuring this in the stdlib already 🙂

  - cc `@rust-lang/wg-async-foundations` \[EDIT: this doesn't seem to have pinged anybody 😩, thanks `@yoshuawuyts` for the real ping\]

r? `@joshtriplett`

___

# Docs preview

https://user-images.githubusercontent.com/9920355/150605731-1f45c2eb-c9b0-4ce3-b17f-2784fb75786e.mp4

___

# Implementation

The implementation ends up being dead simple (so much it's embarrassing):

```rust
pub macro pin($value:expr $(,)?) {
    Pin { pointer: &mut { $value } }
}
```

_and voilà_!

  - The key for it working lies in [the rules governing the scope of anonymous temporaries](https://doc.rust-lang.org/1.58.1/reference/destructors.html#temporary-lifetime-extension).

<details><summary>Comments and context</summary>

This is `Pin::new_unchecked(&mut { $value })`, so, for starters, let's
review such a hypothetical macro (that any user-code could define):
```rust
macro_rules! pin {( $value:expr ) => (
    match &mut { $value } { at_value => unsafe { // Do not wrap `$value` in an `unsafe` block.
        $crate::pin::Pin::<&mut _>::new_unchecked(at_value)
    }}
)}
```

Safety:
  - `type P = &mut _`. There are thus no pathological `Deref{,Mut}` impls that would break `Pin`'s invariants.
  - `{ $value }` is braced, making it a _block expression_, thus **moving** the given `$value`, and making it _become an **anonymous** temporary_.
    By virtue of being anonynomous, it can no longer be accessed, thus preventing any attemps to `mem::replace` it or `mem::forget` it, _etc._

This gives us a `pin!` definition that is sound, and which works, but only in certain scenarios:

  - If the `pin!(value)` expression is _directly_ fed to a function call:
    `let poll = pin!(fut).poll(cx);`

  - If the `pin!(value)` expression is part of a scrutinee:

    ```rust
    match pin!(fut) { pinned_fut => {
        pinned_fut.as_mut().poll(...);
        pinned_fut.as_mut().poll(...);
    }} // <- `fut` is dropped here.
    ```

Alas, it doesn't work for the more straight-forward use-case: `let` bindings.

```rust
let pinned_fut = pin!(fut); // <- temporary value is freed at the end of this statement
pinned_fut.poll(...) // error[E0716]: temporary value dropped while borrowed
                     // note: consider using a `let` binding to create a longer lived value
```

  - Issues such as this one are the ones motivating rust-lang/rfcs#66

This makes such a macro incredibly unergonomic in practice, and the reason most macros out there had to take the path of being a statement/binding macro (_e.g._, `pin!(future);`) instead of featuring the more intuitive ergonomics of an expression macro.

Luckily, there is a way to avoid the problem. Indeed, the problem stems from the fact that a temporary is dropped at the end of its enclosing statement when it is part of the parameters given to function call, which has precisely been the case with our `Pin::new_unchecked()`!

For instance,

```rust
let p = Pin::new_unchecked(&mut <temporary>);
```

becomes:

```rust
let p = { let mut anon = <temporary>; &mut anon };
```

However, when using a literal braced struct to construct the value, references to temporaries can then be taken. This makes Rust change the lifespan of such temporaries so that they are, instead, dropped _at the end of the enscoping block_.

For instance,
```rust
let p = Pin { pointer: &mut <temporary> };
```

becomes:

```rust
let mut anon = <temporary>;
let p = Pin { pointer: &mut anon };
```

which is *exactly* what we want.

Finally, we don't hit problems _w.r.t._ the privacy of the `pointer` field, or the unqualified `Pin` name, thanks to `decl_macro`s being _fully_ hygienic (`def_site` hygiene).

</details>

___

# TODO

  - [x] Add compile-fail tests with attempts to break the `Pin` invariants thanks to the macro (_e.g._, try to access the private `.pointer` field, or see what happens if such a pin is used outside its enscoping scope (borrow error));
  - [ ] Follow-up stuff:
      - [ ] Try to experiment with adding `pin!` to the prelude: this may require to be handled with some extra care, as it may lead to issues reminiscent of those of `assert_matches!`: rust-lang#82913
      - [x] Create the tracking issue.

Author: bors

compiler/rustc_feature/src/active.rs

@@ -161,6 +161,9 @@ declare_features! (
     (active, staged_api, "1.0.0", None, None),
     /// Added for testing E0705; perma-unstable.
     (active, test_2018_feature, "1.31.0", None, Some(Edition::Edition2018)),
+    /// Allows non-`unsafe` —and thus, unsound— access to `Pin` constructions.
+    /// Marked `incomplete` since perma-unstable and unsound.
+    (incomplete, unsafe_pin_internals, "1.60.0", None, None),
     /// Use for stable + negative coherence and strict coherence depending on trait's
     /// rustc_strict_coherence value.
     (active, with_negative_coherence, "1.60.0", None, None),

compiler/rustc_span/src/symbol.rs

@@ -1455,6 +1455,7 @@ symbols! {
         unsafe_block_in_unsafe_fn,
         unsafe_cell,
         unsafe_no_drop_flag,
+        unsafe_pin_internals,
         unsize,
         unsized_fn_params,
         unsized_locals,

library/core/src/pin.rs

@@ -406,7 +406,14 @@ use crate::ops::{CoerceUnsized, Deref, DerefMut, DispatchFromDyn, Receiver};
 #[repr(transparent)]
 #[derive(Copy, Clone)]
 pub struct Pin<P> {
-    pointer: P,
+    // FIXME(#93176): this field is made `#[unstable] #[doc(hidden)] pub` to:
+    //   - deter downstream users from accessing it (which would be unsound!),
+    //   - let the `pin!` macro access it (such a macro requires using struct
+    //     literal syntax in order to benefit from lifetime extension).
+    // Long-term, `unsafe` fields or macro hygiene are expected to offer more robust alternatives.
+    #[unstable(feature = "unsafe_pin_internals", issue = "none")]
+    #[doc(hidden)]
+    pub pointer: P,
 }
 
 // The following implementations aren't derived in order to avoid soundness
@@ -909,3 +916,243 @@ impl<P, U> CoerceUnsized<Pin<U>> for Pin<P> where P: CoerceUnsized<U> {}
 
 #[stable(feature = "pin", since = "1.33.0")]
 impl<P, U> DispatchFromDyn<Pin<U>> for Pin<P> where P: DispatchFromDyn<U> {}
+
+/// Constructs a <code>[Pin]<[&mut] T></code>, by pinning[^1] a `value: T` _locally_[^2].
+///
+/// Unlike [`Box::pin`], this does not involve a heap allocation.
+///
+/// [^1]: If the (type `T` of the) given value does not implement [`Unpin`], then this
+/// effectively pins the `value` in memory, where it will be unable to be moved.
+/// Otherwise, <code>[Pin]<[&mut] T></code> behaves like <code>[&mut] T</code>, and operations such
+/// as [`mem::replace()`][crate::mem::replace] will allow extracting that value, and therefore,
+/// moving it.
+/// See [the `Unpin` section of the `pin` module][self#unpin] for more info.
+///
+/// [^2]: This is usually dubbed "stack"-pinning. And whilst local values are almost always located
+/// in the stack (_e.g._, when within the body of a non-`async` function), the truth is that inside
+/// the body of an `async fn` or block —more generally, the body of a generator— any locals crossing
+/// an `.await` point —a `yield` point— end up being part of the state captured by the `Future` —by
+/// the `Generator`—, and thus will be stored wherever that one is.
+///
+/// ## Examples
+///
+/// ### Basic usage
+///
+/// ```rust
+/// #![feature(pin_macro)]
+/// # use core::marker::PhantomPinned as Foo;
+/// use core::pin::{pin, Pin};
+///
+/// fn stuff(foo: Pin<&mut Foo>) {
+///     // …
+///     # let _ = foo;
+/// }
+///
+/// let pinned_foo = pin!(Foo { /* … */ });
+/// stuff(pinned_foo);
+/// // or, directly:
+/// stuff(pin!(Foo { /* … */ }));
+/// ```
+///
+/// ### Manually polling a `Future` (wihout `Unpin` bounds)
+///
+/// ```rust
+/// #![feature(pin_macro)]
+/// use std::{
+///     future::Future,
+///     pin::pin,
+///     task::{Context, Poll},
+///     thread,
+/// };
+/// # use std::{sync::Arc, task::Wake, thread::Thread};
+///
+/// # /// A waker that wakes up the current thread when called.
+/// # struct ThreadWaker(Thread);
+/// #
+/// # impl Wake for ThreadWaker {
+/// #     fn wake(self: Arc<Self>) {
+/// #         self.0.unpark();
+/// #     }
+/// # }
+/// #
+/// /// Runs a future to completion.
+/// fn block_on<Fut: Future>(fut: Fut) -> Fut::Output {
+///     let waker_that_unparks_thread = // …
+///         # Arc::new(ThreadWaker(thread::current())).into();
+///     let mut cx = Context::from_waker(&waker_that_unparks_thread);
+///     // Pin the future so it can be polled.
+///     let mut pinned_fut = pin!(fut);
+///     loop {
+///         match pinned_fut.as_mut().poll(&mut cx) {
+///             Poll::Pending => thread::park(),
+///             Poll::Ready(res) => return res,
+///         }
+///     }
+/// }
+/// #
+/// # assert_eq!(42, block_on(async { 42 }));
+/// ```
+///
+/// ### With `Generator`s
+///
+/// ```rust
+/// #![feature(generators, generator_trait, pin_macro)]
+/// use core::{
+///     ops::{Generator, GeneratorState},
+///     pin::pin,
+/// };
+///
+/// fn generator_fn() -> impl Generator<Yield = usize, Return = ()> /* not Unpin */ {
+///  // Allow generator to be self-referential (not `Unpin`)
+///  // vvvvvv        so that locals can cross yield points.
+///     static || {
+///         let foo = String::from("foo"); // --+
+///         yield 0;                         // | <- crosses yield point!
+///         println!("{}", &foo); // <----------+
+///         yield foo.len();
+///     }
+/// }
+///
+/// fn main() {
+///     let mut generator = pin!(generator_fn());
+///     match generator.as_mut().resume(()) {
+///         GeneratorState::Yielded(0) => {},
+///         _ => unreachable!(),
+///     }
+///     match generator.as_mut().resume(()) {
+///         GeneratorState::Yielded(3) => {},
+///         _ => unreachable!(),
+///     }
+///     match generator.resume(()) {
+///         GeneratorState::Yielded(_) => unreachable!(),
+///         GeneratorState::Complete(()) => {},
+///     }
+/// }
+/// ```
+///
+/// ## Remarks
+///
+/// Precisely because a value is pinned to local storage, the resulting <code>[Pin]<[&mut] T></code>
+/// reference ends up borrowing a local tied to that block: it can't escape it.
+///
+/// The following, for instance, fails to compile:
+///
+/// ```rust,compile_fail
+/// #![feature(pin_macro)]
+/// use core::pin::{pin, Pin};
+/// # use core::{marker::PhantomPinned as Foo, mem::drop as stuff};
+///
+/// let x: Pin<&mut Foo> = {
+///     let x: Pin<&mut Foo> = pin!(Foo { /* … */ });
+///     x
+/// }; // <- Foo is dropped
+/// stuff(x); // Error: use of dropped value
+/// ```
+///
+/// <details><summary>Error message</summary>
+///
+/// ```console
+/// error[E0716]: temporary value dropped while borrowed
+///   --> src/main.rs:9:28
+///    |
+/// 8  | let x: Pin<&mut Foo> = {
+///    |     - borrow later stored here
+/// 9  |     let x: Pin<&mut Foo> = pin!(Foo { /* … */ });
+///    |                            ^^^^^^^^^^^^^^^^^^^^^ creates a temporary which is freed while still in use
+/// 10 |     x
+/// 11 | }; // <- Foo is dropped
+///    | - temporary value is freed at the end of this statement
+///    |
+///    = note: consider using a `let` binding to create a longer lived value
+/// ```
+///
+/// </details>
+///
+/// This makes [`pin!`] **unsuitable to pin values when intending to _return_ them**. Instead, the
+/// value is expected to be passed around _unpinned_ until the point where it is to be consumed,
+/// where it is then useful and even sensible to pin the value locally using [`pin!`].
+///
+/// If you really need to return a pinned value, consider using [`Box::pin`] instead.
+///
+/// On the other hand, pinning to the stack[<sup>2</sup>](#fn2) using [`pin!`] is likely to be
+/// cheaper than pinning into a fresh heap allocation using [`Box::pin`]. Moreover, by virtue of not
+/// even needing an allocator, [`pin!`] is the main non-`unsafe` `#![no_std]`-compatible [`Pin`]
+/// constructor.
+///
+/// [`Box::pin`]: ../../std/boxed/struct.Box.html#method.pin
+#[unstable(feature = "pin_macro", issue = "93178")]
+#[rustc_macro_transparency = "semitransparent"]
+#[allow_internal_unstable(unsafe_pin_internals)]
+pub macro pin($value:expr $(,)?) {
+    // This is `Pin::new_unchecked(&mut { $value })`, so, for starters, let's
+    // review such a hypothetical macro (that any user-code could define):
+    //
+    // ```rust
+    // macro_rules! pin {( $value:expr ) => (
+    //     match &mut { $value } { at_value => unsafe { // Do not wrap `$value` in an `unsafe` block.
+    //         $crate::pin::Pin::<&mut _>::new_unchecked(at_value)
+    //     }}
+    // )}
+    // ```
+    //
+    // Safety:
+    //   - `type P = &mut _`. There are thus no pathological `Deref{,Mut}` impls
+    //     that would break `Pin`'s invariants.
+    //   - `{ $value }` is braced, making it a _block expression_, thus **moving**
+    //     the given `$value`, and making it _become an **anonymous** temporary_.
+    //     By virtue of being anonynomous, it can no longer be accessed, thus
+    //     preventing any attemps to `mem::replace` it or `mem::forget` it, _etc._
+    //
+    // This gives us a `pin!` definition that is sound, and which works, but only
+    // in certain scenarios:
+    //   - If the `pin!(value)` expression is _directly_ fed to a function call:
+    //     `let poll = pin!(fut).poll(cx);`
+    //   - If the `pin!(value)` expression is part of a scrutinee:
+    //     ```rust
+    //     match pin!(fut) { pinned_fut => {
+    //         pinned_fut.as_mut().poll(...);
+    //         pinned_fut.as_mut().poll(...);
+    //     }} // <- `fut` is dropped here.
+    //     ```
+    // Alas, it doesn't work for the more straight-forward use-case: `let` bindings.
+    // ```rust
+    // let pinned_fut = pin!(fut); // <- temporary value is freed at the end of this statement
+    // pinned_fut.poll(...) // error[E0716]: temporary value dropped while borrowed
+    //                      // note: consider using a `let` binding to create a longer lived value
+    // ```
+    //   - Issues such as this one are the ones motivating https://github.com/rust-lang/rfcs/pull/66
+    //
+    // This makes such a macro incredibly unergonomic in practice, and the reason most macros
+    // out there had to take the path of being a statement/binding macro (_e.g._, `pin!(future);`)
+    // instead of featuring the more intuitive ergonomics of an expression macro.
+    //
+    // Luckily, there is a way to avoid the problem. Indeed, the problem stems from the fact that a
+    // temporary is dropped at the end of its enclosing statement when it is part of the parameters
+    // given to function call, which has precisely been the case with our `Pin::new_unchecked()`!
+    // For instance,
+    // ```rust
+    // let p = Pin::new_unchecked(&mut <temporary>);
+    // ```
+    // becomes:
+    // ```rust
+    // let p = { let mut anon = <temporary>; &mut anon };
+    // ```
+    //
+    // However, when using a literal braced struct to construct the value, references to temporaries
+    // can then be taken. This makes Rust change the lifespan of such temporaries so that they are,
+    // instead, dropped _at the end of the enscoping block_.
+    // For instance,
+    // ```rust
+    // let p = Pin { pointer: &mut <temporary> };
+    // ```
+    // becomes:
+    // ```rust
+    // let mut anon = <temporary>;
+    // let p = Pin { pointer: &mut anon };
+    // ```
+    // which is *exactly* what we want.
+    //
+    // See https://doc.rust-lang.org/1.58.1/reference/destructors.html#temporary-lifetime-extension
+    // for more info.
+    $crate::pin::Pin::<&mut _> { pointer: &mut { $value } }
+}

library/core/tests/lib.rs

@@ -45,6 +45,7 @@
 #![feature(inline_const)]
 #![feature(is_sorted)]
 #![feature(pattern)]
+#![feature(pin_macro)]
 #![feature(sort_internals)]
 #![feature(slice_take)]
 #![feature(maybe_uninit_uninit_array)]
@@ -122,6 +123,7 @@ mod ops;
 mod option;
 mod pattern;
 mod pin;
+mod pin_macro;
 mod ptr;
 mod result;
 mod simd;

library/core/tests/pin_macro.rs

@@ -0,0 +1,33 @@
+// edition:2021
+use core::{
+    marker::PhantomPinned,
+    mem::{drop as stuff, transmute},
+    pin::{pin, Pin},
+};
+
+#[test]
+fn basic() {
+    let it: Pin<&mut PhantomPinned> = pin!(PhantomPinned);
+    stuff(it);
+}
+
+#[test]
+fn extension_works_through_block() {
+    let it: Pin<&mut PhantomPinned> = { pin!(PhantomPinned) };
+    stuff(it);
+}
+
+#[test]
+fn extension_works_through_unsafe_block() {
+    // "retro-type-inference" works as well.
+    let it: Pin<&mut PhantomPinned> = unsafe { pin!(transmute(())) };
+    stuff(it);
+}
+
+#[test]
+fn unsize_coercion() {
+    let slice: Pin<&mut [PhantomPinned]> = pin!([PhantomPinned; 2]);
+    stuff(slice);
+    let dyn_obj: Pin<&mut dyn Send> = pin!([PhantomPinned; 2]);
+    stuff(dyn_obj);
+}

src/test/rustdoc-js-std/typed-query.js

@@ -8,5 +8,7 @@ const EXPECTED = {
         { 'path': 'std', 'name': 'eprint' },
         { 'path': 'std', 'name': 'println' },
         { 'path': 'std', 'name': 'eprintln' },
+        { 'path': 'std::pin', 'name': 'pin' },
+        { 'path': 'core::pin', 'name': 'pin' },
     ],
 };

src/test/ui/feature-gates/feature-gate-unsafe_pin_internals.rs

@@ -0,0 +1,17 @@
+// edition:2018
+#![forbid(incomplete_features, unsafe_code)]
+#![feature(unsafe_pin_internals)]
+//~^ ERROR the feature `unsafe_pin_internals` is incomplete and may not be safe to use
+
+use core::{marker::PhantomPinned, pin::Pin};
+
+/// The `unsafe_pin_internals` is indeed unsound.
+fn non_unsafe_pin_new_unchecked<T>(pointer: &mut T) -> Pin<&mut T> {
+    Pin { pointer }
+}
+
+fn main() {
+    let mut self_referential = PhantomPinned;
+    let _: Pin<&mut PhantomPinned> = non_unsafe_pin_new_unchecked(&mut self_referential);
+    core::mem::forget(self_referential); // move and disable drop glue!
+}

src/test/ui/feature-gates/feature-gate-unsafe_pin_internals.stderr

@@ -0,0 +1,14 @@
+error: the feature `unsafe_pin_internals` is incomplete and may not be safe to use and/or cause compiler crashes
+  --> $DIR/feature-gate-unsafe_pin_internals.rs:3:12
+   |
+LL | #![feature(unsafe_pin_internals)]
+   |            ^^^^^^^^^^^^^^^^^^^^
+   |
+note: the lint level is defined here
+  --> $DIR/feature-gate-unsafe_pin_internals.rs:2:11
+   |
+LL | #![forbid(incomplete_features, unsafe_code)]
+   |           ^^^^^^^^^^^^^^^^^^^
+
+error: aborting due to previous error
+

src/test/ui/pin-macro/cant_access_internals.rs

@@ -0,0 +1,13 @@
+// edition:2018
+#![feature(pin_macro)]
+
+use core::{
+    marker::PhantomPinned,
+    mem,
+    pin::{pin, Pin},
+};
+
+fn main() {
+    let mut phantom_pinned = pin!(PhantomPinned);
+    mem::take(phantom_pinned.pointer); //~ ERROR use of unstable library feature 'unsafe_pin_internals'
+}

src/test/ui/pin-macro/cant_access_internals.stderr

@@ -0,0 +1,11 @@
+error[E0658]: use of unstable library feature 'unsafe_pin_internals'
+  --> $DIR/cant_access_internals.rs:12:15
+   |
+LL |     mem::take(phantom_pinned.pointer);
+   |               ^^^^^^^^^^^^^^^^^^^^^^
+   |
+   = help: add `#![feature(unsafe_pin_internals)]` to the crate attributes to enable
+
+error: aborting due to previous error
+
+For more information about this error, try `rustc --explain E0658`.