Using std::io::{Read, Write, Cursor} in a nostd environment

[roblabla]

I'm surprised there's no ticket for this, so here we go.

It isn't possible in nostd to use std::io::Cursor. That seems to be mostly because it requires std::io::Read, Write, and Seek. However, I'd argue that those traits should also be part of libcore, and only the specific implementations be in libstd. After all, the Read/Write/Seek traits and the Cursor struct don't seem to rely on anything that's part of the stdlib: they don't need allocation, etc...

My use-case is, I have to write data into an array, and treating that array in a file would lead to much more idiomatic code. Furthermore it'd allow using crates like byteorder in nostd environment.

[retep998]

The problem with this, which has been covered in previous issues, is that Read and Write have std::io::Error hardcoded in their method signatures which requires quite a bit of std machinery including the ability to allocate.

[roblabla]

Do you have a link to previous issues ? I tried looking for it, but nothing came up :(.

[retep998]

I... can't find the previous issues. Maybe it was just scattered comments over the place and IRC discussions...

[jD91mZM2]

These traits being absent is the reason rust-core_io exists. Relibc would thank you if core::io became a thing, and I could then continue on my newly started quest to make more crates support no_std.

[burdges]

Anything here requires the std::error::Error trait, which requires std::backtrace::Backtrace, but maybe all that only requires core+alloc. Assuming so..

We should explore replacing std::io::Error with some more flexible error type, vaguely inspired by the SmallError type suggested in rust-lang/rfcs#2820 (comment) that encodes the io::error::Repr enum with some vtable pointer.

In essence, it should act like Box<dyn Error+Send+Sync+'static> except small error types get passed by value.

We could maybe preserve mem::size_of::<SmallError>() being only twice mem::size_of::<usize>() too.

Also, I'm unsure if std::error::Error could feature gate the std::backtrace::Backtrace methods on alloc, but maybe. We must encode Drop inside the errorr::Error trait though, which I suspect either requires magic similar to Box, or else makes Drop a supertrait for errorr::Error. We'd wind up with an Error trait like

pub trait Error: Drop + Debug + Display {
    fn description(&self) -> &str { "description() is deprecated; use Display" }
    fn cause(&self) -> Option<&dyn Error> { self.source() }
    fn source(&self) -> Option<&(dyn Error + 'static)> { None }
    fn type_id(&self, _: private::Internal) -> TypeId where Self: 'static { TypeId::of::<Self>() }
    #[cfg(feature = "alloc")]
    fn backtrace(&self) -> Option<&Backtrace> { None }
}

We must then use wrapper types that handle drop correctly.

#[derive(Clone)]
pub struct BoxError {
    inner: Box<dyn Error+Send+Sync+'static>,
}
// delegate Deref+DerefMut+Drop to inner

impl Drop for BoxError {
    fn drop(&mut self) {
        self.deref_mut().drop()
    }
}

use core::mem;
use core::raw::TraitObject;

const SMALL_ERROR_SIZE : usize = mem::size_of::<usize>();

#[derive(Clone)]
pub struct SmallError {
    vtable: *mut (),
    data: [u8; SMALL_ERROR_SIZE],
}

impl Drop for SmallError {
    fn drop(&mut self) {
        self.deref_mut().drop()
    }
}
impl Send for SmallError {}
impl Sync for SmallError {}

impl SmallError {
    pub fn new<E: Error+Clone+Send+Sync>(err: E) -> SmallError 
    // where mem::size_of::<E>() <= SMALL_ERROR_SIZE  // sigh
    {
        assert!(mem::size_of::<E>() <= SMALL_ERROR_SIZE);
        let data = [0u8; SMALL_ERROR_SIZE],
        core::intrinsics::copy(&err as *const E, &mut data as *mut [u8; SMALL_ERROR_SIZE] as *mut E , 1);
        let vtable = unsafe { mem::transmute::<&mut dyn Error,TraitObject>(&mut dyn err).vtable };
        // mem::forget(err);
        SmallError { vtable, data }
    }
}

impl Deref for SmallError {
    type Target = dyn Error+Send+Sync;  // Should we add +Clone when mutli-trait objects exist
    fn deref(&self) -> &Self::Target {
        let SmallError { vtable, ref data } = self;
        mem::transmute::<TraitObject,&mut dyn Error>(TraitObject { vtable, data })
    }
}

impl DerefMut for SmallError {
    fn deref(&mut self) -> &mut Self::Target {
        let SmallError { vtable, ref data } = self;
        mem::transmute::<TraitObject,&mut dyn Error>(TraitObject { vtable, data })
    }
}

We then need some outer type that calls Drop for BoxError and SmallError, but forwards Error methods, like

pub struct DerefError(&dyn Deref<Target = dyn Error+Send+Sync+'static>+DerefMut+Drop)
impl Drop for DerefError {
    fn drop(&mut self) { self.drop() }
}
impl Error for DerefError {
    // delegate all methods to self.deref()
}

[roblabla]

Backtrace cannot really exist outside std. It is highly dependent on OS features. For instance, to get function names (symbolication step of printing a backtrace), it requires opening the executable as a file and parsing it to access the debug information table, e.g. it requires doing open(argv[0]). Or it might require dlresolve or other kind of libc stuff.

In an ideal world, binaries compiled in no_std could just have an "empty" implementation of Backtraces. I don't think that's possible however, since libstd simply reexports everything in libcore verbatim. It's not possible for libstd to use a different implementation of things than libcore AFAICT.

Unfortunately, moving backtraces to be an exposed interface of std::error::Error seems to permanently kill any possibility of std::error and std::io being available in libcore.

[burdges]

Could we replace &Backtrace with some opaque pub struct BacktraceRef(usize); type that exists libcore? We'd then provide inherent methods for that type only in std, assuming that flies with whatever magic.

[roblabla]

Std cannot (currently) add inherent impls for structs it does not itself define. It plays by the same rule as the rest of the ecosystem when it comes to coherence, there's no magic going on here (modulo some escape hatches for fundamental types that don't apply here). Libstd can implement libstd traits on libcore structs, or implement libcore traits for libstd structs. But it can't add its own inherent impls for (libcore) structs, only the crate defining a type can do that.

The best I can figure out is changing the backtrace function to return a &'a dyn BacktraceTrait and do away entirely with the Backtrace structs (they'd be an implementation detail). This would prevent adding new functionality to backtraces (since they're now a trait that anyone can impl) but it means libcore's Error could return an empty backtrace impl.

Except that's also undesirable because now all the errors living in libcore won't get to use backtraces - only errors in libstd would use the "right" backtrace implementation.

Maybe that'd be fixable using the same mechanism global allocators use? Have some kind of backtrace factory, have libstd give out a default implementation, and ask libcore users to provide their own (probably stubbed) implementation.

[burdges]

I suppose std cannot use features for dependencies like many crates do if we want std to ever actually be the standard library on some future operating system. In other words, if features are additive then std should always have all its features enabled.

I think &dyn BacktraceTrait sounds fine because std::backtrace::Backtrace indicates it'd resemble

trait BacktraceTrait : Debug + Display {
    fn status(&self) -> BacktraceStatus;
}

I still think BacktraceRef could be defined in core, but you define BacktraceTrait in std. Also we've several proposals for extensions like (a) doing inherent methods across distinguished crate boundaries, or (b) allowing similar opaque extern types like you gets from C header files, but your &dyn BacktraceTrait works well enough here I think.

As for implementation, there are several options here:

• We could treat Error special with the trait object tricks I wrote above.
• We could implement those same tricks via some proc macro, and some SmallBoxDyn type, so that other trait objects could benefit form a small Copy types being passed, and do not require even alloc, while larger types get boxed, requiring alloc.
• We could do some SmallBox type that handled the forwarding via compiler magic for the forwarding vtables, but also worked without always passing through trait objects.

[burdges]

I think anyhow::Error already provides no_std wrapper for std::io::Error. At present, it requires alloc but it looks like a reasonable starting point if we want to make this dependency optional.

We'd need to remove the Box in https://docs.rs/anyhow/1.0.26/src/anyhow/lib.rs.html#322-324 which increases the size from one to two usizes.

It appears anyhow::Error already manually implements its trait object internally, which I considered overkill but if they needed it then maybe my suggestions here also do.

We should reimplement the std::error::Error trait so that its new method does not depend upon std.

I think anyhow::Error addresses the Backtrace machinery by depending upon the backtrace crate, so maybe that already works no_std but requires alloc. We need a BacktraceTrait that avoids even needing alloc, as discussed above.

We should also remove the backtrace that anyhow::Error attaches https://docs.rs/anyhow/1.0.25/src/anyhow/error.rs.html#672 because it requires alloc. We could however permit layering one internally when using alloc via another internal wrapper type.

After all this we could then reimplement all traits from std::io and their async cousins. Appears doable although not sure until you try.

[c410-f3r]

#99917 has been merged so is anyone going to create a PR?

[thomcc]

I think it needs a lot more discussion. It returns std::io::Error, which cannot trivially be moved into core, even now that std::error::Error can.

See rust-lang/project-error-handling#11 on some of the challenges there, and discussion if doing so is even desirable. My personal stance is that it should be closer to something like core::io::Write<Error = ...>, as I mentioned in rust-lang/project-error-handling#11 (comment). That would allow keeping useful information in std::io::Error, without regressing performance, etc.

[c410-f3r]

Thank you for the explanation @thomcc.

Here goes a copy-and-paste of library/std/src/io/error.rs with the new error_in_core feature -> https://play.rust-lang.org/?version=nightly&mode=debug&edition=2021&gist=e30835b743ae49d77feb0c1f7c89d934

As seen above, the only thing preventing the inclusion of std::io structures and traits into the alloc crate (not core but that would still enable #[no_std]) is the ErrorData::Os(i32) variant and its std::sys::* calls.

Not sure if the the new provider API can help here but if not, then core::io::Write<Error = ...> seems like the only alternative.

[RaitoBezarius]

I think it needs a lot more discussion. It returns std::io::Error, which cannot trivially be moved into core, even now that std::error::Error can.

See rust-lang/project-error-handling#11 on some of the challenges there, and discussion if doing so is even desirable. My personal stance is that it should be closer to something like core::io::Write<Error = ...>, as I mentioned in rust-lang/project-error-handling#11 (comment). That would allow keeping useful information in std::io::Error, without regressing performance, etc.

I feel like, there could be already an alloc feature, then we can go further and have the core::io::Write<_> thing?

[Altair-Bueno]

Now that core::error::Error lives on core... is there anything blocking this?

[Lokathor]

Changing the error type would be a break. Moving the existing error type likely still has problems, which are solvable but which aren't actually solved yet.

[LunarLambda]

Yeah, the issue is the io::Error type, not the error::Error trait.

Currently, the former requires alloc/Box, making it difficult to move into core.

[crlf0710]

Just out of curiosity, everything will work fine if they're moved into alloc crate, right?

[dpc]

The fact that io::{Write, Read} do not live in core, makes it impossible to write general purpose io-touching code that would work for both std and no_std applications, and is a huge structural rift in the ecosystem in this regard.

I'm surprised that there are not that many people complaining about it.

In 2025, 7 years after this issue was opened, what do people to workaround it? What's the go to solution?

Are there any sights on addressing this issue? Any conclusions on what approach would be ideal? Could we maybe get new core::io::{Write, Read} traits that e.g. would be generic over the error type so one could write their nostd/std code around that, and then have std::io be special case of that (possibly even with some hacker/special casing to make it work)? Or would that not work / what else could? I'm trying to collect all the apspects and status of the problem as it is causing great suffering to projects I'm involved in.

[programmerjake]

Are there any sights on addressing this issue? Any conclusions on what approach would be ideal?

I did create #116685 as a demo of how std::io::Error could be moved to core, which would allow moving all the core I/O traits to core. If that's the approach we want to take, feel free to create your own PR based on that PR.

[fintelia]

Read::read_to_end takes a Vec as an argument so it'll require alloc at a minimum

[dpc]

Another reason to just have separate io traits in core.

What no_std code needs is an ability to express generic and composable read and write primitives, generic over the error, which would avoiding touching allocation. I don't think std::io is suitable for that, because it's a OS-level IO primitive.

[fogti]

I haven't seen ciborium mentioned yet, which has its own I/O abstraction layer for no_std/no_alloc occasions. https://github.com/enarx/ciborium/tree/main/ciborium-io

[StephanvanSchaik]

I'm surprised that there are not that many people complaining about it.

I might have missed this issue when I looked for it in the past, not sure. Part of the problem is that I needed a solution now (and about three years ago), but I would be very grateful to see some progress on core::io.

In 2025, 7 years after this issue was opened, what do people to workaround it? What's the go to solution?

So I am actually dealing with interesting environments that either need a custom std or that need to use no_std, but have enough to implement most of what is in the Rust standard library.

One such environment is Intel SGX, where you don't have actual I/O inside enclaves, but can implement OCALLs to interface the operating system indirectly through the untrusted userspace application loading your enclave. Thus, you can expose enough of the libc API to the enclave to implement most of the Rust standard library. From what I know Fortanix went the route of having a custom target with its own implementation of the Rust standard library (thus avoiding the need for no_std). However, if you want to build enclaves directly with the Intel SGX SDK, you currently have to do this with no_std. I have some small examples of how to do this without Rust nightly, and I think Teaclave went this route as well but might still depend on Rust nightly.

Another such environment is DynamoRIO (and Intel Pin), which is an instrumentation framework. This means that you can write DynamoRIO clients which can trace programs as they execute and even manipulate the code. However, as the program should not be aware of the existence of such DynamoRIO clients, the clients have to adhere to certain restrictions, including that they cannot use libc directly (thus, by extension the Rust standard library). Also, because using libc directly can alter the program's behavior and memory layout. However, as you often need libc in DynamoRIO clients, DynamoRIO provides its own API that allows you to access the file system, create threads, etc. without possibly altering the program's behavior.

I have successfully implemented DynamoRIO bindings for Rust using no_std and I have abstracted away DynamoRIO's API in a drstd crate. For the I/O-part, I originally used acid_io. Unfortunately, that crate seems to pull in Rust's standard library (probably due to lack of maintenance) which makes it unusable for dynamorio-rs, so I recently switched to no_std_io instead.

I have also looked at embedded-io and ciborium-io, but they are just less complete compared to the no_std_io crate. This might be fine for embedded I/O, but it doesn't make as much sense if you can actually implement most of Rust's standard library and just want to offer a batteries-included experience. This is also why I like to create awareness of such environments where you can actually implement a standard library (albeit with some caveats), but might need no_std because the original standard library cannot be used (or a custom target with its own implementation of the standard library), and it wouldn't surprise me if more such environments exists (e.g., it might be possible to implement some part of Rust's standard library for UEFI applications, but there might be some caveats too). As many people, understandably, tend to think of actual no_std environments like embedded hardware instead. Just adding my few cents FWIW.

[Altair-Bueno]

I haven't seen ciborium mentioned yet, which has its own I/O abstraction layer for no_std/no_alloc occasions. enarx/ciborium@main/ciborium-io

Relevant xkcd

We do not want to use an alternative. We want the standard implementation to be actually standard.