rustdoc’s approach to producing output needs to be rethought

[chris-morgan]

rustdoc was built under a single-crate-universe assumption: that the documentation would be just for that crate. While that assumption held, Rust-to-HTML generation worked fine.

This assumption is now broken. We have the implementors/**/*.js technique for trait implementatons; search pulls its information for many crates from one; the order in which you build the documentation of crates matters and—for the Rust repository—make docs produces different output to make -j docs.

It is precept upon precept,
precept upon precept;
line upon line,
line upon line;
here a little,
there a little.

It’s time to stop writing new workarounds and address the basic problem that we have here: rustdoc needs to handle this multiple-crates world intelligently. This means ditching the simple and, alas, simplistic single-pass Rust-to-HTML approach.

I propose in its stead this model:

• First pass, done per crate: extract all the metadata and write it in serialized form, not HTML/JavaScript/whatever. I don’t care what format.
• Second pass, which need only be done after all crates but can be done more frequently without harm: using the metadata, write all the HTML and whatever JavaScript is still necessary (this way, I think it will be for the search only, not the crate-level sidebar and trait implementations stuff).

By default, the second pass should probably still be run, but it can be turned off with a flag like --no-html for efficiency when you do have multiple crates. Or you can run the HTML pass by itself, which will allow meaningful Makefile dependencies in this blasé-with-respect-to-detail example, allowing efficient, consistent and correct make -j behaviour:

doc/index.html: doc/a.json doc/b.json doc/c.json
    rustdoc --html doc/{a,b,c}.json

doc/%.json: src/%/lib.rs
    rustdoc --no-html src/%/lib.rs

[alexcrichton]

The current design of rustdoc has explicitly avoided a second metadata blob which is required for interim processing. The reason for this is that we all of a sudden need to start distributing std.json along with all other libraries, or rustdoc foo.rs doesn't work out-of-the-box.

This problem can be worked around, but this is my largest concern if moving to a interim-json system.

Also, can you explain more what this model buys us? I'm not particularly sold on deterministic make -j output, and it's not clear to me what this would buy us in terms of an improved cross-crate experience.

[steveklabnik]

@chris-morgan can you respond to @alexcrichton 's concerns here?

[tomjakubowski]

Would you still get some benefits by specifying the universe of crates from the get-go, without writing that problematic blob of metadata to disk? It would simplify rustdoc's rendering of cross-crate trait impls and generating a search index I suppose.

[mackwic]

I can't speak for chris, but here are the advantages I can see from emitting a metadata blob:

• it can be used for various other tools other than HTML emitting (eg: automatic blanket impl code generation in editor à la IntellijIdea pop to mind, or completion with embedded documentation à la VisualStudio)
• it can also be used by crates.io in order to check any API breakage and enforce semver, or by the community to evaluate documentation coverage
• it can be used in any organization to embed the rust doc in any other platform (see devdocs.io as an example)
• it enable the community to build new tools which consume the json rather than having to hack on the rustc tree. I think most of the competent frontends guys might be a little bit scared by the rustc repo. :)
• it can help with Separate templates from rustdoc source #29323, Allow documentation to be read from an external file #15470

In order to to smooth the process, rustdoc could embed the std.doc.json and core.doc.json in the binary so that it would work as long as no cli argument provide other jsons. It need the rustc tree to be compiled anyway...

[steveklabnik]

Triage: this is a pain point that we still feel today, especially on Windows. That being said, keeping an issue open to track it isn't really helping; we may or may not change rustdoc to fix this, or a successor to rustdoc might. I don't think this issue is very useful for tracking that, so I'm going to give it a close. Thanks!