Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] rustdoc testing: Further improve chapters and sections #2298

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from
Draft

[WIP] rustdoc testing: Further improve chapters and sections #2298

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 4 additions & 17 deletions src/rustdoc-internals.md
View file
Original file line number Diff line number Diff line change
@@ -272,19 +272,10 @@ Some extra reading about `make_test` can be found

## Dotting i's And Crossing t's

So that's `rustdoc`'s code in a nutshell, but there's more things in the
compiler that deal with it. Since we have the full `compiletest` suite at hand,
there's a set of tests in `tests/rustdoc` that make sure the final `HTML` is
what we expect in various situations. These tests also use a supplementary
script, `src/etc/htmldocck.py`, that allows it to look through the final `HTML`
using `XPath` notation to get a precise look at the output. The full
description of all the commands available to `rustdoc` tests (e.g. [`@has`] and
[`@matches`]) is in [`htmldocck.py`].

To use multiple crates in a `rustdoc` test, add `// aux-build:filename.rs`
to the top of the test file. `filename.rs` should be placed in an `auxiliary`
directory relative to the test file with the comment. If you need to build
docs for the auxiliary file, use `// build-aux-docs`.
<!-- FIXME(fmease):
* Make this section make sense again
* Deduplicate with section in ../rustdoc.md#tests
-->

In addition, there are separate tests for the search index and `rustdoc`'s
ability to query it. The files in `tests/rustdoc-js` each contain a
@@ -295,10 +286,6 @@ that features results in all tabs can be found in `basic.js`. The basic idea is
that you match a given `QUERY` with a set of `EXPECTED` results, complete with
the full item path of each item.

[`@has`]: https://github.com/rust-lang/rust/blob/master/src/etc/htmldocck.py#L39
[`@matches`]: https://github.com/rust-lang/rust/blob/master/src/etc/htmldocck.py#L44
[`htmldocck.py`]: https://github.com/rust-lang/rust/blob/master/src/etc/htmldocck.py

## Testing Locally

Some features of the generated `HTML` documentation might require local
69 changes: 51 additions & 18 deletions src/rustdoc-internals/rustdoc-test-suite.md
View file
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: may or may not be helpful for someone looking to adjust how tests/rustdoc works / change htmldocck/compiletest for tests/rustdoc purposes... but here's a very coarse TL;DR on the test infra bits involved on running tests/rustdoc:

Test infra setup

The test infra setup for tests/rustdoc involves the following bits:

Original file line number Diff line number Diff line change
@@ -4,21 +4,19 @@ This page is specifically about the test suite named `rustdoc`, for other test s

The `rustdoc` test suite is specifically used to test the HTML output of rustdoc.

This is achived by means of `htmldocck.py`, a custom checker script that leverages [XPath].
This is achieved by means of [`htmldocck.py`],
a supplementary checker script invoked by compiletest that leverages [XPath].

[`htmldocck.py`]: https://github.com/rust-lang/rust/blob/master/src/etc/htmldocck.py
[XPath]: https://en.wikipedia.org/wiki/XPath

## Directives
Directives to htmldocck are similar to those given to `compiletest` in that they take the form of `//@` comments.
## Htmldocck-Specific Directives

In addition to the directives listed here,
`rustdoc` tests also support most
[compiletest directives](../tests/directives.html).
Directives to htmldocck are similar to those given to `compiletest` in that they take the form of `//@` comments.
Comment on lines +13 to +15
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: even though it's an implementation detail, I think it's worth hammering in the distinction that htmldocck directives are handled by htmldocck and not compiletest.


All `PATH`s in directives are relative to the the rustdoc output directory (`build/TARGET/test/rustdoc/TESTNAME`),
so it is conventional to use a `#![crate_name = "foo"]` attribute to avoid
having to write a long crate name multiple times.
To avoid repetion, `-` can be used in any `PATH` argument to re-use the previous `PATH` argument.
It is conventional to use a `#![crate_name = "foo"]` attribute to avoid
having to write a long crate name multiple times.

All arguments take the form of quoted strings
(both single and double quotes are supported),
@@ -34,33 +32,41 @@ In this case, the start of the next line should be `//`, with no `@`.

For example, `//@ !has 'foo/struct.Bar.html'` checks that crate `foo` does not have a page for a struct named `Bar` in the crate root.

<!-- FIXME(fmease): Mention that the regexes match case-sensitively and in single-line mode? -->
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussion: may also be worth mentioning if the regexes have any string and/or regex escape rules. For instance, I know some compiletest regex directives don't handle this very well.


### `has`

Usage 1: `//@ has PATH`
Usage 2: `//@ has PATH XPATH PATTERN`

In the first form, `has` checks that a given file exists.

In the second form, `has` is an alias for `matches`,
In the second form, `has` is the same as `matches`,
except `PATTERN` is a whitespace-normalized[^1] string instead of a regex.
<!-- FIXME(fmease): It's more important to note *here* that the file under test gets normalized too (PATTERN is in 99% cases already normalized) -->

### `matches`

Usage: `//@ matches PATH XPATH PATTERN`

Checks that the text of each element selected by `XPATH` in `PATH` matches the python-flavored regex `PATTERN`.
Checks that the text of each element selected by `XPATH` in `PATH` matches the Python-flavored regex `PATTERN`.

### `matchesraw`

Usage: `//@ matchesraw PATH PATTERN`
Usage: `//@ matchesraw PATH XPATH PATTERN`

Checks that the contents of the file `PATH` matches the regex `PATTERN`.

<!-- FIXME(fmease): This previously didn't mention XPATH, mention it in prose -->

### `hasraw`

Usage: `//@ hasraw PATH PATTERN`
Usage: `//@ hasraw PATH XPATH PATTERN`

Same as `matchesraw`, except `PATTERN` is a whitespace-normalized[^1] string instead of a regex.
<!-- FIXME(fmease): It's more important to note *here* that the file under test gets normalized too (PATTERN is in 99% cases already normalized) -->

<!-- FIXME(fmease): This previously didn't mention XPATH, mention it in prose -->

### `count`

@@ -78,34 +84,61 @@ determined by the XPath, and compares it to a pre-recorded value
in a file. The file's name is the test's name with the `.rs` extension
replaced with `.NAME.html`, where NAME is the snapshot's name.

htmldocck supports the `--bless` option to accept the current subtree
Htmldocck supports the `--bless` option to accept the current subtree
Comment on lines -81 to +87
Copy link
Member

@jieyouxu jieyouxu Mar 20, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: IIRC the --bless flow requires html-tidy to be installed and available somewhere. Not sure if this is mentioned already. Maybe worth explicitly remarking or linking to that?

as expected, saving it to the file determined by the snapshot's name.
compiletest's `--bless` flag is forwarded to htmldocck.

<!-- FIXME(fmease): Also mention that we normalize certain URLS
both when and checking and when normalizing
-->

### `has-dir`

Usage: `//@ has-dir PATH`

Checks for the existance of directory `PATH`.
Checks for the existence of directory `PATH`.

### `files`

Usage: `//@ files PATH ENTRIES`

Checks that the directory `PATH` contains exactly `ENTRIES`.
`ENTRIES` is a python list of strings inside a quoted string,
`ENTRIES` is a Python list of strings inside a quoted string,
as if it were to be parsed by `eval`.
(note that the list is actually parsed by `shlex.split`,
so it cannot contain arbitrary python expressions).
so it cannot contain arbitrary Python expressions).

Example: `//@ files "foo/bar" '["index.html", "sidebar-items.js"]'`

[^1]: Whitespace normalization means that all spans of consecutive whitespace are replaced with a single space. The files themselves are also whitespace-normalized.

## Compiletest Directives

In addition to the directives listed here,
`rustdoc` tests also support most
[compiletest directives](../tests/directives.html).

<!-- FIXME(fmease):
Should definitely also mention `//@ aux-crate` and `//@ proc-macro`
UNLESS we nuke this paragraph entirely and refer to the compiletest section(s)?
-->
Comment on lines +121 to +124
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remark (not-a-review-comment): I think this is where I would like compiletest to support some kind of directive handling "plugin" and allow htmldocck to register plugins. Then we could phrase this as "htmldocck registers additional directive handling to compiletest" either here, or even put this bit under main compiletest chapter.

To use multiple crates in a `rustdoc` test, add `//@ aux-build:filename.rs`
to the top of the test file. `filename.rs` should be placed in an `auxiliary`
directory relative to the test file with the comment.

<!-- FIXME(fmease): We might want to explain why this exists / what this actually means -->
If you need to build docs for the auxiliary file, use `//@ build-aux-docs`.

<!-- FIXME(fmease): Mention `//@ doc-flags`! -->

## Limitations
`htmldocck.py` uses the xpath implementation from the standard library.

Htmldocck uses the XPath implementation from the Python standard library.
This leads to several limitations:

* All `XPATH` arguments must start with `//` due to a flaw in the implemention.
* Many XPath features (functions, axies, etc.) are not supported.
* Only well-formed HTML can be parsed (hopefully rustdoc doesn't output mismatched tags).

<!-- FIXME(fmease): Maybe link to revisions? -->
Furthmore, compiletest revisions are not supported.
2 changes: 2 additions & 0 deletions src/rustdoc.md
View file
Original file line number Diff line number Diff line change
@@ -81,6 +81,8 @@ does is call the `main()` that's in this crate's `lib.rs`, though.)

## Tests

<!-- FIXME(fmease): Also mention `tests/ui/rustdoc/`! -->

* Tests on search engine and index are located in `tests/rustdoc-js` and `tests/rustdoc-js-std`.
The format is specified
[in the search guide](rustdoc-internals/search.md#testing-the-search-engine).