diff --git a/docs/AGENTS.md b/docs/AGENTS.md index 456e4b7..3d0360b 100644 --- a/docs/AGENTS.md +++ b/docs/AGENTS.md @@ -70,6 +70,12 @@ dogfood the tool they describe; a broken example is a failing test. ```` ```{doctest} ```` directive — the finder collects both, plus bare doctest blocks in reST. Use ```` ```console ```` for shell commands at a `$` prompt. +- When the block is written to be pasted, drop the prompt and fence it + as ```` ```{testcode} ````, with ```` ```{testoutput} ```` for what it + prints and `:hide:` for a block that asserts without rendering. A + page's `{testcode}` blocks are named for the page and share one + namespace; a `>>>` block joins them only at document scope, so keep + each page to one form unless it runs at that scope. - `ELLIPSIS` and `NORMALIZE_WHITESPACE` are on globally via `doctest_optionflags`, so variable output can elide with `...` without a per-example flag. diff --git a/docs/modules/doctest_docutils/examples.md b/docs/modules/doctest_docutils/examples.md index 0f55c93..e3b7069 100644 --- a/docs/modules/doctest_docutils/examples.md +++ b/docs/modules/doctest_docutils/examples.md @@ -23,6 +23,46 @@ examples aligned with {class}`doctest_docutils.DocutilsDocTestFinder`. ['md', 'rst'] ``` +A directive keeps a reader's view of the example clean: the rendered page drops +the `# doctest: +NORMALIZE_WHITESPACE` written below, while the run still +applies it, so the two spaces in the printed output match the one below them. + +```{doctest} +>>> print("a b") # doctest: +NORMALIZE_WHITESPACE +a b +``` + +## Blocks that share a namespace + +Blocks naming the same group collect as one test, so the second reads what the +first bound: + +```python +>>> import doctest_docutils +>>> finder = doctest_docutils.DocutilsDocTestFinder() +>>> source = ( +... "```{doctest} intro\n>>> greeting = 'hello'\n```\n" +... "\nProse between the blocks.\n\n" +... "```{doctest} intro\n>>> greeting.upper()\n'HELLO'\n```\n" +... ) +>>> tests = finder.find(source, "example.md") +>>> [(test.name, len(test.examples)) for test in tests] +[('intro', 2)] +``` + +Blocks naming no group keep a namespace each, until you ask for the page: + +```python +>>> import doctest_docutils +>>> page = "```python\n>>> alone = 1\n```\n\n```python\n>>> alone\n1\n```\n" +>>> apart = doctest_docutils.DocutilsDocTestFinder() +>>> [test.name for test in apart.find(page, "example.md")] +['example.md[0]', 'example.md[1]'] +>>> shared = doctest_docutils.DocutilsDocTestFinder(namespace_scope="document") +>>> [test.name for test in shared.find(page, "example.md")] +['example.md'] +``` + ## Finder result names {class}`~doctest_docutils.DocutilsDocTestFinder` names collected examples with diff --git a/docs/modules/doctest_docutils/how-to.md b/docs/modules/doctest_docutils/how-to.md index da5be6f..8fbec58 100644 --- a/docs/modules/doctest_docutils/how-to.md +++ b/docs/modules/doctest_docutils/how-to.md @@ -26,6 +26,135 @@ Pass `-v` for verbose standard-library doctest output: $ python -m doctest_docutils README.md -v ``` +## Let a page build one example across several blocks + +Every block runs against a namespace of its own, so a name bound in one block is +gone by the next and any block can be run on its own. When a page is one session +told in pieces, widen the namespace to the whole page: + +```console +$ python -m doctest_docutils README.md --namespace-scope document +``` + +Blocks that name a group share that group's namespace at either setting, because +naming a group is the author asking for it. A group is named as the directive's +argument, `.. doctest:: intro` in reStructuredText and its `{doctest} intro` +fence in Markdown. `--namespace-scope document` also pools the blocks that name +none. + +One name a group cannot take is the one the page would generate for a block that +declares none — the page's own name at `--namespace-scope document`, the page and +the block's position at the default. Both would answer to one namespace and one +node id, so a page spelling both stops with +{exc}`~doctest_docutils.NamespaceNameCollisionError` rather than merging them. +Rename the group; a page whose every block names one generates nothing to collide +with, so `.. doctest:: README.md` on such a page is only a style choice. + +Sharing costs you the guarantee that a block stands alone: a block that reads an +earlier binding fails when it is read, or run, by itself. See +{ref}`the pytest plugin's how-to ` for the same +choice under pytest, spelled `--doctest-docutils-namespace-scope` there, and for +what sharing costs a test run. + +A shared page is reported as one item by default. Ask for one item per block, +each named for where the block sits, when you want to read the run block by +block: + +```console +$ python -m doctest_docutils README.md --namespace-scope document --namespace-items per-block -v +``` + +A passing page prints nothing without `-v`. What changes without it is a +failure's heading, which names the block — `in README.md[1]` rather than +`in README.md`. + +Nothing here schedules the blocks apart, so they share the namespace either way. +Under pytest they can be scheduled apart, which is what +{ref}`the plugin's how-to ` covers. + +## Write a block a reader is meant to paste + +A `>>>` prompt is for a session a reader reads. When the block is there to be +copied into a file, the prompt is in the way, and an expected-output line beneath +it puts an assertion into whatever the reader pasted. Such a page carries no +prompt at all — and a finder that goes looking for `>>>` cannot see it. + +Write those blocks as `{testcode}`, the directive {mod}`sphinx.ext.doctest` +defines. The body is plain Python, run the way a module body runs, so it takes as +many statements as it likes and a bare expression on the last line prints +nothing: + +```{testcode} +greeting = "hello" +shouted = greeting.upper() +``` + +A `{testcode}` expects to print nothing. When it does print, say what with a +`{testoutput}` block under it: + +```{testcode} +print(shouted) +``` + +```{testoutput} +HELLO +``` + +The two blocks above share a namespace, so the second reads what the first bound. +A `{testcode}` that names no group is named for its page, because sharing the +page is the whole point of the form — a visible block and the hidden one +asserting on it have to meet somewhere. Name a group as the directive's +argument, `{testcode} intro`, to keep two runs on one page apart. + +A `>>>` block is named for its page only where the scope above says so. So at +`--namespace-scope document` the two forms land in the same namespace and read +each other's names, and at the default they do not. Write a page that mixes them +at document scope, or keep each page to one form. + +A page written this way sets up the same way, with no prompt: + +````markdown +```{testsetup} +base = 40 +``` +```` + +A `{testsetup}` and `{testcleanup}` may still be written with prompts, which is +how the rest of these docs write them; the prompt decides how the body is read. +A page holding a `{testcode}` names its unnamed setup for the page too, so the +setup a prompt-free page writes reaches the code it is for. + +That is what lets a page assert without showing its assertions. Mark a block +`:hide:` and it runs while every builder drops it, so the reader meets only the +block written to be pasted: + +````markdown +```{testcode} +:hide: + +assert shouted == "HELLO" +``` +```` + +```{testcode} +:hide: + +assert shouted == "HELLO" +``` + +The page you are reading has that hidden block in it, immediately above. + +`{testoutput}` takes `:options:` for the doctest flags the comparison runs under, +and both directives take `:skipif:`. `:pyversion:` parses, because Sphinx +declares it here, but neither Sphinx nor this runner acts on it outside +`{doctest}` — the page says so when you use it. Guard a block with `:skipif:` +instead. + +The cost of the prompt-free form is that there is no interleaving: one block is +one example, so a `{testoutput}` says what the block prints in total rather than +what any line in it prints. A failure quotes the block entire, so the reader sees +where they are. + ## Compare with stdlib doctest Use the stdlib command when you are checking Python modules or plain text that diff --git a/docs/modules/doctest_docutils/index.md b/docs/modules/doctest_docutils/index.md index e63866e..2bb46d7 100644 --- a/docs/modules/doctest_docutils/index.md +++ b/docs/modules/doctest_docutils/index.md @@ -23,7 +23,8 @@ Run your first documentation doctest from a Markdown page. :::{grid-item-card} How-to :link: how-to :link-type: doc -Choose files, run verbose output, and map the command to stdlib doctest. +Choose files, share a namespace across blocks, run verbose output, and map +the command to stdlib doctest. ::: :::{grid-item-card} Examples diff --git a/docs/modules/pytest_doctest_docutils/fixtures.md b/docs/modules/pytest_doctest_docutils/fixtures.md index 9e5fb8e..635a535 100644 --- a/docs/modules/pytest_doctest_docutils/fixtures.md +++ b/docs/modules/pytest_doctest_docutils/fixtures.md @@ -29,6 +29,13 @@ Then the documentation page can use the helper by name: add(2, 3) ``` +A helper like this one holds nothing, so how long it lives never comes up. +Seeding a *resource* — a server, a connection, a temporary directory — is where +it does, because the fixture's scope decides how long the object a page saved +stays usable. See {ref}`what per-block items cost +` before carrying one across several +blocks of a page. + ## Autouse fixtures Autouse fixtures in a visible `conftest.py` are parsed for `.rst` and `.md` diff --git a/docs/modules/pytest_doctest_docutils/how-to.md b/docs/modules/pytest_doctest_docutils/how-to.md index aaeef3c..79b538b 100644 --- a/docs/modules/pytest_doctest_docutils/how-to.md +++ b/docs/modules/pytest_doctest_docutils/how-to.md @@ -32,6 +32,363 @@ Disable Python-module collection explicitly with $ py.test src/ --no-doctest-docutils-modules ``` +## Let a page build one example across several blocks + +Every block on a page runs against a namespace of its own. A name bound in one +block is gone by the next, which is what lets a reader copy any single block out +of the page and run it. Most pages want that, and it is what you get with no +configuration at all. + +A narrative page often wants the opposite: the prose walks through one session a +piece at a time. Name a group on the blocks that belong together, as the +directive's argument: + +````markdown +```{doctest} intro +>>> greeting = "hello" +``` + +Prose between the two blocks. + +```{doctest} intro +>>> greeting.upper() +'HELLO' +``` +```` + +reStructuredText names a group the same way: + +```rst +.. doctest:: intro + + >>> greeting = "hello" +``` + +A group collects as one item — `page.md::intro` — holding every block that named +it, in page order. A group reaches as far as the page it is written on: the same +name on a second page is a second namespace. + +A block can name several groups at once, comma separated, and joins each of +them — it runs once per group, against that group's namespace. `*` stands for +every group the page declares, which is how you write one setup for all of them: + +```rst +.. testsetup:: * + + >>> import math +``` + +`.. testsetup::` and `.. testcleanup::` run before and after the rest of their +group whatever order the page writes them in, so you can move them out of a +reader's way. Their output is still checked, unlike in Sphinx, so a setup that +raises is reported rather than swallowed. A failing example ends its namespace, +which means that namespace's cleanup does not run. + +Only the directive form can name a group. A plain ```` ```python ```` fence, a +bare fence, an indented block, and a reStructuredText doctest block have nowhere +to write one. To share state between those, widen the page for a single run: + +```console +$ pytest docs/ --doctest-docutils-namespace-scope=document +``` + +Or settle it for the project: + +```ini +[pytest] +doctest_docutils_namespace_scope = document +``` + +Under `document`, the blocks that name no group share one namespace per page, +named for the page. Named groups still partition it — declaring a group is the +author asking for sharing, so a group is its own namespace at either setting. + +### What sharing costs + +A namespace is one item. That is what keeps a shared page correct under +`pytest -n auto`: no worker is ever handed half of a session. It also means: + +- The namespace passes or fails as a single line, and a failure stops the + examples after it unless you pass `--doctest-continue-on-failure`. Stopping + is what keeps a half-built namespace out of the blocks below: they never run. + With `--doctest-continue-on-failure` they do run, against a namespace missing + whatever the failed example would have bound, so one broken line can report + as a first failure followed by a run of {exc}`NameError`s that are not + independent. +- A function-scoped fixture sets up once per namespace instead of once per + block. A page whose blocks each expect a fresh fixture belongs at `block`. + A block gated end to end is the exception: it is its own item, and it is + marked skipped before setup, so it neither shares that setup nor pays for + one of its own. +- A block whose every example is skipped is the one thing a namespace does not + hold. It binds nothing the other blocks could read, so it is lifted back out + and collects as an item of its own — `page.rst::intro[1]` — and still + reports `SKIPPED`. The page collects one item more than it has namespaces + for each such block, which is one more line in `--collect-only` and one more + entry in a JUnit report. +- The report's numbered gutter spans the whole namespace, so the prose between + two blocks shows up as blank numbered lines above the failing prompt. + +One page shape moves further. docutils numbers a bare reStructuredText doctest +block by its *last* line, so that block's examples report lines below where it +sits whether or not it shares anything. Sharing extends the reach of that: a +block written close underneath one is pushed past it, by as many lines as the +two overlap. The gutter still ends on the failing prompt. Write those blocks as +`.. doctest::` directives when the exact line matters — a directive is numbered +by the line it opens on. + +### Keep a node id for every block + +The item a namespace collects as is the thing you can point pytest at. When a +namespace is one item, that reach stops at the page: `--lf` re-runs the whole +page rather than the block that failed, `-k` and `--deselect` cannot single a +block out, a JUnit report names the page, and there is no id to paste back +while you iterate on one block. + +Those reaches are worth most where each block has a namespace of its own, the +default scope. A block that reads what an earlier one bound is a fragment of a +session, so selecting it alone — by id, by `-k`, or by `--lf` after it failed — +runs it without the block it depends on. See +{ref}`what per-block items cost `. + +Take the other trade when you want those, and a fixture per block, more than +you want the single line: + +```console +$ pytest docs/ --doctest-docutils-namespace-items=per-block +``` + +Or settle it for the project: + +```ini +[pytest] +doctest_docutils_namespace_items = per-block +``` + +Every block is an item again, under the id it carries when nothing is shared — +`page.md::page.md[1]`, or `page.rst::intro[1]` inside a group — and the blocks +of one namespace are handed the *same* globals rather than a copy each. A +function-scoped fixture is back to setting up once per block, which is what a +project promising a fresh fixture for every example needs. + +The two settings answer different questions, and both still apply: the scope +says what shares a namespace, this one says whether sharing costs the blocks +their ids. At the default scope no page state is shared either way — but +selecting `per-block` is still the opt-in to a live shared mapping, because a +page that declares a group shares one whatever the scope. + +A run that keeps a node id for every block says so in its header, so you can +tell from the report which one you got. The scope rides along on the same line: + +```text +doctest-docutils: namespace items: per-block, namespace scope: document +``` + +A run that only widens the scope is not announced; the default layout reports +nothing, so the header of a project that never touched this setting reads as it +always has. + +(pytest_doctest_docutils-per-block-costs)= + +### What per-block items cost + +A live namespace is a Python object, so it neither crosses a process boundary +nor outlives the fixtures that filled it. The cost shows up in five places. + +Under `pytest-xdist`, two blocks of one namespace landing on different workers +would leave the second reading a namespace the first never built. Which +scheduler you get decides whether that can happen, and `-n` on its own does not +choose one: `pytest-xdist` fills it in with `--dist load`, which distributes by +item. + +So the plugin fills it in first, with file-level scheduling. `pytest docs/ -n +auto` keeps every page on one worker and your shared pages pass: + +```console +$ pytest docs/ -n auto +``` + +Nothing about the run changes otherwise — the node ids stay the ones the layout +collects, and `-v` names the scheduler that ran if you want to see it. File +level is as fine-grained as this can go. `loadgroup` would suit the +`xdist_group` marker the plugin emits per namespace, but that group reaches a +scheduler through a node-id suffix the *worker* writes, from the worker's own +`--dist` value, so no substitute made on the controller can use it. + +Name a scheduler yourself and it is yours. `--dist loadfile`, `--dist +loadgroup`, `--dist loadscope` and `--dist each` all keep a namespace whole, +and `loadgroup` is the finer grained of the two obvious ones: `loadfile` pins a +whole file to one worker, while the group is the file plus the namespace, so a +page holding several namespaces still spreads. + +```console +$ pytest docs/ -n auto --dist loadgroup +``` + +Ask for `--dist load` or `--dist worksteal` — by flag or through `addopts` — +and the session stops instead, naming the page it would have split. Overruling +a scheduler you asked for by name would be the plugin deciding it knows better; +reporting a page that is only wrong because of how it was scheduled would be +worse: + +```text +ERROR: doctest_docutils_namespace_items = per-block can hand a namespace's +blocks one globals mapping between them — a page declaring a group does, +whatever the scope — and a mapping cannot cross processes. --dist worksteal +hands a file's items to whichever worker is free, so it can send docs/page.md's +blocks to different workers. Run with --dist loadgroup or --dist loadfile, or +set doctest_docutils_namespace_items = merged. Dropping --dist leaves -n free +to keep each page on one worker. +``` + +That reads the run, not the setting. A run holding no page whose blocks split — +a suite of Python tests, a single-block page, `--collect-only`, one worker — +keeps its workers whatever it asked for, so carrying the layout in your ini +never costs `-n` to a session that has no namespace to protect. + +A test-retry plugin repeats a single item, which a live namespace cannot +survive. Under `merged` a retry re-runs the namespace from its first block, so +the run rebuilds what it needs and a real failure stays a failure. Under +`per-block` the retry re-runs only the block that failed, against the mapping +that block already changed — so an example whose expectation happens to come +true on the second attempt would be reported as a pass. There is no way to +rebuild the namespace for one block, so the repeat is refused instead: + +```text +Failed: page.rst::demo[1] was run twice against a namespace laid out per +block. A repeated block runs against the globals it already changed, so its +result cannot be trusted. Drop --reruns (and anything else that repeats an +item), or set doctest_docutils_namespace_items = merged, which re-runs a +namespace from its first block. +``` + +A block that passes first time is never repeated, so a green run under +`--reruns` is unaffected. + +Running one block by its id has the same shape: `pytest page.md::page.md[1]` +runs that block and nothing else, so a block reading a name an earlier one +bound reports the `NameError` it earns. `-k`, `--deselect` and `--lf` +reach a block the same way and cost the same thing — a `--lf` re-run of a +failure in a shared page reports the missing binding rather than the diff you +were chasing. That is inherent to running a fragment of a session, not +something the setting can hide. + +A namespace shares the mapping, not the lifetime of what a fixture put in it. +Each block is its own item, so a function-scoped fixture tears down between +blocks. The name that fixture fills is rebound fresh for the next block, but a +name a block derived from it is not: it still holds the finalized object, and a +finalized object usually answers rather than raising. A page reads a plausible +wrong value — a cached attribute that looks right beside a connection that is +already closed — with nothing in the report to say so. + +Give a fixture a page carries across its blocks `scope="module"`. A page is +what module scope means here — the collector for a `.md` or `.rst` page is a +{class}`pytest.Module`, as pytest's own text-doctest collector is — so the +fixture sets up once for the page and tears down when the page ends, and the +object a block saves stays the object the next block reads. + +Name `module` rather than reaching for anything wider. `class` has no node to +attach to on a page, so it silently falls back to setting up per block, and +`package` anchors to the directory holding the `conftest.py` that *defines* the +fixture, and only when that directory is an importable package — a fixture +defined further up resolves to the whole run however the page's own directory +looks. Only `function`, `module` and `session` mean what they say here. + +Two things follow from a page being a module for scope and nothing else. A +module-scoped fixture cannot request a function-scoped one, so reaching for +{ref}`tmp_path ` stops the page with pytest's `ScopeMismatch` +— ask for `tmp_path_factory` instead. And no +module object stands behind a page, +so `request.module` is `None`; a `conftest.py` shared with `.py` tests that +reads it works on those and breaks here. `request.path` names the page. + +Across worker processes it is a page per worker, not a page per run. Leaving +`-n` unadorned keeps a page whole, and so does `--dist loadfile`; asking for +`--dist loadgroup` while every block is its own namespace groups by block +instead, which hands one page to two workers and sets the fixture up in each. + +And because the blocks share one mapping, they share whatever lives in it — +including `__future__` flags, which {mod}`doctest` derives from the namespace at +run time. A `from __future__ import ...` in one block is in force for the rest +of its namespace. + +## Set options for a whole block + +A `{doctest}` directive can carry the flags its examples would otherwise repeat. +`:options:` takes the same names as the inline `# doctest:` comment, and an +example that writes its own flag wins over the directive's: + +```rst +.. doctest:: + :options: +ELLIPSIS + + >>> print("hello world") + hello ... +``` + +`:skipif:` skips a block on a condition the page works out for itself, so you +don't have to write `+SKIP` by hand for one interpreter or one platform. It +takes a Python expression, which is **evaluated** when the page is read, and a +true result marks the block `+SKIP`: + +```rst +.. doctest:: + :skipif: sys.version_info < (3, 12) + + >>> "a modern interpreter" + 'a modern interpreter' +``` + +That is the same flag `:options: +SKIP` sets, so the two spellings report +alike: the block still collects, still counts, and still answers to its own +node id. The reason pytest prints is the one it prints for any skipped +example, which names the flag rather than your condition: + +```console +$ pytest page.rst -rs +``` + +```text +SKIPPED [1] page.rst: page.rst:6: every example skipped +1 skipped +``` + +Where `:options:` sets defaults an example can override, a condition is a gate +it cannot: an example writing `# doctest: -SKIP` inside a gated block stays +skipped. Sphinx drops such a block before reading it at all, and an example that +could reopen it would run on exactly the interpreter or platform the condition +named. The expression sees `sys` and the globals the document starts +with, not anything the page's own examples bound — it is answered while the +page is being read, before any of them run. Naming anything else stops the page +with {exc}`~doctest_docutils.SkipifExpressionError`, which reports the file, +line, and expression to go fix. + +Reading a page is all `--collect-only` does, so listing a page's items runs its +`:skipif:` expressions; keep them free of side effects. + +Skipping one block of a group leaves the group's other blocks running, and the +skipped one still reports. Because a block with nothing left to run binds no +name the group could read, it does not need to share the group's item: it +collects as one of its own, named for the group and for the block's position on +the page, counted from zero across every doctest block. The second block of +`page.rst` in group `intro` is `page.rst::intro[1]`; on a page merged by +`--doctest-docutils-namespace-scope=document` it is `page.rst::page.rst[1]`, +which is the name that block already carries when every block keeps its own +namespace. So the node id a reader pastes back to pytest does not move with the +scope. + +Two cases stay where they are. A block whose examples are only *partly* +skipped is not a skipped block — it has something left to run, and it reports +with its namespace like any other item. And when *every* block of a namespace +is gated, there is nothing for them to be silent beside: the namespace keeps +them all and reports skipped once, as one item, rather than once per block. + +A skipped block is still parsed, so malformed doctest source in one reports +as an error rather than passing unnoticed — the same as for `:options: +SKIP`. + +`:skipif:` works the same on `.. testsetup::` and `.. testcleanup::`, which +declare the option too. + ## Hide a setup line from rendered docs Mark a prompt line with `# doctest: +HIDE` when your suite should run it but a @@ -48,10 +405,9 @@ The marker changes nothing about the run: the line still executes and its output is still checked. It only tags the line so a documentation renderer can drop it from the rendered page while pytest keeps testing it from source. -The plugin registers the marker as pytest configures, so `# doctest: +HIDE` -parses in `.rst`, `.md`, and Python-module doctests. The standalone -`python -m doctest_docutils` command does not register it, so use the marker -when you run examples through pytest. +Importing {mod}`doctest_docutils` registers the marker, so `# doctest: +HIDE` +parses in `.rst`, `.md`, and Python-module doctests under pytest and under the +standalone `python -m doctest_docutils` command alike. ## Keep pytest's built-in doctest plugin disabled diff --git a/docs/modules/pytest_doctest_docutils/index.md b/docs/modules/pytest_doctest_docutils/index.md index 309c68d..f6c7407 100644 --- a/docs/modules/pytest_doctest_docutils/index.md +++ b/docs/modules/pytest_doctest_docutils/index.md @@ -23,7 +23,8 @@ Run documentation doctests through pytest. :::{grid-item-card} How-to :link: how-to :link-type: doc -Collect docs, Python modules, and option-flagged examples. +Collect docs, Python modules, and option-flagged examples, and let a page +build one session across several blocks. ::: :::{grid-item-card} Fixtures diff --git a/pyproject.toml b/pyproject.toml index e3426f0..aa4ec13 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -58,6 +58,7 @@ dev = [ "pytest-rerunfailures", "pytest-mock", "pytest-watcher", + "pytest-xdist", # Coverage "codecov", "coverage", @@ -81,6 +82,7 @@ testing = [ "pytest-rerunfailures", "pytest-mock", "pytest-watcher", + "pytest-xdist", ] coverage =[ "codecov", diff --git a/src/doctest_docutils.py b/src/doctest_docutils.py index f8f2cde..5236639 100644 --- a/src/doctest_docutils.py +++ b/src/doctest_docutils.py @@ -2,14 +2,14 @@ from __future__ import annotations +import copy import doctest -import linecache import logging import os import pathlib -import pprint import re import sys +import types import typing as t import docutils @@ -21,8 +21,6 @@ from docutils_compat import findall if t.TYPE_CHECKING: - import types - from docutils.nodes import Node, TextElement logger = logging.getLogger(__name__) @@ -34,6 +32,234 @@ # Allow optional leading whitespace before doctest directive comments. doctestopt_re = re.compile(r"[ \t]*#\s*doctest:.+$", re.MULTILINE) +#: How wide a namespace the blocks of one document share when they name no +#: group. +NamespaceScope = t.Literal["block", "document"] + +#: Accepted :data:`NamespaceScope` names, narrowest first. +NAMESPACE_SCOPES: tuple[NamespaceScope, ...] = ("block", "document") + +#: Scope used when a caller names none: every ungrouped block starts empty. +DEFAULT_NAMESPACE_SCOPE: NamespaceScope = "block" + +#: Whether the blocks of one namespace become a single test or stay one test +#: each, sharing the globals mapping between them. +NamespaceItems = t.Literal["merged", "per-block"] + +#: Accepted :data:`NamespaceItems` names, fewest tests first. +NAMESPACE_ITEMS: tuple[NamespaceItems, ...] = ("merged", "per-block") + +#: Layout used when a caller names none: a namespace is one test. +DEFAULT_NAMESPACE_ITEMS: NamespaceItems = "merged" + +#: Group a ``.. doctest::`` written without an argument lands in, as in +#: :mod:`sphinx.ext.doctest`. It means the author named no group. +_DEFAULT_GROUP = "default" + +#: Group name meaning "every group this document declares", as in +#: :mod:`sphinx.ext.doctest`. It resolves only once the page has been read. +_WILDCARD_GROUP = "*" + +#: Block types that keep :mod:`sphinx.ext.doctest`'s implicit ``default`` +#: group: the visible block a reader pastes and the hidden one asserting on it +#: have to share a namespace without either of them naming a group. +_GROUPED_BLOCK_TYPES = frozenset({"testcode", "testoutput"}) + +#: Block types that join the implicit ``default`` group only once a page holds +#: a ``{testcode}``. Sphinx puts an unnamed ``{testsetup}`` in that group +#: always; gp-libs gives a page's prompt blocks namespaces of their own, and an +#: unnamed setup has always followed them there. Widening it unconditionally +#: would take the setup away from those blocks, so it widens only for the pages +#: that need it — the ones written in Sphinx's prompt-free style. +_PHASE_BLOCK_TYPES = frozenset({"testsetup", "testcleanup"}) + +#: Matches the ``>>>`` opening an interactive example, which tells a +#: ``{testsetup}`` written for :class:`doctest.DocTestParser` apart from one +#: written the way :mod:`sphinx.ext.doctest` writes it. +_PROMPT_RE = re.compile(r"^[ \t]*>>>(?:[ \t]|$)", re.MULTILINE) + +#: ``HIDE`` marks a prompt that rendered documentation drops and a test run +#: keeps. It changes no output check, but a page carrying it fails to parse +#: wherever the name is unregistered, so registration happens on import rather +#: than at any one entry point's setup: ``python -m doctest_docutils`` reaches +#: no further than this module, and pytest's own ``DoctestModule`` parses .py +#: docstrings without ever consulting the plugin's flag lookup. +_HIDE_FLAG = doctest.register_optionflag("HIDE") + + +class NamespaceScopeError(ValueError): + """Raised when a namespace scope is not one of :data:`NAMESPACE_SCOPES`. + + Examples + -------- + >>> print(NamespaceScopeError("per-file")) + Unknown namespace scope: 'per-file'. Expected one of: block, document + """ + + def __init__(self, value: str) -> None: + super().__init__( + f"Unknown namespace scope: {value!r}. " + f"Expected one of: {', '.join(NAMESPACE_SCOPES)}", + ) + + +class NamespaceItemsError(ValueError): + """Raised when a namespace layout is not one of :data:`NAMESPACE_ITEMS`. + + Examples + -------- + >>> print(NamespaceItemsError("one-each")) + Unknown namespace items: 'one-each'. Expected one of: merged, per-block + """ + + def __init__(self, value: str) -> None: + super().__init__( + f"Unknown namespace items: {value!r}. " + f"Expected one of: {', '.join(NAMESPACE_ITEMS)}", + ) + + +class NamespaceNameCollisionError(ValueError): + """Raised when a declared group takes a name the page generates for itself. + + A page generates names of its own: for a block that declares no group — + the page itself at ``"document"`` scope, the page and the block's position + at ``"block"`` scope — and for a gated block lifted out of the namespace + it was declared in. A group spelling one of those asks for a name the page + has already given away, and the two would run as one: state crossing the + partition the author drew, under a single node id. Neither meaning can be + kept over the other, so the page stops rather than merging them quietly. + + Parameters + ---------- + group : str + Group the page declared. + document_name : str + Base name of the document, without its directory. + generated_for : str + What the page generates the same name for. + + Examples + -------- + >>> print( + ... NamespaceNameCollisionError( + ... "page.rst", "page.rst", "a block declaring none at 'document' scope" + ... ) + ... ) + page.rst: group 'page.rst' takes the name this page generates for a block + declaring none at 'document' scope, so the two would share state and one + node id. Rename the group. + + A gated block lifted out of its group is named the same way, so a group + can take that name too: + + >>> print( + ... NamespaceNameCollisionError( + ... "alpha[1]", "page.rst", "a block lifted out of 'alpha'" + ... ) + ... ) + page.rst: group 'alpha[1]' takes the name this page generates for a block + lifted out of 'alpha', so the two would share state and one node id. + Rename the group. + """ + + def __init__(self, group: str, document_name: str, generated_for: str) -> None: + super().__init__( + f"{document_name}: group {group!r} takes the name this page" + f" generates for {generated_for}, so the two would share state and" + " one node id. Rename the group.", + ) + + +class SkipifExpressionError(ValueError): + """Raised when a block's ``:skipif:`` expression cannot be evaluated. + + Examples + -------- + >>> error = NameError("name 'platform' is not defined") + >>> print(SkipifExpressionError("platform.system()", "page.rst", 4, error)) + page.rst:4: :skipif: 'platform.system()' failed: name 'platform' is not defined + """ + + def __init__( + self, + expression: str, + filename: str, + line: int, + error: BaseException, + ) -> None: + super().__init__( + f"{filename}:{line}: :skipif: {expression!r} failed: {error}", + ) + + +def _parse_namespace_scope(value: str) -> NamespaceScope: + """Return `value` as a :data:`NamespaceScope`, rejecting anything else. + + Parameters + ---------- + value : str + Scope name to validate. + + Returns + ------- + NamespaceScope + The scope, unchanged. + + Raises + ------ + NamespaceScopeError + If `value` names no known scope. + + Examples + -------- + >>> _parse_namespace_scope("document") + 'document' + + >>> try: + ... _parse_namespace_scope("per-file") + ... except NamespaceScopeError as exc: + ... print(exc) + Unknown namespace scope: 'per-file'. Expected one of: block, document + """ + if value not in NAMESPACE_SCOPES: + raise NamespaceScopeError(value) + return value + + +def _parse_namespace_items(value: str) -> NamespaceItems: + """Return `value` as a :data:`NamespaceItems`, rejecting anything else. + + Parameters + ---------- + value : str + Layout name to validate. + + Returns + ------- + NamespaceItems + The layout, unchanged. + + Raises + ------ + NamespaceItemsError + If `value` names no known layout. + + Examples + -------- + >>> _parse_namespace_items("per-block") + 'per-block' + + >>> try: + ... _parse_namespace_items("one-each") + ... except NamespaceItemsError as exc: + ... print(exc) + Unknown namespace items: 'one-each'. Expected one of: merged, per-block + """ + if value not in NAMESPACE_ITEMS: + raise NamespaceItemsError(value) + return value + def is_allowed_version(version: str, spec: str) -> bool: """Check `spec` satisfies `version` or not. @@ -53,6 +279,196 @@ def is_allowed_version(version: str, spec: str) -> bool: return Version(version) in SpecifierSet(spec) +class _ExecSource(str): + r"""Example source that runs as a module body, not as one prompt. + + :meth:`doctest.DocTestRunner.run` compiles every example in ``"single"`` + mode, which rejects a body of more than one statement and echoes a bare + expression. A ``{testcode}`` body is plain Python and neither applies, so + it needs ``"exec"``. + + The mode rides on the source because the source is the only part of an + example the compile call is handed: marking it here keeps the choice on + the data, where :func:`copy.copy` and the merge carry it along, instead of + on a mode flag some runner has to be holding at the right moment. + + Examples + -------- + >>> source = _ExecSource("value = 41\n") + >>> isinstance(source, str), source.splitlines() + (True, ['value = 41']) + """ + + +def _compile_source( + source: str, + filename: str, + mode: str, + flags: int = 0, + dont_inherit: bool = False, + optimize: int = -1, +) -> types.CodeType: + r"""Compile one example, letting an :class:`_ExecSource` overrule `mode`. + + Stands in for the built-in :func:`compile` inside the runner loop + :func:`_exec_mode_run` builds. Every other source compiles exactly as the + runner asked, so a ``>>>`` example keeps the echo it was written for. + + Parameters + ---------- + source : str + Example source. An :class:`_ExecSource` asked for ``"single"`` + compiles in ``"exec"``. + filename : str + Name the compiled code reports itself under. + mode : str + Mode the caller asked for. + flags : int + ``__future__`` and compiler flags. + dont_inherit : bool + Whether to ignore the calling frame's ``__future__`` flags. + optimize : int + Optimization level. + + Returns + ------- + types.CodeType + Compiled example. + + Examples + -------- + >>> import contextlib, io + >>> def run(source): + ... captured = io.StringIO() + ... with contextlib.redirect_stdout(captured): + ... exec(_compile_source(source, "", "single"), {"value": 41}) + ... return captured.getvalue() + + A bare expression echoes under the mode the runner asks for, and stays + quiet once the source says it is a ``{testcode}`` body: + + >>> run("value\n") + '41\n' + >>> run(_ExecSource("value\n")) + '' + + ``"single"`` takes one statement; a ``{testcode}`` body takes as many as + it likes: + + >>> namespace = {} + >>> body = _ExecSource("first = 1\nsecond = first + 1\n") + >>> exec(_compile_source(body, "", "single"), namespace) + >>> namespace["second"] + 2 + """ + if mode == "single" and isinstance(source, _ExecSource): + mode = "exec" + # A variable ``mode`` widens the built-in's return type; the runner never + # asks for an AST, so the answer is always a code object. + return t.cast( + types.CodeType, + compile(source, filename, mode, flags, dont_inherit, optimize), + ) + + +def _exec_mode_run() -> types.FunctionType | None: + r"""Return CPython's runner loop reading :func:`_compile_source` as ``compile``. + + ``DocTestRunner.__run`` hard-codes ``"single"`` and resolves ``compile`` + as a global of :mod:`doctest`, which is why :mod:`sphinx.ext.doctest` + rebinds ``doctest.compile`` for the whole process. gp-libs ships as a + ``pytest11`` plugin loaded into every test session, so it rebinds the name + for one function object instead: the code object is CPython's, unread and + uncopied, and only the globals mapping it looks names up in differs. + + The seam is two facts about a private method, so it is checked rather than + assumed. gp-libs is loaded into every session that has it installed, and an + interpreter that moved the method must not take down the test runs of people + who never wrote a ``{testcode}``: a missing seam leaves CPython's own loop + in place and logs why. What that costs is ``{testcode}`` itself, which falls + back to ``single`` mode and fails on any body of more than one statement — + so the seam is also pinned by a test, where the loud failure belongs. + + Returns + ------- + types.FunctionType or None + Copy of CPython's runner loop, still unbound. `None` where this + interpreter does not resolve ``compile`` the way the loop needs. + + Examples + -------- + >>> run = _exec_mode_run() + >>> run.__code__ is doctest.DocTestRunner._DocTestRunner__run.__code__ + True + >>> run.__globals__["compile"] is _compile_source + True + + :mod:`doctest` itself keeps the built-in: + + >>> "compile" in vars(doctest) + False + """ + original = getattr(doctest.DocTestRunner, "_DocTestRunner__run", None) + # A closure would need its cells rebuilt; a loop that stopped reading + # ``compile`` as a global would silently ignore the swap. + if ( + original is None + or original.__code__.co_freevars + or "compile" not in original.__code__.co_names + ): + logger.error( + "doctest runner seam is missing; testcode blocks fall back to " + "single-statement mode", + extra={"doctest_block_type": "testcode"}, + ) + return None + return types.FunctionType( + original.__code__, + {**vars(doctest), "compile": _compile_source}, + original.__name__, + ) + + +class _ExecModeRunnerMixin: + r"""Runner mixin that runs a ``{testcode}`` body the way Sphinx does. + + Mix in ahead of :class:`doctest.DocTestRunner` or + :class:`doctest.DebugRunner`. Everything else about the run — reporting, + the debugger, ``SKIP``, ``FAIL_FAST`` — is CPython's own loop. + + Examples + -------- + >>> import io + >>> example = doctest.Example("value = 41\nvalue\n", "") + >>> example.source = _ExecSource(example.source) + >>> test = doctest.DocTest([example], {}, "page.md", "page.md", 0, None) + >>> _ExecModeRunner().run(test, out=io.StringIO().write) + TestResults(failed=0, attempted=1) + + The stock runner rejects the same example, and says why: + + >>> report = io.StringIO() + >>> doctest.DocTestRunner().run(test, out=report.write) + TestResults(failed=1, attempted=1) + >>> "multiple statements" in report.getvalue() + True + """ + + # Left unset where this interpreter has no seam, so the mixin inherits + # CPython's loop instead of shadowing it with nothing. + if (_exec_mode_run_override := _exec_mode_run()) is not None: + _DocTestRunner__run = _exec_mode_run_override + del _exec_mode_run_override + + +class _ExecModeRunner(_ExecModeRunnerMixin, doctest.DocTestRunner): + """:class:`doctest.DocTestRunner` that honours a ``{testcode}`` body.""" + + +class _ExecModeDebugRunner(_ExecModeRunnerMixin, doctest.DebugRunner): + """:class:`doctest.DebugRunner` that honours a ``{testcode}`` body.""" + + class TestDirective(Directive): """Base class for doctest-related directives.""" @@ -76,7 +492,6 @@ def run(self) -> list[Node]: code = "\n".join(self.content) test = None - logger.debug(f"directive run: self.name {self.name}") if self.name == "doctest": if "" in code: # convert s to ordinary blank lines for presentation @@ -95,7 +510,7 @@ def run(self) -> list[Node]: if self.arguments: groups = [x.strip() for x in self.arguments[0].split(",")] else: - groups = ["default"] + groups = [_DEFAULT_GROUP] node = nodetype(code, code, testnodetype=self.name, groups=groups) self.set_source_info(node) if test is not None: @@ -103,8 +518,13 @@ def run(self) -> list[Node]: node["test"] = test if self.name == "doctest": node["language"] = "pycon3" + elif self.name == "testcode": + node["language"] = "python" + elif self.name == "testoutput": + # don't try to highlight output + node["language"] = "none" node["options"] = {} - if self.name in ("doctest") and "options" in self.options: + if self.name in {"doctest", "testoutput"} and "options" in self.options: # parse doctest-like output comparison flags option_strings = self.options["options"].replace(",", " ").split() for option in option_strings: @@ -123,11 +543,23 @@ def run(self) -> list[Node]: continue flag = doctest.OPTIONFLAGS_BY_NAME[option[1:]] node["options"][flag] = option[0] == "+" + if self.name in _GROUPED_BLOCK_TYPES and "pyversion" in self.options: + # sphinx.ext.doctest declares :pyversion: on these directives and + # acts on it only for ``doctest``. Diverging either way costs more + # than it buys: honouring it would pass a page Sphinx fails, and + # rejecting it would fail a page Sphinx renders. Say so instead. + self.state.document.reporter.warning( + f"'pyversion' has no effect on '{self.name}'; " + "guard the block with ':skipif:' instead.", + line=self.lineno, + ) if self.name == "doctest" and "pyversion" in self.options: try: spec = self.options["pyversion"] python_version = ".".join([str(v) for v in sys.version_info[:3]]) - if not is_allowed_version(spec, python_version): + # Sphinx, which this was ported from, spells the signature + # (spec, version); gp-libs reversed it. The version goes first. + if not is_allowed_version(python_version, spec): flag = doctest.OPTIONFLAGS_BY_NAME["SKIP"] node["options"][flag] = True # Skip the test except InvalidSpecifier: @@ -141,6 +573,7 @@ def run(self) -> list[Node]: node["trim_flags"] = True elif "no-trim-doctest-flags" in self.options: node["trim_flags"] = False + logger.debug("parsed directive", extra={"doctest_block_type": self.name}) return [node] @@ -168,6 +601,38 @@ class DoctestDirective(TestDirective): } +class TestcodeDirective(TestDirective): + """Test code directive. + + Its body is plain Python a reader can select and paste: no prompt, and no + expected output unless a :class:`TestoutputDirective` follows it. + """ + + option_spec: t.ClassVar = { + "hide": directives.flag, + "no-trim-doctest-flags": directives.flag, + "pyversion": directives.unchanged_required, + "skipif": directives.unchanged_required, + "trim-doctest-flags": directives.flag, + } + + +class TestoutputDirective(TestDirective): + """Test output directive. + + Says what the :class:`TestcodeDirective` block above it prints. + """ + + option_spec: t.ClassVar = { + "hide": directives.flag, + "no-trim-doctest-flags": directives.flag, + "options": directives.unchanged, + "pyversion": directives.unchanged_required, + "skipif": directives.unchanged_required, + "trim-doctest-flags": directives.flag, + } + + class MockTabDirective(TestDirective): """Mock tab directive.""" @@ -185,6 +650,8 @@ def setup() -> dict[str, t.Any]: directives.register_directive("testsetup", TestsetupDirective) directives.register_directive("testcleanup", TestcleanupDirective) directives.register_directive("doctest", DoctestDirective) + directives.register_directive("testcode", TestcodeDirective) + directives.register_directive("testoutput", TestoutputDirective) # Third party mock directive: sphinx-inline-tabs @ 2022.01.02.beta11 directives.register_directive("tab", MockTabDirective) @@ -197,7 +664,14 @@ def setup() -> dict[str, t.Any]: parser = doctest.DocTestParser() _DIRECTIVES_READY = False -_REQUIRED_DIRECTIVES = ("doctest", "testsetup", "testcleanup", "tab") +_REQUIRED_DIRECTIVES = ( + "doctest", + "testsetup", + "testcleanup", + "testcode", + "testoutput", + "tab", +) def _directive_registry() -> dict[str, t.Any]: @@ -216,6 +690,921 @@ def _ensure_directives_registered() -> None: _DIRECTIVES_READY = True +def _node_groups(node: nodes.Element) -> list[str]: + """Return every doctest group a block declares, in the order written. + + Only the directive forms carry a ``groups`` attribute: ``.. doctest:: name`` + in reStructuredText and the ``{doctest} name`` fence in Markdown. Declaring + a group is the author asking blocks to share a namespace, so it holds at + every :data:`NamespaceScope`. + + ``default`` is the group a directive lands in when its author wrote no + argument, so it names nothing the author chose. A block that wants it is + asking for its page, which :func:`_page_scoped` answers. + + Parameters + ---------- + node : docutils.nodes.Element + Node a doctest was collected from. + + Returns + ------- + list[str] + Group names, empty for a block that named none. + + Examples + -------- + >>> from docutils import nodes + >>> _node_groups(nodes.literal_block("", "", groups=["intro"])) + ['intro'] + + A comma list names every group the block joins: + + >>> _node_groups(nodes.literal_block("", "", groups=["alpha", "beta"])) + ['alpha', 'beta'] + + A directive written without an argument names no group, and a plain fence + or a reStructuredText doctest block has nowhere to write one: + + >>> _node_groups(nodes.literal_block("", "", groups=["default"])) + [] + >>> _node_groups(nodes.doctest_block("", "")) + [] + + A ``{testcode}`` is read the same way, whatever it lands in: + + >>> _node_groups( + ... nodes.literal_block( + ... "", "", testnodetype="testcode", groups=["default"] + ... ) + ... ) + [] + """ + groups = node.get("groups") + if not isinstance(groups, list): + return [] + names = [str(group).strip() for group in groups] + return [name for name in names if name and name != _DEFAULT_GROUP] + + +def _page_scoped(node: nodes.Element, grouped_types: frozenset[str]) -> bool: + """Say whether a block shares its page whatever the scope says. + + A ``{testcode}`` exists so a visible block and the ``:hide:`` block + asserting on it read one namespace, which is the page. Sphinx spells that + the ``default`` group; here it is the name an ungrouped block already + carries at ``"document"`` scope, so the two forms meet on one page instead + of on a name no author wrote. + + Parameters + ---------- + node : docutils.nodes.Element + Node a doctest was collected from. + grouped_types : frozenset[str] + Block types that share their page. See :data:`_PHASE_BLOCK_TYPES` for + why a page can widen it. + + Returns + ------- + bool + `True` when the block shares its page. + + Examples + -------- + >>> from docutils import nodes + >>> code = nodes.literal_block("", "", testnodetype="testcode") + >>> _page_scoped(code, _GROUPED_BLOCK_TYPES) + True + + A prompt block keeps the scope the run asked for: + + >>> _page_scoped(nodes.doctest_block("", ""), _GROUPED_BLOCK_TYPES) + False + + A ``{testsetup}`` shares the page only where the page asked: + + >>> setup = nodes.literal_block("", "", testnodetype="testsetup") + >>> _page_scoped(setup, _GROUPED_BLOCK_TYPES) + False + >>> _page_scoped(setup, _GROUPED_BLOCK_TYPES | _PHASE_BLOCK_TYPES) + True + """ + return node.get("testnodetype") in grouped_types + + +def _namespace_name( + group: str | None, + scope: NamespaceScope, + document_name: str, + index: int, +) -> str: + """Return the name of the namespace a block runs in. + + The name is also the key blocks merge under and the pytest node id they + collect as, so two blocks share a namespace exactly when they share a name. + + Parameters + ---------- + group : str or None + Group the block declared, from :func:`_node_groups`. + scope : NamespaceScope + Scope chosen for blocks that declared no group. + document_name : str + Base name of the document, without its directory. + index : int + Position of the block in the document, counted from zero. + + Returns + ------- + str + Namespace name. + + Examples + -------- + A declared group names its own namespace at every scope: + + >>> _namespace_name("intro", "block", "page.md", 0) + 'intro' + >>> _namespace_name("intro", "document", "page.md", 0) + 'intro' + + A block that declared none is named for its position, or for the page when + the document shares one namespace: + + >>> _namespace_name(None, "block", "page.md", 3) + 'page.md[3]' + >>> _namespace_name(None, "document", "page.md", 3) + 'page.md' + """ + if group is not None: + return group + if scope == "document": + return document_name + return f"{document_name}[{index}]" + + +def _node_line(node: nodes.Element) -> int: + """Return the file line a block reports itself against. + + docutils leaves ``line`` unset on a doctest block nested inside a + directive, a list item, or a block quote. The node holding it still carries + one, which puts the block within a few lines of its prompts instead of at + the top of the page. + + Parameters + ---------- + node : docutils.nodes.Element + Node the block was collected from. + + Returns + ------- + int + Line to position and report the block against, ``0`` when nothing up + the tree carries one. + + Examples + -------- + >>> from docutils import nodes + >>> block = nodes.doctest_block("", "") + >>> block.line = 6 + >>> _node_line(block) + 6 + + A block the parser left unpositioned borrows the line of whatever holds it: + + >>> nested = nodes.doctest_block("", "") + >>> admonition = nodes.note("", nested) + >>> admonition.line = 7 + >>> _node_line(nested) + 7 + + >>> _node_line(nodes.doctest_block("", "")) + 0 + """ + current: Node | None = node + while current is not None: + if current.line: + return int(current.line) + current = current.parent + return 0 + + +def _skipif(expression: str, globs: dict[str, t.Any]) -> bool: + """Return whether a block's ``:skipif:`` expression asks to skip the block. + + The expression is Python source read from the document and **evaluated**, + the contract :mod:`sphinx.ext.doctest` documents. It sees a copy of the + globals the document starts with — the `globs` handed to + :meth:`DocutilsDocTestFinder.find` — and nothing the page's own examples + bound, because it is answered while the page is being read, before any of + them run. + + Sphinx seeds that namespace from its ``doctest_global_setup`` setting; + gp-libs has no such setting, so it binds :mod:`sys` instead unless the + document bound the name itself. Without it the option could not answer the + two questions it is written for, the Python version and the platform. + + Parameters + ---------- + expression : str + Python expression from the directive's ``:skipif:`` option. + globs : dict[str, typing.Any] + Globals the document starts with. + + Returns + ------- + bool + Whether the block's examples are marked :data:`doctest.SKIP`. + + Examples + -------- + >>> _skipif("True", {}) + True + >>> _skipif("False", {}) + False + + :mod:`sys` answers for the interpreter running the page: + + >>> _skipif("sys.version_info < (3, 10)", {}) + False + + The document's starting globals are in scope, and win: + + >>> _skipif("greeting == 'hello'", {"greeting": "hello"}) + True + """ + # eval is the option's contract, not an oversight: sphinx.ext.doctest + # defines :skipif: as a Python expression. The expression comes from a + # document the project already runs as tests, so it grants no reach the + # page's own examples do not have. It is evaluated while the document is + # collected, which means ``--collect-only`` runs it too. + return bool(eval(expression, {"sys": sys, **globs})) + + +def _gated(node: nodes.Element, filename: str, globs: dict[str, t.Any]) -> bool: + """Return whether a block's ``:skipif:`` asks for it to be passed over. + + Parameters + ---------- + node : docutils.nodes.Element + Node a block was collected from. + filename : str + Path a failed expression is reported against. + globs : dict[str, typing.Any] + Globals the document starts with. + + Returns + ------- + bool + Whether the block is gated. A block that wrote no ``:skipif:`` is not. + + Raises + ------ + SkipifExpressionError + If the expression cannot be evaluated. + + Examples + -------- + >>> from docutils import nodes + >>> _gated(nodes.literal_block("", ""), "page.rst", {}) + False + >>> _gated(nodes.literal_block("", "", skipif="True"), "page.rst", {}) + True + + A broken expression names the block it was written on: + + >>> try: + ... _gated(nodes.literal_block("", "", skipif="nope"), "page.rst", {}) + ... except SkipifExpressionError as exc: + ... print(exc) + page.rst:0: :skipif: 'nope' failed: name 'nope' is not defined + """ + skipif = node.get("skipif") + if skipif is None: + return False + try: + return _skipif(skipif, globs) + except Exception as exc: + raise SkipifExpressionError(skipif, filename, _node_line(node), exc) from exc + + +def _pair_testoutput( + block_nodes: list[nodes.Element], + filename: str, + globs: dict[str, t.Any], +) -> tuple[list[nodes.Element], dict[int, nodes.Element]]: + r"""Hand each ``{testoutput}`` to the ``{testcode}`` it follows. + + A ``{testoutput}`` is not a block of its own: it says what an earlier block + prints, as in :mod:`sphinx.ext.doctest`. Pairing is tracked per group, the + way :meth:`sphinx.ext.doctest.TestGroup.add_code` tracks it: the output + joins the last block its groups saw, and only when that block is a + ``{testcode}`` still waiting for one. Two groups can therefore run their + blocks interleaved and still each get their own output, while any other + block of the group in between — a ``{doctest}``, a second ``{testcode}`` — + closes the pairing. A stray is dropped with a warning rather than collected + as a test that checks nothing. + + A second ``{testoutput}`` for one ``{testcode}`` replaces the first, as + :mod:`sphinx.ext.doctest` does, so a page reads the same here as it builds + there. Unlike Sphinx it says so: the page kept two answers to one question + and only one of them ran. + + A gated ``{testoutput}`` is dropped as well, which leaves its + ``{testcode}`` expecting no output — what :mod:`sphinx.ext.doctest` does + when a ``:skipif:`` takes the node out of the doctree. + + Parameters + ---------- + block_nodes : list[docutils.nodes.Element] + Every block the page holds, in document order. + filename : str + Path warnings and failed expressions are reported against. + globs : dict[str, typing.Any] + Globals the document starts with. + + Returns + ------- + tuple[list[docutils.nodes.Element], dict[int, docutils.nodes.Element]] + Blocks left to collect, and the output node each ``{testcode}`` was + given, keyed by :func:`id`. + + Examples + -------- + >>> from docutils import nodes + >>> def block(kind, text, *groups): + ... return nodes.literal_block( + ... text, text, testnodetype=kind, groups=list(groups) or ["default"] + ... ) + >>> code, output = block("testcode", "print(1)"), block("testoutput", "1") + >>> blocks, wants = _pair_testoutput([code, output], "page.md", {}) + >>> [held["testnodetype"] for held in blocks] + ['testcode'] + >>> wants[id(code)].astext() + '1' + + A second one replaces the first, and the page hears about it: + + >>> blocks, wants = _pair_testoutput( + ... [code, output, block("testoutput", "2")], "page.md", {} + ... ) + >>> len(blocks), wants[id(code)].astext() + (1, '2') + + An intervening block closes the pairing, so a later output is a stray: + + >>> _, wants = _pair_testoutput( + ... [code, block("doctest", ">>> 1\n1"), block("testoutput", "2")], + ... "page.md", + ... {}, + ... ) + >>> wants + {} + + Two groups can run interleaved and still each be answered: + + >>> alpha = block("testcode", "print('a')", "alpha") + >>> beta = block("testcode", "print('b')", "beta") + >>> _, wants = _pair_testoutput( + ... [alpha, beta, block("testoutput", "a", "alpha")], "page.md", {} + ... ) + >>> wants[id(alpha)].astext(), id(beta) in wants + ('a', False) + """ + blocks: list[nodes.Element] = [] + wants: dict[int, nodes.Element] = {} + # Last block each group saw, and whether it is still open to an output. + # ``None`` records a group whose latest block cannot take one. + pending: dict[str, nodes.Element | None] = {} + for node in block_nodes: + groups = _node_groups(node) or [_DEFAULT_GROUP] + if node.get("testnodetype") != "testoutput": + blocks.append(node) + open_to_output = node.get("testnodetype") == "testcode" + for group in groups: + pending[group] = node if open_to_output else None + continue + if _gated(node, filename, globs): + continue + above = next( + (pending[group] for group in groups if pending.get(group) is not None), + None, + ) + if above is None: + logger.warning( + "testoutput block follows no testcode of its group", + extra={ + "doctest_source_file": filename, + "doctest_block_type": "testoutput", + }, + ) + continue + if id(above) in wants: + logger.warning( + "testoutput block replaces the one above it", + extra={ + "doctest_source_file": filename, + "doctest_block_type": "testoutput", + }, + ) + # Left open, so a third replaces the second, as sphinx.ext.doctest does. + wants[id(above)] = node + return blocks, wants + + +def _runs_as_exec(block_type: str, source: str) -> bool: + r"""Report whether a block's body runs as a module body rather than prompts. + + A ``{testcode}`` always does. A ``{testsetup}`` or ``{testcleanup}`` does + when it carries no ``>>>``, which is how :mod:`sphinx.ext.doctest` writes + one — Sphinx runs those bodies through the same ``exec`` its ``{testcode}`` + uses, and rejects a prompt outright. gp-libs has always read them with + :class:`doctest.DocTestParser` instead, so both spellings have to work: the + prompt decides which. + + Parameters + ---------- + block_type : str + Block type, from the node's ``testnodetype``. + source : str + Body of the block. + + Returns + ------- + bool + Whether to build the block as one ``exec``-mode example. + + Examples + -------- + >>> _runs_as_exec("testcode", "value = 41") + True + + A setup block written either way is read the way it was written: + + >>> _runs_as_exec("testsetup", "base = 40") + True + >>> _runs_as_exec("testsetup", ">>> base = 40") + False + + Nothing else changes mode, and an empty body has no statement to run: + + >>> _runs_as_exec("doctest", "base = 40"), _runs_as_exec("testsetup", " ") + (False, False) + """ + if block_type == "testcode": + return True + if block_type not in _PHASE_BLOCK_TYPES or not source.strip(): + return False + return _PROMPT_RE.search(source) is None + + +def _testcode_test( + source: str, + want: str, + options: dict[int, bool], + name: str, + filename: str, + lineno: int, + globs: dict[str, t.Any], +) -> doctest.DocTest: + r"""Return the test one ``{testcode}`` block runs as. + + The block is a single example: its body is the source, the + ``{testoutput}`` below it is the expected output, and the source is marked + :class:`_ExecSource` so the runner compiles it as a module body. + ```` is off unless the page turns it back on, as in + :mod:`sphinx.ext.doctest` — a blank line in a ``{testoutput}`` block is + just a blank line. + + Parameters + ---------- + source : str + Body of the ``{testcode}`` block. + want : str + Output the block is expected to print, empty when none was given. + options : dict[int, bool] + Doctest flags from the ``{testoutput}`` block's ``:options:``. + name : str + Name the test collects under. + filename : str + Path failures are reported against. + lineno : int + Line the block sits on. + globs : dict[str, typing.Any] + Globals the namespace starts with. + + Returns + ------- + doctest.DocTest + One example, ready to merge with the rest of its namespace. + + Examples + -------- + >>> test = _testcode_test("print(2 + 2)", "4", {}, "page.md", "page.md", 7, {}) + >>> test.name, test.lineno, len(test.examples) + ('page.md', 7, 1) + >>> example = test.examples[0] + >>> example.source, example.want + ('print(2 + 2)\n', '4\n') + >>> isinstance(example.source, _ExecSource) + True + + A ``{testoutput}`` spelling a traceback checks the exception instead: + + >>> raiser = _testcode_test( + ... "raise ValueError('boom')", + ... "Traceback (most recent call last):\n ...\nValueError: boom", + ... {}, + ... "page.md", + ... "page.md", + ... 0, + ... {}, + ... ) + >>> raiser.examples[0].exc_msg + 'ValueError: boom\n' + """ + # Normalized before the match so the captured message ends in the newline + # ``traceback.format_exception_only`` puts on the line it is checked against. + if want and not want.endswith("\n"): + want += "\n" + match = doctest.DocTestParser._EXCEPTION_RE.match(want) # type: ignore[attr-defined] + example = doctest.Example( + source, + want, + exc_msg=match.group("msg") if match else None, + options={doctest.DONT_ACCEPT_BLANKLINE: True, **options}, + ) + example.source = _ExecSource(example.source) + # A reader is shown ``lines[example.lineno - 9 : example.lineno + 1]`` of the + # block and sent to ``test.lineno + example.lineno + 1``. One example holding + # a whole block would show its first line only and send the reader there, so + # the example sits on the block's last line: the report then quotes the block + # entire and lands inside it rather than above the failure. + example.lineno = max(len(source.splitlines()) - 1, 0) + return doctest.DocTest([example], globs, name, filename, lineno, source) + + +def _merge_blocks( + blocks: list[doctest.DocTest], + name: str, + filename: str, + globs: dict[str, t.Any], + keep: list[doctest.DocTest] | None = None, +) -> doctest.DocTest: + r"""Merge one namespace's blocks into a single test. + + Each block keeps the line docutils reported for it, with blank lines + standing in for the prose between two blocks, so a merged example reports + the line it reports on its own and the ``%03d`` gutter of a failure still + counts up to the failing ``>>>``. + + A block the lines above already reach follows them instead. Two blocks can + claim overlapping lines: an ``.. include::`` numbers its nodes against the + included file, and a reStructuredText doctest block reports its *last* + line, so its examples already report lines further down the page than the + block occupies. An included block can therefore hold the lowest line number + of the namespace and anchor the merged test on the included file's + coordinates, which pads the page's own blocks out to their distance from + it. + + Parameters + ---------- + blocks : list[doctest.DocTest] + Blocks of one namespace, each parsed on its own, in the order they run. + Each kept block's ``example.lineno`` is shifted **in place**, so no + block may be merged twice while `keep` holds it — which is why + :meth:`DocutilsDocTestFinder._find` leaves a lifted block out of `keep` + before merging that block on its own. + name : str + Namespace name, which becomes the test name. + filename : str + Path failures are reported against. + globs : dict[str, typing.Any] + Globals the namespace starts with. + keep : list[doctest.DocTest] or None + Blocks whose examples the merged test runs, compared by identity. + `None`, the default, keeps every block. A block left out still + contributes its source and its spacing, so the blocks around it report + the lines they reported before and a failure's gutter still shows what + was passed over — only its examples are dropped. + + Returns + ------- + doctest.DocTest + One test, holding the examples of every block in `keep`, laid out + across the page the blocks came from. + + Examples + -------- + Two blocks six lines apart keep that distance, and each example reports the + line its prompt sits on: + + >>> parser = doctest.DocTestParser() + >>> def block(line, source): + ... return parser.get_doctest(source, {}, "page.md", "page.md", line) + >>> merged = _merge_blocks( + ... [block(3, ">>> greeting = 'hello'"), + ... block(9, ">>> greeting.upper()\n'HELLO'")], + ... "page.md", + ... "page.md", + ... {}, + ... ) + >>> merged.name, merged.lineno + ('page.md', 3) + >>> merged.docstring.splitlines() + [">>> greeting = 'hello'", '', '', '', '', '', '>>> greeting.upper()', "'HELLO'"] + >>> [merged.lineno + example.lineno + 1 for example in merged.examples] + [4, 10] + + A block whose line the one above already covers is appended after it: + + >>> merged = _merge_blocks( + ... [block(3, ">>> one = 1\n>>> two = 2\n>>> three = 3"), + ... block(4, ">>> one + two")], + ... "page.md", + ... "page.md", + ... {}, + ... ) + >>> merged.docstring.splitlines() + ['>>> one = 1', '>>> two = 2', '>>> three = 3', '>>> one + two'] + """ + # Laid out by where each block sits on the page, but run in the order + # given: a namespace hands its blocks over as setup, tests, cleanup, which + # is rarely the order a reader meets them. Anchoring the text on the caller + # ordering would report every example against whichever block happened to + # come first in that sequence. + in_page_order = sorted(blocks, key=lambda block: block.lineno or 0) + origin = in_page_order[0].lineno or 0 + lines: list[str] = [] + offsets: dict[int, int] = {} + for block in in_page_order: + offset = max((block.lineno or 0) - origin, len(lines)) + lines.extend([""] * (offset - len(lines))) + lines.extend((block.docstring or "").splitlines()) + offsets[id(block)] = offset + examples: list[doctest.Example] = [] + for block in blocks: + # A dropped block still pads and still shows its source, so the blocks + # after it keep the lines they reported before and a failure's gutter + # still shows what was passed over. Its examples are left out entirely: + # whoever takes them next positions them itself. + if keep is not None and not any(block is kept for kept in keep): + continue + for example in block.examples: + # Positioned on a copy, so merging reads its blocks rather than + # consuming them: a block may be merged again — into a second group + # it named, or on its own once lifted — and still report the line it + # sits on. ``options`` is copied too, since a shallow copy would + # hand both merges the same mutable mapping. + shifted = copy.copy(example) + shifted.options = dict(example.options) + shifted.lineno += offsets[id(block)] + examples.append(shifted) + return doctest.DocTest( + examples, + globs, + name, + filename, + origin, + "\n".join(lines), + ) + + +class _CollectedBlock(t.NamedTuple): + """One block of a page, parsed against one namespace. + + Attributes + ---------- + position : int + Where the block sits in the document, counted from zero. It is the + number a block's name carries at ``"block"`` scope, and the number a + block lifted out of its namespace is named for. Spelled ``position`` + rather than ``index`` because a :class:`tuple` already has an + ``index``. + block_type : str + ``doctest``, ``testsetup``, ``testcleanup``, or the node's tag name. + test : doctest.DocTest + The block's examples, with its directive options already merged in. + """ + + position: int + block_type: str + test: doctest.DocTest + + +def _all_examples_skipped(test: doctest.DocTest) -> bool: + r"""Return whether every example of `test` carries :data:`doctest.SKIP`. + + This is the question ``_pytest.doctest._check_all_skipped`` asks of an item + before running it, and the answer decides whether pytest reports the item + ``SKIPPED``. Asking it of a single block says whether that block would + report, were it an item of its own. + + A block holding no examples answers ``False``: there is nothing in it to + skip, and nothing for a reader to be told about. + + Parameters + ---------- + test : doctest.DocTest + Examples of one block. + + Returns + ------- + bool + Whether none of the block's examples is left to run. + + Examples + -------- + >>> parser = doctest.DocTestParser() + >>> def block(source): + ... return parser.get_doctest(source, {}, "page.rst", "page.rst", 0) + + >>> _all_examples_skipped(block(">>> 1 / 0 # doctest: +SKIP\n")) + True + >>> _all_examples_skipped(block(">>> 2 + 2\n4\n")) + False + + A block only half of whose examples are gated still has one to run: + + >>> _all_examples_skipped( + ... block(">>> 1 / 0 # doctest: +SKIP\n>>> 2 + 2\n4\n") + ... ) + False + + >>> _all_examples_skipped(block("Prose, and no prompts at all.\n")) + False + """ + return bool(test.examples) and all( + example.options.get(doctest.SKIP, False) for example in test.examples + ) + + +def _split_skipped_blocks( + blocks: list[_CollectedBlock], +) -> tuple[list[_CollectedBlock], list[_CollectedBlock]]: + r"""Split a namespace's blocks into the ones it keeps and the ones it lifts out. + + A block whose every example is skipped binds nothing, so the namespace + reaches the same state with it or without it. Merged in, though, it is + silent: pytest reports a namespace skipped only when *no* example in it is + left to run, so one gated block among running ones reports as a pass. + Lifting it back out gives it an item of its own, which reports. + + A namespace with nothing left to run keeps every block, so it reports + skipped once as a namespace rather than once per block. + + Parameters + ---------- + blocks : list[_CollectedBlock] + Every block of one namespace, in the order it runs. + + Returns + ------- + tuple[list[_CollectedBlock], list[_CollectedBlock]] + Blocks the namespace keeps, and blocks that become items of their own. + The first is never empty: the namespace lifts a block out only when + another one is left to run. + + Examples + -------- + >>> parser = doctest.DocTestParser() + >>> def block(index, source): + ... return _CollectedBlock( + ... index, + ... "doctest", + ... parser.get_doctest(source, {}, "page.rst", "page.rst", index), + ... ) + + The gated block of a namespace that still runs is lifted out: + + >>> kept, lifted = _split_skipped_blocks([ + ... block(0, ">>> value = 1\n"), + ... block(1, ">>> value = 999 # doctest: +SKIP\n"), + ... block(2, ">>> value\n1\n"), + ... ]) + >>> [held.position for held in kept], [held.position for held in lifted] + ([0, 2], [1]) + + A namespace with nothing left to run keeps its blocks, so the one item it + collects as reports skipped once: + + >>> kept, lifted = _split_skipped_blocks([ + ... block(0, ">>> 1 / 0 # doctest: +SKIP\n"), + ... block(1, ">>> 2 / 0 # doctest: +SKIP\n"), + ... ]) + >>> [held.position for held in kept], [held.position for held in lifted] + ([0, 1], []) + """ + runnable = any( + not example.options.get(doctest.SKIP, False) + for held in blocks + for example in held.test.examples + ) + if not runnable: + return list(blocks), [] + return ( + [held for held in blocks if not _all_examples_skipped(held.test)], + [held for held in blocks if _all_examples_skipped(held.test)], + ) + + +def _lifted_name(namespace: str, position: int) -> str: + """Return the name a block lifted out of `namespace` collects under. + + It is the namespace's own name with the block's document position, the + same ``name[n]`` shape a block that names no group already carries at + ``"block"`` scope. So a gated block reads the same in ``--collect-only`` + and answers to the same node id whether the page shares a namespace or + not, and it cannot collide with the namespace it came out of. + + Parameters + ---------- + namespace : str + Namespace the block was lifted out of. + position : int + Where the block sits in the document, counted from zero. + + Returns + ------- + str + Name for the block's own test. + + Examples + -------- + >>> _lifted_name("intro", 3) + 'intro[3]' + + A page sharing one namespace names its blocks as ``"block"`` scope would: + + >>> _lifted_name("page.md", 3) + 'page.md[3]' + """ + return f"{namespace}[{position}]" + + +def _block_name(namespace: str, document_name: str, position: int) -> str: + """Return the name one block collects under when its namespace keeps it apart. + + A namespace laid out ``"per-block"`` is many tests, so each needs a name. + It is the namespace's own name with the block's document position — the + same ``name[n]`` shape :func:`_lifted_name` gives a gated block — so the + node id a reader pastes back to pytest does not move with the layout. A + namespace already named for this one block adds nothing. + + Parameters + ---------- + namespace : str + Namespace the block runs in. + document_name : str + Base name of the document, without its directory. + position : int + Where the block sits in the document, counted from zero. + + Returns + ------- + str + Name for the block's own test. + + Examples + -------- + A group numbers its blocks by where they sit on the page: + + >>> _block_name("intro", "page.md", 1) + 'intro[1]' + + A page sharing one namespace numbers them the same way: + + >>> _block_name("page.md", "page.md", 1) + 'page.md[1]' + + A block that has a namespace to itself already carries the number: + + >>> _block_name("page.md[1]", "page.md", 1) + 'page.md[1]' + """ + if namespace == _namespace_name(None, "block", document_name, position): + return namespace + return _lifted_name(namespace, position) + + +class _CollectedTest(t.NamedTuple): + """One test a page collected, and the namespace it was collected into. + + Attributes + ---------- + namespace : str + Namespace the test's examples run against. Under + :data:`NAMESPACE_ITEMS` ``"per-block"`` a namespace's tests hold one + globals mapping between them, which makes the namespace the unit a + caller distributing tests across processes cannot split. + test : doctest.DocTest + Examples, ready to run. + """ + + namespace: str + test: doctest.DocTest + + class DocTestFinderNameDoesNotExist(ValueError): """Raised with doctest lookup name not provided.""" @@ -227,17 +1616,71 @@ def __init__(self, string: str) -> None: class DocutilsDocTestFinder: - """DocTestFinder for doctest-docutils. + r"""DocTestFinder for doctest-docutils. Class used to extract the DocTests relevant to a docutils file. Doctests are extracted from the following directive types: doctest_block (doctest), DocTestDirective. Myst-parser is also supported for parsing markdown files. + + Blocks that name the same group — ``.. doctest:: intro`` in + reStructuredText, ``{doctest} intro`` in Markdown — are one test, so a name + bound in the group's first block is still bound in its last. Blocks that + name no group get a namespace each unless `namespace_scope` widens them to + the page. + + A block whose every example is skipped is the exception: it binds nothing, + so it comes back as a test of its own, named for its namespace and for + where it sits on the page, and reports as the skip it is instead of + vanishing into a namespace that runs. + + Examples + -------- + Two blocks in group ``intro`` come back as one test named for the group: + + >>> page = "\n".join([ + ... "```{doctest} intro", + ... ">>> greeting = 'hello'", + ... "```", + ... "", + ... "Narrative prose between the blocks.", + ... "", + ... "```{doctest} intro", + ... ">>> greeting.upper()", + ... "'HELLO'", + ... "```", + ... ]) + >>> tests = DocutilsDocTestFinder().find(page, "page.md") + >>> [(test.name, len(test.examples)) for test in tests] + [('intro', 2)] + + A gated block between them is its own test, named for where it sits: + + >>> gated = page.replace( + ... "Narrative prose between the blocks.", + ... "```{doctest} intro\n>>> greeting = 'nope' # doctest: +SKIP\n```", + ... ) + >>> [(test.name, len(test.examples)) for test in + ... DocutilsDocTestFinder().find(gated, "page.md")] + [('intro', 2), ('intro[1]', 1)] + + `namespace_items` decides whether that sharing costs the blocks their own + tests. Under ``"per-block"`` the group is two tests again, holding one + globals mapping between them: + + >>> per_block = DocutilsDocTestFinder(namespace_items="per-block") + >>> tests = per_block.find(page, "page.md") + >>> [(test.name, len(test.examples)) for test in tests] + [('intro[0]', 1), ('intro[1]', 1)] + >>> tests[0].globs is tests[1].globs + True """ def __init__( self, verbose: bool = False, parser: doctest.DocTestParser = parser, + namespace_scope: NamespaceScope = DEFAULT_NAMESPACE_SCOPE, + namespace_items: NamespaceItems = DEFAULT_NAMESPACE_ITEMS, ) -> None: """Create a new doctest finder. @@ -245,10 +1688,33 @@ def __init__( to create new DocTest objects (or objects that implement the same interface as DocTest). The signature for this factory function should match the signature of the DocTest constructor. + + Parameters + ---------- + verbose : bool + Log each document as it is searched. + parser : doctest.DocTestParser + Parser that turns a block's source into a :class:`doctest.DocTest`. + namespace_scope : NamespaceScope + Namespace a block that names no group runs in: ``"block"`` gives it + one of its own, ``"document"`` shares one across the page. + namespace_items : NamespaceItems + What a namespace comes back as: ``"merged"`` gives one test holding + every block's examples, ``"per-block"`` gives one test per block, + each handed the namespace's globals mapping rather than a copy. + + Raises + ------ + NamespaceScopeError + If `namespace_scope` names no known scope. + NamespaceItemsError + If `namespace_items` names no known layout. """ _ensure_directives_registered() self._parser = parser self._verbose = verbose + self._namespace_scope = _parse_namespace_scope(namespace_scope) + self._namespace_items = _parse_namespace_items(namespace_items) def find( self, @@ -257,13 +1723,103 @@ def find( globs: dict[str, t.Any] | None = None, extraglobs: dict[str, t.Any] | None = None, ) -> list[doctest.DocTest]: - """Return list of the DocTests defined by given string (its parsed directives). - - The globals for each DocTest is formed by combining `globs` and `extraglobs` - (bindings in `extraglobs` override bindings in `globs`). A new copy of the - globals dictionary is created for each DocTest. If `globs` is not specified, + r"""Return list of the DocTests defined by given string (its parsed directives). + + One DocTest comes back per namespace, plus one for each fully skipped + block a running namespace lifted out: the blocks that share a namespace + are merged into one, and the rest stand alone. The globals for each + DocTest is formed by combining `globs` and `extraglobs` (bindings in + `extraglobs` override bindings in `globs`). A new copy of the globals + dictionary is created for each DocTest. If `globs` is not specified, then it defaults to the module's `__dict__`, if specified, or {} otherwise. If `extraglobs` is not specified, then it defaults to {}. + + A finder built with `namespace_items` ``"per-block"`` merges nothing: + one DocTest comes back per block, and the blocks of one namespace are + handed that namespace's globals rather than a copy each, so a caller + running them has to pass ``clear_globs=False``. + + Namespaces come back in the order a reader meets the first block of + each, and a namespace's own tests in setup, test, cleanup order — which + is document order exactly when no two namespaces interleave. + + Examples + -------- + >>> page = "\n".join(f"```python\n>>> {n}\n{n}\n```\n" for n in range(11)) + >>> [test.name for test in DocutilsDocTestFinder().find(page, "page.md")][-2:] + ['page.md[9]', 'page.md[10]'] + + A test is named for the page it came from, never for the path it was + collected under, so its pytest node id reads the same on every machine: + + >>> finder = DocutilsDocTestFinder() + >>> [test.name for test in finder.find(">>> 2 + 2\n4\n", "docs/page.rst")] + ['page.rst[0]'] + + A page sharing one namespace names a gated block the same way ``block`` + scope would, so the node id that selects it does not move with the + scope: + + >>> page = "\n".join([ + ... "```python", ">>> value = 1", "```", "", + ... "```python", ">>> value = 999 # doctest: +SKIP", "```", "", + ... "```python", ">>> value", "1", "```", + ... ]) + >>> shared = DocutilsDocTestFinder(namespace_scope="document") + >>> [test.name for test in shared.find(page, "page.md")] + ['page.md', 'page.md[1]'] + >>> [test.name for test in DocutilsDocTestFinder().find(page, "page.md")] + ['page.md[0]', 'page.md[1]', 'page.md[2]'] + """ + return [ + collected.test + for collected in self._collect(string, name, globs, extraglobs) + ] + + def _collect( + self, + string: str, + name: str | None = None, + globs: dict[str, t.Any] | None = None, + extraglobs: dict[str, t.Any] | None = None, + ) -> list[_CollectedTest]: + r"""Return every test a page holds, each beside the namespace it runs in. + + :meth:`find` is this without the namespaces. A caller that has to keep + one namespace's tests together needs the name they share: under + :data:`NAMESPACE_ITEMS` ``"per-block"`` they hold one globals mapping + between them, and a mapping is a Python object, so it does not cross + processes. + + Parameters + ---------- + string : str + Page source. + name : str or None + Path the page was read from, whose suffix picks the parser. + globs : dict[str, typing.Any] or None + Globals every namespace starts from. + extraglobs : dict[str, typing.Any] or None + Globals overriding `globs`. + + Returns + ------- + list[_CollectedTest] + Tests, each naming its namespace, namespaces in the order a reader + meets the first block of each and a namespace's own tests in setup, + test, cleanup order. + + Examples + -------- + >>> page = "\n".join([ + ... "```{doctest} intro", ">>> greeting = 'hello'", "```", "", + ... "```python", ">>> 2 + 2", "4", "```", + ... ]) + >>> finder = DocutilsDocTestFinder(namespace_items="per-block") + >>> [(held.namespace, held.test.name) for held in finder._collect( + ... page, "page.md" + ... )] + [('intro', 'intro[0]'), ('page.md[1]', 'page.md[1]')] """ # If name was not specified, then extract it from the string. if name is None: @@ -271,12 +1827,6 @@ def find( if name is None: raise DocTestFinderNameDoesNotExist(string=string) - # No access to a loader, so assume it's a normal - # filesystem path - source_lines = linecache.getlines(name) or None - if not source_lines: - source_lines = None - # Initialize globals, and merge in extraglobs. globs = {} if globs is None else globs.copy() if extraglobs is not None: @@ -284,54 +1834,32 @@ def find( if "__name__" not in globs: globs["__name__"] = "__main__" # provide a default module name - tests: list[doctest.DocTest] = [] + tests: list[_CollectedTest] = [] source_path: pathlib.Path | None = ( pathlib.Path(name) if name is not None else None ) - self._find(tests, string, name, source_lines, globs, {}, source_path) - # Sort the tests by alpha order of names, for consistency in - # verbose-mode output. This was a feature of doctest in Pythons - # <= 2.3 that got lost by accident in 2.4. It was repaired in - # 2.4.4 and 2.5. - tests.sort() + self._find(tests, string, name, globs, {}, source_path) + # ``_find`` appends in document-traversal order; leave it that way. + # ``DocTest.__lt__`` compares names, and a name carries its block index + # as text, so sorting runs ``page.md[10]`` ahead of ``page.md[1]``. return tests def _find( self, - tests: list[doctest.DocTest], + tests: list[_CollectedTest], string: str, name: str, - source_lines: list[str] | None, globs: dict[str, t.Any], seen: dict[int, int], source_path: pathlib.Path | None = None, ) -> None: """Find tests for the given string, and add them to `tests`.""" - if self._verbose: - logger.info(f"Finding tests in {name}") - # If we've already processed this string, then ignore it. if id(string) in seen: return seen[id(string)] = 1 - # Find a test for this string, and add it to the list of tests. - logger.debug( - "_find({})".format( - pprint.pformat( - { - "tests": tests, - "string": string, - "name": name, - "source_lines": source_lines, - "globs": globs, - "seen": seen, - }, - ), - ), - ) ext = pathlib.Path(name).suffix - logger.debug(f"parse, ext: {ext}") if ext == ".md": import myst_parser.parsers.docutils_ from myst_parser.config.main import MdParserConfig @@ -382,24 +1910,258 @@ def condition(node: Node) -> bool: or isinstance(node, nodes.doctest_block) ) - for idx, node in enumerate(findall(doc)(condition)): - logger.debug(f"() node: {node.astext()}") - assert isinstance(node, nodes.Element) - test_name = node.get("groups") - if isinstance(test_name, list): - test_name = test_name[0] - if test_name is None or test_name == "default": - test_name = f"{name}[{idx}]" - logger.debug(f"() node: {test_name}") - test = self._get_test( - string=node.astext(), - name=test_name, - filename=name, - globs=globs, - source_lines=[str(node.line)], + document_name = pathlib.Path(name).name + # Namespaces keep insertion order, so the merged tests come back in the + # order the reader meets each namespace's first block. Each holds its + # setup, test, and cleanup blocks apart: sphinx.ext.doctest runs a + # group's setup before its tests and its cleanup after, whatever order + # the page wrote them in, and a testsetup exists to be movable. + namespaces: dict[str, dict[str, list[_CollectedBlock]]] = {} + + found: list[nodes.Element] = [ + node for node in findall(doc)(condition) if isinstance(node, nodes.Element) + ] + # A page written in Sphinx's prompt-free style puts its setup in the + # same implicit group as its code. A page of prompt blocks keeps the + # namespaces it has always had, so the widening follows the testcode. + grouped_types = _GROUPED_BLOCK_TYPES + if any(node.get("testnodetype") == "testcode" for node in found): + grouped_types = _GROUPED_BLOCK_TYPES | _PHASE_BLOCK_TYPES + block_nodes, wants = _pair_testoutput(found, name, globs) + declared = [_node_groups(node) for node in block_nodes] + # A prompt-free block shares its page, so it is named for the page the + # way an ungrouped block already is at document scope. That is what + # lets the two forms meet there instead of on a name no author wrote. + scopes: list[NamespaceScope] = [ + "document" if _page_scoped(node, grouped_types) else self._namespace_scope + for node in block_nodes + ] + + def generated_name(idx: int) -> str: + return _namespace_name(None, scopes[idx], document_name, idx) + + # A block joins every group it names. ``*`` means every group the + # document declares, so it can only be resolved once the page has been + # read; a page whose only blocks are wildcards has no group to join, so + # each keeps its own namespace. + memberships: list[list[str]] = [ + [] if _WILDCARD_GROUP in groups else (groups or [generated_name(idx)]) + for idx, groups in enumerate(declared) + ] + ordered: list[str] = [] + for names in memberships: + for candidate in names: + if candidate not in ordered: + ordered.append(candidate) + for idx, groups in enumerate(declared): + if _WILDCARD_GROUP in groups: + memberships[idx] = list(ordered) or [generated_name(idx)] + + # A generated name and a declared one are the same string to everything + # downstream: the namespace mapping keys on it and the test is named + # for it. So a page that spells both has to say which it meant, and + # cannot. Checked against the names actually generated, not the shape + # they take, so a group named for a page whose blocks all declare one + # is left alone. A page of nothing but wildcards declares no name to + # collide with, which is why the fallback above needs no check. + generated = { + generated_name(idx) for idx, groups in enumerate(declared) if not groups + } + for group in sorted( + {name for groups in declared for name in groups} & generated, + ): + raise NamespaceNameCollisionError( + group, + document_name, + f"a block declaring none at {self._namespace_scope!r} scope", + ) + + for idx, node in enumerate(block_nodes): + block_type = str(node.get("testnodetype", node.tagname)) + lineno = _node_line(node) + # The block's own flags, before its examples get a say. A true + # ``:skipif:`` joins them as ``+SKIP``: one spelling of "do not run + # this" that a reader can predict from the other, and one path + # through the runner, which keeps the block collected, reported and + # selectable by node id instead of vanishing from the page. + options = dict(node.get("options") or {}) + gated = _gated(node, name, globs) + if gated: + logger.debug( + "doctest block skipped by skipif", + extra={ + "doctest_source_file": name, + "doctest_block_type": block_type, + }, + ) + # ``node["test"]`` is the source before the directive trimmed + # ``# doctest:`` flags out of the code a reader sees. Both + # spellings have the same line count, so either positions the + # block the same way. + source = str(node.get("test") or node.astext()) + output = wants.get(id(node)) + for namespace in memberships[idx]: + logger.debug( + "doctest block collected into namespace %s", + namespace, + extra={ + "doctest_source_file": name, + "doctest_block_type": block_type, + }, + ) + # Parsed once per namespace: _merge_blocks shifts + # ``example.lineno`` in place, so two namespaces sharing one + # block's examples would shift them twice. + test_name = ( + namespace + if self._namespace_items == "merged" + else _block_name(namespace, document_name, idx) + ) + test = ( + _testcode_test( + source=source, + want=output.astext() if output is not None else "", + options=dict(output.get("options") or {}) + if output is not None + else {}, + name=test_name, + filename=name, + lineno=lineno, + globs=globs, + ) + if _runs_as_exec(block_type, source) + else self._get_test( + string=source, + name=test_name, + filename=name, + globs=globs, + lineno=lineno, + ) + ) + if options or gated: + for example in test.examples: + # A directive's ``:options:`` set the block's defaults; + # an example's own inline flags win, as in + # sphinx.ext.doctest. + merged = dict(options) + merged.update(example.options) + if gated: + # A ``:skipif:`` is a gate, not a default. + # sphinx.ext.doctest drops the block before its + # source is ever read, so nothing written inside it + # can turn the gate off — and an example that did + # would run on exactly the interpreter or platform + # the condition was guarding against. + merged[doctest.SKIP] = True + example.options = merged + phases = namespaces.setdefault( + namespace, + {"testsetup": [], "test": [], "testcleanup": []}, + ) + phases[block_type if block_type in phases else "test"].append( + _CollectedBlock(idx, block_type, test), + ) + + # Anchored on the document position of the first block each test holds, + # so the tests come back in the order a reader meets them. A namespace + # anchors where it now starts, which is where it started before if it + # lifted nothing out. Ties — one block joining two groups — keep the + # order the page declared them in, which a stable sort preserves. + anchored: list[tuple[int, int, _CollectedTest]] = [] + for namespace, phases in namespaces.items(): + in_phase_order = [ + *phases["testsetup"], + *phases["test"], + *phases["testcleanup"], + ] + if self._namespace_items == "per-block": + # One mapping for the namespace, handed to every block of it. + # ``DocTest.__init__`` copies the globals it is given, so the + # mapping is assigned afterwards, as sphinx.ext.doctest does; + # whoever runs these tests has to leave it uncleared for the + # sharing to reach the block below. + shared = dict(globs) + anchor = min(held.position for held in in_phase_order) + for order, held in enumerate(in_phase_order): + held.test.globs = shared + anchored.append( + ( + # The namespace anchors as a whole, and its blocks + # keep phase order inside it: a testsetup written + # at the foot of the page still runs first. + anchor, + order, + _CollectedTest(namespace, held.test), + ), + ) + continue + kept, lifted = _split_skipped_blocks(in_phase_order) + anchored.append( + ( + # Every block anchors its namespace, lifted or not, so + # lifting the first one cannot let another namespace + # declared below it collect first. + min(held.position for held in in_phase_order), + # Ties with a block this namespace lifted break toward the + # block: it sits at that line, the namespace resumes later. + min(held.position for held in kept), + # Merged over every block, so the padding a lifted block + # contributed stays and the blocks after it keep the lines + # they reported before it was lifted. + _CollectedTest( + namespace, + _merge_blocks( + [held.test for held in in_phase_order], + namespace, + name, + globs, + keep=[held.test for held in kept], + ), + ), + ), + ) + for held in lifted: + lifted_name = _lifted_name(namespace, held.position) + # Lifting generates a name the same way declaring a group + # does, so it can land on one the page already declared. + if lifted_name in namespaces: + raise NamespaceNameCollisionError( + lifted_name, + document_name, + f"a block lifted out of {namespace!r}", + ) + logger.debug( + "skipped doctest block lifted out of namespace %s as %s", + namespace, + lifted_name, + extra={ + "doctest_source_file": name, + "doctest_block_type": held.block_type, + }, + ) + anchored.append( + ( + held.position, + held.position, + _CollectedTest( + namespace, + _merge_blocks([held.test], lifted_name, name, globs), + ), + ), + ) + anchored.sort(key=lambda entry: (entry[0], entry[1])) + tests.extend(collected for _, _, collected in anchored) + logger.debug( + "parsed document into %d test(s)", + len(anchored), + extra={"doctest_source_file": name}, + ) + if self._verbose: + logger.info( + "found %d test(s)", + len(anchored), + extra={"doctest_source_file": name}, ) - if test is not None: - tests.append(test) def _get_test( self, @@ -407,12 +2169,9 @@ def _get_test( name: str, filename: str, globs: dict[str, t.Any], - source_lines: list[str], + lineno: int, ) -> doctest.DocTest: - """Return a DocTest for given string, or return None.""" - lineno = int(source_lines[0]) - - # Return a DocTest for this string. + """Return a DocTest for one block's source.""" return self._parser.get_doctest(string, globs, name, filename, lineno) @@ -438,10 +2197,60 @@ def testdocutils( raise_on_error: bool = False, parser: doctest.DocTestParser = parser, encoding: str | None = None, + namespace_scope: NamespaceScope = DEFAULT_NAMESPACE_SCOPE, + namespace_items: NamespaceItems = DEFAULT_NAMESPACE_ITEMS, ) -> doctest.TestResults: - """Docutils-based test entrypoint. + r"""Docutils-based test entrypoint. Based on doctest.testfile at python 3.10 + + Parameters + ---------- + namespace_scope : NamespaceScope + Namespace the blocks that name no group run in. See + :class:`DocutilsDocTestFinder`; the other parameters follow + :func:`doctest.testfile`. + namespace_items : NamespaceItems + Whether a namespace runs as one test or as one test per block. Running + a file has no scheduler to split the blocks across, so ``"per-block"`` + shares state here as it does in a serial pytest run. + + Returns + ------- + doctest.TestResults + Failed examples, and examples attempted. + + Examples + -------- + A page whose second block reads a name the first one bound fails while each + block keeps its own namespace, and passes once the page shares one: + + >>> import contextlib, io, pathlib, tempfile + >>> directory = tempfile.TemporaryDirectory() + >>> page = pathlib.Path(directory.name) / "page.rst" + >>> _ = page.write_text( + ... ">>> greeting = 'hello'\n\n>>> greeting.upper()\n'HELLO'\n", + ... encoding="utf-8", + ... ) + + >>> def run(**kwargs): + ... with contextlib.redirect_stdout(io.StringIO()): + ... return testdocutils( + ... str(page), module_relative=False, report=False, **kwargs + ... ) + + >>> run() + TestResults(failed=1, attempted=2) + + >>> run(namespace_scope="document") + TestResults(failed=0, attempted=2) + + Keeping each block a test of its own shares the page just the same: + + >>> run(namespace_scope="document", namespace_items="per-block") + TestResults(failed=0, attempted=2) + + >>> directory.cleanup() """ global master @@ -470,17 +2279,23 @@ def testdocutils( globs["__name__"] = "__main__" # Find, parse, and run all tests in the given module. - finder = DocutilsDocTestFinder() + finder = DocutilsDocTestFinder( + namespace_scope=namespace_scope, + namespace_items=namespace_items, + ) runner: doctest.DebugRunner | doctest.DocTestRunner if raise_on_error: - runner = doctest.DebugRunner(verbose=verbose, optionflags=optionflags) + runner = _ExecModeDebugRunner(verbose=verbose, optionflags=optionflags) else: - runner = doctest.DocTestRunner(verbose=verbose, optionflags=optionflags) + runner = _ExecModeRunner(verbose=verbose, optionflags=optionflags) + # A namespace laid out per block hands its tests one mapping between them, + # which the runner would otherwise empty after running the first of them. + clear_globs = namespace_items != "per-block" for test in finder.find(text, filename, globs=globs, extraglobs=extraglobs): - runner.run(test) + runner.run(test, clear_globs=clear_globs) if report: runner.summarize() @@ -544,6 +2359,28 @@ def _test() -> int: action="store_true", help=("Force parsing using docutils (reStructuredText, markdown)"), ) + p.add_argument( + "--namespace-scope", + action="store", + choices=NAMESPACE_SCOPES, + default=DEFAULT_NAMESPACE_SCOPE, + help=( + "namespace the blocks that name no group run in: block (default," + " one each) or document (one for the page); blocks that name a" + " group always share that group's namespace" + ), + ) + p.add_argument( + "--namespace-items", + action="store", + choices=NAMESPACE_ITEMS, + default=DEFAULT_NAMESPACE_ITEMS, + help=( + "what a namespace runs as: merged (default, one item holding every" + " block of it) or per-block (one item per block, sharing the" + " namespace between them)" + ), + ) p.add_argument("file", nargs="+", help="file containing the tests to run") args = p.parse_args() @@ -569,6 +2406,8 @@ def _test() -> int: module_relative=False, verbose=verbose, optionflags=options, + namespace_scope=args.namespace_scope, + namespace_items=args.namespace_items, ) elif filename.endswith(".py"): # It is a module -- insert its dir into sys.path and try to diff --git a/src/pytest_doctest_docutils.py b/src/pytest_doctest_docutils.py index 13c2db0..1c13bf7 100644 --- a/src/pytest_doctest_docutils.py +++ b/src/pytest_doctest_docutils.py @@ -12,9 +12,11 @@ from __future__ import annotations import bdb +import collections import doctest import io import logging +import pathlib import sys import typing as t @@ -23,12 +25,26 @@ from _pytest import outcomes from _pytest.outcomes import OutcomeException -from doctest_docutils import DocutilsDocTestFinder, _ensure_directives_registered +from doctest_docutils import ( + _HIDE_FLAG, + DEFAULT_NAMESPACE_ITEMS, + DEFAULT_NAMESPACE_SCOPE, + NAMESPACE_ITEMS, + NAMESPACE_SCOPES, + DocutilsDocTestFinder, + NamespaceItems, + NamespaceItemsError, + NamespaceScope, + NamespaceScopeError, + _ensure_directives_registered, + _ExecModeRunnerMixin, + _parse_namespace_items, + _parse_namespace_scope, +) if t.TYPE_CHECKING: - import pathlib import types - from collections.abc import Iterable + from collections.abc import Generator, Iterable, Sequence from doctest import _Out from _pytest.config.argparsing import Parser @@ -43,6 +59,43 @@ # Lazy definition of runner class RUNNER_CLASS = None +#: Namespace scope resolved once at configure time, read back during collection. +_NAMESPACE_SCOPE_KEY = pytest.StashKey[NamespaceScope]() + +#: Namespace layout resolved once at configure time, read back during collection. +_NAMESPACE_ITEMS_KEY = pytest.StashKey[NamespaceItems]() + +#: Whether the run asked for a ``--dist`` scheduler by name, captured before +#: pytest-xdist rewrites the value ``-n`` alone leaves behind. +_DIST_NAMED_KEY = pytest.StashKey[bool]() + +_NAMESPACE_HELP = ( + "namespace the doctest blocks of one .rst/.md file run in when they name" + " no group: block (default, one each) or document (one for the page);" + " blocks naming a group always share that group's namespace" +) + +_ITEMS_HELP = ( + "what a namespace collects as: merged (default, one item holding every" + " block of it) or per-block (one item per block, keeping their node ids" + " and sharing the namespace between them)" +) + +#: The ``--dist`` values that keep every item of one file on one worker, which +#: is what a shared namespace needs: a globals mapping is a Python object, so +#: it does not cross processes. Named as an allowlist rather than a list of +#: splitting schedulers so that a scheduler pytest-xdist adds later is handled +#: before it is trusted, instead of silently splitting a namespace. +#: ``load`` and ``worksteal`` hand a file's items to whichever worker is free; +#: ``-n`` without ``--dist`` resolves to ``load``. +_WHOLE_NAMESPACE_SCHEDULERS = frozenset( + {"no", "each", "loadfile", "loadgroup", "loadscope"}, +) + +#: What a page is when nothing says otherwise, matching the collector: a +#: ``.rst`` or ``.md`` file is one whatever ``--doctest-glob`` says. +_PAGE_SUFFIXES = frozenset({".rst", ".md"}) + def pytest_addoption(parser: Parser) -> None: """Add options to py.test for doctest_docutils.""" @@ -60,6 +113,152 @@ def pytest_addoption(parser: Parser) -> None: help="disable doctest-doctests in .py modules (pass-through to pytest-doctest)", dest="doctestmodules", ) + group.addoption( + "--doctest-docutils-namespace-scope", + action="store", + choices=NAMESPACE_SCOPES, + default=None, + help=( + f"{_NAMESPACE_HELP}; overrides the doctest_docutils_namespace_scope" + " ini option" + ), + dest="doctest_docutils_namespace_scope", + ) + parser.addini( + "doctest_docutils_namespace_scope", + _NAMESPACE_HELP, + default=DEFAULT_NAMESPACE_SCOPE, + ) + group.addoption( + "--doctest-docutils-namespace-items", + action="store", + choices=NAMESPACE_ITEMS, + default=None, + help=( + f"{_ITEMS_HELP}; overrides the doctest_docutils_namespace_items ini option" + ), + dest="doctest_docutils_namespace_items", + ) + parser.addini( + "doctest_docutils_namespace_items", + _ITEMS_HELP, + default=DEFAULT_NAMESPACE_ITEMS, + ) + + +def _resolve_namespace_scope( + cli_value: str | None, + ini_value: str | None, +) -> NamespaceScope: + """Resolve the namespace scope: command line first, then ini, then default. + + Parameters + ---------- + cli_value : str | None + Value of ``--doctest-docutils-namespace-scope``, `None` when unset. + ini_value : str | None + Value of the ``doctest_docutils_namespace_scope`` ini option. + + Returns + ------- + doctest_docutils.NamespaceScope + Scope to build the finder with. + + Raises + ------ + pytest.UsageError + If either value names a scope that does not exist. + + Examples + -------- + >>> _resolve_namespace_scope(None, None) + 'block' + + >>> _resolve_namespace_scope(None, "document") + 'document' + + One run can narrow a project that shares each page, without editing the + configuration everyone else reads: + + >>> _resolve_namespace_scope("block", "document") + 'block' + + A name that no scope answers to stops the session once, rather than + failing every file it collects, and says where the name was written — + argparse already names the flag, so only the ini file needs saying: + + >>> try: + ... _resolve_namespace_scope(None, "per-file") + ... except pytest.UsageError as exc: + ... print(exc) + Unknown namespace scope: 'per-file'. Expected one of: block, document + Set by the doctest_docutils_namespace_scope ini option. + """ + value = cli_value or ini_value or DEFAULT_NAMESPACE_SCOPE + try: + return _parse_namespace_scope(value) + except NamespaceScopeError as exc: + message = str(exc) + if value == ini_value: + message += "\nSet by the doctest_docutils_namespace_scope ini option." + raise pytest.UsageError(message) from exc + + +def _resolve_namespace_items( + cli_value: str | None, + ini_value: str | None, +) -> NamespaceItems: + """Resolve the namespace layout: command line first, then ini, then default. + + Parameters + ---------- + cli_value : str | None + Value of ``--doctest-docutils-namespace-items``, `None` when unset. + ini_value : str | None + Value of the ``doctest_docutils_namespace_items`` ini option. + + Returns + ------- + doctest_docutils.NamespaceItems + Layout to build the finder with. + + Raises + ------ + pytest.UsageError + If either value names a layout that does not exist. + + Examples + -------- + >>> _resolve_namespace_items(None, None) + 'merged' + + >>> _resolve_namespace_items(None, "per-block") + 'per-block' + + One run can merge a project that keeps its blocks apart, without editing + the configuration everyone else reads: + + >>> _resolve_namespace_items("merged", "per-block") + 'merged' + + A name that no layout answers to stops the session once, and says where + the name was written: + + >>> try: + ... _resolve_namespace_items(None, "one-each") + ... except pytest.UsageError as exc: + ... print(exc) + Unknown namespace items: 'one-each'. Expected one of: merged, per-block + Set by the doctest_docutils_namespace_items ini option. + """ + value = cli_value or ini_value or DEFAULT_NAMESPACE_ITEMS + try: + return _parse_namespace_items(value) + except NamespaceItemsError as exc: + message = str(exc) + if value == ini_value: + message += "\nSet by the doctest_docutils_namespace_items ini option." + raise pytest.UsageError(message) from exc def pytest_configure(config: pytest.Config) -> None: @@ -67,15 +266,441 @@ def pytest_configure(config: pytest.Config) -> None: Todo: Find a way to make these plugins cooperate without collecting twice. """ - # Register HIDE eagerly, before collection parses any docstring. The .py - # path delegates to pytest's own DoctestModule (which never calls our - # _get_flag_lookup), so registering it here is what lets a docstring carry - # ``# doctest: +HIDE`` without raising ``invalid option`` at parse time. - _get_hide_flag() + # Resolved once, so a misspelled scope stops the session here instead of + # erroring on every file collected. + config.stash[_NAMESPACE_SCOPE_KEY] = _resolve_namespace_scope( + config.getoption("doctest_docutils_namespace_scope", None), + config.getini("doctest_docutils_namespace_scope"), + ) + config.stash[_NAMESPACE_ITEMS_KEY] = _resolve_namespace_items( + config.getoption("doctest_docutils_namespace_items", None), + config.getini("doctest_docutils_namespace_items"), + ) + # Registered whether or not anything will carry it, so that a project + # running --strict-markers passes without opting into the layout that + # emits the marker. Only when pytest-xdist is absent, though: xdist + # registers the same name itself, and registering it twice lists it twice + # in ``pytest --markers`` for every project, opted in or not. + if not config.pluginmanager.hasplugin("xdist"): + config.addinivalue_line( + "markers", + "xdist_group(name): keep a namespace's blocks on one pytest-xdist" + " worker under --dist loadgroup", + ) if config.pluginmanager.has_plugin("doctest"): config.pluginmanager.set_blocked("doctest") +def _worker_count(specs: Iterable[str]) -> int: + """Count the execution environments a run's ``--tx`` specifications ask for. + + A specification may stand for more than one environment: ``2*popen`` is + two. pytest-xdist expands the multiplier in ``parse_tx_spec_config`` and + sizes every scheduler on the result, so counting the specifications + themselves would undercount a run that used the shorthand and read a + two-worker session as a one-worker one. + + Reproduced rather than imported because the upstream helper raises when + a run names no environment at all, which is pytest-xdist's question to + answer. Reproduced exactly, quirks included: counting a specification + differently than the run does would size this guard against a session + pytest-xdist laid out another way. + + Parameters + ---------- + specs : Iterable[str] + ``--tx`` specifications, as argparse collected them. + + Returns + ------- + int + Environments the run has behind it. + + Examples + -------- + One specification is usually one environment: + + >>> _worker_count(["popen", "popen"]) + 2 + + A multiplier stands for as many as it says: + + >>> _worker_count(["2*popen"]) + 2 + + >>> _worker_count(["2*popen", "3*popen"]) + 5 + + A specification whose ``*`` is not a count keeps the whole of itself: + + >>> _worker_count(["popen//python=python3.13"]) + 1 + + >>> _worker_count(["popen//chdir=a*b"]) + 1 + + A count asking for no environment takes none away from the run: + + >>> _worker_count(["0*popen"]) + 0 + + >>> _worker_count(["-1*popen", "2*popen"]) + 2 + + >>> _worker_count([]) + 0 + """ + total = 0 + for spec in specs: + # ``find``, not ``partition``: no ``*`` answers -1, and upstream + # reads the count from ``spec[:-1]``. + marker = spec.find("*") + try: + count = int(spec[:marker]) + except ValueError: + total += 1 + else: + # ``[spec] * count`` is empty at or below zero. + total += max(count, 0) + return total + + +def _splitting_scheduler( + items: NamespaceItems, + scheduler: str, + workers: int, +) -> str | None: + """Name the scheduler that would hand one namespace to two workers. + + Parameters + ---------- + items : doctest_docutils.NamespaceItems + Resolved namespace layout. + scheduler : str + Resolved ``--dist`` value. + workers : int + Number of execution environments the run has behind it. + + Returns + ------- + str or None + The scheduler's name when it would split a namespace, else `None`. + + Examples + -------- + >>> _splitting_scheduler("per-block", "load", 2) + 'load' + + >>> _splitting_scheduler("per-block", "worksteal", 4) + 'worksteal' + + A merged namespace is one item, which no scheduler can cut in half: + + >>> _splitting_scheduler("merged", "load", 2) is None + True + + Neither can a scheduler that keeps a file, a group or a scope whole: + + >>> _splitting_scheduler("per-block", "loadfile", 2) is None + True + + >>> _splitting_scheduler("per-block", "loadgroup", 2) is None + True + + Nor a run with nothing to split a namespace between: + + >>> _splitting_scheduler("per-block", "load", 1) is None + True + """ + if items != "per-block": + return None + if workers < 2: + return None + if scheduler in _WHOLE_NAMESPACE_SCHEDULERS: + return None + return scheduler + + +def _shared_page(ids: Iterable[str], globs: Sequence[str]) -> str | None: + """Name the first page a run collected more than one item from. + + A namespace never reaches past the page it was read from, so a page + collecting one item holds its namespace whole and no scheduler can split + it. Two items from one page is the shape a shared mapping needs, and it + is the only shape a controller can see: it is handed node ids, not the + namespaces behind them. + + Parameters + ---------- + ids : Iterable[str] + Node ids a worker collected. + globs : Sequence[str] + ``--doctest-glob`` patterns, which decide what this plugin collects + as a page. + + Returns + ------- + str or None + Path of the first page holding several items, or `None` when a + namespace cannot be split however the run is scheduled. + + Examples + -------- + >>> globs = ["*.rst", "*.md"] + >>> _shared_page(["docs/page.md::page.md[0]", "docs/page.md::page.md[1]"], globs) + 'docs/page.md' + + A page collecting a single item has nothing to hand a second worker: + + >>> _shared_page(["docs/page.md::page.md"], globs) is None + True + + A suite of Python tests holds no page at all, however many items one + module collects — which is what keeps ``-n`` for a project that carries + the layout in its ini and no documentation in its suite: + + >>> _shared_page(["tests/t.py::test_one", "tests/t.py::test_two"], globs) is None + True + + A project that renamed what a page is says so through ``--doctest-glob``: + + >>> _shared_page( + ... ["docs/page.txt::page.txt[0]", "docs/page.txt::page.txt[1]"], + ... ["*.txt"], + ... ) + 'docs/page.txt' + """ + counts = collections.Counter( + page + for page in (node_id.split("::", 1)[0] for node_id in ids) + if _is_page(page, globs) + ) + return next((page for page, held in counts.items() if held > 1), None) + + +def _is_page(path: str, globs: Sequence[str]) -> bool: + """Say whether a node id's path is a file this plugin collects as a page. + + Mirrors what the collector accepts, so the two cannot drift into a run + scheduled as though a page were a Python module. + + Parameters + ---------- + path : str + Path part of a node id, always written with forward slashes. + globs : Sequence[str] + ``--doctest-glob`` patterns. + + Returns + ------- + bool + `True` when the path names a page. + + Examples + -------- + >>> _is_page("docs/page.md", ["*.rst", "*.md"]) + True + + >>> _is_page("tests/test_plugin.py", ["*.rst", "*.md"]) + False + + reStructuredText and Markdown stay pages whatever the patterns say, + because a file named on the command line is collected on its suffix + alone: + + >>> _is_page("docs/page.rst", ["*.txt"]) + True + + >>> _is_page("docs/page.txt", ["*.txt"]) + True + """ + page = pathlib.PurePosixPath(path) + if page.suffix in _PAGE_SUFFIXES: + return True + return any(page.match(glob) for glob in globs) + + +@pytest.hookimpl(hookwrapper=True) +def pytest_cmdline_main(config: pytest.Config) -> Generator[None, None, None]: + """Record whether the run named a ``--dist`` scheduler, before xdist rewrites it. + + pytest-xdist promotes ``-n`` to ``--dist load`` inside its own + ``pytest_cmdline_main``, after which a promoted ``load`` and a typed + ``--dist load`` are the same string and nothing downstream can tell them + apart. Reading the value first is what separates them, and reading it + from a wrapper is what makes that reliable: pluggy enters every wrapper + before it calls any implementation, so this does not race pytest-xdist + for the value. Marking it ``tryfirst`` would only tie — pytest-xdist + marks its own implementation ``tryfirst`` too, leaving plugin + registration order to break it. + + What it sees is argparse's own result, which is ``no`` unless the run + asked for a scheduler. That covers both places a run can ask from: + pytest splices ini ``addopts`` into the arguments before parsing them, + so a ``--dist`` written there reaches argparse exactly as one typed on + the command line does — unlike ``sys.argv``, which never shows it. + + ``-d`` is the same request spelled short, and is read as one. ``--dist + no`` alongside ``-n`` is not a choice pytest-xdist keeps, so it is not + one this keeps either. + + Parameters + ---------- + config : pytest.Config + Configuration whose options argparse has filled in. + + Yields + ------ + None + Once, to run the implementations this wraps. + """ + config.stash[_DIST_NAMED_KEY] = config.getoption("dist", "no") != "no" or bool( + config.getoption("distload", False), + ) + yield + + +@pytest.hookimpl(optionalhook=True) +def pytest_xdist_make_scheduler(config: pytest.Config, log: t.Any) -> t.Any: + """Keep a shared namespace whole when the run left the scheduler open. + + ``-n`` on its own is a request for workers, not for a way of filling + them, and pytest-xdist answers it with ``--dist load``, which hands a + file's items to whichever worker is free. Under ``per-block`` that can + put the block binding a name and the block reading it on different + workers. Where the run expressed no preference, this fills it in with + file-level scheduling rather than failing: a namespace never reaches + past its page, so keeping a page whole keeps every namespace in it + whole. + + Only a page is kept whole. Everything + else keeps a scope of its own, so a suite whose Python tests happen to + share a file still spreads across the workers it asked for. + + ``loadgroup`` is not the substitute to make: it reads a group off the node + id, and that suffix is written by the *worker*, from the worker's own + ``--dist`` value, so nothing the controller decides here reaches it. + File-level scheduling also leaves node ids untouched, which a group suffix + would not. + + A run that named its scheduler is left alone, whatever it named. + + Parameters + ---------- + config : pytest.Config + Configuration carrying the resolved layout and ``--dist`` value. + log : Any + pytest-xdist ``Producer`` the scheduler logs through. + + Returns + ------- + Any + A scheduler keeping each page whole when it is standing in, else + `None` to leave the choice to pytest-xdist. + """ + if config.stash.get(_DIST_NAMED_KEY, True): + return None + if not _splitting_scheduler( + config.stash[_NAMESPACE_ITEMS_KEY], + config.getoption("dist", "no"), + _worker_count(config.getoption("tx", None) or []), + ): + return None + from xdist.scheduler import ( # type: ignore[import-untyped,unused-ignore] + LoadScopeScheduling, + ) + + globs = config.getoption("doctestglob") or ["*.rst", "*.md"] + + class _PageScheduling(LoadScopeScheduling): # type: ignore[misc] + """Keep a page's items together, and spread everything else.""" + + def _split_scope(self, nodeid: str) -> str: + path = nodeid.split("::", 1)[0] + return path if _is_page(path, globs) else nodeid + + return _PageScheduling(config, log) + + +@pytest.hookimpl(optionalhook=True) +def pytest_xdist_node_collection_finished(node: t.Any, ids: Sequence[str]) -> None: + """Stop a run whose named scheduler would split a page this suite holds. + + Reached only when the run asked for ``--dist load`` or ``--dist + worksteal`` itself. Choosing a scheduler by name is not something a + plugin should quietly overrule, so the session stops and says why rather + than reporting a page that is only wrong because of how it was + scheduled — a shared globals mapping is a Python object, and half a + namespace on a worker reads as a ``NameError`` in the page. + + Read here because here is the first moment a controller knows what the + run actually holds: it never collects itself, and a worker's collection + arrives as node ids. A suite with no page among them has no namespace to + protect, so it keeps its workers. + + Parameters + ---------- + node : Any + pytest-xdist ``WorkerController`` that finished collecting. + ids : Sequence[str] + Node ids it collected. + + Raises + ------ + pytest.UsageError + If a page the run holds would be split between workers. + """ + config = node.config + if not config.stash.get(_DIST_NAMED_KEY, True): + # Left open, so a scheduler that keeps a page whole stood in. + return + scheduler = _splitting_scheduler( + config.stash[_NAMESPACE_ITEMS_KEY], + config.getoption("dist", "no"), + _worker_count(config.getoption("tx", None) or []), + ) + if scheduler is None: + return + page = _shared_page(ids, config.getoption("doctestglob") or ["*.rst", "*.md"]) + if page is None: + return + message = ( + "doctest_docutils_namespace_items = per-block can hand a namespace's" + " blocks one globals mapping between them — a page declaring a group" + " does, whatever the scope — and a mapping cannot cross processes." + f" --dist {scheduler} hands a file's items to whichever worker is" + f" free, so it can send {page}'s blocks to different workers. Run" + " with --dist loadgroup or --dist loadfile, or set" + " doctest_docutils_namespace_items = merged. Dropping --dist leaves" + " -n free to keep each page on one worker." + ) + raise pytest.UsageError(message) + + +def pytest_report_header(config: pytest.Config) -> str | None: + """Say how namespaces are laid out, when they are not laid out as usual. + + A run that changed nothing reports nothing, so the header of an + unconfigured project reads as it always has. + + Parameters + ---------- + config : pytest.Config + Configuration holding the resolved settings. + + Returns + ------- + str or None + One line naming the layout and the scope, or `None` under the default + layout. + """ + items = config.stash[_NAMESPACE_ITEMS_KEY] + if items == DEFAULT_NAMESPACE_ITEMS: + return None + scope = config.stash[_NAMESPACE_SCOPE_KEY] + return f"doctest-docutils: namespace items: {items}, namespace scope: {scope}" + + def _unblock_doctest(config: pytest.Config) -> bool: """Unblock doctest plugin (pytest 8.1+ only). @@ -160,7 +785,7 @@ def _is_doctest( def _init_runner_class() -> type[doctest.DocTestRunner]: import doctest - class PytestDoctestRunner(doctest.DebugRunner): + class PytestDoctestRunner(_ExecModeRunnerMixin, doctest.DebugRunner): """Runner to collect failures. Note that the out variable in this case is a list instead of a @@ -173,9 +798,55 @@ def __init__( verbose: bool | None = None, optionflags: int = 0, continue_on_failure: bool = True, + share_globs: bool = False, ) -> None: super().__init__(checker=checker, verbose=verbose, optionflags=optionflags) self.continue_on_failure = continue_on_failure + self.share_globs = share_globs + self._already_run: set[int] = set() + + def run( + self, + test: doctest.DocTest, + compileflags: int | None = None, + out: _Out | None = None, + clear_globs: bool = True, + ) -> doctest.TestResults: + """Run one test, keeping its globals when its namespace shares them. + + ``clear_globs`` empties ``test.globs`` once the test is done, which + is what stops one item's bindings reaching the next. A namespace + laid out per block wants exactly that reach: its items hold one + mapping between them, so the block below reads what this one bound. + + That reach is also why a block cannot be run twice. Anything that + repeats one item — a retry plugin, ``--count`` — would run it again + against the mapping it already changed, and an expectation that + comes true the second time would be reported as a pass. There is no + way to rebuild the namespace for one block alone, so the repeat is + refused instead. + """ + if self.share_globs: + if id(test) in self._already_run: + import pytest + + pytest.fail( + f"{test.name} was run twice against a namespace laid " + "out per block. A repeated block runs against the " + "globals it already changed, so its result cannot be " + "trusted. Drop --reruns (and anything else that " + "repeats an item), or set " + "doctest_docutils_namespace_items = merged, which " + "re-runs a namespace from its first block.", + pytrace=False, + ) + self._already_run.add(id(test)) + return super().run( + test, + compileflags, + out, + clear_globs and not self.share_globs, + ) def report_failure( self, @@ -238,17 +909,13 @@ def _get_number_flag() -> int: def _get_hide_flag() -> int: - """Register and return the HIDE flag. + """Return the HIDE flag, registered by importing :mod:`doctest_docutils`. ``HIDE`` is a no-op for execution: the output checker never consults it. It marks a doctest example that documentation tooling should drop from the - rendered output while still running it as a test. Registering it here means - ``# doctest: +HIDE`` parses instead of raising ``ValueError: invalid - option`` at collection time. + rendered output while still running it as a test. """ - import doctest - - return doctest.register_optionflag("HIDE") + return _HIDE_FLAG def _get_flag_lookup() -> dict[str, int]: @@ -300,6 +967,7 @@ def _get_runner( verbose: bool | None = None, optionflags: int = 0, continue_on_failure: bool = True, + share_globs: bool = False, ) -> doctest.DocTestRunner: # We need this in order to do a lazy import on doctest global RUNNER_CLASS @@ -312,10 +980,11 @@ def _get_runner( verbose=verbose, optionflags=optionflags, continue_on_failure=continue_on_failure, + share_globs=share_globs, ) -class DocutilsDocTestRunner(doctest.DocTestRunner): +class DocutilsDocTestRunner(_ExecModeRunnerMixin, doctest.DocTestRunner): """DocTestRunner for doctest_docutils.""" def summarize( # type: ignore @@ -354,6 +1023,58 @@ def _DocTestRunner__patched_linecache_getlines( return self.save_linecache_getlines(filename, module_globals) # type: ignore +def _wholly_skipped_reason(test: doctest.DocTest) -> str | None: + r"""Return why a test is skipped outright, or `None` when it runs something. + + Parameters + ---------- + test : doctest.DocTest + Collected test, one namespace or one block lifted out of it. + + Returns + ------- + str or None + Reason naming the page and the first line skipped, or `None`. + + Examples + -------- + >>> import doctest + >>> parser = doctest.DocTestParser() + >>> running = parser.get_doctest(">>> 2 + 2\n4\n", {}, "page", "page.rst", 3) + >>> _wholly_skipped_reason(running) is None + True + + >>> gated = parser.get_doctest( + ... ">>> 2 + 2 # doctest: +SKIP\n4\n", {}, "page", "page.rst", 3 + ... ) + >>> _wholly_skipped_reason(gated) + 'page.rst:4: every example skipped' + + The page is named, not the path it resolves, which pytest prints beside the + reason already: + + >>> nested = parser.get_doctest( + ... ">>> 2 + 2 # doctest: +SKIP\n4\n", {}, "page", "docs/a/page.rst", 3 + ... ) + >>> _wholly_skipped_reason(nested) + 'page.rst:4: every example skipped' + + A block holding no example at all skips nothing, which is not the same + answer as every example being skipped — ``all([])`` is `True`: + + >>> empty = parser.get_doctest("prose only\n", {}, "page", "page.rst", 3) + >>> _wholly_skipped_reason(empty) is None + True + """ + if not test.examples: + return None + if not all(example.options.get(doctest.SKIP, False) for example in test.examples): + return None + line = (test.lineno or 0) + test.examples[0].lineno + 1 + page = pathlib.Path(test.filename or "").name + return f"{page}:{line}: every example skipped" + + class DocTestDocutilsFile(pytest.Module): """Pytest module for doctest_docutils.""" @@ -366,8 +1087,14 @@ def collect(self) -> Iterable[DoctestItem]: encoding = self.config.getini("doctest_encoding") text = self.path.read_text(encoding) + namespace_items = self.config.stash[_NAMESPACE_ITEMS_KEY] + per_block = namespace_items == "per-block" + # Uses internal doctest module parsing mechanism. - finder = DocutilsDocTestFinder() + finder = DocutilsDocTestFinder( + namespace_scope=self.config.stash[_NAMESPACE_SCOPE_KEY], + namespace_items=namespace_items, + ) # While doctests in .rst/.md files don't support fixtures directly, # we still need to pick up autouse fixtures. @@ -382,17 +1109,39 @@ def collect(self) -> Iterable[DoctestItem]: optionflags=optionflags, checker=_pytest.doctest._get_checker(), continue_on_failure=_pytest.doctest._get_continue_on_failure(self.config), + share_globs=per_block, ) from _pytest.doctest import DoctestItem - for test in finder.find( + for collected in finder._collect( text, str(self.path), ): + test = collected.test if test.examples: # skip empty doctests - yield DoctestItem.from_parent( + item = DoctestItem.from_parent( self, # type: ignore name=test.name, runner=runner, dtest=test, ) + if per_block: + # pytest-xdist reads this on the worker and suffixes the + # node id with the group, so --dist loadgroup keeps a + # namespace whole. Only that scheduler reads it: the + # suffix is written from the worker's own --dist value, + # so a controller standing a scheduler in cannot rely on + # it and groups by file instead. + item.add_marker( + pytest.mark.xdist_group( + f"{self.nodeid}::{collected.namespace}", + ), + ) + reason = _wholly_skipped_reason(test) + if reason is not None: + # Marked rather than left to _check_all_skipped, which only + # fires once the item is running: by then its fixtures have + # set up for a test that executes nothing. A marker is read + # before setup, and it carries a reason naming the block. + item.add_marker(pytest.mark.skip(reason=reason)) + yield item diff --git a/tests/regressions/test_autouse_fixtures.py b/tests/regressions/test_autouse_fixtures.py index 1058619..d887e7d 100644 --- a/tests/regressions/test_autouse_fixtures.py +++ b/tests/regressions/test_autouse_fixtures.py @@ -134,3 +134,61 @@ def get_value(): result = pytester.runpytest(str(test_file)) result.assert_outcomes(passed=1) + + +def test_a_module_scoped_fixture_spans_a_shared_page( + pytester: _pytest.pytester.Pytester, +) -> None: + """A page is what ``scope="module"`` means, across every block of it. + + :class:`~pytest_doctest_docutils.DocTestDocutilsFile` collects a page as a + :class:`pytest.Module`, which is the node pytest resolves module scope + against. That is what lets a page carry an object its blocks derived from a + fixture: one setup for the page, so the object block one saved is still the + object block two reads. + + The case above runs a single block, which passes whether the fixture spans + the page or sets up per item. This one fails if the lifetime ever narrows + back to the block. + """ + pytester.plugins = ["pytest_doctest_docutils"] + pytester.makeconftest( + textwrap.dedent( + """ +import pytest + + +@pytest.fixture(scope="module") +def resource(): + yield object() + + +@pytest.fixture(autouse=True) +def seed(doctest_namespace, resource): + doctest_namespace["resource"] = resource + """, + ), + ) + (pytester.path / "page.md").write_text( + textwrap.dedent( + """ +``` +>>> saved = resource +``` + +``` +>>> saved is resource +True +``` + """, + ), + encoding="utf-8", + ) + + result = pytester.runpytest( + str(pytester.path / "page.md"), + "--doctest-docutils-namespace-scope=document", + "--doctest-docutils-namespace-items=per-block", + ) + + result.assert_outcomes(passed=2) diff --git a/tests/test_doctest_docutils.py b/tests/test_doctest_docutils.py index 42807e1..623de48 100644 --- a/tests/test_doctest_docutils.py +++ b/tests/test_doctest_docutils.py @@ -2,7 +2,10 @@ from __future__ import annotations +import contextlib import doctest +import io +import logging import textwrap import typing as t @@ -252,6 +255,63 @@ def test_DocutilsDocTestFinder( doctest.DebugRunner(verbose=False).run(test) +class DocumentOrderFixture(t.NamedTuple): + """Page of eleven numbered blocks, enough for name order to diverge. + + Attributes + ---------- + test_id : str + pytest parametrize id. + file_name : str + Page name, whose suffix picks the parser. + page : str + Page content: block ``n`` evaluates to ``n``. + """ + + test_id: str + file_name: str + page: str + + +DOCUMENT_ORDER_FIXTURES = [ + DocumentOrderFixture( + test_id="MyST-fences", + file_name="example.md", + page="\n".join(f"```python\n>>> {n}\n{n}\n```\n" for n in range(11)), + ), + DocumentOrderFixture( + test_id="reST-doctest_blocks", + file_name="example.rst", + page="\n".join(f">>> {n}\n{n}\n" for n in range(11)), + ), +] + + +@pytest.mark.parametrize( + DocumentOrderFixture._fields, + DOCUMENT_ORDER_FIXTURES, + ids=[f.test_id for f in DOCUMENT_ORDER_FIXTURES], +) +def test_finder_collects_in_document_order( + tmp_path: pathlib.Path, + test_id: str, + file_name: str, + page: str, +) -> None: + """Blocks come back in the order a reader meets them, not in name order. + + Sorting by name put ``page.md[10]`` ahead of ``page.md[1]``. + """ + page_path = tmp_path / file_name + page_path.write_text(page, encoding="utf-8") + + tests = doctest_docutils.DocutilsDocTestFinder().find(page, str(page_path)) + + assert [test.examples[0].source.strip() for test in tests] == [ + str(n) for n in range(11) + ] + + class DoctestOptReTestCase(t.NamedTuple): """Test fixture for doctestopt_re regex. @@ -336,3 +396,2663 @@ def test_docutils_package_relative_error_message() -> None: exc = doctest_docutils.TestDocutilsPackageRelativeError() assert str(exc) == "Package may only be specified for module-relative paths." + + +def test_inline_flags_survive_a_directive(tmp_path: pathlib.Path) -> None: + """A ``# doctest:`` flag applies even where the rendered code drops it. + + ``.. doctest::`` trims the flag out of the code a reader sees and keeps the + original on the node, so the finder has to read the original. + """ + page = textwrap.dedent( + """ +.. doctest:: + + >>> print("a b") # doctest: +NORMALIZE_WHITESPACE + a b + """, + ) + page_path = tmp_path / "page.rst" + page_path.write_text(page, encoding="utf-8") + + (test,) = doctest_docutils.DocutilsDocTestFinder().find(page, str(page_path)) + runner = doctest.DocTestRunner(verbose=False) + runner.run(test, out=lambda _: None) + + assert test.examples[0].options[doctest.NORMALIZE_WHITESPACE] is True + assert runner.failures == 0 + + +class DirectiveOptionFixture(t.NamedTuple): + """Directive whose options reach the examples it holds. + + Attributes + ---------- + test_id : str + pytest parametrize id. + page : str + reStructuredText page holding one ``.. doctest::`` directive. + flag : int + Option flag to read off the collected example. + enabled : bool + Whether the flag is expected on. + """ + + test_id: str + page: str + flag: int + enabled: bool + + +DIRECTIVE_OPTION_FIXTURES = [ + DirectiveOptionFixture( + test_id="directive-options-reach-the-example", + page=".. doctest::\n :options: +ELLIPSIS\n\n >>> 2 + 2\n 4\n", + flag=doctest.ELLIPSIS, + enabled=True, + ), + DirectiveOptionFixture( + test_id="an-inline-flag-beats-the-directive", + page=".. doctest::\n :options: +ELLIPSIS\n\n" + " >>> 2 + 2 # doctest: -ELLIPSIS\n 4\n", + flag=doctest.ELLIPSIS, + enabled=False, + ), +] + + +@pytest.mark.parametrize( + DirectiveOptionFixture._fields, + DIRECTIVE_OPTION_FIXTURES, + ids=[f.test_id for f in DIRECTIVE_OPTION_FIXTURES], +) +def test_directive_options_apply_per_example( + test_id: str, + page: str, + flag: int, + enabled: bool, +) -> None: + """``:options:`` sets a block's defaults; an example's own flags win.""" + (test,) = doctest_docutils.DocutilsDocTestFinder().find(page, "page.rst") + + assert test.examples[0].options[flag] is enabled + + +OUT_OF_ORDER_LINES_REST = [ + ( + "nested-in-a-directive", + textwrap.dedent( + """ +Title +===== + +>>> outer = 1 + +.. note:: + + >>> outer + 1 + 2 + """, + ), + ), + ( + "nested-in-list-items", + textwrap.dedent( + """ +Title +===== + +- First item: + + >>> counted = 1 + +- Second item: + + >>> counted + 1 + 2 + """, + ), + ), +] + + +@pytest.mark.parametrize( + ("test_id", "page"), + OUT_OF_ORDER_LINES_REST, + ids=[test_id for test_id, _ in OUT_OF_ORDER_LINES_REST], +) +def test_a_nested_block_collects( + tmp_path: pathlib.Path, + test_id: str, + page: str, +) -> None: + """A doctest block nested in another node is collected, not fatal. + + docutils leaves ``line`` unset on a block inside a directive, a list item, + or a block quote, and reading it as a number took the whole page down. + """ + page_path = tmp_path / "page.rst" + page_path.write_text(page, encoding="utf-8") + + tests = doctest_docutils.DocutilsDocTestFinder().find(page, str(page_path)) + + linenos = [test.lineno or 0 for test in tests] + + assert linenos == sorted(linenos) + assert all(lineno > 0 for lineno in linenos) + + +class SkipifFixture(t.NamedTuple): + """Directive whose ``:skipif:`` decides whether its block runs. + + Attributes + ---------- + test_id : str + pytest parametrize id. + expression : str + Expression written on the directive's ``:skipif:`` option. + skipped : bool + Whether the block is expected to carry ``SKIP``. + """ + + test_id: str + expression: str + skipped: bool + + +SKIPIF_FIXTURES = [ + SkipifFixture(test_id="true-skips-the-block", expression="True", skipped=True), + SkipifFixture(test_id="false-runs-the-block", expression="False", skipped=False), + SkipifFixture( + test_id="expression-sees-the-starting-globals", + expression="__name__ == 'nonesuch'", + skipped=False, + ), + SkipifFixture( + test_id="expression-sees-sys", + expression="sys.version_info < (3, 10)", + skipped=False, + ), +] + + +@pytest.mark.parametrize( + SkipifFixture._fields, + SKIPIF_FIXTURES, + ids=[f.test_id for f in SKIPIF_FIXTURES], +) +def test_skipif_marks_its_block_skip( + test_id: str, + expression: str, + skipped: bool, +) -> None: + """A true ``:skipif:`` marks its block ``SKIP`` rather than dropping it. + + Both spellings of "do not run this" land on the same flag, so the block + stays collectable, countable, and selectable by node id either way. + """ + page = f".. doctest::\n :skipif: {expression}\n\n >>> 2 + 2\n 4\n" + + (test,) = doctest_docutils.DocutilsDocTestFinder().find(page, "page.rst") + + assert test.examples[0].options.get(doctest.SKIP, False) is skipped + + +GATED_MIDDLE_BLOCK_REST = textwrap.dedent( + """ + .. doctest:: intro + + >>> greeting = "hello" + + .. doctest:: intro + :skipif: True + + >>> raise AssertionError("the skipped block ran") + + .. doctest:: intro + + >>> greeting.upper() + 'HELLO' + """, +) + + +def test_skipif_skips_only_its_own_block_of_a_group() -> None: + """A group's other blocks keep running when one of them is skipped. + + The gated block binds nothing its group could read, so it comes back on + its own rather than merged into a group that runs without it. What is + left of the group carries no flag, and the block that does still holds + the source it would have run. + """ + group, gated = doctest_docutils.DocutilsDocTestFinder().find( + GATED_MIDDLE_BLOCK_REST, + "page.rst", + ) + + assert [test.name for test in (group, gated)] == ["intro", "intro[1]"] + assert [example.options.get(doctest.SKIP, False) for example in group.examples] == [ + False, + False, + ] + assert [example.options[doctest.SKIP] for example in gated.examples] == [True] + + +def test_lifting_a_gated_block_moves_no_reported_line() -> None: + """Every example reports the line it reports with the gate turned off. + + The lifted block is positioned where docutils put it, so a reader told it + was skipped is pointed at the same place the group would have pointed. + """ + finder = doctest_docutils.DocutilsDocTestFinder() + + def reported(page: str) -> list[int]: + return sorted( + (test.lineno or 0) + example.lineno + 1 + for test in finder.find(page, "page.rst") + for example in test.examples + ) + + assert reported(GATED_MIDDLE_BLOCK_REST) == reported( + GATED_MIDDLE_BLOCK_REST.replace(":skipif: True", ":skipif: False"), + ) + + +class GateSpellingFixture(t.NamedTuple): + """One way of writing "do not run this block", and the block it writes. + + Attributes + ---------- + test_id : str + pytest parametrize id. + block : str + Middle block of a three-block group, gated its own way. + """ + + test_id: str + block: str + + +GATE_SPELLING_FIXTURES = [ + GateSpellingFixture( + test_id="skipif-condition", + block=".. doctest:: intro\n :skipif: True\n\n >>> 1 / 0\n", + ), + GateSpellingFixture( + test_id="directive-options-flag", + block=".. doctest:: intro\n :options: +SKIP\n\n >>> 1 / 0\n", + ), + GateSpellingFixture( + test_id="inline-flag", + block=".. doctest:: intro\n\n >>> 1 / 0 # doctest: +SKIP\n", + ), + GateSpellingFixture( + test_id="every-example-inline", + block=( + ".. doctest:: intro\n\n >>> 1 / 0 # doctest: +SKIP\n" + " >>> 2 / 0 # doctest: +SKIP\n" + ), + ), +] + + +@pytest.mark.parametrize( + GateSpellingFixture._fields, + GATE_SPELLING_FIXTURES, + ids=[f.test_id for f in GATE_SPELLING_FIXTURES], +) +def test_every_spelling_of_a_gate_lifts_its_block_out( + test_id: str, + block: str, +) -> None: + """A block is lifted out for what its examples carry, not how it says it. + + A condition, a directive flag, and an inline comment all land on + :data:`doctest.SKIP`, so a reader who knows one can predict the others. + """ + page = ( + ".. doctest:: intro\n\n >>> greeting = 'hello'\n\n" + f"{block}\n" + ".. doctest:: intro\n\n >>> greeting.upper()\n 'HELLO'\n" + ) + + group, gated = doctest_docutils.DocutilsDocTestFinder().find(page, "page.rst") + runner = doctest.DocTestRunner(verbose=False) + for test in (group, gated): + runner.run(test, out=lambda _: None) + + assert [test.name for test in (group, gated)] == ["intro", "intro[1]"] + assert runner.failures == 0 + + +def test_a_half_gated_block_stays_in_its_namespace() -> None: + """A block with one example left to run is not a skipped block. + + Its silence is the silence pytest keeps for any partly skipped item, and + the example that runs may bind a name the rest of the group reads. + """ + page = ( + ".. doctest:: intro\n\n" + " >>> greeting = 'hello' # doctest: +SKIP\n" + " >>> greeting = 'hi'\n\n" + ".. doctest:: intro\n\n >>> greeting\n 'hi'\n" + ) + + tests = doctest_docutils.DocutilsDocTestFinder().find(page, "page.rst") + runner = doctest.DocTestRunner(verbose=False) + for test in tests: + runner.run(test, out=lambda _: None) + + assert [test.name for test in tests] == ["intro"] + assert runner.failures == 0 + + +def test_a_namespace_gated_end_to_end_stays_whole() -> None: + """A namespace with nothing left to run keeps every block it holds. + + One test reports the skip once. Lifting each block out would report the + same page N times, which is noise, not information. + """ + page = ( + ".. doctest:: solo\n :skipif: True\n\n >>> 1 / 0\n\n" + ".. doctest:: solo\n :options: +SKIP\n\n >>> 2 / 0\n" + ) + + tests = doctest_docutils.DocutilsDocTestFinder().find(page, "page.rst") + + assert [test.name for test in tests] == ["solo"] + assert all( + example.options[doctest.SKIP] for test in tests for example in test.examples + ) + + +def test_a_shared_page_names_a_gated_block_as_block_scope_does() -> None: + """The node id that selects a gated block does not move with the scope. + + Under ``document`` the page is one namespace named for the page, so a + block lifted back out of it lands on the name it carries when every block + keeps its own namespace. + """ + page = textwrap.dedent( + """ + ```python + >>> value = 1 + ``` + + ```python + >>> value = 999 # doctest: +SKIP + ``` + + ```python + >>> value + 1 + ``` + """, + ) + + shared = doctest_docutils.DocutilsDocTestFinder(namespace_scope="document") + apart = doctest_docutils.DocutilsDocTestFinder(namespace_scope="block") + + assert [test.name for test in shared.find(page, "page.md")] == [ + "page.md", + "page.md[1]", + ] + assert [test.name for test in apart.find(page, "page.md")] == [ + "page.md[0]", + "page.md[1]", + "page.md[2]", + ] + + +def test_an_inline_flag_cannot_reopen_a_true_skipif() -> None: + """An example's own ``-SKIP`` loses to a condition, unlike to ``:options:``. + + ``sphinx.ext.doctest`` drops a gated block before its source is read, so + nothing written inside one can turn the gate off. An example that could + would run on exactly the interpreter or platform it was guarded against. + """ + page = ".. doctest::\n :skipif: True\n\n >>> 2 + 2 # doctest: -SKIP\n 4\n" + + (test,) = doctest_docutils.DocutilsDocTestFinder().find(page, "page.rst") + + assert test.examples[0].options[doctest.SKIP] is True + + +SKIPPED_SETUP_DIRECTIVES = [ + ("testsetup", "testsetup"), + ("testcleanup", "testcleanup"), +] + + +@pytest.mark.parametrize( + ("test_id", "directive"), + SKIPPED_SETUP_DIRECTIVES, + ids=[test_id for test_id, _ in SKIPPED_SETUP_DIRECTIVES], +) +def test_skipif_marks_setup_and_cleanup_blocks_skip( + test_id: str, + directive: str, +) -> None: + """``:skipif:`` reaches the setup and cleanup directives that declare it. + + Both list ``skipif`` in their ``option_spec``, so the option is not a + ``.. doctest::`` exclusive and has to behave the same on all three. A + gated one comes back on its own, as any gated block does, so a group whose + setup never ran says so. + """ + page = ( + f".. {directive}:: fixture\n :skipif: True\n\n" + " >>> raise AssertionError('the skipped block ran')\n\n" + ".. doctest:: fixture\n\n >>> 2 + 2\n 4\n" + ) + + gated, group = doctest_docutils.DocutilsDocTestFinder().find(page, "page.rst") + + assert [test.name for test in (gated, group)] == ["fixture[0]", "fixture"] + assert [example.options.get(doctest.SKIP, False) for example in group.examples] == [ + False, + ] + assert [example.options[doctest.SKIP] for example in gated.examples] == [True] + + +SKIPIF_STANDALONE_PAGE = textwrap.dedent( + """ + Standalone + ========== + + .. doctest:: + :skipif: True + + >>> 1 / 0 + + .. doctest:: + + >>> 2 + 2 + 4 + """, +) + + +def test_skipif_under_testdocutils(tmp_path: pathlib.Path) -> None: + """The standalone runner skips the block instead of never seeing it. + + :class:`doctest.DocTestRunner` honours ``SKIP`` itself, so the library + stays usable without pytest and the skipped example is never executed. + """ + page = tmp_path / "page.rst" + page.write_text(SKIPIF_STANDALONE_PAGE, encoding="utf-8") + + results = doctest_docutils.testdocutils( + str(page), + module_relative=False, + report=False, + ) + + assert results.failed == 0 + + +class StandaloneExitFixture(t.NamedTuple): + """Page run through the ``python -m doctest_docutils`` entry point. + + Attributes + ---------- + test_id : str + pytest parametrize id. + page : str + reStructuredText source written to the temporary page. + exit_code : int + Status ``doctest_docutils._test`` is expected to return. + """ + + test_id: str + page: str + exit_code: int + + +STANDALONE_EXIT_FIXTURES = [ + StandaloneExitFixture( + test_id="a-skipped-block-alone-passes", + page=SKIPIF_STANDALONE_PAGE, + exit_code=0, + ), + StandaloneExitFixture( + test_id="a-real-failure-beside-it-still-fails", + page=SKIPIF_STANDALONE_PAGE.replace( + " >>> 2 + 2\n 4\n", " >>> 2 + 2\n 5\n" + ), + exit_code=1, + ), +] + + +@pytest.mark.parametrize( + StandaloneExitFixture._fields, + STANDALONE_EXIT_FIXTURES, + ids=[f.test_id for f in STANDALONE_EXIT_FIXTURES], +) +def test_skipif_exit_code_from_the_command( + tmp_path: pathlib.Path, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], + test_id: str, + page: str, + exit_code: int, +) -> None: + """``python -m doctest_docutils`` exits non-zero only on a real failure.""" + page_path = tmp_path / "page.rst" + page_path.write_text(page, encoding="utf-8") + monkeypatch.setattr("sys.argv", ["doctest_docutils", str(page_path)]) + + assert doctest_docutils._test() == exit_code + + assert "ZeroDivisionError" not in capsys.readouterr().out + + +def test_skipif_that_cannot_be_evaluated_names_its_block() -> None: + """An expression naming something out of reach reports as that block. + + The namespace a ``:skipif:`` sees is small on purpose, so reaching outside + it is an ordinary mistake; the report has to say which block to go fix. + """ + page = ( + "Title\n=====\n\n.. doctest::\n" + ' :skipif: platform.system() == "Windows"\n\n >>> 2 + 2\n 4\n' + ) + + with pytest.raises(doctest_docutils.SkipifExpressionError) as excinfo: + doctest_docutils.DocutilsDocTestFinder().find(page, "page.rst") + + assert str(excinfo.value) == ( + "page.rst:4: :skipif: 'platform.system() == \"Windows\"' failed: " + "name 'platform' is not defined" + ) + + +def test_hide_optionflag_parses_without_pytest() -> None: + """``+HIDE`` parses wherever :mod:`doctest_docutils` is imported. + + The flag is gp-libs' own, and a page carrying an unregistered name fails to + parse, so registering it only as pytest configures left the standalone + ``python -m doctest_docutils`` command unable to read the repo's own pages. + """ + page = ( + ".. doctest::\n\n >>> base = 40 # doctest: +HIDE\n" + " >>> base + 2\n 42\n" + ) + + (test,) = doctest_docutils.DocutilsDocTestFinder().find(page, "page.rst") + + assert test.examples[0].options[doctest_docutils._HIDE_FLAG] is True + + +STATE_MD = textwrap.dedent( + """ +# Title + +```python +>>> greeting = "hello" +>>> greeting +'hello' +``` + +Narrative prose between the two blocks. + +```python +>>> greeting.upper() +'HELLO' +``` + """, +) + +SHARED_GROUP_REST = textwrap.dedent( + """ +Title +===== + +.. doctest:: intro + + >>> greeting = "hello" + +Narrative prose. + +.. doctest:: intro + + >>> greeting.upper() + 'HELLO' + """, +) + +DISTINCT_GROUPS_REST = textwrap.dedent( + """ +Title +===== + +.. doctest:: alpha + + >>> alpha_only = 1 + +.. doctest:: beta + + >>> alpha_only + Traceback (most recent call last): + NameError: name 'alpha_only' is not defined + """, +) + + +class NamespaceFixture(t.NamedTuple): + """Page whose blocks land in one namespace or in several. + + Attributes + ---------- + test_id : str + pytest parametrize id. + file_name : str + Page name, whose suffix picks the parser. + page : str + Page content. + namespace_scope : doctest_docutils.NamespaceScope + Scope the finder is built with. + test_names : list[str] + Test names ``find`` returns, in order. + example_sources : list[list[str]] + Example sources per returned test, in document order. + """ + + test_id: str + file_name: str + page: str + namespace_scope: doctest_docutils.NamespaceScope + test_names: list[str] + example_sources: list[list[str]] + + +NAMESPACE_FIXTURES = [ + NamespaceFixture( + test_id="ungrouped-fences-stay-apart-by-default", + file_name="page.md", + page=STATE_MD, + namespace_scope="block", + test_names=["page.md[0]", "page.md[1]"], + example_sources=[['greeting = "hello"', "greeting"], ["greeting.upper()"]], + ), + NamespaceFixture( + test_id="ungrouped-fences-share-the-page-under-document", + file_name="page.md", + page=STATE_MD, + namespace_scope="document", + test_names=["page.md"], + example_sources=[ + ['greeting = "hello"', "greeting", "greeting.upper()"], + ], + ), + NamespaceFixture( + test_id="group-shares-by-default", + file_name="page.rst", + page=SHARED_GROUP_REST, + namespace_scope="block", + test_names=["intro"], + example_sources=[['greeting = "hello"', "greeting.upper()"]], + ), + NamespaceFixture( + test_id="group-shares-under-document", + file_name="page.rst", + page=SHARED_GROUP_REST, + namespace_scope="document", + test_names=["intro"], + example_sources=[['greeting = "hello"', "greeting.upper()"]], + ), + NamespaceFixture( + test_id="distinct-groups-partition-the-page", + file_name="page.rst", + page=DISTINCT_GROUPS_REST, + namespace_scope="document", + test_names=["alpha", "beta"], + example_sources=[["alpha_only = 1"], ["alpha_only"]], + ), +] + + +@pytest.mark.parametrize( + NamespaceFixture._fields, + NAMESPACE_FIXTURES, + ids=[f.test_id for f in NAMESPACE_FIXTURES], +) +def test_finder_merges_a_namespace_into_one_test( + tmp_path: pathlib.Path, + test_id: str, + file_name: str, + page: str, + namespace_scope: doctest_docutils.NamespaceScope, + test_names: list[str], + example_sources: list[list[str]], +) -> None: + """A namespace is one test holding its blocks' examples in document order. + + Naming a group is the author asking two blocks to share, so a group shares + at every scope; blocks that name none follow the scope. + """ + page_path = tmp_path / file_name + page_path.write_text(page, encoding="utf-8") + + finder = doctest_docutils.DocutilsDocTestFinder(namespace_scope=namespace_scope) + tests = finder.find(page, str(page_path)) + + assert [test.name for test in tests] == test_names + assert [ + [example.source.strip() for example in test.examples] for test in tests + ] == example_sources + + +class NamespaceStateFixture(t.NamedTuple): + """Page run end to end, counting the examples that fail. + + Attributes + ---------- + test_id : str + pytest parametrize id. + file_name : str + Page name, whose suffix picks the parser. + page : str + Page content. + namespace_scope : doctest_docutils.NamespaceScope + Scope the finder is built with. + failures : int + Examples expected to fail once every test has run. + """ + + test_id: str + file_name: str + page: str + namespace_scope: doctest_docutils.NamespaceScope + failures: int + + +NAMESPACE_STATE_FIXTURES = [ + NamespaceStateFixture( + test_id="second-fence-cannot-read-the-first-by-default", + file_name="page.md", + page=STATE_MD, + namespace_scope="block", + failures=1, + ), + NamespaceStateFixture( + test_id="second-fence-reads-the-first-under-document", + file_name="page.md", + page=STATE_MD, + namespace_scope="document", + failures=0, + ), + NamespaceStateFixture( + test_id="group-reads-what-its-first-block-bound", + file_name="page.rst", + page=SHARED_GROUP_REST, + namespace_scope="block", + failures=0, + ), + NamespaceStateFixture( + test_id="groups-stay-isolated-from-each-other", + file_name="page.rst", + page=DISTINCT_GROUPS_REST, + namespace_scope="document", + failures=0, + ), +] + + +@pytest.mark.parametrize( + NamespaceStateFixture._fields, + NAMESPACE_STATE_FIXTURES, + ids=[f.test_id for f in NAMESPACE_STATE_FIXTURES], +) +def test_namespace_scope_decides_what_a_block_can_read( + tmp_path: pathlib.Path, + test_id: str, + file_name: str, + page: str, + namespace_scope: doctest_docutils.NamespaceScope, + failures: int, +) -> None: + """State reaches exactly as far as the namespace it was bound in. + + The isolated page proves it by expecting the ``NameError`` its own examples + document. + """ + page_path = tmp_path / file_name + page_path.write_text(page, encoding="utf-8") + + finder = doctest_docutils.DocutilsDocTestFinder(namespace_scope=namespace_scope) + runner = doctest.DocTestRunner(verbose=False) + for test in finder.find(page, str(page_path)): + runner.run(test, out=lambda _: None) + + assert runner.failures == failures + + +class MergedLineNumberFixture(t.NamedTuple): + """Page a namespace merges, in each block form docutils positions apart. + + Attributes + ---------- + test_id : str + pytest parametrize id. + file_name : str + Page name, whose suffix picks the parser. + page : str + Page content. + """ + + test_id: str + file_name: str + page: str + + +LONG_THEN_SHORT_REST = textwrap.dedent( + """ +Title +===== + +>>> first = 1 +>>> second = 2 +>>> third = 3 +>>> fourth = 4 + +Prose short enough that placing by ``node.line`` would overlap the blocks. + +>>> first + fourth +5 + """, +) + +MERGED_LINE_NUMBER_FIXTURES = [ + MergedLineNumberFixture( + test_id="MyST-fences", + file_name="page.md", + page=STATE_MD, + ), + MergedLineNumberFixture( + test_id="reST-doctest_directives", + file_name="page.rst", + page=SHARED_GROUP_REST, + ), + MergedLineNumberFixture( + test_id="reST-doctest_blocks", + file_name="page.rst", + page=LONG_THEN_SHORT_REST, + ), +] + + +@pytest.mark.parametrize( + MergedLineNumberFixture._fields, + MERGED_LINE_NUMBER_FIXTURES, + ids=[f.test_id for f in MERGED_LINE_NUMBER_FIXTURES], +) +def test_merged_examples_keep_their_gutter( + tmp_path: pathlib.Path, + test_id: str, + file_name: str, + page: str, +) -> None: + """The line a failure prints is the line the failing prompt sits on. + + pytest counts the ``%03d`` gutter from ``test.lineno + 1`` through the + merged source, so the blank lines standing in for prose have to match the + prose they replace, block after block. + """ + page_path = tmp_path / file_name + page_path.write_text(page, encoding="utf-8") + + finder = doctest_docutils.DocutilsDocTestFinder(namespace_scope="document") + (merged,) = finder.find(page, str(page_path)) + + gutter = (merged.docstring or "").splitlines() + assert [gutter[example.lineno] for example in merged.examples] == [ + f">>> {example.source.splitlines()[0]}" for example in merged.examples + ] + + +def _reported_lines( + page: str, + page_path: pathlib.Path, + scope: doctest_docutils.NamespaceScope, + items: doctest_docutils.NamespaceItems = "merged", +) -> list[int]: + """Return the file line every example on `page` reports, at `scope`.""" + finder = doctest_docutils.DocutilsDocTestFinder( + namespace_scope=scope, + namespace_items=items, + ) + return [ + (test.lineno or 0) + example.lineno + 1 + for test in finder.find(page, str(page_path)) + for example in test.examples + ] + + +@pytest.mark.parametrize( + ("test_id", "file_name", "page"), + [ + ("MyST-fences", "page.md", STATE_MD), + ("reST-doctest_directives", "page.rst", SHARED_GROUP_REST), + ("reST-doctest_blocks", "page.rst", LONG_THEN_SHORT_REST), + ], + ids=["MyST-fences", "reST-doctest_directives", "reST-doctest_blocks"], +) +def test_merging_moves_no_reported_line( + tmp_path: pathlib.Path, + test_id: str, + file_name: str, + page: str, +) -> None: + """A merged example reports the line it reports on its own. + + Every block form is placed at the line docutils gave it, so a page whose + blocks stand clear of each other reads the same merged as it does apart. + """ + page_path = tmp_path / file_name + page_path.write_text(page, encoding="utf-8") + + assert _reported_lines(page, page_path, "document") == _reported_lines( + page, + page_path, + "block", + ) + + +CROWDED_REST = textwrap.dedent( + """ +Title +===== + +>>> one = 1 +>>> two = 2 +>>> three = 3 +>>> four = 4 +>>> five = 5 +>>> six = 6 + +Prose. + +.. doctest:: + + >>> one + six + 7 + """, +) + + +def test_a_crowded_block_follows_the_one_above_it(tmp_path: pathlib.Path) -> None: + """A block the lines above already reach reports further down the page. + + docutils reports a reStructuredText doctest block's *last* line, so its own + examples already report lines below the block: a six-line block starting on + line 5 reports lines 10 to 15. A directive two lines further down has to + follow those, and moves by the overlap. The gutter still shows the failing + prompt, which is what a reader reads the report for. + """ + page_path = tmp_path / "page.rst" + page_path.write_text(CROWDED_REST, encoding="utf-8") + + apart = _reported_lines(CROWDED_REST, page_path, "block") + merged = _reported_lines(CROWDED_REST, page_path, "document") + + assert merged[:-1] == apart[:-1] + assert merged[-1] - apart[-1] == 2 + + finder = doctest_docutils.DocutilsDocTestFinder(namespace_scope="document") + (test,) = finder.find(CROWDED_REST, str(page_path)) + gutter = (test.docstring or "").splitlines() + assert gutter[test.examples[-1].lineno] == ">>> one + six" + + +@pytest.mark.parametrize( + ("test_id", "page"), + OUT_OF_ORDER_LINES_REST, + ids=[test_id for test_id, _ in OUT_OF_ORDER_LINES_REST], +) +def test_merging_survives_a_block_docutils_left_unpositioned( + tmp_path: pathlib.Path, + test_id: str, + page: str, +) -> None: + """A doctest block nested in another node still merges and runs. + + docutils leaves ``line`` unset on a block inside a directive, a list item, + or a block quote, so placing every block by that value alone would stack + them all at the top of the page. + """ + page_path = tmp_path / "page.rst" + page_path.write_text(page, encoding="utf-8") + + finder = doctest_docutils.DocutilsDocTestFinder(namespace_scope="document") + (merged,) = finder.find(page, str(page_path)) + runner = doctest.DocTestRunner(verbose=False) + runner.run(merged, out=lambda _: None) + + linenos = [example.lineno for example in merged.examples] + assert linenos == sorted(set(linenos)) + assert runner.failures == 0 + + +def test_a_group_survives_an_include(tmp_path: pathlib.Path) -> None: + """A group split across an ``.. include::`` merges and runs. + + docutils numbers the included page's nodes against that page, so the second + block claims a line the first one already covers. + """ + (tmp_path / "part.rst").write_text( + "Part\n----\n\nProse.\n\n.. doctest:: intro\n\n" + " >>> greeting.upper()\n 'HELLO'\n", + encoding="utf-8", + ) + page = ( + "Title\n=====\n\n.. doctest:: intro\n\n" + " >>> greeting = 'hello'\n\n.. include:: part.rst\n" + ) + page_path = tmp_path / "main.rst" + page_path.write_text(page, encoding="utf-8") + + finder = doctest_docutils.DocutilsDocTestFinder() + (merged,) = finder.find(page, str(page_path)) + runner = doctest.DocTestRunner(verbose=False) + runner.run(merged, out=lambda _: None) + + assert [example.source.strip() for example in merged.examples] == [ + "greeting = 'hello'", + "greeting.upper()", + ] + assert runner.failures == 0 + + +def test_markdown_failures_point_at_the_prompt(tmp_path: pathlib.Path) -> None: + """A merged Markdown page reports the file line each ``>>>`` sits on.""" + page_path = tmp_path / "page.md" + page_path.write_text(STATE_MD, encoding="utf-8") + + finder = doctest_docutils.DocutilsDocTestFinder(namespace_scope="document") + (test,) = finder.find(STATE_MD, str(page_path)) + + lines = STATE_MD.splitlines() + reported = [(test.lineno or 0) + example.lineno + 1 for example in test.examples] + assert [lines[lineno - 1] for lineno in reported] == [ + '>>> greeting = "hello"', + ">>> greeting", + ">>> greeting.upper()", + ] + + +def test_collection_logs_the_namespace_each_block_joined( + caplog: pytest.LogCaptureFixture, +) -> None: + """Collection records the namespace, source file, and block type. + + ``doctest_source_file`` and ``doctest_block_type`` are the structured keys + a log processor filters on, so assert the schema, not the message. + """ + finder = doctest_docutils.DocutilsDocTestFinder() + with caplog.at_level(logging.DEBUG, logger="doctest_docutils"): + finder.find(SHARED_GROUP_REST, "page.rst") + + collected = [ + record + for record in caplog.records + if record.msg == "doctest block collected into namespace %s" + ] + assert [record.args for record in collected] == [("intro",), ("intro",)] + assert {record.__dict__["doctest_block_type"] for record in collected} == { + "doctest", + } + assert {record.__dict__["doctest_source_file"] for record in collected} == { + "page.rst", + } + + +def test_verbose_finder_records_what_a_page_yielded( + caplog: pytest.LogCaptureFixture, +) -> None: + """``verbose=True`` reports how many tests a page produced, and from where. + + ``doctest_source_file`` is the structured key, so the page is filtered on + rather than read out of the message. + """ + page = ".. doctest::\n\n >>> 2 + 2\n 4\n" + + finder = doctest_docutils.DocutilsDocTestFinder(verbose=True) + with caplog.at_level(logging.INFO, logger="doctest_docutils"): + finder.find(page, "page.rst") + + found = [record for record in caplog.records if record.msg == "found %d test(s)"] + assert [record.args for record in found] == [(1,)] + assert [record.__dict__["doctest_source_file"] for record in found] == ["page.rst"] + + +def test_namespace_scope_rejects_an_unknown_name() -> None: + """An unknown scope names the values it could have been.""" + with pytest.raises(doctest_docutils.NamespaceScopeError) as excinfo: + doctest_docutils.DocutilsDocTestFinder( + namespace_scope=t.cast("doctest_docutils.NamespaceScope", "per-file"), + ) + + assert str(excinfo.value) == ( + "Unknown namespace scope: 'per-file'. Expected one of: block, document" + ) + + +TAKEN_DOCUMENT_NAME_REST = """ +Page +==== + +.. doctest:: page.rst + + >>> declared = "in the group the author named" + +A block declaring nothing, which document scope names for the page: + + >>> declared + Traceback (most recent call last): + NameError: name 'declared' is not defined +""" + +TAKEN_BLOCK_NAME_REST = """ +Page +==== + + >>> ungrouped = "in the block that declared nothing" + +.. doctest:: page.rst[0] + + >>> ungrouped + Traceback (most recent call last): + NameError: name 'ungrouped' is not defined +""" + +ALL_BLOCKS_GROUPED_REST = """ +Page +==== + +.. doctest:: page.rst + + >>> value = 1 + +.. doctest:: page.rst + + >>> value + 1 +""" + + +def test_a_group_may_not_take_the_name_a_page_generates() -> None: + """A page generating a name a group declared cannot say which it meant. + + At ``"document"`` scope a block declaring no group is named for the page, + so a group of the same name asks for a namespace already given away. The + two would share state and collect under one node id, which is a wrong + answer either way — so the page says so instead of picking one. + """ + with pytest.raises(doctest_docutils.NamespaceNameCollisionError) as excinfo: + doctest_docutils.DocutilsDocTestFinder(namespace_scope="document").find( + TAKEN_DOCUMENT_NAME_REST, + "page.rst", + ) + + assert "group 'page.rst' takes the name this page generates" in str(excinfo.value) + assert "a block declaring none at 'document' scope" in str(excinfo.value) + assert "Rename the group" in str(excinfo.value) + + +def test_a_group_may_not_take_a_generated_block_name() -> None: + """``block`` scope generates ``page[n]``, which a group can spell too. + + The default scope names an ungrouped block for its position, so the + collision reaches a page that configured nothing. + """ + with pytest.raises(doctest_docutils.NamespaceNameCollisionError) as excinfo: + doctest_docutils.DocutilsDocTestFinder().find( + TAKEN_BLOCK_NAME_REST, + "page.rst", + ) + + assert "group 'page.rst[0]' takes the name this page generates" in str( + excinfo.value, + ) + + +TAKEN_LIFTED_NAME_REST = """ +Page +==== + +.. doctest:: alpha + + >>> a = 1 + +.. doctest:: alpha + + >>> a # doctest: +SKIP + 1 + +.. doctest:: alpha[1] + + >>> b = 2 +""" + + +def test_a_group_may_not_take_a_lifted_block_name() -> None: + """Lifting names a block the same way declaring a group does. + + A block gated end to end is lifted out of its namespace as ``name[n]``, + which is a name a group can spell. Two tests answering to one node id is + the same wrong answer whichever half of the page generated it, so the + refusal covers the lifted name as well as the declared one. + """ + with pytest.raises(doctest_docutils.NamespaceNameCollisionError) as excinfo: + doctest_docutils.DocutilsDocTestFinder().find( + TAKEN_LIFTED_NAME_REST, + "page.rst", + ) + + assert "group 'alpha[1]' takes the name this page generates" in str(excinfo.value) + assert "a block lifted out of 'alpha'" in str(excinfo.value) + + +@pytest.mark.parametrize("scope", ["block", "document"]) +def test_a_group_named_for_its_page_is_left_alone( + scope: doctest_docutils.NamespaceScope, +) -> None: + """Only a name the page actually generates is taken. + + A page whose every block declares a group generates no name at all, so + naming a group after the file it sits in is a style choice and not a + collision. Checking the generated names rather than their shape is what + keeps this page collecting. + """ + tests = doctest_docutils.DocutilsDocTestFinder(namespace_scope=scope).find( + ALL_BLOCKS_GROUPED_REST, + "page.rst", + ) + runner = doctest.DocTestRunner(verbose=False) + for test in tests: + runner.run(test, out=lambda _: None) + + assert [test.name for test in tests] == ["page.rst"] + assert runner.failures == 0 + + +class PyversionFixture(t.NamedTuple): + """Directive whose ``:pyversion:`` decides whether its block runs. + + Attributes + ---------- + test_id : str + pytest parametrize id. + spec : str + PEP-440 specifier written on the ``:pyversion:`` option. + skipped : bool + Whether the block is expected to carry ``SKIP``. + """ + + test_id: str + spec: str + skipped: bool + + +PYVERSION_FIXTURES = [ + PyversionFixture(test_id="satisfied-runs", spec=">=3.10", skipped=False), + PyversionFixture(test_id="unsatisfied-skips", spec=">=99.0", skipped=True), + PyversionFixture(test_id="upper-bound-skips", spec="<3.0", skipped=True), +] + + +@pytest.mark.parametrize( + PyversionFixture._fields, + PYVERSION_FIXTURES, + ids=[f.test_id for f in PYVERSION_FIXTURES], +) +def test_pyversion_skips_the_block_it_excludes( + test_id: str, + spec: str, + skipped: bool, +) -> None: + """``:pyversion:`` compares the running interpreter against the specifier. + + The arguments were reversed, so every specifier was parsed as a version and + the page died on ``InvalidVersion`` before the option could decide anything. + """ + page = f".. doctest::\n :pyversion: {spec}\n\n >>> 2 + 2\n 4\n" + + (test,) = doctest_docutils.DocutilsDocTestFinder().find(page, "page.rst") + + assert test.examples[0].options.get(doctest.SKIP, False) is skipped + + +def test_pyversion_warns_on_a_malformed_specifier( + capsys: pytest.CaptureFixture[str], +) -> None: + """A ``:pyversion:`` that is no PEP-440 specifier warns and leaves the block. + + The option decides whether a block is for this interpreter. A value it + cannot parse answers neither way, so the block is left runnable and the + page reports the option rather than dying on it. + """ + page = ".. doctest::\n :pyversion: not a spec\n\n >>> 2 + 2\n 4\n" + + (test,) = doctest_docutils.DocutilsDocTestFinder().find(page, "page.rst") + + assert test.examples[0].options.get(doctest.SKIP, False) is False + assert "'not a spec' is not a valid pyversion option" in capsys.readouterr().err + + +BLANKLINE_REST = textwrap.dedent( + """ +Title +===== + +.. doctest:: + + >>> print("a\\n\\nb") + a + + b + """, +) + +SETUP_GROUP_REST = textwrap.dedent( + """ +Title +===== + +.. testsetup:: demo + + >>> import math + +.. doctest:: demo + + >>> math.floor(2.5) + 2 + +.. testcleanup:: demo + + >>> del math + """, +) + + +class DirectiveSourceFixture(t.NamedTuple): + """Page whose blocks only run once the directive's own source is read. + + Attributes + ---------- + test_id : str + pytest parametrize id. + page : str + reStructuredText page. + collected : int + Tests expected back from the finder. + """ + + test_id: str + page: str + collected: int + + +DIRECTIVE_SOURCE_FIXTURES = [ + DirectiveSourceFixture( + test_id="blankline-marker-inside-a-directive", + page=BLANKLINE_REST, + collected=1, + ), + DirectiveSourceFixture( + test_id="testsetup-and-testcleanup-share-a-group", + page=SETUP_GROUP_REST, + collected=1, + ), +] + + +@pytest.mark.parametrize( + DirectiveSourceFixture._fields, + DIRECTIVE_SOURCE_FIXTURES, + ids=[f.test_id for f in DIRECTIVE_SOURCE_FIXTURES], +) +def test_directive_blocks_run_from_their_own_source( + test_id: str, + page: str, + collected: int, +) -> None: + """Directives run the source they stored, not the code they render. + + ``.. doctest::`` rewrites a ```` marker into a real blank line + for the page and keeps the marker on the node, so reading the rendered + text instead compared against a blank line and failed. A ``testsetup`` + naming a group is only useful once that group is one namespace. + """ + tests = doctest_docutils.DocutilsDocTestFinder().find(page, "page.rst") + runner = doctest.DocTestRunner(verbose=False) + for test in tests: + runner.run(test, out=lambda _: None) + + assert len(tests) == collected + assert runner.failures == 0 + + +def test_a_failing_block_still_fails_beside_a_skipped_one() -> None: + """Skipping one block of a group does not excuse the rest of it. + + Lifting the gated block out must not lift the group's coverage out with + it: a skip that quietly took the whole namespace along would turn a broken + page green. + """ + page = textwrap.dedent( + """ +Title +===== + +.. doctest:: demo + :skipif: True + + >>> 1 / 0 + +.. doctest:: demo + + >>> 2 + 2 + 5 + """, + ) + + tests = doctest_docutils.DocutilsDocTestFinder().find(page, "page.rst") + runner = doctest.DocTestRunner(verbose=False) + for test in tests: + runner.run(test, out=lambda _: None) + + assert [test.name for test in tests] == ["demo[0]", "demo"] + assert runner.failures == 1 + + +def test_a_skipped_block_must_still_parse() -> None: + """A skipped block is parsed, so malformed doctest source still reports. + + Dropping the block hid its syntax; marking it ``SKIP`` does not. That + matches ``:options: +SKIP``, whose blocks have always had to parse. + """ + page = ".. doctest::\n :skipif: True\n\n >>>print(2)\n" + + with pytest.raises(ValueError, match="lacks blank after >>>"): + doctest_docutils.DocutilsDocTestFinder().find(page, "page.rst") + + +OUT_OF_ORDER_PHASES_REST = textwrap.dedent( + """ +Title +===== + +.. testcleanup:: demo + + >>> del value + +.. doctest:: demo + + >>> value + 1 + +.. testsetup:: demo + + >>> value = 1 + """, +) + +TWO_SETUPS_REST = textwrap.dedent( + """ +Title +===== + +.. testsetup:: demo + + >>> order = ["first"] + +.. testsetup:: demo + + >>> order.append("second") + +.. doctest:: demo + + >>> order + ['first', 'second'] + """, +) + + +def test_a_group_runs_setup_first_and_cleanup_last() -> None: + """Phase beats page order, so a hidden block can sit anywhere. + + ``testsetup`` and ``testcleanup`` render as comments, so an author moves + them out of the reader's way; running them where they sit bound names too + late and tore them down too early. + """ + (test,) = doctest_docutils.DocutilsDocTestFinder().find( + OUT_OF_ORDER_PHASES_REST, + "page.rst", + ) + runner = doctest.DocTestRunner(verbose=False) + runner.run(test, out=lambda _: None) + + assert [example.source.strip() for example in test.examples] == [ + "value = 1", + "value", + "del value", + ] + assert runner.failures == 0 + + +def test_two_setups_keep_their_page_order() -> None: + """Blocks of one phase run in the order the page wrote them.""" + (test,) = doctest_docutils.DocutilsDocTestFinder().find( + TWO_SETUPS_REST, + "page.rst", + ) + runner = doctest.DocTestRunner(verbose=False) + runner.run(test, out=lambda _: None) + + assert runner.failures == 0 + + +COMMA_GROUPS_REST = textwrap.dedent( + """ +Title +===== + +.. doctest:: alpha, beta + + >>> shared = 1 + +.. doctest:: beta + + >>> shared + 1 + """, +) + +WILDCARD_GROUP_REST = textwrap.dedent( + """ +Title +===== + +.. testsetup:: * + + >>> import math + +.. doctest:: alpha + + >>> math.floor(2.5) + 2 + +.. doctest:: beta + + >>> math.ceil(2.5) + 3 + """, +) + + +def test_a_block_joins_every_group_it_names() -> None: + """A comma list is every group the block belongs to, not just the first.""" + tests = doctest_docutils.DocutilsDocTestFinder().find( + COMMA_GROUPS_REST, + "page.rst", + ) + runner = doctest.DocTestRunner(verbose=False) + for test in tests: + runner.run(test, out=lambda _: None) + + assert [(test.name, len(test.examples)) for test in tests] == [ + ("alpha", 1), + ("beta", 2), + ] + assert runner.failures == 0 + + +def test_a_wildcard_joins_every_group_the_page_declares() -> None: + """``*`` is how a page writes one setup block for all of its groups.""" + tests = doctest_docutils.DocutilsDocTestFinder().find( + WILDCARD_GROUP_REST, + "page.rst", + ) + runner = doctest.DocTestRunner(verbose=False) + for test in tests: + runner.run(test, out=lambda _: None) + + assert [test.name for test in tests] == ["alpha", "beta"] + assert all(len(test.examples) == 2 for test in tests) + assert runner.failures == 0 + + +def test_a_shared_block_reports_one_line_in_every_group() -> None: + """Merging shifts example line numbers in place, so each copy is its own. + + A block joining two namespaces that shared its examples would have them + shifted twice, and the second group would report failures against a line + the block does not sit on. + """ + alpha, beta = doctest_docutils.DocutilsDocTestFinder().find( + COMMA_GROUPS_REST, + "page.rst", + ) + + def reported(test: doctest.DocTest, index: int) -> int: + return (test.lineno or 0) + test.examples[index].lineno + 1 + + assert reported(alpha, 0) == reported(beta, 0) + + +def test_out_of_order_phases_report_their_own_lines() -> None: + """A phase written away from its group still reports where it sits. + + A namespace hands its blocks over as setup, tests, cleanup, which is + rarely page order. Anchoring the merged text on that sequence reported + every example against whichever block came first in it, and could point + past the end of the file. + """ + finder = doctest_docutils.DocutilsDocTestFinder() + + def reported(page: str) -> list[tuple[str, int]]: + # Pairs rather than a dict keyed on the source: two examples can share + # a source, and one would then overwrite the other's line silently. + # Sorted because a merged namespace hands its blocks over in phase + # order and three separate ones come back in page order. + return sorted( + (example.source.strip(), (test.lineno or 0) + example.lineno + 1) + for test in finder.find(page, "page.rst") + for example in test.examples + ) + + merged = reported(OUT_OF_ORDER_PHASES_REST) + alone = reported(OUT_OF_ORDER_PHASES_REST.replace(":: demo", "::")) + + assert merged == alone + assert max(line for _, line in merged) <= len( + OUT_OF_ORDER_PHASES_REST.splitlines(), + ) + + +class NamespaceItemsFixture(t.NamedTuple): + """Page whose namespaces keep one test per block. + + Attributes + ---------- + test_id : str + pytest parametrize id. + file_name : str + Page name, whose suffix picks the parser. + page : str + Page content. + namespace_scope : doctest_docutils.NamespaceScope + Scope the finder is built with. + test_names : list[str] + Test names ``find`` returns, in order. + namespaces : list[str] + Namespace each returned test runs in, in the same order. + """ + + test_id: str + file_name: str + page: str + namespace_scope: doctest_docutils.NamespaceScope + test_names: list[str] + namespaces: list[str] + + +NAMESPACE_ITEMS_FIXTURES = [ + NamespaceItemsFixture( + test_id="ungrouped-fences-keep-the-names-block-scope-gives-them", + file_name="page.md", + page=STATE_MD, + namespace_scope="block", + test_names=["page.md[0]", "page.md[1]"], + namespaces=["page.md[0]", "page.md[1]"], + ), + NamespaceItemsFixture( + test_id="a-shared-page-keeps-those-names-too", + file_name="page.md", + page=STATE_MD, + namespace_scope="document", + test_names=["page.md[0]", "page.md[1]"], + namespaces=["page.md", "page.md"], + ), + NamespaceItemsFixture( + test_id="a-group-numbers-its-blocks-by-page-position", + file_name="page.rst", + page=SHARED_GROUP_REST, + namespace_scope="block", + test_names=["intro[0]", "intro[1]"], + namespaces=["intro", "intro"], + ), + NamespaceItemsFixture( + test_id="distinct-groups-stay-distinct", + file_name="page.rst", + page=DISTINCT_GROUPS_REST, + namespace_scope="document", + test_names=["alpha[0]", "beta[1]"], + namespaces=["alpha", "beta"], + ), +] + + +@pytest.mark.parametrize( + NamespaceItemsFixture._fields, + NAMESPACE_ITEMS_FIXTURES, + ids=[f.test_id for f in NAMESPACE_ITEMS_FIXTURES], +) +def test_per_block_keeps_a_test_per_block( + tmp_path: pathlib.Path, + test_id: str, + file_name: str, + page: str, + namespace_scope: doctest_docutils.NamespaceScope, + test_names: list[str], + namespaces: list[str], +) -> None: + """Every block comes back as its own test, named for where it sits. + + The name is the one a block already answers to at ``block`` scope, so a + node id does not move with the layout, and the tests of one namespace hold + one globals mapping rather than a copy each. + """ + page_path = tmp_path / file_name + page_path.write_text(page, encoding="utf-8") + + finder = doctest_docutils.DocutilsDocTestFinder( + namespace_scope=namespace_scope, + namespace_items="per-block", + ) + collected = finder._collect(page, str(page_path)) + + assert [held.test.name for held in collected] == test_names + assert [held.namespace for held in collected] == namespaces + by_namespace: dict[str, list[int]] = {} + for held in collected: + by_namespace.setdefault(held.namespace, []).append(id(held.test.globs)) + assert all(len(set(ids)) == 1 for ids in by_namespace.values()) + + +class NamespaceItemsStateFixture(t.NamedTuple): + """Page run block by block, counting the examples that fail. + + Attributes + ---------- + test_id : str + pytest parametrize id. + page : str + Page content. + namespace_scope : doctest_docutils.NamespaceScope + Scope the finder is built with. + failures : int + Examples expected to fail once every test has run. + """ + + test_id: str + page: str + namespace_scope: doctest_docutils.NamespaceScope + failures: int + + +NAMESPACE_ITEMS_STATE_FIXTURES = [ + NamespaceItemsStateFixture( + test_id="a-block-still-reads-nothing-by-default", + page=STATE_MD, + namespace_scope="block", + failures=1, + ), + NamespaceItemsStateFixture( + test_id="a-shared-page-reaches-the-block-below", + page=STATE_MD, + namespace_scope="document", + failures=0, + ), +] + + +@pytest.mark.parametrize( + NamespaceItemsStateFixture._fields, + NAMESPACE_ITEMS_STATE_FIXTURES, + ids=[f.test_id for f in NAMESPACE_ITEMS_STATE_FIXTURES], +) +def test_per_block_state_reaches_as_far_as_its_namespace( + tmp_path: pathlib.Path, + test_id: str, + page: str, + namespace_scope: doctest_docutils.NamespaceScope, + failures: int, +) -> None: + """A shared mapping carries names between tests; a scope still bounds it. + + ``DocTestRunner.run`` empties ``test.globs`` when it is done, so a caller + running these tests has to pass ``clear_globs=False`` for the sharing to + outlive the first block. That is the contract the pytest plugin's runner + holds up for it. + """ + page_path = tmp_path / "page.md" + page_path.write_text(page, encoding="utf-8") + + finder = doctest_docutils.DocutilsDocTestFinder( + namespace_scope=namespace_scope, + namespace_items="per-block", + ) + runner = doctest.DocTestRunner(verbose=False) + for test in finder.find(page, str(page_path)): + runner.run(test, out=lambda _: None, clear_globs=False) + + assert runner.failures == failures + + +def test_a_cleared_namespace_forgets_between_blocks(tmp_path: pathlib.Path) -> None: + """Sharing needs the runner's cooperation, and the shape says so. + + The mapping is handed over whole, but the stdlib empties it after each + test unless told otherwise, so a caller that forgets gets isolated blocks + back rather than a silently half-shared page. + """ + page_path = tmp_path / "page.md" + page_path.write_text(STATE_MD, encoding="utf-8") + + finder = doctest_docutils.DocutilsDocTestFinder( + namespace_scope="document", + namespace_items="per-block", + ) + runner = doctest.DocTestRunner(verbose=False) + for test in finder.find(STATE_MD, str(page_path)): + runner.run(test, out=lambda _: None) + + assert runner.failures == 1 + + +def test_per_block_reports_the_lines_block_scope_reports( + tmp_path: pathlib.Path, +) -> None: + """A block never merges, so it reports where docutils put it. + + Merging pads a namespace's text so its examples keep the lines they report + alone; keeping the blocks apart has nothing to pad, which is the same + answer by a shorter route. + """ + page_path = tmp_path / "page.rst" + page_path.write_text(CROWDED_REST, encoding="utf-8") + + assert _reported_lines( + CROWDED_REST, + page_path, + "document", + items="per-block", + ) == _reported_lines(CROWDED_REST, page_path, "block") + + +def test_per_block_runs_setup_first_and_cleanup_last() -> None: + """Phase still beats page order when a namespace is many tests. + + Collection order is run order for a caller that walks the list, so the + tests of one namespace come back setup first and cleanup last however the + page arranged them. + """ + finder = doctest_docutils.DocutilsDocTestFinder(namespace_items="per-block") + tests = finder.find(OUT_OF_ORDER_PHASES_REST, "page.rst") + runner = doctest.DocTestRunner(verbose=False) + for test in tests: + runner.run(test, out=lambda _: None, clear_globs=False) + + assert [test.name for test in tests] == ["demo[2]", "demo[1]", "demo[0]"] + assert [example.source.strip() for test in tests for example in test.examples] == [ + "value = 1", + "value", + "del value", + ] + assert runner.failures == 0 + + +def test_per_block_gates_a_block_and_leaves_the_rest_running() -> None: + """A gate is read the same way whatever a namespace collects as. + + Merged, a gated block is lifted into a test of its own so it still + reports; per block it already is one, and the blocks either side of it run + and can still fail. + """ + finder = doctest_docutils.DocutilsDocTestFinder(namespace_items="per-block") + tests = finder.find(GATED_MIDDLE_BLOCK_REST, "page.rst") + + gated = [test for test in tests if doctest_docutils._all_examples_skipped(test)] + assert [test.name for test in gated] == ["intro[1]"] + + runner = doctest.DocTestRunner(verbose=False) + for test in tests: + runner.run(test, out=lambda _: None, clear_globs=False) + + assert runner.failures == 0 + + +PER_BLOCK_MEMBERSHIP_FIXTURES = [ + ( + "comma-groups", + COMMA_GROUPS_REST, + [("alpha", "alpha[0]"), ("beta", "beta[0]"), ("beta", "beta[1]")], + ), + ( + "wildcard-group", + WILDCARD_GROUP_REST, + [ + ("alpha", "alpha[0]"), + ("beta", "beta[0]"), + ("alpha", "alpha[1]"), + ("beta", "beta[2]"), + ], + ), +] + + +@pytest.mark.parametrize( + ("test_id", "page", "collected"), + PER_BLOCK_MEMBERSHIP_FIXTURES, + ids=[test_id for test_id, _, _ in PER_BLOCK_MEMBERSHIP_FIXTURES], +) +def test_per_block_gives_a_shared_block_a_test_in_each_group( + test_id: str, + page: str, + collected: list[tuple[str, str]], +) -> None: + """A block joining two groups runs once per group, against that namespace. + + Its test is named for the group it is running in, so two groups holding + one block do not collide, and each group still gets its setup before its + tests. + """ + finder = doctest_docutils.DocutilsDocTestFinder(namespace_items="per-block") + held = finder._collect(page, "page.rst") + runner = doctest.DocTestRunner(verbose=False) + for one in held: + runner.run(one.test, out=lambda _: None, clear_globs=False) + + assert [(one.namespace, one.test.name) for one in held] == collected + assert runner.failures == 0 + + +def test_per_block_keeps_a_directive_option() -> None: + """``:options:`` reach a block's examples whatever it collects as.""" + finder = doctest_docutils.DocutilsDocTestFinder(namespace_items="per-block") + (test,) = finder.find( + ".. doctest::\n :options: +ELLIPSIS\n\n" + ' >>> print("hello world")\n hello ...\n', + "page.rst", + ) + runner = doctest.DocTestRunner(verbose=False) + runner.run(test, out=lambda _: None, clear_globs=False) + + assert test.examples[0].options[doctest.ELLIPSIS] is True + assert runner.failures == 0 + + +def test_namespace_items_rejects_an_unknown_name() -> None: + """An unknown layout names the values it could have been.""" + with pytest.raises(doctest_docutils.NamespaceItemsError) as excinfo: + doctest_docutils.DocutilsDocTestFinder( + namespace_items=t.cast("doctest_docutils.NamespaceItems", "one-each"), + ) + + assert str(excinfo.value) == ( + "Unknown namespace items: 'one-each'. Expected one of: merged, per-block" + ) + + +def test_merged_collection_names_the_namespace_of_every_test() -> None: + """A merged page names its namespaces too, lifted blocks included. + + The namespace is what a caller distributing tests has to keep together, + and a block lifted out of one belongs to it as much as the merged test + does. + """ + finder = doctest_docutils.DocutilsDocTestFinder() + collected = finder._collect(GATED_MIDDLE_BLOCK_REST, "page.rst") + + assert [(held.namespace, held.test.name) for held in collected] == [ + ("intro", "intro"), + ("intro", "intro[1]"), + ] + + +def test_per_block_under_testdocutils(tmp_path: pathlib.Path) -> None: + """Running a file has no scheduler, so per block simply shares. + + ``testdocutils`` owns its runner, so it is the one that has to leave the + namespace uncleared between blocks. + """ + page = tmp_path / "page.md" + page.write_text(STATE_MD, encoding="utf-8") + + with contextlib.redirect_stdout(io.StringIO()): + shared = doctest_docutils.testdocutils( + str(page), + module_relative=False, + report=False, + namespace_scope="document", + namespace_items="per-block", + ) + apart = doctest_docutils.testdocutils( + str(page), + module_relative=False, + report=False, + namespace_items="per-block", + ) + + assert shared == doctest.TestResults(failed=0, attempted=3) + assert apart == doctest.TestResults(failed=1, attempted=3) + + +def test_merging_reads_its_blocks_rather_than_consuming_them() -> None: + """Merging a block twice positions it the same way both times. + + A block is merged more than once whenever it names two groups, and again + when a gated block is lifted out of a namespace and merged on its own. + Shifting the block's own examples would move them further every time. + """ + parser = doctest.DocTestParser() + blocks = [ + parser.get_doctest(">>> 1 + 1\n2\n", {}, "n", "page.rst", 3), + parser.get_doctest(">>> 2 + 2\n4\n", {}, "n", "page.rst", 9), + ] + originals = [example.lineno for block in blocks for example in block.examples] + + first = doctest_docutils._merge_blocks(blocks, "n", "page.rst", {}) + second = doctest_docutils._merge_blocks(blocks, "n", "page.rst", {}) + + assert [example.lineno for example in first.examples] == [ + example.lineno for example in second.examples + ] + assert [ + example.lineno for block in blocks for example in block.examples + ] == originals + + +TESTCODE_PAGE_MD = textwrap.dedent( + """ + # Page + + Visible, pasteable, no prompt: + + ```{testcode} + value = 41 + ``` + + Hidden assertion the reader never sees: + + ```{testcode} + :hide: + + assert value == 41 + ``` + + Visible with expected output: + + ```{testcode} + print(value + 1) + ``` + + ```{testoutput} + 42 + ``` + """, +) + + +def _run_page( + tmp_path: pathlib.Path, source: str, **kwargs: t.Any +) -> doctest.TestResults: + """Run one page through ``testdocutils`` with its report swallowed.""" + page = tmp_path / kwargs.pop("filename", "page.md") + page.write_text(source, encoding="utf-8") + with contextlib.redirect_stdout(io.StringIO()): + return doctest_docutils.testdocutils( + str(page), + module_relative=False, + report=False, + **kwargs, + ) + + +def test_a_page_without_a_prompt_collects() -> None: + """A page a reader can paste out of is one namespace of three examples. + + A ``{testcode}`` carries no ``>>>``, which is the whole reason the pages + it is written for were invisible to the finder. + """ + tests = doctest_docutils.DocutilsDocTestFinder().find(TESTCODE_PAGE_MD, "page.md") + + assert [(test.name, len(test.examples)) for test in tests] == [("page.md", 3)] + + +def test_a_page_without_a_prompt_passes(tmp_path: pathlib.Path) -> None: + """The hidden block reads what the visible one bound, and asserts on it.""" + assert _run_page(tmp_path, TESTCODE_PAGE_MD) == doctest.TestResults( + failed=0, + attempted=3, + ) + + +def test_a_hidden_testcode_asserts_for_real(tmp_path: pathlib.Path) -> None: + """The hidden block is a test, not decoration: a false one fails the page.""" + broken = TESTCODE_PAGE_MD.replace("assert value == 41", "assert value == 999") + + assert _run_page(tmp_path, broken).failed == 1 + + +def test_a_hidden_testcode_leaves_the_rendered_page() -> None: + """``:hide:`` turns the block into a comment, as in :mod:`sphinx.ext.doctest`. + + Every builder drops a comment, so the reader meets only the block written + to be pasted. + """ + import docutils.core + from docutils import nodes + + doctest_docutils._ensure_directives_registered() + page = ( + ".. testcode::\n\n value = 41\n\n" + ".. testcode::\n :hide:\n\n assert value == 41\n" + ) + + doctree = docutils.core.publish_doctree(page) + + assert [ + (node.tagname, node.astext()) + for node in doctree.findall(nodes.Element) + if node.get("testnodetype") + ] == [ + ("literal_block", "value = 41"), + ("comment", "assert value == 41"), + ] + + +def test_a_bare_expression_in_testcode_reports_no_output( + tmp_path: pathlib.Path, +) -> None: + """``exec`` mode echoes nothing, so a bare expression is not a failure. + + ``single`` mode would print the value and report it as output the block + never said to expect. + """ + page = "```{testcode}\nvalue = 41\nvalue\n```\n" + + assert _run_page(tmp_path, page) == doctest.TestResults(failed=0, attempted=1) + + +def test_a_multi_statement_testcode_body_runs(tmp_path: pathlib.Path) -> None: + """``single`` mode takes one statement; a pasteable block takes many.""" + page = ( + "```{testcode}\nfirst = 1\nsecond = first + 1\nprint(second + 1)\n```\n" + "\n```{testoutput}\n3\n```\n" + ) + + assert _run_page(tmp_path, page) == doctest.TestResults(failed=0, attempted=1) + + +ECHO_FIXTURES = [ + ("a-block-that-expects-the-echo", "```python\n>>> 2 + 2\n4\n```\n", 0), + ("a-block-that-expects-nothing", "```python\n>>> 2 + 2\n```\n", 1), +] + + +@pytest.mark.parametrize( + ("test_id", "page", "failed"), + ECHO_FIXTURES, + ids=[fixture[0] for fixture in ECHO_FIXTURES], +) +def test_a_prompt_keeps_its_echo( + tmp_path: pathlib.Path, + test_id: str, + page: str, + failed: int, +) -> None: + """A ``>>>`` example still compiles in ``single`` mode. + + The second page proves the echo is real rather than merely tolerated: a + bare expression that printed nothing would pass it. + """ + assert _run_page(tmp_path, page).failed == failed + + +def test_testoutput_checks_the_block_above_it(tmp_path: pathlib.Path) -> None: + """The output a ``{testcode}`` prints is compared against the block below.""" + page = "```{testcode}\nprint(41 + 1)\n```\n\n```{testoutput}\n99\n```\n" + + assert _run_page(tmp_path, page).failed == 1 + + +def test_testoutput_options_reach_the_example(tmp_path: pathlib.Path) -> None: + """``:options:`` on the output block set the flags the check runs under.""" + page = ( + "```{testcode}\nprint('a long line of output')\n```\n\n" + "```{testoutput}\n:options: +ELLIPSIS\n\na long ... output\n```\n" + ) + + assert _run_page(tmp_path, page) == doctest.TestResults(failed=0, attempted=1) + + +def test_testoutput_can_expect_an_exception(tmp_path: pathlib.Path) -> None: + """A traceback in the output block is checked as an exception, not as text.""" + page = ( + "```{testcode}\nraise ValueError('boom')\n```\n\n" + "```{testoutput}\nTraceback (most recent call last):\n" + " ...\nValueError: boom\n```\n" + ) + + assert _run_page(tmp_path, page) == doctest.TestResults(failed=0, attempted=1) + + +def test_a_stray_testoutput_is_dropped(caplog: pytest.LogCaptureFixture) -> None: + """Output with no block above it checks nothing, and says so. + + Collecting it as a test of its own would report a pass for an expectation + nothing ever produced. + """ + page = "```{testoutput}\n42\n```\n" + + with caplog.at_level(logging.WARNING, logger="doctest_docutils"): + tests = doctest_docutils.DocutilsDocTestFinder().find(page, "page.md") + + assert tests == [] + assert [ + record.doctest_block_type + for record in caplog.records + if hasattr(record, "doctest_block_type") + ] == ["testoutput"] + + +TESTCODE_NAMESPACE_FIXTURES = [ + ("block-merged", "block", "merged", ["page.md"]), + ("document-merged", "document", "merged", ["page.md"]), + ("block-per-block", "block", "per-block", ["page.md[0]", "page.md[1]"]), + ("document-per-block", "document", "per-block", ["page.md[0]", "page.md[1]"]), +] + + +@pytest.mark.parametrize( + ("test_id", "scope", "items", "names"), + TESTCODE_NAMESPACE_FIXTURES, + ids=[fixture[0] for fixture in TESTCODE_NAMESPACE_FIXTURES], +) +def test_testcode_shares_its_page_at_every_setting( + test_id: str, + scope: str, + items: str, + names: list[str], +) -> None: + """A ``{testcode}`` shares its page whatever the scope says. + + The scope names the namespace of a block that declared no group; a + ``{testcode}`` is written so the visible block and the hidden one + asserting on it stay together, which is the page. It is named for the + page rather than for Sphinx's ``default``, so the id reads like every + other one this finder hands out. + """ + page = "```{testcode}\nvalue = 41\n```\n\n```{testcode}\nassert value == 41\n```\n" + + finder = doctest_docutils.DocutilsDocTestFinder( + namespace_scope=t.cast("t.Any", scope), + namespace_items=t.cast("t.Any", items), + ) + + assert [test.name for test in finder.find(page, "page.md")] == names + + +MIXED_FORMS_MD = textwrap.dedent( + """ + ``` + >>> base = 40 + ``` + + ```{testcode} + print(base + 2) + ``` + + ```{testoutput} + 42 + ``` + """, +) + + +def test_document_scope_joins_both_forms(tmp_path: pathlib.Path) -> None: + """At document scope a ``{testcode}`` reads what a prompt block bound. + + Both forms are named for the page there, which is the one namespace + :mod:`sphinx.ext.doctest` gives every block that declares no group. + """ + finder = doctest_docutils.DocutilsDocTestFinder(namespace_scope="document") + + assert [test.name for test in finder.find(MIXED_FORMS_MD, "page.md")] == ["page.md"] + assert _run_page( + tmp_path, + MIXED_FORMS_MD, + namespace_scope="document", + ) == doctest.TestResults(0, 2) + + +def test_block_scope_keeps_both_forms_apart() -> None: + """At block scope a prompt block is its own namespace, as it always was. + + Only the prompt-free blocks share, so the page collects two tests and the + ``{testcode}`` cannot read the prompt block's name. + """ + finder = doctest_docutils.DocutilsDocTestFinder() + + assert [test.name for test in finder.find(MIXED_FORMS_MD, "page.md")] == [ + "page.md[0]", + "page.md", + ] + + +def test_a_second_testoutput_replaces_the_first( + tmp_path: pathlib.Path, + caplog: pytest.LogCaptureFixture, +) -> None: + """The last answer wins, as in Sphinx, and the page hears about it. + + :meth:`sphinx.ext.doctest.TestGroup.add_code` replaces the output a + ``{testcode}`` already had, so a page reads the same here as it builds + there. Saying so is the part Sphinx leaves out. + """ + page = textwrap.dedent( + """ + ```{testcode} + print("second") + ``` + + ```{testoutput} + first + ``` + + ```{testoutput} + second + ``` + """, + ) + + with caplog.at_level(logging.WARNING, logger="doctest_docutils"): + results = _run_page(tmp_path, page) + + assert results == doctest.TestResults(0, 1) + assert [ + record.message + for record in caplog.records + if hasattr(record, "doctest_block_type") + ] == ["testoutput block replaces the one above it"] + + +def test_skipif_gates_a_testcode(tmp_path: pathlib.Path) -> None: + """A gated ``{testcode}`` is skipped rather than run and failed.""" + page = "```{testcode}\n:skipif: True\n\nraise AssertionError('never run')\n```\n" + + assert _run_page(tmp_path, page) == doctest.TestResults(failed=0, attempted=1) + + +def test_a_testsetup_of_the_group_runs_before_a_testcode( + tmp_path: pathlib.Path, +) -> None: + """Phase order holds with a ``{testcode}`` in the namespace. + + The setup block is written below the code it sets up, and still runs + first. It is written with prompts, which a page may keep doing. + """ + page = ( + "```{testcode} demo\nassert base == 40\nprint(base + 2)\n```\n\n" + "```{testoutput} demo\n42\n```\n\n" + "```{testsetup} demo\n>>> base = 40\n```\n" + ) + + assert _run_page(tmp_path, page) == doctest.TestResults(failed=0, attempted=2) + + +def test_testcode_reaches_the_command( + tmp_path: pathlib.Path, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + """``python -m doctest_docutils`` runs a prompt-free page too. + + The command never loads pytest, so the mode a ``{testcode}`` runs under + cannot come from the plugin. + """ + page = tmp_path / "page.md" + page.write_text(TESTCODE_PAGE_MD, encoding="utf-8") + monkeypatch.setattr("sys.argv", ["doctest_docutils", str(page)]) + + assert doctest_docutils._test() == 0 + + broken = tmp_path / "broken.md" + broken.write_text( + TESTCODE_PAGE_MD.replace("assert value == 41", "assert value == 999"), + encoding="utf-8", + ) + monkeypatch.setattr("sys.argv", ["doctest_docutils", str(broken)]) + + assert doctest_docutils._test() == 1 + assert "AssertionError" in capsys.readouterr().out + + +def test_the_exec_mode_seam_leaves_the_doctest_module_alone() -> None: + """The mode rides on one function object, not on :mod:`doctest`. + + :mod:`sphinx.ext.doctest` rebinds ``doctest.compile`` for the process and + never puts it back. gp-libs loads into every pytest session through its + ``pytest11`` entry point, so the rebinding stays inside the runner it was + made for — and the seam it needs is pinned here, because a CPython that + stopped resolving ``compile`` as a global would break it silently. + """ + stock = doctest.DocTestRunner._DocTestRunner__run # type: ignore[attr-defined] + + assert "compile" not in vars(doctest) + assert "compile" in stock.__code__.co_names + # A closure would need its cells rebuilt, and rebuilding a function without + # them raises nothing — it just misbehaves. + assert stock.__code__.co_freevars == () + assert ( + doctest_docutils._ExecModeRunner._DocTestRunner__run.__globals__["compile"] + is doctest_docutils._compile_source + ) + + +def test_two_groups_running_interleaved_each_get_their_output() -> None: + """A ``{testoutput}`` answers its own group, not whichever block sits above. + + :meth:`sphinx.ext.doctest.TestGroup.add_code` keeps a list per group and + pairs an output with that group's latest block, so a page may run two + groups' blocks alternately. + """ + page = textwrap.dedent( + """ + ```{testcode} alpha + print("A") + ``` + + ```{testcode} beta + print("B") + ``` + + ```{testoutput} alpha + A + ``` + + ```{testoutput} beta + B + ``` + """, + ) + finder = doctest_docutils.DocutilsDocTestFinder() + + tests = finder.find(page, "page.md") + + assert {test.name: test.examples[0].want for test in tests} == { + "alpha": "A\n", + "beta": "B\n", + } + + +def test_a_block_between_a_testcode_and_its_output_closes_the_pairing() -> None: + """Only the group's latest block takes an output, as under Sphinx. + + ``sphinx-build -b doctest`` fails this page: the ``{doctest}`` block lands + in the same group and leaves the ``{testcode}`` expecting nothing. + """ + page = textwrap.dedent( + """ + ```{testcode} + print("A") + ``` + + ```{doctest} + >>> 1 + 1 + 2 + ``` + + ```{testoutput} + A + ``` + """, + ) + finder = doctest_docutils.DocutilsDocTestFinder() + + tests = finder.find(page, "page.md") + + assert [example.want for test in tests for example in test.examples] == [ + "", + "2\n", + ] + + +def test_a_prompt_free_testsetup_runs(tmp_path: pathlib.Path) -> None: + """A page copied out of the Sphinx docs works, prompts and all absent. + + :mod:`sphinx.ext.doctest` runs a ``{testsetup}`` body through the same + ``exec`` its ``{testcode}`` uses and rejects a ``>>>`` outright, so the + canonical page carries no prompt anywhere. + """ + page = textwrap.dedent( + """ + ```{testsetup} + base = 40 + ``` + + ```{testcode} + print(base + 2) + ``` + + ```{testoutput} + 42 + ``` + """, + ) + + assert _run_page(tmp_path, page) == doctest.TestResults(failed=0, attempted=2) + + +def test_a_prompt_style_testsetup_reaches_an_unnamed_testcode( + tmp_path: pathlib.Path, +) -> None: + """Both spellings of a setup body feed the same page. + + gp-libs has always written a ``{testsetup}`` with prompts, so the prompt + decides how the body is read rather than the directive. + """ + page = textwrap.dedent( + """ + ```{testsetup} + >>> base = 40 + ``` + + ```{testcode} + print(base + 2) + ``` + + ```{testoutput} + 42 + ``` + """, + ) + + assert _run_page(tmp_path, page) == doctest.TestResults(failed=0, attempted=2) + + +def test_a_page_of_prompt_blocks_keeps_the_setup_it_had() -> None: + """An unnamed ``{testsetup}`` follows the prompt blocks when no testcode does. + + The implicit ``default`` group widens to the setup phase only for the pages + that need it, so a page written before ``{testcode}`` existed collects + exactly the namespaces it always did. + """ + page = textwrap.dedent( + """ + ```{testsetup} + >>> base = 40 + ``` + + ```python + >>> base + 2 + 42 + ``` + """, + ) + finder = doctest_docutils.DocutilsDocTestFinder(namespace_scope="document") + + (test,) = finder.find(page, "page.md") + + assert test.name == "page.md" + assert len(test.examples) == 2 + + +def test_a_failing_testcode_quotes_the_whole_block() -> None: + """The report shows the block and lands inside it, not on its opening line. + + pytest quotes ``lines[example.lineno - 9 : example.lineno + 1]`` and sends + the reader to ``test.lineno + example.lineno + 1``. + """ + page = ( + "# Page\n\n```{testcode}\na = 1\nb = 2\nc = 3\nraise ValueError('boom')\n```\n" + ) + finder = doctest_docutils.DocutilsDocTestFinder() + + (test,) = finder.find(page, "page.md") + (example,) = test.examples + + assert example.lineno == 3 + assert (test.lineno or 0) + example.lineno + 1 == 7 + + +def test_pyversion_on_a_testcode_says_it_does_nothing( + capsys: pytest.CaptureFixture[str], +) -> None: + """A declared option that is ignored has to say so. + + :mod:`sphinx.ext.doctest` declares ``:pyversion:`` on ``{testcode}`` and + acts on it only for ``{doctest}``. Honouring it here would pass a page + Sphinx fails; refusing it would fail a page Sphinx renders. + """ + page = ".. testcode::\n :pyversion: < 3.0\n\n ran = True\n" + + (test,) = doctest_docutils.DocutilsDocTestFinder().find(page, "page.rst") + + assert test.examples[0].options.get(doctest.SKIP, False) is False + assert "'pyversion' has no effect on 'testcode'" in capsys.readouterr().err + + +def test_the_exec_mode_seam_degrades_instead_of_failing_to_import() -> None: + """A moved private method must not break unrelated sessions. + + gp-libs loads through its ``pytest11`` entry point into every session that + has it installed, so an interpreter without the seam leaves CPython's loop + in place. ``{testcode}`` is what stops working, and + ``test_the_exec_mode_seam_leaves_the_doctest_module_alone`` is where that + is caught loudly. + """ + stock = doctest.DocTestRunner._DocTestRunner__run # type: ignore[attr-defined] + try: + del doctest.DocTestRunner._DocTestRunner__run # type: ignore[attr-defined] + + assert doctest_docutils._exec_mode_run() is None + finally: + doctest.DocTestRunner._DocTestRunner__run = stock # type: ignore[attr-defined] + + assert doctest_docutils._exec_mode_run() is not None diff --git a/tests/test_doctest_options.py b/tests/test_doctest_options.py index aa9cd68..eda994a 100644 --- a/tests/test_doctest_options.py +++ b/tests/test_doctest_options.py @@ -144,6 +144,80 @@ class DoctestOptionCase(t.NamedTuple): expected_outcome="skipped", description="Inline +SKIP directive works in .md files", ), + DoctestOptionCase( + test_id="skipif-false-runs-the-block-rst", + file_ext=".rst", + ini_options="", + doctest_content=textwrap.dedent( + """ + Example + ======= + + .. doctest:: + :skipif: False + + >>> 2 + 2 + 4 + """, + ), + expected_outcome="passed", + description=":skipif: False leaves the block collected", + ), + DoctestOptionCase( + test_id="skipif-true-reports-as-skipped-rst", + file_ext=".rst", + ini_options="", + doctest_content=textwrap.dedent( + """ + Example + ======= + + .. doctest:: + :skipif: True + + >>> 1 / 0 + """, + ), + expected_outcome="skipped", + description=":skipif: True reports the same way as :options: +SKIP", + ), + DoctestOptionCase( + test_id="skipif-true-reports-as-skipped-md", + file_ext=".md", + ini_options="", + doctest_content=textwrap.dedent( + """ + # Example + + ```{doctest} + :skipif: True + + >>> 1 / 0 + ``` + """, + ), + expected_outcome="skipped", + description=":skipif: True reports as skipped in a Markdown fence too", + ), + DoctestOptionCase( + test_id="inline-flag-cannot-reopen-a-true-skipif-rst", + file_ext=".rst", + ini_options="", + doctest_content=textwrap.dedent( + """ + Example + ======= + + .. doctest:: + :skipif: True + + >>> 2 + 2 # doctest: -SKIP + 4 + """, + ), + expected_outcome="skipped", + description="An example's own flag cannot reopen a true :skipif:", + ), # Inline ELLIPSIS directive DoctestOptionCase( test_id="inline-ellipsis-directive-rst", @@ -178,6 +252,80 @@ class DoctestOptionCase(t.NamedTuple): expected_outcome="passed", description="Inline +ELLIPSIS directive works in .md files", ), + DoctestOptionCase( + test_id="directive-options-normalize-whitespace-rst", + file_ext=".rst", + ini_options="", + doctest_content=textwrap.dedent( + """ + Example + ======= + + .. doctest:: + :options: +NORMALIZE_WHITESPACE + + >>> print("a b") + a b + """, + ), + expected_outcome="passed", + description=":options: applies to the block's examples", + ), + DoctestOptionCase( + test_id="directive-options-skip-rst", + file_ext=".rst", + ini_options="", + doctest_content=textwrap.dedent( + """ + Example + ======= + + .. doctest:: + :options: +SKIP + + >>> 1 / 0 + """, + ), + expected_outcome="skipped", + description=":options: +SKIP skips the block's examples", + ), + DoctestOptionCase( + test_id="inline-flag-beats-directive-options-rst", + file_ext=".rst", + ini_options="", + doctest_content=textwrap.dedent( + """ + Example + ======= + + .. doctest:: + :options: +SKIP + + >>> 2 + 2 # doctest: -SKIP + 4 + """, + ), + expected_outcome="passed", + description="An example's own flag overrides the directive's options", + ), + DoctestOptionCase( + test_id="inline-flag-inside-directive-rst", + file_ext=".rst", + ini_options="", + doctest_content=textwrap.dedent( + """ + Example + ======= + + .. doctest:: + + >>> print("a b") # doctest: +NORMALIZE_WHITESPACE + a b + """, + ), + expected_outcome="passed", + description="An inline flag applies although the directive trims it", + ), ] @@ -499,3 +647,278 @@ def test_edge_cases( assert "0 items" in stdout or "no tests ran" in stdout or expected_tests == 0 elif expected_outcome == "passed": result.assert_outcomes(passed=expected_tests) + + +THREE_BLOCK_REST = textwrap.dedent( + """ + Example + ======= + + .. doctest:: + :skipif: True + + >>> 1 / 0 + + .. doctest:: + :options: +SKIP + + >>> 1 / 0 + + .. doctest:: + + >>> 2 + 2 + 4 + """, +) + + +def test_skipif_true_reports_like_the_skip_flag( + pytester: _pytest.pytester.Pytester, +) -> None: + """A ``:skipif:`` block collects, counts, and reports as ``+SKIP`` does. + + A page holding all three spellings — a true ``:skipif:``, an + ``:options: +SKIP``, and an ordinary block — collects three items. The two + skipped ones report under ``-rs`` with the same reason, so a reader who + knows either spelling can predict the other. + """ + pytester.plugins = ["pytest_doctest_docutils"] + pytester.makefile(".ini", pytest="[pytest]\naddopts=-p no:doctest") + page = pytester.path / "test_doc.rst" + page.write_text(THREE_BLOCK_REST, encoding="utf-8") + + collected = pytester.runpytest(str(page), "--collect-only", "-q") + + collected.stdout.fnmatch_lines( + [ + "test_doc.rst::test_doc.rst[[]0[]]", + "test_doc.rst::test_doc.rst[[]1[]]", + "test_doc.rst::test_doc.rst[[]2[]]", + ], + consecutive=True, + ) + + result = pytester.runpytest(str(page), "-rs") + + result.assert_outcomes(passed=1, skipped=2) + result.stdout.fnmatch_lines( + [ + "SKIPPED [[]1[]] *: test_doc.rst:6: every example skipped", + "SKIPPED [[]1[]] *: test_doc.rst:11: every example skipped", + ], + ) + + +def test_skipif_block_is_selectable_by_node_id( + pytester: _pytest.pytester.Pytester, +) -> None: + """The skipped block keeps a node id a reader can run on its own. + + Dropping it left nothing to select; marking it ``SKIP`` leaves the item + addressable, which is what makes ``-rs`` and IDE test discovery agree. + """ + pytester.plugins = ["pytest_doctest_docutils"] + pytester.makefile(".ini", pytest="[pytest]\naddopts=-p no:doctest") + page = pytester.path / "test_doc.rst" + page.write_text(THREE_BLOCK_REST, encoding="utf-8") + + result = pytester.runpytest(f"{page}::test_doc.rst[0]") + + result.assert_outcomes(skipped=1) + + +GATED_GROUP_REST = textwrap.dedent( + """ + Example + ======= + + .. doctest:: intro + + >>> greeting = "hello" + + .. doctest:: intro + :skipif: True + + >>> raise AssertionError("the skipped block ran") + + .. doctest:: intro + + >>> greeting.upper() + 'HELLO' + """, +) + + +def test_skipif_leaves_the_rest_of_its_group_running( + pytester: _pytest.pytester.Pytester, +) -> None: + """Skipping one block of a group is not skipping the group's item. + + The group's item passes on the strength of the blocks that did run, and + the gated block is an item of its own that reports skipped with a node id + and a reason. The skipped block would raise if it ran, and the last block + needs a name the first one bound, which pins both halves of that claim. + """ + pytester.plugins = ["pytest_doctest_docutils"] + pytester.makefile(".ini", pytest="[pytest]\naddopts=-p no:doctest") + page = pytester.path / "test_doc.rst" + page.write_text(GATED_GROUP_REST, encoding="utf-8") + + result = pytester.runpytest(str(page), "-rs", "-v") + + result.assert_outcomes(passed=1, skipped=1) + result.stdout.fnmatch_lines( + [ + "test_doc.rst::intro PASSED*", + "test_doc.rst::intro[[]1[]] SKIPPED*", + ], + consecutive=True, + ) + result.stdout.fnmatch_lines( + ["SKIPPED [[]1[]] *: test_doc.rst:*: every example skipped"], + ) + + +def test_a_gated_block_of_a_group_is_selectable( + pytester: _pytest.pytester.Pytester, +) -> None: + """The item a gated block collects as answers to its own node id. + + A reader who sees the skip in ``-rs`` can paste the id back to pytest and + get the same one line, which is what makes the report actionable. + """ + pytester.plugins = ["pytest_doctest_docutils"] + pytester.makefile(".ini", pytest="[pytest]\naddopts=-p no:doctest") + page = pytester.path / "test_doc.rst" + page.write_text(GATED_GROUP_REST, encoding="utf-8") + + result = pytester.runpytest(f"{page}::intro[1]", "-rs") + + result.assert_outcomes(skipped=1) + + +def test_a_group_skipped_end_to_end_reports_skipped( + pytester: _pytest.pytester.Pytester, +) -> None: + """A group whose every block is skipped reports as one skipped item. + + The two spellings mix inside a single namespace, and pytest reports the + item skipped exactly when no example in it is left to run. + """ + pytester.plugins = ["pytest_doctest_docutils"] + pytester.makefile(".ini", pytest="[pytest]\naddopts=-p no:doctest") + page = pytester.path / "test_doc.rst" + page.write_text( + textwrap.dedent( + """ + Example + ======= + + .. doctest:: solo + :skipif: True + + >>> 1 / 0 + + .. doctest:: solo + :options: +SKIP + + >>> 1 / 0 + """, + ), + encoding="utf-8", + ) + + result = pytester.runpytest(str(page), "-rs") + + result.assert_outcomes(skipped=1) + + +def test_skipif_reaches_setup_and_cleanup_under_pytest( + pytester: _pytest.pytester.Pytester, +) -> None: + """A skipped ``testsetup`` or ``testcleanup`` does not run its examples. + + Both directives declare ``skipif``, and both would fail the group's item + if their examples ran. Each reports as its own skipped item, so a group + running without the setup it was written with is visible in the report. + """ + pytester.plugins = ["pytest_doctest_docutils"] + pytester.makefile(".ini", pytest="[pytest]\naddopts=-p no:doctest") + page = pytester.path / "test_doc.rst" + page.write_text( + textwrap.dedent( + """ + Example + ======= + + .. testsetup:: fixture + :skipif: True + + >>> raise AssertionError("the skipped testsetup ran") + + .. doctest:: fixture + + >>> 2 + 2 + 4 + + .. testcleanup:: fixture + :skipif: True + + >>> raise AssertionError("the skipped testcleanup ran") + """, + ), + encoding="utf-8", + ) + + result = pytester.runpytest(str(page), "-v") + + result.assert_outcomes(passed=1, skipped=2) + result.stdout.fnmatch_lines( + [ + "test_doc.rst::fixture[[]0[]] SKIPPED*", + "test_doc.rst::fixture PASSED*", + "test_doc.rst::fixture[[]2[]] SKIPPED*", + ], + consecutive=True, + ) + + +def test_collect_only_evaluates_the_skipif_expression( + pytester: _pytest.pytester.Pytester, +) -> None: + """Listing a page's items runs its ``:skipif:`` expressions. + + The option's contract is a Python expression evaluated while the page is + read, and ``--collect-only`` reads the page. Marking the block ``SKIP`` + instead of dropping it changes what the reader sees, not when the + expression is answered — so a page whose expression touches the world + still touches it during discovery. + """ + pytester.plugins = ["pytest_doctest_docutils"] + pytester.makefile(".ini", pytest="[pytest]\naddopts=-p no:doctest") + witness = pytester.path / "collect-only-ran.txt" + # Writes the witness file, then evaluates false, so the block still runs. + expression = ( + f'__import__("pathlib").Path({str(witness)!r}).write_text("ran") and False' + ) + page = pytester.path / "test_doc.rst" + page.write_text( + textwrap.dedent( + f""" + Example + ======= + + .. doctest:: + :skipif: {expression} + + >>> 2 + 2 + 4 + """, + ), + encoding="utf-8", + ) + + result = pytester.runpytest(str(page), "--collect-only", "-q") + + result.stdout.fnmatch_lines(["test_doc.rst::test_doc.rst[[]0[]]"]) + assert witness.read_text(encoding="utf-8") == "ran" diff --git a/tests/test_pytest_doctest_docutils.py b/tests/test_pytest_doctest_docutils.py index 70fbd5a..382e4ee 100644 --- a/tests/test_pytest_doctest_docutils.py +++ b/tests/test_pytest_doctest_docutils.py @@ -520,3 +520,1465 @@ def demo() -> int: result = pytester.runpytest(str(example), "--doctest-docutils-modules") result.assert_outcomes(passed=1) + + +STATE_MD = textwrap.dedent( + """ +# Title + +```python +>>> greeting = "hello" +>>> greeting +'hello' +``` + +Narrative prose between the two blocks. + +```python +>>> greeting.upper() +'HELLO' +``` + """, +) + +SHARED_GROUP_REST = textwrap.dedent( + """ +Title +===== + +.. doctest:: intro + + >>> greeting = "hello" + +Narrative prose. + +.. doctest:: intro + + >>> greeting.upper() + 'HELLO' + """, +) + + +def _write_ini( + pytester: _pytest.pytester.Pytester, + *lines: str, + addopts: str = "", +) -> None: + """Write a pytest.ini that keeps the built-in doctest plugin out. + + ``addopts`` appends to that, for a run whose configuration is the thing + under test. + """ + pytester.makefile( + ".ini", + pytest="\n".join( + ["[pytest]", f"addopts=-p no:doctest {addopts}".rstrip(), *lines], + ), + ) + + +class NamespaceCollectionCase(t.NamedTuple): + """Page and the items it collects. + + Attributes + ---------- + test_id : str + pytest parametrize id. + file_name : str + Page written into the pytester directory. + page : str + Page content. + node_ids : list[str] + Node ids expected, in collection order. + """ + + test_id: str + file_name: str + page: str + node_ids: list[str] + + +NAMESPACE_COLLECTION_CASES = [ + NamespaceCollectionCase( + test_id="group-collects-as-one-item", + file_name="page.rst", + page=SHARED_GROUP_REST, + node_ids=["page.rst::intro"], + ), + NamespaceCollectionCase( + test_id="markdown-group-collects-as-one-item", + file_name="page.md", + page=textwrap.dedent( + """ +# Title + +```{doctest} intro +>>> greeting = "hello" +``` + +Narrative prose. + +```{doctest} intro +>>> greeting.upper() +'HELLO' +``` + """, + ), + node_ids=["page.md::intro"], + ), + NamespaceCollectionCase( + test_id="distinct-groups-collect-separately", + file_name="page.rst", + page=textwrap.dedent( + """ +Title +===== + +.. doctest:: alpha + + >>> alpha_only = 1 + +.. doctest:: beta + + >>> alpha_only + Traceback (most recent call last): + NameError: name 'alpha_only' is not defined + """, + ), + node_ids=["page.rst::alpha", "page.rst::beta"], + ), + NamespaceCollectionCase( + test_id="ungrouped-blocks-collect-one-item-each", + file_name="page.md", + page="\n".join(f"```python\n>>> {n}\n{n}\n```\n" for n in range(12)), + node_ids=[f"page.md::page.md[{n}]" for n in range(12)], + ), +] + + +@pytest.mark.parametrize( + NamespaceCollectionCase._fields, + NAMESPACE_COLLECTION_CASES, + ids=[case.test_id for case in NAMESPACE_COLLECTION_CASES], +) +def test_namespace_collection( + pytester: _pytest.pytester.Pytester, + test_id: str, + file_name: str, + page: str, + node_ids: list[str], +) -> None: + """A page collects one item per namespace, in the order it reads. + + The node id carries the namespace and nothing machine-specific, so it can + be written into a ``--deselect`` and survive the trip to another checkout. + """ + pytester.plugins = ["pytest_doctest_docutils"] + _write_ini(pytester) + (pytester.path / file_name).write_text(page, encoding="utf-8") + + items, _ = pytester.inline_genitems(file_name) + + assert [item.nodeid for item in items] == node_ids + + result = pytester.runpytest(file_name) + result.assert_outcomes(passed=len(node_ids)) + + +def test_node_id_selects_one_namespace( + pytester: _pytest.pytester.Pytester, +) -> None: + """Running a node id runs exactly the namespace it names.""" + pytester.plugins = ["pytest_doctest_docutils"] + _write_ini(pytester) + (pytester.path / "page.rst").write_text(SHARED_GROUP_REST, encoding="utf-8") + + result = pytester.runpytest("page.rst::intro", "-v") + + result.assert_outcomes(passed=1) + result.stdout.fnmatch_lines(["page.rst::intro *"]) + + +def test_group_stops_at_the_document( + pytester: _pytest.pytester.Pytester, +) -> None: + """The same group name on two pages is two namespaces. + + Groups are read per document, as they are in :mod:`sphinx.ext.doctest`, so + one page cannot reach into the state another built. + """ + pytester.plugins = ["pytest_doctest_docutils"] + _write_ini(pytester) + (pytester.path / "first.rst").write_text( + ".. doctest:: intro\n\n >>> only_in_first = 1\n", + encoding="utf-8", + ) + (pytester.path / "second.rst").write_text( + textwrap.dedent( + """ +.. doctest:: intro + + >>> only_in_first + Traceback (most recent call last): + NameError: name 'only_in_first' is not defined + """, + ), + encoding="utf-8", + ) + + result = pytester.runpytest(str(pytester.path)) + + result.assert_outcomes(passed=2) + + +class NamespaceScopeOptionCase(t.NamedTuple): + """Namespace scope driven through the plugin's configuration. + + Attributes + ---------- + test_id : str + pytest parametrize id. + file_name : str + Page written into the pytester directory. + page : str + Page content. + ini_scope : str + Value for the ``doctest_docutils_namespace_scope`` ini option, empty to + leave it unset. + cli_args : list[str] + Extra command-line arguments for the run. + passed : int + Items expected to pass. + failed : int + Items expected to fail. + """ + + test_id: str + file_name: str + page: str + ini_scope: str + cli_args: list[str] + passed: int + failed: int + + +NAMESPACE_SCOPE_OPTION_CASES = [ + NamespaceScopeOptionCase( + test_id="unconfigured-keeps-blocks-apart", + file_name="page.md", + page=STATE_MD, + ini_scope="", + cli_args=[], + passed=1, + failed=1, + ), + NamespaceScopeOptionCase( + test_id="ini-document-shares-the-page", + file_name="page.md", + page=STATE_MD, + ini_scope="document", + cli_args=[], + passed=1, + failed=0, + ), + NamespaceScopeOptionCase( + test_id="cli-document-shares-the-page", + file_name="page.md", + page=STATE_MD, + ini_scope="", + cli_args=["--doctest-docutils-namespace-scope=document"], + passed=1, + failed=0, + ), + NamespaceScopeOptionCase( + test_id="cli-block-overrides-ini-document", + file_name="page.md", + page=STATE_MD, + ini_scope="document", + cli_args=["--doctest-docutils-namespace-scope=block"], + passed=1, + failed=1, + ), + NamespaceScopeOptionCase( + test_id="a-group-shares-whatever-the-scope-says", + file_name="page.rst", + page=SHARED_GROUP_REST, + ini_scope="block", + cli_args=[], + passed=1, + failed=0, + ), +] + + +@pytest.mark.parametrize( + NamespaceScopeOptionCase._fields, + NAMESPACE_SCOPE_OPTION_CASES, + ids=[case.test_id for case in NAMESPACE_SCOPE_OPTION_CASES], +) +def test_namespace_scope_option( + pytester: _pytest.pytester.Pytester, + test_id: str, + file_name: str, + page: str, + ini_scope: str, + cli_args: list[str], + passed: int, + failed: int, +) -> None: + """The scope reaches the finder from the ini file or the flag, flag first.""" + pytester.plugins = ["pytest_doctest_docutils"] + _write_ini( + pytester, + *([f"doctest_docutils_namespace_scope = {ini_scope}"] if ini_scope else []), + ) + (pytester.path / file_name).write_text(page, encoding="utf-8") + + result = pytester.runpytest(file_name, *cli_args) + + result.assert_outcomes(passed=passed, failed=failed) + + +def test_namespace_scope_rejects_an_unknown_ini_value( + pytester: _pytest.pytester.Pytester, +) -> None: + """A misspelled scope stops the session once, naming the values it knows.""" + pytester.plugins = ["pytest_doctest_docutils"] + _write_ini(pytester, "doctest_docutils_namespace_scope = per-file") + (pytester.path / "first.md").write_text(STATE_MD, encoding="utf-8") + (pytester.path / "second.md").write_text(STATE_MD, encoding="utf-8") + + result = pytester.runpytest(str(pytester.path)) + + assert result.ret == pytest.ExitCode.USAGE_ERROR + result.stderr.fnmatch_lines( + ["*Unknown namespace scope: 'per-file'*block, document*"], + ) + assert ( + len( + [line for line in result.stderr.lines if "Unknown namespace scope" in line], + ) + == 1 + ) + + +def test_document_scope_survives_xdist( + pytester: _pytest.pytester.Pytester, +) -> None: + """A shared page passes when pytest splits the session across workers. + + A namespace is one item, so no worker can be handed half of one. This is + the property that decided the design, which is why ``pytest-xdist`` is a + development dependency rather than something to skip around when absent. + """ + pytester.plugins = ["pytest_doctest_docutils"] + _write_ini(pytester, "doctest_docutils_namespace_scope = document") + (pytester.path / "page.md").write_text(STATE_MD, encoding="utf-8") + (pytester.path / "other.md").write_text(STATE_MD, encoding="utf-8") + + result = pytester.runpytest(str(pytester.path), "-n", "2") + + result.assert_outcomes(passed=2) + + +GATED_STATE_MD = textwrap.dedent( + """ +# Title + +```python +>>> greeting = "hello" +``` + +```python +>>> greeting = "nope" # doctest: +SKIP +``` + +```python +>>> greeting.upper() +'HELLO' +``` + """, +) + + +def test_a_shared_page_still_reports_its_gated_block( + pytester: _pytest.pytester.Pytester, +) -> None: + """A page merged end to end still says which of its blocks did not run. + + Under ``document`` a page with no groups is one namespace, which is where + a gated block would otherwise disappear: the item passes on the strength + of the blocks that ran and nothing names the one that did not. + """ + pytester.plugins = ["pytest_doctest_docutils"] + _write_ini(pytester, "doctest_docutils_namespace_scope = document") + (pytester.path / "page.md").write_text(GATED_STATE_MD, encoding="utf-8") + + result = pytester.runpytest("page.md", "-rs", "-v") + + result.assert_outcomes(passed=1, skipped=1) + result.stdout.fnmatch_lines( + ["page.md::page.md PASSED*", "page.md::page.md[[]1[]] SKIPPED*"], + consecutive=True, + ) + result.stdout.fnmatch_lines( + ["SKIPPED [[]1[]] *: page.md:*: every example skipped"], + ) + + +def test_a_gated_block_survives_xdist( + pytester: _pytest.pytester.Pytester, +) -> None: + """The item a gated block collects as distributes like any other. + + It is an ordinary item holding one block's examples, so a worker gets all + of it or none of it, the same property the merged namespace has. + """ + pytester.plugins = ["pytest_doctest_docutils"] + _write_ini(pytester, "doctest_docutils_namespace_scope = document") + (pytester.path / "page.md").write_text(GATED_STATE_MD, encoding="utf-8") + (pytester.path / "other.md").write_text(GATED_STATE_MD, encoding="utf-8") + + result = pytester.runpytest(str(pytester.path), "-n", "2") + + result.assert_outcomes(passed=2, skipped=2) + + +class NamespaceItemsOptionCase(t.NamedTuple): + """Namespace layout driven through the plugin's configuration. + + Attributes + ---------- + test_id : str + pytest parametrize id. + page : str + Page content, written as ``page.md``. + ini_lines : list[str] + Extra lines for the generated ``pytest.ini``. + cli_args : list[str] + Extra command-line arguments for the run. + node_ids : list[str] + Node ids expected, in collection order. + passed : int + Items expected to pass. + failed : int + Items expected to fail. + """ + + test_id: str + page: str + ini_lines: list[str] + cli_args: list[str] + node_ids: list[str] + passed: int + failed: int + + +NAMESPACE_ITEMS_OPTION_CASES = [ + NamespaceItemsOptionCase( + test_id="unconfigured-merges-a-shared-page", + page=STATE_MD, + ini_lines=["doctest_docutils_namespace_scope = document"], + cli_args=[], + node_ids=["page.md::page.md"], + passed=1, + failed=0, + ), + NamespaceItemsOptionCase( + test_id="ini-per-block-keeps-both-node-ids", + page=STATE_MD, + ini_lines=[ + "doctest_docutils_namespace_scope = document", + "doctest_docutils_namespace_items = per-block", + ], + cli_args=[], + node_ids=["page.md::page.md[0]", "page.md::page.md[1]"], + passed=2, + failed=0, + ), + NamespaceItemsOptionCase( + test_id="cli-per-block-keeps-both-node-ids", + page=STATE_MD, + ini_lines=["doctest_docutils_namespace_scope = document"], + cli_args=["--doctest-docutils-namespace-items=per-block"], + node_ids=["page.md::page.md[0]", "page.md::page.md[1]"], + passed=2, + failed=0, + ), + NamespaceItemsOptionCase( + test_id="cli-merged-overrides-ini-per-block", + page=STATE_MD, + ini_lines=[ + "doctest_docutils_namespace_scope = document", + "doctest_docutils_namespace_items = per-block", + ], + cli_args=["--doctest-docutils-namespace-items=merged"], + node_ids=["page.md::page.md"], + passed=1, + failed=0, + ), + NamespaceItemsOptionCase( + test_id="per-block-alone-shares-nothing", + page=STATE_MD, + ini_lines=["doctest_docutils_namespace_items = per-block"], + cli_args=[], + node_ids=["page.md::page.md[0]", "page.md::page.md[1]"], + passed=1, + failed=1, + ), +] + + +@pytest.mark.parametrize( + NamespaceItemsOptionCase._fields, + NAMESPACE_ITEMS_OPTION_CASES, + ids=[case.test_id for case in NAMESPACE_ITEMS_OPTION_CASES], +) +def test_namespace_items_option( + pytester: _pytest.pytester.Pytester, + test_id: str, + page: str, + ini_lines: list[str], + cli_args: list[str], + node_ids: list[str], + passed: int, + failed: int, +) -> None: + """The layout reaches the finder from the ini file or the flag, flag first. + + Scope and layout are separate questions: the scope says what shares a + namespace, the layout says whether sharing costs the blocks their node + ids. Setting only the layout shares nothing, because the default scope + still gives each block a namespace of its own. + """ + pytester.plugins = ["pytest_doctest_docutils"] + _write_ini(pytester, *ini_lines) + (pytester.path / "page.md").write_text(page, encoding="utf-8") + + items, _ = pytester.inline_genitems("page.md", *cli_args) + assert [item.nodeid for item in items] == node_ids + + result = pytester.runpytest("page.md", *cli_args) + result.assert_outcomes(passed=passed, failed=failed) + + +def test_namespace_items_rejects_an_unknown_ini_value( + pytester: _pytest.pytester.Pytester, +) -> None: + """A misspelled layout stops the session once, naming the values it knows.""" + pytester.plugins = ["pytest_doctest_docutils"] + _write_ini(pytester, "doctest_docutils_namespace_items = one-each") + (pytester.path / "first.md").write_text(STATE_MD, encoding="utf-8") + (pytester.path / "second.md").write_text(STATE_MD, encoding="utf-8") + + result = pytester.runpytest(str(pytester.path)) + + assert result.ret == pytest.ExitCode.USAGE_ERROR + result.stderr.fnmatch_lines( + ["*Unknown namespace items: 'one-each'*merged, per-block*"], + ) + + +def test_a_per_block_node_id_runs_one_block( + pytester: _pytest.pytester.Pytester, +) -> None: + """A node id reaches one block, and says plainly what running it alone costs. + + Reaching a block is the whole point of the layout — ``--lf``, ``-k``, a + JUnit report and a re-run all work through the id. A block that reads what + the block above it bound cannot run alone, because nothing bound it: that + limitation is inherent to running a fragment of a session, so it reports as + the ``NameError`` it is. + """ + pytester.plugins = ["pytest_doctest_docutils"] + _write_ini( + pytester, + "doctest_docutils_namespace_scope = document", + "doctest_docutils_namespace_items = per-block", + ) + (pytester.path / "page.md").write_text(STATE_MD, encoding="utf-8") + + first = pytester.runpytest("page.md::page.md[0]", "-v") + first.assert_outcomes(passed=1) + first.stdout.fnmatch_lines(["page.md::page.md[[]0[]] *"]) + + second = pytester.runpytest("page.md::page.md[1]") + + second.assert_outcomes(failed=1) + second.stdout.fnmatch_lines(["*NameError: name 'greeting' is not defined*"]) + + +def test_per_block_marks_each_namespace_for_loadgroup( + pytester: _pytest.pytester.Pytester, +) -> None: + """Every block carries the group its namespace distributes under. + + The plugin can emit the marker but cannot pick the scheduler, so the + marker is what makes ``--dist loadgroup`` usable. The group is the file + plus the namespace, because a namespace never reaches past its page. + """ + pytester.plugins = ["pytest_doctest_docutils"] + _write_ini( + pytester, + "doctest_docutils_namespace_scope = document", + "doctest_docutils_namespace_items = per-block", + ) + (pytester.path / "page.md").write_text(STATE_MD, encoding="utf-8") + + items, _ = pytester.inline_genitems("page.md") + + markers = [item.get_closest_marker("xdist_group") for item in items] + assert [marker.args[0] for marker in markers if marker is not None] == [ + "page.md::page.md", + "page.md::page.md", + ] + + +def test_merged_marks_nothing_for_loadgroup( + pytester: _pytest.pytester.Pytester, +) -> None: + """A merged namespace is one item, which no scheduler can split.""" + pytester.plugins = ["pytest_doctest_docutils"] + _write_ini(pytester, "doctest_docutils_namespace_scope = document") + (pytester.path / "page.md").write_text(STATE_MD, encoding="utf-8") + + items, _ = pytester.inline_genitems("page.md") + + assert [item.get_closest_marker("xdist_group") for item in items] == [None] + + +class SplittingSchedulerCase(t.NamedTuple): + """Invocation naming a scheduler that distributes a namespace by item. + + Attributes + ---------- + test_id : str + pytest parametrize id. + args : list[str] + Arguments appended to the run. + addopts : str + Arguments the ini file carries instead. + named : str + Scheduler the refusal is expected to name. + """ + + test_id: str + args: list[str] + addopts: str + named: str + + +SPLITTING_SCHEDULER_CASES = [ + SplittingSchedulerCase( + test_id="load-distributes-by-item", + args=["-n", "2", "--dist", "load"], + addopts="", + named="load", + ), + SplittingSchedulerCase( + test_id="worksteal-distributes-then-rebalances", + args=["-n", "2", "--dist", "worksteal"], + addopts="", + named="worksteal", + ), + SplittingSchedulerCase( + test_id="addopts-names-the-scheduler-too", + args=["-n", "2"], + addopts="--dist load", + named="load", + ), + SplittingSchedulerCase( + test_id="multiplied-tx-spells-more-than-one-worker", + args=["--tx", "2*popen", "--dist", "load"], + addopts="", + named="load", + ), + SplittingSchedulerCase( + test_id="multiplied-tx-adds-up-across-specifications", + args=["--tx", "1*popen", "--tx", "1*popen", "--dist", "load"], + addopts="", + named="load", + ), + SplittingSchedulerCase( + test_id="a-count-asking-for-none-takes-none-away", + args=["--tx", "-1*popen", "--tx", "2*popen", "--dist", "load"], + addopts="", + named="load", + ), +] + + +@pytest.mark.parametrize( + SplittingSchedulerCase._fields, + SPLITTING_SCHEDULER_CASES, + ids=[case.test_id for case in SPLITTING_SCHEDULER_CASES], +) +def test_per_block_refuses_a_named_splitting_scheduler( + pytester: _pytest.pytester.Pytester, + test_id: str, + args: list[str], + addopts: str, + named: str, +) -> None: + """Naming a scheduler that distributes by item stops the run. + + ``load`` hands a file's items to whichever worker is free and + ``worksteal`` does the same, then re-balances. A shared globals mapping + is a Python object and does not cross processes, so the session stops + rather than reporting a page that is only wrong because of how it was + scheduled. Asking for one by name is a choice to answer, not to overrule. + + A scheduler asked for through ini ``addopts`` is asked for just as much + as one typed on the command line — pytest folds ``addopts`` into the + arguments before parsing them, which is why reading the parsed value + finds both. It is also why ``sys.argv`` cannot be read instead. + """ + pytester.plugins = ["pytest_doctest_docutils"] + _write_ini( + pytester, + "doctest_docutils_namespace_scope = document", + "doctest_docutils_namespace_items = per-block", + addopts=addopts, + ) + (pytester.path / "page.md").write_text(STATE_MD, encoding="utf-8") + + result = pytester.runpytest(str(pytester.path), *args) + + assert result.ret == pytest.ExitCode.USAGE_ERROR + result.stderr.fnmatch_lines([f"*--dist {named} hands a file's items*"]) + result.stderr.fnmatch_lines(["*--dist loadgroup or --dist loadfile*"]) + + +def test_per_block_keeps_a_single_multiplied_worker( + pytester: _pytest.pytester.Pytester, +) -> None: + """One worker cannot split a namespace, however the run spelled it. + + ``--tx 1*popen`` asks for the same single environment ``--tx popen`` + does. Counting the multiplier has to leave that run alone, or reading + the shorthand correctly would cost every one-worker run its scheduler. + """ + pytester.plugins = ["pytest_doctest_docutils"] + _write_ini( + pytester, + "doctest_docutils_namespace_scope = document", + "doctest_docutils_namespace_items = per-block", + ) + (pytester.path / "page.md").write_text(STATE_MD, encoding="utf-8") + + result = pytester.runpytest( + str(pytester.path), + "--tx", + "1*popen", + "--dist", + "load", + ) + + result.assert_outcomes(passed=2) + + +def test_per_block_keeps_workers_for_a_suite_holding_no_page( + pytester: _pytest.pytester.Pytester, +) -> None: + """A suite with no page keeps its workers, whatever scheduler it named. + + The layout is a project-wide setting, so a project can carry it in its + ini while a given run collects only Python tests. Nothing there holds a + namespace, so there is nothing a scheduler could split and no reason to + take ``-n`` away — which is why the refusal reads the run's collection + rather than the setting. + """ + pytester.plugins = ["pytest_doctest_docutils"] + _write_ini( + pytester, + "doctest_docutils_namespace_scope = document", + "doctest_docutils_namespace_items = per-block", + ) + pytester.makepyfile( + test_python=""" + def test_one() -> None: + assert True + + + def test_two() -> None: + assert True + """, + ) + + left_open = pytester.runpytest(str(pytester.path), "-n", "2") + left_open.assert_outcomes(passed=2) + + named = pytester.runpytest(str(pytester.path), "-n", "2", "--dist", "worksteal") + + named.assert_outcomes(passed=2) + + +def test_per_block_fills_in_a_scheduler_the_run_left_open( + pytester: _pytest.pytester.Pytester, +) -> None: + """``-n`` alone asks for workers, not for a way of filling them. + + pytest-xdist answers it with ``--dist load``, which splits a page. The + run said nothing about distribution, so file-level scheduling is filled + in and the page comes through whole beside the Python tests that share + the session. + + The node ids stay the ones the layout collects. ``loadgroup`` would suit + the marker the plugin emits, but the group is appended to a node id by + the worker, from the worker's own ``--dist`` value, so a controller + cannot reach it — and substituting that scheduler would leave every item + in a scope of its own and split the page after all. + """ + pytester.plugins = ["pytest_doctest_docutils"] + _write_ini( + pytester, + "doctest_docutils_namespace_scope = document", + "doctest_docutils_namespace_items = per-block", + ) + (pytester.path / "page.md").write_text(STATE_MD, encoding="utf-8") + (pytester.path / "other.md").write_text(STATE_MD, encoding="utf-8") + pytester.makepyfile( + test_python=""" + def test_one() -> None: + assert True + """, + ) + + result = pytester.runpytest(str(pytester.path), "-n", "2", "-v") + + result.assert_outcomes(passed=5) + result.stdout.fnmatch_lines(["*scheduling tests via _PageScheduling*"]) + assert not [line for line in result.stdout.lines if "@page.md" in line] + + +def test_merged_survives_the_splitting_scheduler( + pytester: _pytest.pytester.Pytester, +) -> None: + """The refusal reaches only the layout that needs it.""" + pytester.plugins = ["pytest_doctest_docutils"] + _write_ini(pytester, "doctest_docutils_namespace_scope = document") + (pytester.path / "page.md").write_text(STATE_MD, encoding="utf-8") + (pytester.path / "other.md").write_text(STATE_MD, encoding="utf-8") + + result = pytester.runpytest(str(pytester.path), "-n", "2") + + result.assert_outcomes(passed=2) + + +class PerBlockSchedulerCase(t.NamedTuple): + """Distributed run a per-block page comes through whole. + + Attributes + ---------- + test_id : str + pytest parametrize id. + args : list[str] + Arguments the run is made with. + passed : int + Examples expected to pass across every worker. + """ + + test_id: str + args: list[str] + passed: int + + +PER_BLOCK_SCHEDULER_CASES = [ + PerBlockSchedulerCase( + test_id="loadfile-keeps-a-file-whole", + args=["-n", "2", "--dist", "loadfile"], + passed=4, + ), + PerBlockSchedulerCase( + test_id="loadgroup-keeps-a-namespace-whole", + args=["-n", "2", "--dist", "loadgroup"], + passed=4, + ), + PerBlockSchedulerCase( + test_id="loadscope-keeps-a-file-whole", + args=["-n", "2", "--dist", "loadscope"], + passed=4, + ), + PerBlockSchedulerCase( + test_id="each-repeats-the-suite-per-worker", + args=["-n", "2", "--dist", "each"], + passed=8, + ), + PerBlockSchedulerCase( + test_id="one-worker-has-nothing-to-split-against", + args=["-n", "1"], + passed=4, + ), + PerBlockSchedulerCase( + test_id="n-alone-leaves-the-scheduler-to-fill-in", + args=["-n", "2"], + passed=4, + ), + PerBlockSchedulerCase( + test_id="dist-without-workers-never-distributes", + args=["--dist", "load"], + passed=4, + ), +] + + +@pytest.mark.parametrize( + PerBlockSchedulerCase._fields, + PER_BLOCK_SCHEDULER_CASES, + ids=[case.test_id for case in PER_BLOCK_SCHEDULER_CASES], +) +def test_per_block_survives_a_scheduler_that_keeps_it_together( + pytester: _pytest.pytester.Pytester, + test_id: str, + args: list[str], + passed: int, +) -> None: + """A state-building page passes wherever its namespace stays on one worker. + + ``loadfile`` and ``loadscope`` split on the node id's path; ``loadgroup`` + reads the ``xdist_group`` marker the plugin emits; ``each`` gives every + worker the whole suite. A run xdist would not distribute at all — one + worker, or a ``--dist`` value with no workers behind it — is not refused + either, because there is nothing for it to split a namespace between. + ``-n`` on its own names no scheduler, so one that keeps a page whole is + filled in. + """ + pytester.plugins = ["pytest_doctest_docutils"] + _write_ini( + pytester, + "doctest_docutils_namespace_scope = document", + "doctest_docutils_namespace_items = per-block", + ) + (pytester.path / "page.md").write_text(STATE_MD, encoding="utf-8") + (pytester.path / "other.md").write_text(STATE_MD, encoding="utf-8") + + result = pytester.runpytest(str(pytester.path), *args) + + result.assert_outcomes(passed=passed) + + +def test_strict_markers_passes_whether_or_not_you_opt_in( + pytester: _pytest.pytester.Pytester, +) -> None: + """``xdist_group`` is registered whatever the layout, and pytest-xdist absent. + + A project that never asks for the layout never meets the marker at all. + One that does, on a machine without pytest-xdist to register the marker + itself, would otherwise have every item rejected as carrying an unknown + marker. + + Run out of process because pytest caches known marker names on the global + ``MarkGenerator``, so an in-process run inherits whatever this suite's own + session registered and could not tell the two cases apart. + """ + _write_ini(pytester) + (pytester.path / "page.md").write_text(STATE_MD, encoding="utf-8") + + unconfigured = pytester.runpytest_subprocess( + "page.md", + "--strict-markers", + "-p", + "no:xdist", + ) + unconfigured.assert_outcomes(passed=1, failed=1) + + result = pytester.runpytest_subprocess( + "page.md", + "--strict-markers", + "-p", + "no:xdist", + "--doctest-docutils-namespace-items=per-block", + ) + + result.assert_outcomes(passed=1, failed=1) + + +def test_xdist_group_is_listed_once(pytester: _pytest.pytester.Pytester) -> None: + """``pytest --markers`` describes the marker once, xdist installed or not. + + pytest-xdist registers ``xdist_group`` itself, so this plugin only fills + the gap it leaves. Registering unconditionally would list the marker twice + for every project that has xdist, opted in or not. + + Run out of process for the same reason as the ``--strict-markers`` case: + marker registration is read back off configuration this suite's own + session has already populated. + """ + _write_ini(pytester) + + with_xdist = pytester.runpytest_subprocess("--markers") + without_xdist = pytester.runpytest_subprocess("--markers", "-p", "no:xdist") + + def listed(result: _pytest.pytester.RunResult) -> list[str]: + return [ + line + for line in result.stdout.lines + if line.startswith("@pytest.mark.xdist_group") + ] + + assert len(listed(with_xdist)) == 1 + assert listed(without_xdist) == [ + ( + "@pytest.mark.xdist_group(name): keep a namespace's blocks on one" + " pytest-xdist worker under --dist loadgroup" + ), + ] + + +def test_per_block_collects_under_collect_only( + pytester: _pytest.pytester.Pytester, +) -> None: + """``--collect-only`` is never refused: xdist skips itself when only collecting. + + A project carrying ``-n`` in its ``addopts`` has to be able to enumerate + its own suite, and no example runs, so no namespace is ever shared. + """ + pytester.plugins = ["pytest_doctest_docutils"] + _write_ini( + pytester, + "doctest_docutils_namespace_scope = document", + "doctest_docutils_namespace_items = per-block", + ) + (pytester.path / "page.md").write_text(STATE_MD, encoding="utf-8") + + result = pytester.runpytest(str(pytester.path), "--collect-only", "-q", "-n", "2") + + assert result.ret == pytest.ExitCode.OK + result.stdout.fnmatch_lines(["page.md::page.md[[]0[]]", "page.md::page.md[[]1[]]"]) + + +def test_per_block_reports_the_layout_it_resolved( + pytester: _pytest.pytester.Pytester, +) -> None: + """The header says which layout ran, and says nothing when it is the usual one.""" + pytester.plugins = ["pytest_doctest_docutils"] + _write_ini(pytester, "doctest_docutils_namespace_scope = document") + (pytester.path / "page.md").write_text(STATE_MD, encoding="utf-8") + + quiet = pytester.runpytest("page.md") + assert not [ + line for line in quiet.stdout.lines if line.startswith("doctest-docutils:") + ] + + result = pytester.runpytest( + "page.md", + "--doctest-docutils-namespace-items=per-block", + ) + + result.stdout.fnmatch_lines( + ["doctest-docutils: namespace items: per-block, namespace scope: document"], + ) + + +DOCTEST_NAMESPACE_CONFTEST = textwrap.dedent( + """ +from typing import Any, Dict +import pytest + +@pytest.fixture(autouse=True) +def add_doctest_fixtures(doctest_namespace: Dict[str, Any]): + doctest_namespace["add"] = lambda a, b: a + b + """, +) + +FIXTURE_USING_MD = textwrap.dedent( + """ +# Title + +```python +>>> add(1, 2) +3 +``` + +Prose between the blocks. + +```python +>>> add(3, 4) +7 +``` + """, +) + + +@pytest.mark.parametrize( + ("test_id", "items", "passed"), + [ + ("merged", "merged", 1), + ("per-block", "per-block", 2), + ], + ids=["merged", "per-block"], +) +def test_doctest_namespace_reaches_every_block( + pytester: _pytest.pytester.Pytester, + test_id: str, + items: str, + passed: int, +) -> None: + """A fixture seeded into the namespace is in scope for every block of it. + + pytest merges the fixture into ``dtest.globs`` at item setup. Merged, that + happens once for the namespace; per block it happens once per block, into + the one mapping they share. + """ + pytester.plugins = ["pytest_doctest_docutils"] + _write_ini( + pytester, + "doctest_docutils_namespace_scope = document", + f"doctest_docutils_namespace_items = {items}", + ) + pytester.makeconftest(DOCTEST_NAMESPACE_CONFTEST) + (pytester.path / "page.md").write_text(FIXTURE_USING_MD, encoding="utf-8") + + result = pytester.runpytest("page.md") + + result.assert_outcomes(passed=passed) + + +TORN_DOWN_FIXTURE_CONFTEST = textwrap.dedent( + """ +from typing import Any, Dict +import pytest + + +class Server: + def __init__(self) -> None: + self.alive = True + + +@pytest.fixture(autouse=True) +def server(doctest_namespace: Dict[str, Any]): + running = Server() + doctest_namespace["server"] = running + yield running + running.alive = False + """, +) + +CARRIED_FIXTURE_MD = textwrap.dedent( + """ +# Title + +```python +>>> kept = server +>>> kept.alive +True +``` + +Prose between the blocks. + +```python +>>> kept.alive +True +``` + """, +) + + +@pytest.mark.parametrize( + ("test_id", "items", "passed", "failed"), + [ + ("merged", "merged", 1, 0), + ("per-block", "per-block", 1, 1), + ], + ids=["merged", "per-block"], +) +def test_per_block_finalizes_a_fixture_between_blocks( + pytester: _pytest.pytester.Pytester, + test_id: str, + items: str, + passed: int, + failed: int, +) -> None: + """A namespace shares the mapping, not the lifetime of what a fixture made. + + Per block, each block is its own item, so a function-scoped fixture tears + down between them. An object one block bound out of that fixture is + finalized before the next block reads it, which merged is a single item + and so never happens. + """ + pytester.plugins = ["pytest_doctest_docutils"] + _write_ini( + pytester, + "doctest_docutils_namespace_scope = document", + f"doctest_docutils_namespace_items = {items}", + ) + pytester.makeconftest(TORN_DOWN_FIXTURE_CONFTEST) + (pytester.path / "page.md").write_text(CARRIED_FIXTURE_MD, encoding="utf-8") + + result = pytester.runpytest("page.md") + + result.assert_outcomes(passed=passed, failed=failed) + + +def test_per_block_still_reports_a_gated_block( + pytester: _pytest.pytester.Pytester, +) -> None: + """A gated block reports skipped, and the block after it still runs. + + Merged, a gated block has to be lifted out of its namespace to report at + all; per block it is already an item, and it is marked before setup either + way, so its fixtures never run for a block that executes nothing. + """ + pytester.plugins = ["pytest_doctest_docutils"] + _write_ini( + pytester, + "doctest_docutils_namespace_scope = document", + "doctest_docutils_namespace_items = per-block", + ) + (pytester.path / "page.md").write_text(GATED_STATE_MD, encoding="utf-8") + + result = pytester.runpytest("page.md", "-rs", "-v") + + result.assert_outcomes(passed=2, skipped=1) + result.stdout.fnmatch_lines( + [ + "page.md::page.md[[]0[]] PASSED*", + "page.md::page.md[[]1[]] SKIPPED*", + "page.md::page.md[[]2[]] PASSED*", + ], + consecutive=True, + ) + result.stdout.fnmatch_lines( + ["SKIPPED [[]1[]] *: page.md:*: every example skipped"], + ) + + +def test_merged_keeps_a_failure_through_a_retry( + pytester: _pytest.pytester.Pytester, +) -> None: + """A retry re-runs a merged namespace whole, so a real failure stands. + + The retry rebuilds the namespace from its first block, which is what makes + the default layout safe to combine with a test-retry plugin. Under + ``per-block`` a retry re-runs only the block that failed, against the + mapping that block already changed. + """ + pytester.plugins = ["pytest_doctest_docutils"] + _write_ini(pytester) + (pytester.path / "page.rst").write_text( + textwrap.dedent( + """ + Title + ===== + + .. doctest:: demo + + >>> seen = [] + + .. doctest:: demo + + >>> seen.append(1) + >>> len(seen) + 2 + """, + ), + encoding="utf-8", + ) + + result = pytester.runpytest("page.rst", "--reruns", "2") + + result.assert_outcomes(failed=1) + result.stdout.fnmatch_lines(["*1 failed*2 rerun*"]) + + +def test_per_block_refuses_a_repeated_block( + pytester: _pytest.pytester.Pytester, +) -> None: + """A block run twice is refused rather than trusted. + + A retry re-runs one block against the globals it already changed, so an + expectation that comes true on the second attempt would report as a pass. + The namespace cannot be rebuilt for one block alone, so the repeat fails + with a message naming the way out. + """ + pytester.plugins = ["pytest_doctest_docutils"] + _write_ini(pytester, "doctest_docutils_namespace_items = per-block") + (pytester.path / "page.rst").write_text( + textwrap.dedent( + """ + Title + ===== + + .. doctest:: demo + + >>> seen = [] + + .. doctest:: demo + + >>> seen.append(1) + >>> len(seen) + 2 + """, + ), + encoding="utf-8", + ) + + result = pytester.runpytest("page.rst", "--reruns", "2") + + result.assert_outcomes(passed=1, failed=1) + result.stdout.fnmatch_lines( + ["*was run twice against a namespace laid out per block*"] + ) + + +TESTCODE_PAGE_MD = textwrap.dedent( + """ + # Page + + Visible, pasteable, no prompt: + + ```{testcode} + value = 41 + ``` + + Hidden assertion the reader never sees: + + ```{testcode} + :hide: + + assert value == 41 + ``` + + Visible with expected output: + + ```{testcode} + print(value + 1) + ``` + + ```{testoutput} + 42 + ``` + """, +) + + +TESTCODE_NAMESPACE_CASES = [ + ("block-merged", "block", "merged", 1), + ("document-merged", "document", "merged", 1), + ("block-per-block", "block", "per-block", 3), + ("document-per-block", "document", "per-block", 3), +] + + +@pytest.mark.parametrize( + ("test_id", "scope", "items", "passed"), + TESTCODE_NAMESPACE_CASES, + ids=[case[0] for case in TESTCODE_NAMESPACE_CASES], +) +def test_a_prompt_free_page_passes_at_every_namespace_setting( + pytester: _pytest.pytester.Pytester, + test_id: str, + scope: str, + items: str, + passed: int, +) -> None: + """The page a reader pastes out of runs under every namespace setting. + + The blocks share the ``default`` group whatever the scope, so the layout + decides only how many items they collect as. + """ + pytester.plugins = ["pytest_doctest_docutils"] + _write_ini( + pytester, + f"doctest_docutils_namespace_scope = {scope}", + f"doctest_docutils_namespace_items = {items}", + ) + (pytester.path / "page.md").write_text(TESTCODE_PAGE_MD, encoding="utf-8") + + result = pytester.runpytest("page.md") + + result.assert_outcomes(passed=passed) + + +def test_a_prompt_free_page_collects_as_its_group( + pytester: _pytest.pytester.Pytester, +) -> None: + """The node id a reader pastes back names the page, not a block index.""" + pytester.plugins = ["pytest_doctest_docutils"] + _write_ini(pytester) + (pytester.path / "page.md").write_text(TESTCODE_PAGE_MD, encoding="utf-8") + + items, _ = pytester.inline_genitems("page.md") + + assert [item.name for item in items] == ["page.md"] + + +def test_a_failing_testoutput_reports_against_its_page( + pytester: _pytest.pytester.Pytester, +) -> None: + """A mismatch reports as an ordinary doctest failure on the page's line.""" + pytester.plugins = ["pytest_doctest_docutils"] + _write_ini(pytester) + (pytester.path / "page.md").write_text( + "```{testcode}\nprint(41 + 1)\n```\n\n```{testoutput}\n99\n```\n", + encoding="utf-8", + ) + + result = pytester.runpytest("page.md") + + result.assert_outcomes(failed=1) + result.stdout.fnmatch_lines(["*Expected:*", "*99*", "*Got:*", "*42*"]) + + +def test_the_canonical_sphinx_page_passes( + pytester: _pytest.pytester.Pytester, +) -> None: + """A page copied out of the Sphinx docs collects and passes as one item. + + Nothing on it carries a prompt: :mod:`sphinx.ext.doctest` runs a + ``{testsetup}`` body through ``exec`` and rejects ``>>>`` outright, so the + setup, the code and the hidden assertion are all plain Python. + """ + pytester.plugins = ["pytest_doctest_docutils"] + _write_ini(pytester) + (pytester.path / "page.md").write_text( + textwrap.dedent( + """ + # Page + + ```{testsetup} + base = 40 + ``` + + ```{testcode} + print(base + 2) + ``` + + ```{testoutput} + 42 + ``` + + ```{testcode} + :hide: + + assert base == 40 + ``` + """, + ), + encoding="utf-8", + ) + + result = pytester.runpytest("page.md") + + result.assert_outcomes(passed=1) + + +def test_a_failing_testcode_reports_the_whole_block( + pytester: _pytest.pytester.Pytester, +) -> None: + """The report quotes every line of the block and lands inside it.""" + pytester.plugins = ["pytest_doctest_docutils"] + _write_ini(pytester) + (pytester.path / "page.md").write_text( + "# Page\n\n```{testcode}\na = 1\nb = 2\nraise ValueError('boom')\n```\n", + encoding="utf-8", + ) + + result = pytester.runpytest("page.md") + + result.assert_outcomes(failed=1) + result.stdout.fnmatch_lines( + ["004 a = 1", "005 b = 2", "006 raise ValueError('boom')"], + ) + result.stdout.fnmatch_lines(["*page.md:6: UnexpectedException*"]) diff --git a/uv.lock b/uv.lock index a5aa06a..0cbdb3a 100644 --- a/uv.lock +++ b/uv.lock @@ -382,6 +382,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/8a/0e/97c33bf5009bdbac74fd2beace167cab3f978feb69cc36f1ef79360d6c4e/exceptiongroup-1.3.1-py3-none-any.whl", hash = "sha256:a7a39a3bd276781e98394987d3a5701d0c4edffb633bb7a5144577f82c773598", size = 16740, upload-time = "2025-11-21T23:01:53.443Z" }, ] +[[package]] +name = "execnet" +version = "2.1.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/bf/89/780e11f9588d9e7128a3f87788354c7946a9cbb1401ad38a48c4db9a4f07/execnet-2.1.2.tar.gz", hash = "sha256:63d83bfdd9a23e35b9c6a3261412324f964c2ec8dcd8d3c6916ee9373e0befcd", size = 166622, upload-time = "2025-11-12T09:56:37.75Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ab/84/02fc1827e8cdded4aa65baef11296a9bbe595c474f0d6d758af082d849fd/execnet-2.1.2-py3-none-any.whl", hash = "sha256:67fba928dd5a544b783f6056f449e5e3931a5c378b128bc18501f7ea79e296ec", size = 40708, upload-time = "2025-11-12T09:56:36.333Z" }, +] + [[package]] name = "gp-furo-theme" version = "0.1.0a37" @@ -426,6 +435,7 @@ dev = [ { name = "pytest-mock" }, { name = "pytest-rerunfailures" }, { name = "pytest-watcher" }, + { name = "pytest-xdist" }, { name = "ruff" }, { name = "sphinx-autobuild", version = "2024.10.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, { name = "sphinx-autobuild", version = "2025.8.25", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11'" }, @@ -452,6 +462,7 @@ testing = [ { name = "pytest-mock" }, { name = "pytest-rerunfailures" }, { name = "pytest-watcher" }, + { name = "pytest-xdist" }, ] [package.metadata] @@ -477,6 +488,7 @@ dev = [ { name = "pytest-mock" }, { name = "pytest-rerunfailures" }, { name = "pytest-watcher" }, + { name = "pytest-xdist" }, { name = "ruff", specifier = ">=0.16.0" }, { name = "sphinx-autobuild" }, { name = "sphinx-autodoc-api-style", specifier = "==0.1.0a37" }, @@ -501,6 +513,7 @@ testing = [ { name = "pytest-mock" }, { name = "pytest-rerunfailures" }, { name = "pytest-watcher" }, + { name = "pytest-xdist" }, ] [[package]] @@ -1033,6 +1046,19 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/fc/3f/172d73600ad2771774cda108efb813fc724fc345e5240a81a1085f1ade5d/pytest_watcher-0.6.3-py3-none-any.whl", hash = "sha256:83e7748c933087e8276edb6078663e6afa9926434b4fd8b85cf6b32b1d5bec89", size = 12431, upload-time = "2026-01-10T23:28:17.64Z" }, ] +[[package]] +name = "pytest-xdist" +version = "3.8.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "execnet" }, + { name = "pytest" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/78/b4/439b179d1ff526791eb921115fca8e44e596a13efeda518b9d845a619450/pytest_xdist-3.8.0.tar.gz", hash = "sha256:7e578125ec9bc6050861aa93f2d59f1d8d085595d6551c2c90b6f4fad8d3a9f1", size = 88069, upload-time = "2025-07-01T13:30:59.346Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ca/31/d4e37e9e550c2b92a9cbc2e4d0b7420a27224968580b5a447f420847c975/pytest_xdist-3.8.0-py3-none-any.whl", hash = "sha256:202ca578cfeb7370784a8c33d6d05bc6e13b4f25b5053c30a152269fd10f0b88", size = 46396, upload-time = "2025-07-01T13:30:56.632Z" }, +] + [[package]] name = "pyyaml" version = "6.0.3"