Reserved-row toolbar: Stage 1 primitives, dependency contracts, and terminal qualification - #1751
Conversation
Groundwork for keeping the bottom toolbar off the scroll path. Not yet wired into toolbar rendering; that is a separate change. A DECSTBM scroll region stops ordinary output from scrolling through the bottom rows, but is not sufficient on its own: ED (ESC [ J), which prompt_toolkit's renderer uses to erase, ignores the scroll margins and clears to the bottom of the display regardless. DL (ESC [ M) is bounded by the margins, and deleting every line from the cursor to the bottom margin produces the same all-blank result, so it substitutes for ED while respecting the reserved rows. DL also needs no knowledge of the cursor's row, which matters because Output does not track one, and it degrades to ED's behaviour when no region is set. The region is anchored at row 1 because a region starting lower orphans the rows above it, which then never scroll and never reach the terminal's scrollback. Verified against real tmux 3.7c: the reserved row survives 120 lines of scrolling and a bounded erase, rows above the cursor are preserved, scrolled lines reach scrollback from the first line, and the reserved row never leaks into history.
The reserved-row toolbar work relies on details of prompt_toolkit that are not public API: the cursor-position-report arithmetic, both destructive erase calls going through the Output interface, and the renderer's diff baseline. A silent change to any of these would break terminal rendering in ways that are hard to attribute, so lock them behaviorally here instead. The Windows cases can only run on Windows, where CI is the only place they get exercised: Windows10_Output is a registered virtual subclass rather than a real one, so capability checks must not rely on inheritance; geometry is delegated to the native backend, so adapting get_size() alone is insufficient; its inner VT output carries a zero-size stub; and legacy Win32Output.erase_down is a separate implementation that a VT sequence replacement never reaches. Verified by mutating prompt_toolkit locally: breaking the CPR formula, removing either erase call, or leaving the diff baseline set each fails these tests.
Temporary diagnostic for qualifying the reserved-row bottom toolbar on Windows. It records what the qualification needs and a checkout cannot otherwise supply: the input and output backend classes actually selected, viewport geometry against the backing console buffer, which object serves each delegated method, the rows-below-cursor value the renderer uses for available height, and the console mode before and after a scroll-region probe. It refuses to run unless stdout and stdin are both terminals. Redirecting or piping selects PlainTextOutput, which would record the wrong backend and silently invalidate the whole inventory. The probe always restores full-screen margins and attributes, so it leaves the terminal as it found it. Intended to be removed once Windows qualification is complete.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## reserved_row_toolbar #1751 +/- ##
========================================================
+ Coverage 99.67% 99.69% +0.01%
========================================================
Files 25 26 +1
Lines 6465 6516 +51
========================================================
+ Hits 6444 6496 +52
+ Misses 21 20 -1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. |
Two defects found in review, neither covered by the existing tests. Entering the context saved only whether the output already had an instance-level erase_down, not what it was. Exiting then left the bounded erase installed and lost the caller's implementation, so lines kept being deleted after the reservation ended. Save the previous attribute and put it back. scroll_region_sequence() accepted a reservation leaving one usable row and emitted ESC[1;1r. DECSTBM requires the bottom margin to exceed the top, so terminals ignore that sequence and keep their previous margins: the caller believes it holds a reservation while output still scrolls through the reserved rows and destroys them. Measured on tmux 3.7c, where the marker was overwritten and 22 rows of output spilled through the region. The failure is total rather than degraded, so callers must release the reservation below this floor instead of narrowing it. Reject the case with a message naming the floor. Both fixes are mutation-checked: restoring the reported behavior, or lowering the floor, fails the new tests.
Vt100_Output.write_raw() only appends to the output's own buffer, so neither the region nor its reset reached the terminal on its own. Verified before fixing: after leaving the context the reset was still sitting in the buffer as ['\x1b[r'], on the normal and the exception path alike. That left the margins restricted whenever the body exited without another renderer operation, so later shell output kept scrolling inside the old region. Entry had the same defect from the other direction: the reservation was only queued, so anything written before the next flush could still scroll through the reserved rows. Flush after writing each sequence, so the reservation is in force once __enter__ returns and restoration has actually happened once __exit__ does. Four tests were passing only because they flushed after the context exited, or asserted outside it; they now assert what a caller would actually observe. Removing either flush fails them.
…rase Two defects found in review. Changing the scroll margins moves the cursor. DECSTBM homes it, and so does the reset: measured on tmux 3.7c, a cursor at row 10 column 7 landed at row 1 column 1 after each. Entering the reservation below existing output therefore sent later rendering to the top of the screen, and leaving it did the same to subsequent shell output, overwriting what was already there. Wrap both margin changes in a save/restore pair. This helper cannot place the cursor when it starts inside the reserved band, because discovering the row needs a cursor-position report and it has no input to read one from. That precondition is now documented; placing the prompt within the usable area belongs to the layer that owns the terminal. Renderer.clear(), which Ctrl-L reaches, calls erase_screen() as well as erase_down(). Only the latter was bounded, so an unbounded ED2 still erased the reserved rows. Bound erase_screen too: home inside the region and delete every usable line, which matches ED2's erase-and-home semantics without touching the reserved rows. Verified against a real terminal rather than only through emitted sequences: with the cursor at row 9 column 5, it is unchanged after entry and after exit, sits at the region's home after a clear, and the reserved row survives Ctrl-L with the region fully cleared. All fifteen mutations of this module fail their tests.
Three claims in this module were wrong or overstated. The opening paragraphs said DL leaves "the same all-blank result" as ED and was safe to leave installed. It is not: ED preserves the part of the cursor's line before the cursor, while DL deletes the whole line. Measured on tmux 3.7c, running DL from a nonzero column destroyed committed text to the left of the cursor. The substitution is sound only from column zero, which the three renderer paths reaching erase_down all normalize to first. That is a precondition to enforce, not a property to assume. The bounded full-screen erase was described as reproducing ED2 semantics. ED2 does not move the cursor -- measured, a cursor at row 12 column 33 was unchanged across it -- and Renderer.clear() homes separately afterwards. The bounded form homes first because DL clears downward from the cursor, so covering the usable area means starting at its top. That is a chosen contract, now described as one. No behavior changes.
_call_in_ui() polls the pending future with a 0.1s timeout and, on expiry, re-raises when the future is already done. That branch exists because concurrent.futures.TimeoutError is TimeoutError on Python 3.11+, so a callback raising a timeout of its own cannot be told apart by type from the poll expiring. Re-raising the caught exception conflates the two. When the callback completes in the window between the poll expiring and the future being inspected, the caller is told the call timed out even though it succeeded. Ask the future for its outcome instead: a callback that raised a timeout still propagates it, and one that produced a value now returns it. Found while investigating an intermittent failure of test_command_toolbar_ui_call_propagates_failures, which reproduced once in 60 runs before this change and not once in 120 after. The added regression test drives the interleaving deterministically rather than relying on timing.
|
@kmvanbrunt @bambu Can one of your please follow the instructions towards the bottom of the description under the The purpose of this branch and PR is to do some proof-of-concept testing for a new architectural approach where the bottom row (or 2 or whatever) is truly reserved just for the status bar and is completely separate from the normal app. The concept appears viable for both Linux and Mac, but needs some manual testing validation on Windows. The basic thing to run is: uv run python scripts/windows_toolbar_inventory.pyThen there is a checklist of things to look for. I sent you both an email with a status document explaining a little more. |
# Conflicts: # CHANGELOG.md
|
Windows Terminal Run. Here is what I was in the terminal to start: PS C:\Users\toddleonhardt\src\python\cmd2> uv run .\scripts\windows_toolbar_inventory.py
=== Environment inventory ===
captured_at 2026-09-09T16:00:06.582220+00:00
platform Windows-11-10.0.26200-SP0
os_release 11
python 3.14.7 (tags/v3.14.7:823f032, Aug 5 2026, 11:30:42) [MSC v.1944 64 bit (ARM64)]
python_executable C:\Users\toddleonhardt\src\python\cmd2\.venv\Scripts\python.exe
prompt_toolkit 3.0.53
TERM None
WT_SESSION d875e8dd-78e2-4fb5-8636-7a6f3fafe6ef
MSYSTEM None
TERM_PROGRAM None
output_class prompt_toolkit.output.windows10.Windows10_Output
input_class prompt_toolkit.input.win32.Win32Input
stdout_isatty True
stdin_isatty True
viewport_rows 49
viewport_columns 90
win32_buffer_size {'X': 91, 'Y': 49}
win32_window {'Left': 0, 'Top': 0, 'Right': 90, 'Bottom': 48}
win32_cursor {'X': 0, 'Y': 1}
viewport_differs_from_buffer False
rows_below_cursor_position 48
delegate::erase_down prompt_toolkit.output.vt100.Vt100_Output
delegate::erase_screen prompt_toolkit.output.vt100.Vt100_Output
delegate::get_size prompt_toolkit.output.win32.Win32Output
delegate::get_rows_below_cursor_position prompt_toolkit.output.win32.Win32Output
delegate::flush prompt_toolkit.output.windows10.Windows10_Output
console_mode_before 0x0007 (VIRTUAL_TERMINAL_PROCESSING=on, WRAP_AT_EOL=on)
=== DECSTBM probe (visual confirmation required) ===
About to reserve the bottom row, scroll past it, then reset.
WATCH THE BOTTOM ROW. Press Enter when ready.After I pressed enter I saw some stuff and TOOLBARMARKER was at the bottom. But at no point did I see a bottom bar at all and saw (with a lot of scrollback): probe line 0001ell
probe line 0002icrosoft Corporation. All rights reserved.
probe line 0003
probe line 0004dleonhardt> cd .\src\python\cmd2\
probe line 0005dleonhardt\src\python\cmd2> uv run .\scripts\windows_toolbar_inventory.py
probe line 0006
probe line 0007 inventory ===
probe line 0008 2026-09-09T16:05:07.412343+00:00
probe line 0009 Windows-11-10.0.26200-SP0
probe line 0010 11
probe line 0011 3.14.7 (tags/v3.14.7:823f032, Aug 5 2026, 11:30:4
probe line 001264 bit (ARM64)]
probe line 0013able C:\Users\toddleonhardt\src\python\cmd2\.venv\Scrip
probe line 0014
probe line 0015t 3.0.53
probe line 0016 None
probe line 0017 7ebb8bf0-daaa-4ac2-8891-0c284bfe590c
probe line 0018 None
probe line 0019 None
probe line 0020 prompt_toolkit.output.windows10.Windows10_Output
probe line 0021 prompt_toolkit.input.win32.Win32Input
probe line 0022 True
probe line 0023 True
probe line 0024 49
probe line 0025mns 90
probe line 0026size {'X': 91, 'Y': 49}
probe line 0027 {'Left': 0, 'Top': 0, 'Right': 90, 'Bottom': 48}
probe line 0028 {'X': 0, 'Y': 5}
probe line 0029ers_from_buffer False
probe line 0030rsor_position 44
probe line 0031se_down prompt_toolkit.output.vt100.Vt100_Output
probe line 0032se_screen prompt_toolkit.output.vt100.Vt100_Output
probe line 0033_size prompt_toolkit.output.win32.Win32Output
probe line 0034_rows_below_cursor_position prompt_toolkit.output.win32.Win32Output
probe line 0035sh prompt_toolkit.output.windows10.Windows10_Output
probe line 0036before 0x0007 (VIRTUAL_TERMINAL_PROCESSING=on, WRAP_AT_EO
probe line 0037
probe line 0038
probe line 0039be (visual confirmation required) ===
probe line 0040rve the bottom row, scroll past it, then reset.
probe line 0041TOM ROW. Press Enter when ready.
probe line 0042
probe line 0043
probe line 0044
probe line 0045
probe line 0046
probe line 0047
probe line 0048
probe line 0049
probe line 0050
probe line 0051
probe line 0052
probe line 0053
probe line 0054
probe line 0055
probe line 0056
probe line 0057
probe line 0058
probe line 0059
probe line 0060
probe line 0061
probe line 0062
probe line 0063
probe line 0064
probe line 0065
probe line 0066
probe line 0067
probe line 0068
probe line 0069
probe line 0070
probe line 0071
probe line 0072
probe line 0073
probe line 0074
probe line 0075
probe line 0076
probe line 0077
probe line 0078
probe line 0079
probe line 0080
probe line 0081
probe line 0082
probe line 0083
probe line 0084
probe line 0085
probe line 0086
probe line 0087
probe line 0088
probe line 0089
probe line 0090
probe line 0091
probe line 0092
probe line 0093
probe line 0094
probe line 0095
probe line 0096
probe line 0097
probe line 0098
probe line 0099
probe line 0100
=== Report ===1
console mode before : 0x0007 (VIRTUAL_TERMINAL_PROCESSING=on, WRAP_AT_EOL=on)
console mode after : 0x0007 (VIRTUAL_TERMINAL_PROCESSING=on, WRAP_AT_EOL=on)
modes match : True
probe line 0105
Answer in the result template:
1. Did TOOLBARMARKER stay on the bottom row for the whole scroll?
2. Is the scrollback complete (probe line 0001 upward) and in order?
3. Does TOOLBARMARKER appear anywhere in the scrollback? (it must not)
4. After this program exits, does the shell prompt behave normally?
probe line 0111
machine-readable record written to: C:\Users\toddleonhardt\src\python\cmd2\scripts\windows_toolbar_inventory_TODDLEONHARCC3D.json
Attach that file to the result template.
PS C:\Users\toddleonhardt\src\python\cmd2>
probe line 0116
probe line 0117
probe line 0118
probe line 0119
probe line 0120
probe line 0121
probe line 0122
probe line 0123
probe line 0124
probe line 0125
probe line 0126
probe line 0127
probe line 0128
probe line 0129
probe line 0130
probe line 0131
probe line 0132
probe line 0133
probe line 0134
probe line 0135
probe line 0136
probe line 0137
probe line 0138
probe line 0139
probe line 0140
probe line 0141
probe line 0142
probe line 0143
probe line 0144
probe line 0145
probe line 0146
TOOLBARMARKERAt no point did I ever see an actual bottom bar. Scrollback looked funky. The application had exited and dropped me back to the shell but in the middle of the screen. The shell prompt behaved normally afterwards and the cursor was visible after exit. I don't understand how this was supposed to work, but I can't imagine this is what "right" looks like. Here is that JSON output: |
|
Git Bash shell. Experience essentially identical to Windows terminal. Here is the JSON output: |
|
Conhost beheavior nearly identical to others except I was able to scroll where TOOLBARMARKER did not remain on the bottom. For both Windows Terminal and Git Bash, TOOLBARMARKER always remained on the bottom. JSON output attached |
The probe's inventory, DECSTBM reserved-row check and bounded-erase check are terminal properties, not Windows ones, and Stage 1's POSIX results were all collected through tmux -- which owns its own scrollback and scroll regions, so they qualify tmux rather than any host terminal. Renaming it to scripts/toolbar_inventory.py and dropping the Windows framing lets the same script close the bare-terminal gap on POSIX. Two probe properties matter for the answers to mean anything, and both are now guaranteed rather than incidental: every line is flushed individually with a --delay pause so the scroll is actually watched instead of arriving as one atomic repaint, and every probe line erases to end of line so surviving tails of prior content cannot read as scrollback corruption.
|
The recent commit fixed a couple bugs in the test script, generalized it to work on all platforms, and added a short pause inbetween each line of output to make it easier to observe the scrolling behavior. |
|
Updated Windows Terminal Run. Everything looked fine. |
|
Re-run of Git Bash terminal run. Everything looked fine. |
Every terminal tested -- iTerm2, Windows Terminal, Git Bash/mintty -- shows the toolbar leaving the bottom row when the user scrolls up. This is expected and worth saying plainly, because the alternative reads as a bug. An application cannot paint into the terminal's scrollback view: it is not notified of the scroll, cannot query it, and has no sequence addressing it. A toolbar that stayed visible over history would have to have been written into the scrollback buffer, leaving copies of itself wedged between the output lines being read. The programs that do pin a status line for a whole session buy it with the alternate screen, where there is no scrollback to scroll.
Stage 1 qualification work for the reserved-row bottom toolbar. Ready — Windows qualification is complete and Stage 1 is closed. This existed to run the
Windows CI matrix and give the tester a known commit to check out; both are done. It
changes no runtime behaviour: nothing here is wired into toolbar rendering.
Targets the long-lived
reserved_row_toolbarintegration branch.Background
Pressing Enter makes the bottom toolbar blink, rated objectionable by a user on three
real-terminal runs. The cause is now understood: prompt-toolkit only rewrites rows that
differ from
Renderer._last_screen, andrenderer.erase()discards that baseline, forcinga full repaint. Three paths trigger it per command. The repaint that restores the toolbar
is the same event that erases it, so removing the erases cannot work — proven by positive
control.
The design (specified separately) keeps the toolbar on a DECSTBM-reserved bottom row that
ordinary output never scrolls through, with the renderer's destructive operations bounded to
the region above it.
What's here
cmd2/scroll_region.py+ tests — DECSTBM sequences and a margin-bounded replacement forerase_down.EDignores the scroll margins and would destroy the reserved row;DLisbounded by them and needs no knowledge of the cursor's row, which matters because
Outputdoes not track one. 15 tests, 100% line coverage, five mutations each killing their test.
tests/test_prompt_toolkit_contracts.py— locks the non-public prompt-toolkit detailsthe design depends on, so a dependency upgrade fails loudly here instead of silently
breaking terminal rendering: the CPR available-height arithmetic, both destructive erase
calls going through the
Outputinterface, and the renderer's diff baseline.Four of these run only on Windows, and CI is the only place they can run:
Windows10_Outputis a registered virtual subclass rather than a real one, so capabilitychecks must not rely on inheritance; geometry is delegated natively, so adapting
get_size()alone is insufficient; its inner VT output carries a zero-size stub; and legacyWin32Output.erase_downis a separate implementation a VT replacement never reaches.scripts/toolbar_inventory.py— temporary diagnostic for the Windows tester,included so the tool and the code under test come from the same commit. It refuses to run
unless stdout and stdin are both terminals, because redirection selects
PlainTextOutputand would record the wrong backend. Its probe always restores margins and attributes. To be
removed once Windows qualification is done.
Evidence so far
POSIX primitives were qualified through real tmux 3.7c at 12, 24 and 40 rows — 15 probes
including controls, all agreeing with expectation. Highlights: a region anchored at row 1
preserves scrollback and protects the reserved row; anchoring at row 2 orphans the first
line; a one-usable-row region is not honoured at all;
DLat a nonzero column destroys theline prefix, so column-zero normalization must be enforced rather than assumed.
Qualification results — Stage 1 CLOSED
Every environment that gated Stage 1 has now been run and passes. Console modes were restored
exactly on both Windows runs.
Vt100_Output/Vt100Input1;65rWindows10_Output/Win32Input1;20rMSYSTEM=CLANGARM64)Windows10_Output/Win32Input1;24rTOOLBARMARKERstayed on the bottom row for the whole scrollTOOLBARMARKER0x0007, VT processing on, both Windows runs)The bare iTerm2 run also closes a gap the POSIX evidence could not: Stage 1's primitives were
collected through tmux, and tmux owns its own scrollback and scroll regions, so those passes
qualified tmux rather than any host terminal.
Split delegation, now observed rather than assumed
Both Windows runs show one object with three different owners:
Geometry comes from the native Win32 side while rendering goes out over VT — the arrangement the
contract tests exist to pin.
rows_below_cursor_positionis implemented there (1 and 18) whereVt100 raises
NotImplementedError.Superseded first round
The earlier Windows attempts in this thread failed: no bar was ever visible and the scrollback
looked interleaved. That was the probe's own defect — it buffered the whole run and flushed once,
painting it as a single repaint, so nothing scrolled and the bar was never drawn where it could be
seen. Fixed by per-line flush plus a
--delaypause. Those first artifacts qualify nothing.Known limitation: the toolbar is not pinned during scrollback
Seen on every terminal, POSIX and Windows: scrolling up moves the bar out of view rather than
holding it at the bottom. This is expected and not fixable within the design, and it is now
documented in
docs/features/prompt.md.An application cannot paint into the terminal's scrollback view — it is not notified of the scroll,
cannot query it, and has no escape sequence addressing it. A toolbar that stayed visible over
history would have to have been written into the scrollback buffer, which is exactly the hard
failure the "free of
TOOLBARMARKER" check exists to catch. The property that makes this designcorrect and the property being asked for are the same property in opposite directions.
lessandvimpin a status line for a whole session only by using the alternate screen, where there is noscrollback at all.
What this still does not establish
viewport_differs_from_bufferwas
Falseon both Windows runs, so that hazard is untested rather than ruled out.fallback covers it.
integrated implementation and belong to later stages.
scripts/toolbar_inventory.pywas to be removed once Windows qualification finished. It is keptfor now because the remaining stages still need real-terminal runs; it should be removed before
this feature reaches
main.