This adds opt-in continuation tracking to `terminal.Stream` that allows
any caller to call `writeContinuation` in order to get the minimum bytes
necessary from a grounded parser state to the identical state.
This enables reliable stream restart across serialization states, which
could be used for local restart, networked terminals, etc. For me, this
is used for multiplexers. :)
## Implementation
The implementation of this was really carefully done to avoid any
negative performance impact particularly when continuation tracking is
_off_.
The way this work is simple:
1. ESC is the only char that leaves the ground state and most
ESC sequences are short. So if we're in a non-ground state, we
do a backwards vectorized search to find the last `ESC` in the
input slice. If one doesn't exist, we assume we found it previously
and store the whole slice (rare, since ESC sequences are usually
short like I said).
2. If we're in the ground state that means we only have a potential
incomplete UTF-8 codepoint, so we find the lead UTF-8 byte.
3. When writing, we normalize the suffix to drop things like BEL
commands that would've already been handled to avoid
double-calling.
## Performance
Via `ghostty-bench +terminal-stream`
Corpus main tracking off tracking on
plain ASCII (256 MiB) 175.4ms 175.8ms 175.5ms
UTF-8 (32 MiB) 268.4ms 270.0ms 269.9ms
5% invalid UTF-8 (32 MiB) 316.4ms 317.8ms 320.2ms
CSI-heavy (32 MiB) 145.0ms 146.3ms 145.9ms
OSC (32 MiB) 1621.0ms 1627.5ms 1638.3ms
Kitty APC (128 MiB) 95.5ms 96.9ms 96.3ms
mixed traffic (32 MiB) 172.4ms 172.0ms 172.3ms
giant APC (128 MiB) 38.3ms 38.3ms 40.8ms
The masked-compare scan that finds bulk-copyable cell runs still
processed one cell per iteration and remained the largest single
cost in a column reflow.
Scan whole groups of cells at a time using the group variants of the
Mask helper: a group that fully matches the run pattern (and, for
text runs, contains no Kitty virtual placeholder, via eqlAny)
extends the run by the whole group, and any mismatch falls through
to the scalar loop which finds the exact end of the run within it.
The group length comes from the shared simd.lanes helper where the
target has SIMD support and falls back to a plain unrolled group
elsewhere.
1.19x faster on ghostty-bench +terminal-resize --mode=cols (120x80
terminal, 10k-line scrollback, shrink/grow column reflow cycles).
Combined with the preceding reflow optimizations, resize with reflow
is 5.8x faster than before the series.
Finding the length of a bulk-copyable cell run evaluated the
field-wise bulkCopyable predicate plus a style compare per cell,
which compiles to a chain of extracts and branches and had become
the hottest loop in a column reflow.
Once the first cell passes the full predicate, a cell continues the
run iff it matches the first cell in content tag, style id, wide
property, and hyperlink flag, so the continuation test is now a
masked compare of the raw cell bits via the Mask helper, plus a
masked equality test against the Kitty virtual placeholder codepoint
for text runs (placeholders must set a row flag so they take the
slow path). This is slightly stricter than the predicate (a bg-color
cell no longer extends an unstyled text run), which only splits a
copy into multiple runs and remains correct.
1.21x faster on ghostty-bench +terminal-resize --mode=cols (120x80
terminal, 10k-line scrollback, shrink/grow column reflow cycles).
Two small extensions to the Mask helper, both motivated by the
reflow bulk run scan in the next commits.
fieldMask now accepts dot-separated field paths so a mask can cover
a nested field of a packed struct or packed union member, e.g.
"content.codepoint.data" covers exactly the codepoint bits of a cell
without its padding. Packed union members all share bit offset zero.
Mask gains eqlAny, the "any" counterpart to eql: it returns whether
any value in a group has masked fields equal to the expected
pattern. This supports run scans that must stop when a sentinel
value appears anywhere in a group, such as the Kitty virtual
placeholder codepoint which requires slow-path handling.
In `resizeCols`, stash the most recently finished source node
instead of destroying it, so we can recycle it without a bunch of
syscalls.
1.30x faster on ghostty-bench +terminal-resize --mode=cols (120x80
terminal, 10k-line scrollback, shrink/grow column reflow cycles),
with system time dropping from 34ms to 8ms per run.
reflowRow computed the capacity for prospective destination pages on
every source row via Capacity.adjust, which performs a full page
layout calculation to find the available grid space.
The result only depends on the source page, and reflow visits source pages
sequentially and never revisits one, so memoize the adjustment per
source page so we only do this once.
1.06x faster on ghostty-bench +terminal-resize --mode=cols (120x80
terminal, 10k-line scrollback, shrink/grow column reflow cycles).
Reflow copied every cell through a per-cell state machine
(writeCell) that dispatches on content tag, wide property, grapheme,
hyperlink, and style handling, and advances the destination cursor
one cell at a time. The vast majority of cells in practice are
narrow text or bg-color cells with no managed memory that share a
single style across long runs.
reflowRow now scans ahead for the run of such cells bounded by the
remaining space in the destination row, copies the run with a single
memcpy, and adjusts the style ref count once for the whole run via
useMultiple. Wide characters, spacers, graphemes, hyperlinks, Kitty
placeholders, and rows containing tracked pins all take the original
per-cell path, and a style set failure falls back to writeCell which
handles growing page capacity.
2.19x faster on ghostty-bench +terminal-resize --mode=cols (120x80
terminal, 10k-line scrollback, shrink/grow column reflow cycles).
Memoize the most recent style mapping and when there is a reuse
bump the ref with `use()`. This avoids a lookup (`addWithId`) on
every single styled cell.
1.25x faster on ghostty-bench +terminal-resize --mode=cols (120x80
terminal, 10k-line scrollback, shrink/grow column reflow cycles).
Reflow scanned the full tracked pin list for every source cell it
copied, twice per cell in the wide-character case, even though pins
are rare and at most a handful exist. Each check also went through
node.page(), which can restore a compressed page just to compare
pointers.
reflowRow now determines once per row whether any tracked pin is on
the source row and skips the per-cell pin scans entirely when there
is none, which is the overwhelmingly common case. The comparisons
use node identity instead of pages: a node owns exactly one page, so
they are equivalent, and this avoids the restore hazard.
1.09x faster on ghostty-bench +terminal-resize --mode=cols (120x80
terminal, 10k-line scrollback, shrink/grow column reflow cycles).
Expose complete encode and decode entry points directly from terminal.snapshot instead of requiring terminal.snapshot.snapshot. Reorder the decode APIs to accept the allocator and I/O context before the reader.
Publish each screen's logical history extent before READY so clients can size scrollbars while older pages are still arriving. Keep the value advisory and continue deriving native PageList totals from decoded pages.
Reduce HISTORY to its structural screen key and page count, and update the format documentation, Kaitai schema, verifier, and versioned fixtures.
Treat FINISH as the self-delimiting snapshot boundary instead of peeking for end-of-file. Normal decoding now leaves continuation bytes unread, allowing snapshots and live protocol data to share a stream without waiting for closure.
Add decodeExact for bounded files that still require strict exhaustion, and update the Kaitai schema, documentation, and tests for continuation and sequential snapshot decoding.
Stream complete snapshot records to any std.Io.Writer while retaining one reusable payload buffer for length and CRC calculation. Update BLAKE3 incrementally so checkpoints no longer require rehashing an allocating destination.
Wrap decode hashing in StreamReader to enforce exact checkpoint boundaries. Preserve v1 bytes while allowing snapshots to begin at the current writer position and retaining only valid prefixes on failures.
Route HISTORY sequences by their encoded screen key so both keyed sequence groups can arrive in either order. Keep undeclared and duplicate routing strict, separate HISTORY manifest parsing from page restoration, and clear decoder-only generation state before returning the terminal.
Normalize unknown terminal-wide semantic fields during restore while keeping dimensions and screen count structural. Preserve canonical encoding, ignore reserved mode and tab-stop bits, reset invalid color and scrolling state, and clamp finite scrollback policies to the native range.
Treat HISTORY row counts as canonical metadata rather than a reason to reject otherwise usable history. Restore topology from the declared PAGE sequence while keeping record framing, routing keys, and sequence boundaries strict.
Keep the standalone style and hyperlink codecs strict while allowing PAGE and SCREEN decoders to discard invalid optional data at boundaries they own. Normalize invalid styles to defaults, ignore unrepresentable hyperlinks, reuse duplicate entries, and validate hyperlink values before encoding.
Keep SCREEN encoding strict while allowing decoding to recover from unknown or noncanonical semantic state. Cursor positions now clamp to the restored active area, and invalid enum values, reserved bits, and optional state degrade to native defaults.
Keep snapshot grid encoding strict by rejecting malformed wide-cell relationships before they can produce invalid wire data.
Decode untrusted grids liberally while preserving record alignment. Unknown semantic values and content kinds degrade to safe defaults, optional graphemes and hyperlinks are dropped when invalid or over capacity, and malformed wide-cell markers normalize to narrow cells.
Treat Kitty virtual placeholder codepoints as ordinary valid grid content during snapshot restore and derive the native row lookup hint from decoded cells. Image and placement registries remain intentionally omitted.
Cover the behavior with a complete snapshot round trip containing a real virtual placement and its grapheme diacritics.
Describe the complete version 1 snapshot format with a Kaitai schema and make every golden fixture self-describing for automatic discovery. Add a verifier that compiles the schema, parses all fixtures, and checks record checksums, checkpoint digests, and cross-record invariants.
Preserve Kaitai metadata when generating fixture candidates and provide the compiler and Python runtime dependencies through the development shell. Keep the mode registry portable to Kaitai JavaScript targets so the complete fixture also works in the web IDE.
Extracted out the raw `src/terminal` changes needed for the future
snapshot work, 4 separate changes. These are uncontroversial and
relatively simple, summarized below. Tests AI assisted but the rest
including commit messages, this PR message, etc. all organic.
* **Add iterator to ref counted set.** Iterate over live entries and
their IDs. Const, doesn't mutate the set.
* **lib.Enum produces stable enums for Zig.** Basically the same as C
except it uses the smallest fitting integer including the holes.
* **PageList: a couple helpers for manually creating pages.** There is
`PageList.Builder` for creating a new pagelist and
`PageList.allocatePage` for modifying an existing one. This allows
PageList construction from raw pages.
Rename Builder.addPage and PageAllocation.cancel to their consistent allocatePage and deinit forms. Track successful ownership transfers so both builder APIs can use unconditional deferred cleanup without releasing pages transferred to a PageList.
Fixes#13522
Fixes unreachable when reflow dupes a hyperlink into a destination page
whose string allocator is nearly full.
The capacity precondition in ReflowCursor.writeCell performed a single
test allocation of `uri.len + id.len` bytes before duping a hyperlink
into the destination page. But PageEntry.dupe allocates the URI and
the explicit ID as two separate allocations, and the string allocator
rounds every allocation up to its 32-byte chunk size independently, so
the two separate allocations can require one more chunk than the
single combined test allocation.
Write a new helper to make sure we get the right amount of space
using the same allocation pattern of dupe.
Applications cannot infer whether an unfocused terminal remains visible, so
focus reports are insufficient for avoiding expensive rendering while a
view is hidden.
Implement private mode 2033 and the visibility query/report sequences.
Track conservative per-surface visibility, report every effective change
while enabled, and always answer explicit queries and mode enables. Keep
view visibility across terminal resets because it is owned by the host,
not terminal state.
Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019fa965-aa5f-7099-85b4-a9679d2c8bd3
