Single-row erasure and trailing blank-row trimming can remap or remove
stored row coordinates while retaining the same node. External pointer-
plus-generation references could therefore survive these less common
PageList mutation paths.
Renew each affected node inside the traversal that already changes it,
and signal compression activity only when row erasure begins in
history. This adds constant work per touched page without another list
scan or normal append cost.
Compaction and capacity growth publish fresh page generations, so an
active incremental traversal can detect them. They did not update the
separate activity token, however, and a completed compressor could
remain idle after either operation restored or replaced a cold page.
Mark successful replacements as compression activity without resetting
the exact continuation marker. The next scheduled step can retain valid
progress or restart through the existing generation checks.
ScreenSearch reloaded active state before pruning stale flattened
history. Reload could compare a shifted tracked selection against
cached page coordinates and panic before the later validation step ran.
Prune history immediately after dimension reconciliation in
reloadActive, before any cached coordinate is inspected or converted to
a pin. Selection now relies on that ordering and avoids a redundant
second prune.
RenderState applied flattened highlights after releasing the terminal
lock and matched them to copied rows by node address alone. A recycled
address could therefore make a stale asynchronous highlight decorate
unrelated content.
Capture the live node generation with each render row and require both
the pointer and copied generation to match. Validation remains lock-
free and never dereferences a potentially stale node.
ViewportSearch compared only cached node addresses when deciding
whether to reuse its owned search window. An in-place layout change or
pool address reuse could therefore make stale text and coordinates
appear current.
Snapshot each node generation with its pointer and compare only those
captured values. This detects a renewed generation at the same address
without ever dereferencing cached node pointers.
Reverse multi-page highlight construction reordered node and row-
bound columns but left captured page generations in their original
order. Every cross-page result therefore paired each node with another
page generation and could be rejected as stale despite remaining live.
Reverse the serial column with the other flattened chunk metadata and
cover the node-plus-generation pairing in the existing boundary match
test.
PageList.split moved the source suffix into a fresh target but left
the shortened source on its old generation. Cached matches in that
suffix could therefore pass validation against the live source pointer
and reach pin tracking with invalid coordinates.
Renew the source generation only after target cloning succeeds, and
mark compression activity so restored history is reconsidered. The
conservative floor keeps bounded pruning safe even when the renewed
source and fresh target precede older successors.
Partial history erasure shifted retained rows while preserving the
node generation. Cached flattened matches then passed pointer-plus-
generation validation and could be tracked with coordinates beyond the
shortened page.
Renew the generation before reinitializing or shifting a page layout,
and mark compression activity so an incremental pass restarts and
revisits a restored cold page. The conservative serial floor permits
the fresh generation to remain before older live successors without
rejecting them.
Page generations are not ordered with the page list because splits
and in-place replacements insert fresh generations before older live
pages. Advancing page_serial_min while pruning could therefore reject
live successors and fail PageList integrity checks.
Keep the floor as a whole-list invalidation epoch and use the existing
pointer-plus-generation membership check for ordinary removals. Add
bounded pruning coverage for split and replacement ordering, and verify
reset still rejects a stale generation when its node address is reused.
All found by GPT 5.6. I'm still going through manual review of each one
now and will remove or rewrite the ones that are pointless or
unreachable (if any, I prompted it to ignore those too).
Count-limited PageIterator traversal took the minimum of the page and requested lengths, then tested whether that minimum exceeded the request. The condition was impossible, so iteration never crossed a page. Reverse traversal also excluded the current row and subtracted one from row zero, causing a runtime safety panic.
Count the current row in both directions, consume the returned length, and move to an adjacent page only when the request has rows remaining. Returned chunks now preserve their half-open bounds at row zero and across page boundaries.
Search selection could run after the terminal removed a screen but before the next refresh reconciled the cached searchers. Reloading that stale ScreenSearch dereferenced freed PageList nodes, while normal cleanup also tried to untrack pins from the destroyed list.
Reconcile under the terminal lock before selecting, track ScreenSet generations so allocator address reuse cannot hide replacements, and release stale search buffers without touching pins already freed with the screen. Cleanup now takes the same lock when live pins must be untracked.
History pruning only compared cached serials with page_serial_min. Replacing a historical node through compaction left its old serial above that cutoff, so selecting the cached match attempted to track a destroyed node and hit the PageList validity assertion.
Validate every flattened chunk by finding the live node and matching its serial without dereferencing stale pointers. Remove only the invalid result and adjust any later selection index so unrelated older results remain available.
PageListSearch.feed changed only the node of its tracked progress pin. If the preceding history page had fewer rows or columns after a split, the retained bottom-right coordinates fell outside that page and the next PageList integrity check panicked.
Reset both coordinates to the new node's actual bottom-right cell whenever feed advances. The progress pin now remains valid across heterogeneous page sizes.
Partial history erasure can remove a page without marking tracked pins as garbage because they move coherently to the next page. ScreenSearch therefore retained a selection whose cached history result was subsequently removed by pruneHistory. Selecting again indexed an empty history result list and panicked.
Clear the tracked selection when its combined result index falls within the history suffix being pruned. Retained active and newer history selections keep their existing indices.
startHyperlink accepts borrowed URI and ID slices. When those slices came from the current cursor hyperlink, startHyperlinkOnce ended and freed that hyperlink before duplicating the replacement, then dereferenced the released URI and segfaulted.
Duplicate the new hyperlink before ending the prior one. Aliased inputs remain valid through the copy, and allocation failure leaves the existing cursor hyperlink intact.
setTitle can receive the slice returned by getTitle. Clearing the list retained its allocation, so appending that same slice used memcpy with aliased source and destination ranges and panicked in runtime-safe builds.
Resize the list within its reserved capacity and copy the value forward before writing the sentinel. This supports the complete current value and its subslices without weakening allocation-failure atomicity.
setPwd can receive the slice returned by getPwd. Clearing the list retained its allocation, so appending that same slice used memcpy with aliased source and destination ranges and panicked in runtime-safe builds.
Resize the list within its reserved capacity and copy the value forward before writing the sentinel. This supports the complete current value and its subslices without weakening allocation-failure atomicity.
Screen.select accepted an already tracked selection by value. Passing the screen's current selection back into the setter caused the old selection cleanup to free the same pin pair that the replacement retained, so the next selection operation dereferenced stale pool entries and panicked.
When replacing tracked state, release only old pins that are not also owned by the replacement. Exact and partial aliases now retain their shared pins while ordinary replacements still reclaim both old entries.
Terminal.resize deinitialized the current tab stops before allocating their replacement. If a resize beyond the inline tab-stop capacity ran out of memory, the terminal retained an undefined tab-stop value and normal deinitialization dereferenced a poisoned pointer.
Allocate and initialize the replacement first, then release the old tab stops only after allocation succeeds. A failed resize now retains the original tab stops and remains safe to destroy.
setTitle appended the title bytes and terminating NUL with separate fallible operations. If only the terminator allocation failed, it returned OutOfMemory but left a nonempty unterminated buffer that made getTitle panic on its sentinel check.
Reserve checked capacity before clearing the existing title, then append both the title and terminator without further allocation. Allocation failure now leaves the prior valid title intact.
setPwd appended the path bytes and terminating NUL with separate fallible operations. If only the terminator allocation failed, the function returned OutOfMemory but left a nonempty unterminated buffer that made getPwd panic on its sentinel check.
Reserve checked capacity for the complete sentinel-terminated value before clearing the old state, then use infallible appends. Allocation failure now leaves a valid prior value instead of exposing partial data.
Implicit OSC 8 hyperlinks incremented a u32 identifier with checked arithmetic even though the cursor contract allows the sequence to wrap. Reaching the maximum identifier therefore caused a runtime safety panic before the link could be installed.
Use wrapping arithmetic for both the successful increment and the error rollback. The identifier now returns to zero at the boundary, while failed allocation attempts still restore the original value.
With mode 2027 disabled, the printer attaches zero-width codepoints without applying Unicode grapheme boundaries. Enabling the mode later and printing another non-ASCII codepoint asserted that every stored pair was part of one grapheme and panicked when it was not.
Feed all stored codepoints through the grapheme state machine and let it reset at existing boundaries before testing the new codepoint. The deterministic print comparison can now exercise live mode changes without clearing the screen first.
Screen.clearRows sliced backing cells using the desired PageList width.
During incomplete reflow, clearing a narrower stored page extended the
slice past its row and tripped clearCells runtime validation before any
cells were cleared.
Obtain cells from the owning page and use its stored width for whole-row
managed-memory bookkeeping. Mixed-width history rows now clear safely.
Rectangle orientation swaps endpoint columns when a selection is mirrored.
During incomplete reflow, copying a column from a wider page onto the
narrower corner page created an invalid pin that panicked when resolved.
Clamp every swapped column to the page that owns the oriented corner.
Top-left and bottom-right calculations now always return valid pins.
Screen.clone clipped selections with the desired screen width or copied
rectangle columns unchanged. PageList.clone preserves stored page widths,
so a clipped boundary on a narrower page produced an invalid tracked pin
and panicked during runtime validation.
Build fallback pins from the first or last cloned node and clamp rectangle
columns to that node. Clipped selections now remain valid during reflow.
Selection drags converted caller-provided floating-point positions directly
to u32 and multiplied geometry dimensions without checking their range.
Non-finite positions, oversized values, overflowing dimensions, or empty
core geometry could therefore panic in runtime safety builds.
Clamp pixel positions to the representable u32 range, reject empty
geometry, and saturate the pixel span when dimensions overflow. Valid
drags keep their existing threshold behavior.
cursorCellEndOfPrev moved its pin to the previous row but then set the
column from the desired screen width. If incomplete reflow left that
previous page narrower, resolving the cell used an out-of-bounds column
and panicked in runtime safety builds.
Set the column from the page reached by the cursor pin so the returned
cell is always the actual final cell of that row.
containedRowCached built full-row and rectangular selections using
the desired screen width. During incomplete reflow, an intermediate
narrower page received endpoints beyond its cell range, and consumers
could panic when resolving the returned pins.
Use the owning page width for full rows and clamp rectangular bounds to
that width. Every contained-row selection now returns resolvable pins.
Semantic line selection moved to the previous row when the next row
started with different content, then assigned the previous pin an x
coordinate from the next page. During incomplete reflow, a wider next
page produced an out-of-bounds pin and a runtime safety panic.
Set the end column from the page that owns the destination pin. Line
selection now remains valid while crossing mixed-width page boundaries.
Tabstops.reset subtracted one from the column count before iterating
default stops. Although init and resize accept zero columns, resetting
that state with a nonzero interval underflowed and panicked.
Return after clearing when the grid has fewer than two columns. Empty
and single-column tabstop sets now preserve the normal no-stop result.
SelectionGesture passed caller-supplied repeat timestamps directly to
Instant.since. A C API client or non-monotonic timer could provide an
earlier timestamp after a later one, causing a runtime safety panic
while converting negative elapsed seconds to u64.
Compare instants before calculating elapsed time and treat backwards
timestamps as failed repeats. The next press becomes a new single-click
anchor, matching other invalid repeat inputs.
Screen.clearCells accepted a slice but its runtime safety validation
indexed the first and last elements unconditionally. Passing an empty
range therefore panicked before the otherwise valid no-op clear.
Return immediately for an empty slice so validation and managed-memory
bookkeeping only run when there are cells to clear.
Pin movement assumed every page had the same column count. During an
incomplete reflow, crossing into a narrower page could produce an
out-of-bounds x coordinate, while wrapped movement could land on the
wrong row or stop early.
Use destination page widths while moving vertically or wrapping, and
reject points that exceed the resolved page. Add synthetic mixed-width
coverage for movement, wrapping, overflow, and point conversion.
pointFromPin accumulated scrollback rows directly into the u32 Y
field. An unbounded PageList with more than 2^32 rows could overflow
while converting a valid pin and panic in runtime safety builds.
Use checked additions for every cross-page row contribution. If the
pin cannot fit in point.Coordinate, return null through the existing
out-of-range result instead of trapping.
Cell.screenPoint accumulated page row counts in CellCountInt even
though screen point Y coordinates are u32. Once scrollback crossed
65,535 rows, walking back through page metadata overflowed and trapped
in runtime safety builds.
Accumulate directly in u32 so page-local u16 row counts widen before
addition and the result uses the full range promised by point.Coordinate.
Prompt scrolling negated negative deltas to count the requested jumps.
minInt(isize) has no positive signed representation, so a caller could
trigger a runtime safety panic before the search for an earlier prompt
started.
Use @abs to produce the full unsigned magnitude. An extreme negative
request now follows the normal prompt traversal and clamps at the oldest
available prompt.
PageList.scroll negated negative row deltas to obtain their
magnitude. minInt(isize) has no positive signed representation, so
callers could trigger a runtime safety panic before the existing
traversal had a chance to clamp at the top.
Use @abs to calculate an unsigned magnitude that represents every isize
value. The same value now drives both cached-pin and general traversal
paths.
Runtime safety violating scenarios found by GPT 5.6. Verified each one
manually. See each commit.
I'm going to keep searching so not going to merge this yet.
setCursorPos added origin-mode margins to requested row and column
values before clamping them to the scrolling region. A request near
maxInt(usize) overflowed during that addition and crashed instead of
landing on the region boundary.
Use saturating addition for the origin offsets. The existing clamp then
places oversized requests on the bottom-right margin without changing
normal cursor positioning.
Pin.leftWrap and rightWrap calculated the destination using the
remainder after consuming the current row. When that remainder was an
exact multiple of the column count, rightWrap subtracted one from zero
and leftWrap produced a column equal to the width. Dereferencing either
pin could panic. A maximum usize offset on a one-column page also
overflowed the row count.
Base the row and column calculations on the remainder minus one. This
maps exact multiples to the final cell of the correct row and keeps the
maximum offset calculation in range so traversal reports overflow
normally.
Low-level search state accepted an empty needle even though the search
thread normally filters it out. SlidingWindow treated the empty string
as a zero-length match and underflowed while calculating its inclusive
end offset. Active and viewport overlap calculations could also
underflow while loading adjacent pages.
Treat an empty needle as an inactive search with no matches or history,
and saturate the viewport overlap length.
Screen searches only reset cached dimensions while feeding more history.
Selecting or reloading a result immediately after a resize left
flattened highlights pointing at page nodes freed by reflow. The next
selection operation could dereference those stale pointers and crash.
Centralize dimension invalidation and run it before feed, reload, and
selection paths inspect cached state. Add regression coverage for
selecting a cached active match after a column resize.
APC payloads such as Kitty graphics images can be megabytes of base64
data, but every byte was dispatched individually: through the VT state
machine table, an apc_put action, the stream handler, the APC protocol
handler, and finally a per-byte ArrayList append in the Kitty command
parser. Five layers of dispatch per byte made large image transfers far
slower than they needed to be.
Add a bulk fast path alongside the existing CSI fast paths in
consumeUntilGround: scan the longest run of apc_put bytes (stopping at
any byte the parse table doesn't treat as APC payload: CAN, SUB, ESC,
and most C1 bytes exit or abort the string state, and 0xA0-0xFF are
ignored by it) and dispatch the run as a single new apc_put_slice
action. The APC handler identifies the protocol from the first few bytes
as before, then passes the remainder of each slice to the protocol
parser in bulk; the Kitty parser appends payload data with a single
appendSlice. Ignored/unknown APC sequences now drop each slice in O(1)
instead of per-byte dispatch.
The fast path is guarded the same way as the CSI fast paths: handlers
with a vtRaw hook (the inspector) keep receiving per-byte apc_put
actions, and the scalar next() path is unchanged.
Also add benchmark support: a `ghostty-gen +kitty` synthetic generator
emitting well-formed Kitty graphics transmit commands with 4 KiB random
base64 payloads (not valid image data; the corpus exercises the parsing
paths, not image decoding), and a `ghostty-bench +apc-parser` benchmark
that measures the stream -> APC -> Kitty parse path without image
decode/storage.
Benchmarks on a 64 MiB corpus (hyperfine, ReleaseFast, x86_64 Linux,
baseline is identical source with only the fast path disabled):
apc-parser: 1.061 s -> 43 ms (~25x)
terminal-stream (kitty): 1.163 s -> 72 ms (~16x)
terminal-stream (ascii): no change
The ascii case was verified with retired instruction counts (perf stat,
pinned to one core) since wall time on the test machine has 4-7 ms of
noise: 988,030,458 vs 988,045,833 instructions (+0.0016%), a fixed
startup-size delta; the ground-state hot loop never reaches the new
branch.
AI Disclosure: This code was written with the assistance of Fable 5 via
Amp.
Screen searches only reset cached dimensions while feeding more
history. Selecting or reloading a result immediately after a resize
left flattened highlights pointing at page nodes freed by reflow. The
next selection operation could dereference those stale pointers and
crash.
Centralize dimension invalidation and run it before feed, reload, and
selection paths inspect cached state. Add regression coverage for
selecting a cached active match after a column resize.
APC payloads such as Kitty graphics images can be megabytes of base64
data, but every byte was dispatched individually: through the VT state
machine table, an apc_put action, the stream handler, the APC protocol
handler, and finally a per-byte ArrayList append in the Kitty command
parser. Five layers of dispatch per byte made large image transfers
far slower than they needed to be.
Add a bulk fast path alongside the existing CSI fast paths in
consumeUntilGround: scan the longest run of apc_put bytes (stopping
at any byte the parse table doesn't treat as APC payload: CAN, SUB,
ESC, and most C1 bytes exit or abort the string state, and 0xA0-0xFF
are ignored by it) and dispatch the run as a single new apc_put_slice
action. The APC handler identifies the protocol from the first few
bytes as before, then passes the remainder of each slice to the
protocol parser in bulk; the Kitty parser appends payload data with a
single appendSlice. Ignored/unknown APC sequences now drop each slice
in O(1) instead of per-byte dispatch.
The fast path is guarded the same way as the CSI fast paths: handlers
with a vtRaw hook (the inspector) keep receiving per-byte apc_put
actions, and the scalar next() path is unchanged.
Also add benchmark support: a `ghostty-gen +kitty` synthetic generator
emitting well-formed Kitty graphics transmit commands with 4 KiB
random base64 payloads (not valid image data; the corpus exercises
the parsing paths, not image decoding), and a `ghostty-bench
+apc-parser` benchmark that measures the stream -> APC -> Kitty parse
path without image decode/storage.
Benchmarks on a 64 MiB corpus (hyperfine, ReleaseFast, x86_64 Linux,
baseline is identical source with only the fast path disabled):
apc-parser: 1.061 s -> 43 ms (~25x)
terminal-stream (kitty): 1.163 s -> 72 ms (~16x)
terminal-stream (ascii): no change
The ascii case was verified with retired instruction counts (perf
stat, pinned to one core) since wall time on the test machine has
4-7 ms of noise: 988,030,458 vs 988,045,833 instructions (+0.0016%),
a fixed startup-size delta; the ground-state hot loop never reaches
the new branch.
This adds transparent compression for non-active/non-viewport scrollback
pages, reducing physical memory for compressed pages by anywhere from
70% to 90%. Compression is obviously highly dependent on the shape of
the data, but these are the numbers I got for normal scrollback.
Due to compression being available, I bumped the default scrollback
limit from 10MB to 50MB. On average, a full scrollback still uses less
memory than the prior limit due to the compression ratios.
## Demo
Here is a demo video showing me filling my scrollback and using the
inspector so you can see the live compression/decompression activity and
results:
https://github.com/user-attachments/assets/7b9d0383-42f7-47bf-8b3f-853e3f89549c
## Resident vs. Virtual Memory
This PR works by lowering _resident/physicalmemory, but doesn't touch
_virtual_ memory.
Practically what this means is that users need to make sure they're
looking at resident memory to see the change.
We use OS primitives like `MADV_DONTNEED` on Linux or
`MADV_FREE_REUSABLE` on Darwin to discard our physical memory, but
retain our virtual memory allocations. This is awesome because it means
our decompression is infallible: the OS has already given us the memory,
but it just remaps it at that point.
This is baked into the core implementation, so compression only works on
systems that support an OS primitive to retain virtual mappings while
discarding physical. Today, that is macOS and 64-bit Linux. Other
operating systems have support we just haven't coded it up yet.
## Implementation
A major refactor had to happen to PageList to represent nodes as either
resident or compressed. Pins typically accessed node content directly so
we had to add a bunch of helpers to read metadata without decompression
(but some access requires it).
Compression is relatively slow and its important we don't impact IO
performance. So we support incremental compression passes and they only
run when the terminal is idle (250ms timer on the render thread that
resets on any activity). Benchmarks show zero regression in IO
throughput on this change.
In order to maintain the no-libc invariant for libghostty-vt, we use a
hand-written (an AI assisted optimization) LZ4 compression
implementation. The performance and compression ratio is _okay_. Its a
good first step for this. I think in the future I want to look at other
implementations we can bring in based on build configuration.
## Performance
Measured with a saved 7.3 MB corpus made by repeating `gh --help` output
into a 120x80 terminal with a 50 MB scrollback limit on my machine:
| compression measurement | result |
|-------------------------|--------|
| pages compressed | 121 |
| raw page backing | 49.56 MB |
| encoded storage | 3.03 MB (6.11% of raw) |
| estimated physical memory savings | 46.53 MB (93.89%) |
| full compression | 30.3 ms total, 12.2 ms over the 18.1 ms no-op
baseline (~101 µs/page) |
| incremental drain | 29.7 ms total, 11.6 ms over baseline (~96 µs/page)
|
| compress and restore | 33.5 ms total, 3.2 ms over full compression
(~26 µs/page to restore) |
The workload above is especially repetitive, so its 6.11% encoded ratio
is better than the 10% to 30% expected for text-heavy terminal history
in general. Steady-state throughput is unchanged within noise
(`terminal-stream` benchmarks and manual `cat` timings).
## libghostty-vt
The same caller-driven compression controls are exposed to Zig and C.
Note that compression _is not automatic_ for libghostty users. Callers
must determine their own definition of "idle" and when to compress and
call our incremental callback APIs to perform the compression.
Decompression is automatic and as-needed (and will trigger
recompression-required flags so callers are aware).
## LLM Notes
This work was done in concert with Codex. I reviewed and
rewrote/reshaped pretty much every change extensively, particularly
PageList/Terminal. This PR message is written by hand, commit messages
are LLM written but reviewed.
