Commit Graph

16563 Commits

Author SHA1 Message Date
Tim Culverhouse
8c523ed039 terminal: vectorize APC payload scanning
Kitty graphics payloads are dispatched in bulk, but finding each slice
boundary still examines every byte with a scalar loop. This leaves large
direct base64 image transmissions parser-bound.

Scan ordinary APC bytes using the vector width recommended for the compile
target. Keep the scalar scan both as the tail and as the full fallback when
the target has no recommended vector width. Test state-machine boundaries
against byte-at-a-time parsing.

A ReleaseFast APC parser benchmark over the same 64 MiB Kitty graphics
corpus, with 10 warmups and 30 measured runs, produced:

                  mean       median
  scalar        37.6 ms      32.6 ms
  vectorized    22.3 ms      19.0 ms

Hyperfine reports the vectorized version as 1.69 times faster overall, with
the median runtime improving by approximately 42 percent.
2026-07-10 09:29:15 -05:00
Mitchell Hashimoto
3f2b7946d7 Misc runtime safety fixes (#13278)
All found by GPT 5.6. I'm still going through manual review of each one
now and will remove or rewrite the ones that are pointless or
unreachable (if any, I prompted it to ignore those too).
2026-07-10 06:58:02 -07:00
Mitchell Hashimoto
9b466e9c55 terminal: fix count-limited page iteration
Count-limited PageIterator traversal took the minimum of the page and requested lengths, then tested whether that minimum exceeded the request. The condition was impossible, so iteration never crossed a page. Reverse traversal also excluded the current row and subtracted one from row zero, causing a runtime safety panic.

Count the current row in both directions, consume the returned length, and move to an adjacent page only when the request has rows remaining. Returned chunks now preserve their half-open bounds at row zero and across page boundaries.
2026-07-10 06:54:20 -07:00
Mitchell Hashimoto
73ac36fa56 terminal/search: discard destroyed screen state
Search selection could run after the terminal removed a screen but before the next refresh reconciled the cached searchers. Reloading that stale ScreenSearch dereferenced freed PageList nodes, while normal cleanup also tried to untrack pins from the destroyed list.

Reconcile under the terminal lock before selecting, track ScreenSet generations so allocator address reuse cannot hide replacements, and release stale search buffers without touching pins already freed with the screen. Cleanup now takes the same lock when live pins must be untracked.
2026-07-10 06:54:20 -07:00
Mitchell Hashimoto
86e1f7b8b5 terminal/search: reject replaced result pages
History pruning only compared cached serials with page_serial_min. Replacing a historical node through compaction left its old serial above that cutoff, so selecting the cached match attempted to track a destroyed node and hit the PageList validity assertion.

Validate every flattened chunk by finding the live node and matching its serial without dereferencing stale pointers. Remove only the invalid result and adjust any later selection index so unrelated older results remain available.
2026-07-10 06:47:10 -07:00
Mitchell Hashimoto
5d8eb78b73 terminal/search: reset pins while feeding
PageListSearch.feed changed only the node of its tracked progress pin. If the preceding history page had fewer rows or columns after a split, the retained bottom-right coordinates fell outside that page and the next PageList integrity check panicked.

Reset both coordinates to the new node's actual bottom-right cell whenever feed advances. The progress pin now remains valid across heterogeneous page sizes.
2026-07-10 06:47:10 -07:00
Mitchell Hashimoto
d239f98056 terminal/search: drop pruned selections
Partial history erasure can remove a page without marking tracked pins as garbage because they move coherently to the next page. ScreenSearch therefore retained a selection whose cached history result was subsequently removed by pruneHistory. Selecting again indexed an empty history result list and panicked.

Clear the tracked selection when its combined result index falls within the history suffix being pruned. Retained active and newer history selections keep their existing indices.
2026-07-10 06:47:10 -07:00
Mitchell Hashimoto
cfce1cd56c terminal: duplicate hyperlinks before replacement
startHyperlink accepts borrowed URI and ID slices. When those slices came from the current cursor hyperlink, startHyperlinkOnce ended and freed that hyperlink before duplicating the replacement, then dereferenced the released URI and segfaulted.

Duplicate the new hyperlink before ending the prior one. Aliased inputs remain valid through the copy, and allocation failure leaves the existing cursor hyperlink intact.
2026-07-10 06:47:10 -07:00
Mitchell Hashimoto
dace6d1c6d terminal: handle aliased title updates
setTitle can receive the slice returned by getTitle. Clearing the list retained its allocation, so appending that same slice used memcpy with aliased source and destination ranges and panicked in runtime-safe builds.

Resize the list within its reserved capacity and copy the value forward before writing the sentinel. This supports the complete current value and its subslices without weakening allocation-failure atomicity.
2026-07-10 06:47:10 -07:00
Mitchell Hashimoto
a14a619ceb terminal: handle aliased pwd updates
setPwd can receive the slice returned by getPwd. Clearing the list retained its allocation, so appending that same slice used memcpy with aliased source and destination ranges and panicked in runtime-safe builds.

Resize the list within its reserved capacity and copy the value forward before writing the sentinel. This supports the complete current value and its subslices without weakening allocation-failure atomicity.
2026-07-10 06:47:09 -07:00
Mitchell Hashimoto
0c299000f8 terminal: preserve aliased selection pins
Screen.select accepted an already tracked selection by value. Passing the screen's current selection back into the setter caused the old selection cleanup to free the same pin pair that the replacement retained, so the next selection operation dereferenced stale pool entries and panicked.

When replacing tracked state, release only old pins that are not also owned by the replacement. Exact and partial aliases now retain their shared pins while ordinary replacements still reclaim both old entries.
2026-07-10 06:47:09 -07:00
Mitchell Hashimoto
91f0cf67d5 terminal: preserve tabstops on resize failure
Terminal.resize deinitialized the current tab stops before allocating their replacement. If a resize beyond the inline tab-stop capacity ran out of memory, the terminal retained an undefined tab-stop value and normal deinitialization dereferenced a poisoned pointer.

Allocate and initialize the replacement first, then release the old tab stops only after allocation succeeds. A failed resize now retains the original tab stops and remains safe to destroy.
2026-07-10 06:47:09 -07:00
Mitchell Hashimoto
fdd255c782 terminal: reserve title terminator atomically
setTitle appended the title bytes and terminating NUL with separate fallible operations. If only the terminator allocation failed, it returned OutOfMemory but left a nonempty unterminated buffer that made getTitle panic on its sentinel check.

Reserve checked capacity before clearing the existing title, then append both the title and terminator without further allocation. Allocation failure now leaves the prior valid title intact.
2026-07-10 06:47:09 -07:00
Mitchell Hashimoto
1d5b6f90e5 terminal: reserve pwd terminator atomically
setPwd appended the path bytes and terminating NUL with separate fallible operations. If only the terminator allocation failed, the function returned OutOfMemory but left a nonempty unterminated buffer that made getPwd panic on its sentinel check.

Reserve checked capacity for the complete sentinel-terminated value before clearing the old state, then use infallible appends. Allocation failure now leaves a valid prior value instead of exposing partial data.
2026-07-10 06:47:09 -07:00
Mitchell Hashimoto
04810273af terminal: wrap implicit hyperlink identifiers
Implicit OSC 8 hyperlinks incremented a u32 identifier with checked arithmetic even though the cursor contract allows the sequence to wrap. Reaching the maximum identifier therefore caused a runtime safety panic before the link could be installed.

Use wrapping arithmetic for both the successful increment and the error rollback. The identifier now returns to zero at the boundary, while failed allocation attempts still restore the original value.
2026-07-10 06:47:09 -07:00
Mitchell Hashimoto
b287f6d1ab terminal: handle stored grapheme boundaries
With mode 2027 disabled, the printer attaches zero-width codepoints without applying Unicode grapheme boundaries. Enabling the mode later and printing another non-ASCII codepoint asserted that every stored pair was part of one grapheme and panicked when it was not.

Feed all stored codepoints through the grapheme state machine and let it reset at existing boundaries before testing the new codepoint. The deterministic print comparison can now exercise live mode changes without clearing the screen first.
2026-07-10 06:47:09 -07:00
Mitchell Hashimoto
e727a36589 terminal: clear rows using stored page width
Screen.clearRows sliced backing cells using the desired PageList width.
During incomplete reflow, clearing a narrower stored page extended the
slice past its row and tripped clearCells runtime validation before any
cells were cleared.

Obtain cells from the owning page and use its stored width for whole-row
managed-memory bookkeeping. Mixed-width history rows now clear safely.
2026-07-10 06:44:21 -07:00
Mitchell Hashimoto
b6f34be44f terminal: clamp mirrored selection corners
Rectangle orientation swaps endpoint columns when a selection is mirrored.
During incomplete reflow, copying a column from a wider page onto the
narrower corner page created an invalid pin that panicked when resolved.

Clamp every swapped column to the page that owns the oriented corner.
Top-left and bottom-right calculations now always return valid pins.
2026-07-10 06:44:21 -07:00
Mitchell Hashimoto
6071606577 terminal: clamp cloned selections to page width
Screen.clone clipped selections with the desired screen width or copied
rectangle columns unchanged. PageList.clone preserves stored page widths,
so a clipped boundary on a narrower page produced an invalid tracked pin
and panicked during runtime validation.

Build fallback pins from the first or last cloned node and clamp rectangle
columns to that node. Clipped selections now remain valid during reflow.
2026-07-10 06:44:21 -07:00
Mitchell Hashimoto
08e376f239 terminal: bound selection drag geometry
Selection drags converted caller-provided floating-point positions directly
to u32 and multiplied geometry dimensions without checking their range.
Non-finite positions, oversized values, overflowing dimensions, or empty
core geometry could therefore panic in runtime safety builds.

Clamp pixel positions to the representable u32 range, reject empty
geometry, and saturate the pixel span when dimensions overflow. Valid
drags keep their existing threshold behavior.
2026-07-10 06:44:21 -07:00
Mitchell Hashimoto
a55850c981 terminal: use previous page width for cursor cells
cursorCellEndOfPrev moved its pin to the previous row but then set the
column from the desired screen width. If incomplete reflow left that
previous page narrower, resolving the cell used an out-of-bounds column
and panicked in runtime safety builds.

Set the column from the page reached by the cursor pin so the returned
cell is always the actual final cell of that row.
2026-07-09 20:58:14 -07:00
Mitchell Hashimoto
a9f5b7eba2 terminal: clamp selection rows to page width
containedRowCached built full-row and rectangular selections using
the desired screen width. During incomplete reflow, an intermediate
narrower page received endpoints beyond its cell range, and consumers
could panic when resolving the returned pins.

Use the owning page width for full rows and clamp rectangular bounds to
that width. Every contained-row selection now returns resolvable pins.
2026-07-09 20:56:44 -07:00
Mitchell Hashimoto
fa8cae88b2 terminal: use destination width for line selection
Semantic line selection moved to the previous row when the next row
started with different content, then assigned the previous pin an x
coordinate from the next page. During incomplete reflow, a wider next
page produced an out-of-bounds pin and a runtime safety panic.

Set the end column from the page that owns the destination pin. Line
selection now remains valid while crossing mixed-width page boundaries.
2026-07-09 20:54:09 -07:00
Mitchell Hashimoto
3d08161b50 terminal: handle empty tabstop ranges
Tabstops.reset subtracted one from the column count before iterating
default stops. Although init and resize accept zero columns, resetting
that state with a nonzero interval underflowed and panicked.

Return after clearing when the grid has fewer than two columns. Empty
and single-column tabstop sets now preserve the normal no-stop result.
2026-07-09 20:50:23 -07:00
Mitchell Hashimoto
8f1c2fe959 terminal: handle backwards selection timestamps
SelectionGesture passed caller-supplied repeat timestamps directly to
Instant.since. A C API client or non-monotonic timer could provide an
earlier timestamp after a later one, causing a runtime safety panic
while converting negative elapsed seconds to u64.

Compare instants before calculating elapsed time and treat backwards
timestamps as failed repeats. The next press becomes a new single-click
anchor, matching other invalid repeat inputs.
2026-07-09 20:48:58 -07:00
Mitchell Hashimoto
0cb004734e terminal: ignore empty cell clears
Screen.clearCells accepted a slice but its runtime safety validation
indexed the first and last elements unconditionally. Passing an empty
range therefore panicked before the otherwise valid no-op clear.

Return immediately for an empty slice so validation and managed-memory
bookkeeping only run when there are cells to clear.
2026-07-09 20:38:29 -07:00
Mitchell Hashimoto
d6e24d9856 terminal: make pin traversal width-aware
Pin movement assumed every page had the same column count. During an
incomplete reflow, crossing into a narrower page could produce an
out-of-bounds x coordinate, while wrapped movement could land on the
wrong row or stop early.

Use destination page widths while moving vertically or wrapping, and
reject points that exceed the resolved page. Add synthetic mixed-width
coverage for movement, wrapping, overflow, and point conversion.
2026-07-09 20:38:28 -07:00
Mitchell Hashimoto
30b42f42a3 terminal: reject unrepresentable pin coordinates
pointFromPin accumulated scrollback rows directly into the u32 Y
field. An unbounded PageList with more than 2^32 rows could overflow
while converting a valid pin and panic in runtime safety builds.

Use checked additions for every cross-page row contribution. If the
pin cannot fit in point.Coordinate, return null through the existing
out-of-range result instead of trapping.
2026-07-09 20:38:18 -07:00
Mitchell Hashimoto
e6e4a9fdc1 terminal: widen cell screen coordinates
Cell.screenPoint accumulated page row counts in CellCountInt even
though screen point Y coordinates are u32. Once scrollback crossed
65,535 rows, walking back through page metadata overflowed and trapped
in runtime safety builds.

Accumulate directly in u32 so page-local u16 row counts widen before
addition and the result uses the full range promised by point.Coordinate.
2026-07-09 20:23:06 -07:00
Mitchell Hashimoto
afbf5ba156 terminal: handle minimum prompt scroll delta
Prompt scrolling negated negative deltas to count the requested jumps.
minInt(isize) has no positive signed representation, so a caller could
trigger a runtime safety panic before the search for an earlier prompt
started.

Use @abs to produce the full unsigned magnitude. An extreme negative
request now follows the normal prompt traversal and clamps at the oldest
available prompt.
2026-07-09 20:20:46 -07:00
Mitchell Hashimoto
c753fe4a4f terminal: handle minimum row scroll delta
PageList.scroll negated negative row deltas to obtain their
magnitude. minInt(isize) has no positive signed representation, so
callers could trigger a runtime safety panic before the existing
traversal had a chance to clamp at the top.

Use @abs to calculate an unsigned magnitude that represents every isize
value. The same value now drives both cached-pin and general traversal
paths.
2026-07-09 20:19:37 -07:00
Mitchell Hashimoto
35e1a0160c build(deps): bump cachix/install-nix-action from 31.10.6 to 31.10.7 (#13271)
Bumps
[cachix/install-nix-action](https://github.com/cachix/install-nix-action)
from 31.10.6 to 31.10.7.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/cachix/install-nix-action/releases">cachix/install-nix-action's
releases</a>.</em></p>
<blockquote>
<h2>v31.10.7</h2>
<h2>What's Changed</h2>
<ul>
<li>nix: 2.34.7 -&gt; 2.34.8 by <a
href="https://github.com/github-actions"><code>@​github-actions</code></a>[bot]
in <a
href="https://redirect.github.com/cachix/install-nix-action/pull/278">cachix/install-nix-action#278</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/cachix/install-nix-action/compare/v31.10.6...v31.10.7">https://github.com/cachix/install-nix-action/compare/v31.10.6...v31.10.7</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="a49548c11d"><code>a49548c</code></a>
Merge pull request <a
href="https://redirect.github.com/cachix/install-nix-action/issues/278">#278</a>
from cachix/create-pull-request/patch</li>
<li><a
href="147e749b5f"><code>147e749</code></a>
nix: 2.34.7 -&gt; 2.34.8</li>
<li><a
href="23cf0fec1d"><code>23cf0fe</code></a>
Merge pull request <a
href="https://redirect.github.com/cachix/install-nix-action/issues/276">#276</a>
from cachix/dependabot/github_actions/actions/checkout-7</li>
<li><a
href="8bdfc70a3e"><code>8bdfc70</code></a>
chore(deps): bump actions/checkout from 6 to 7</li>
<li>See full diff in <a
href="8aa03977d8...a49548c11d">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=cachix/install-nix-action&package-manager=github_actions&previous-version=31.10.6&new-version=31.10.7)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>
2026-07-09 20:04:02 -07:00
Mitchell Hashimoto
6236d3859c Misc runtime safety fixes (#13275)
Runtime safety violating scenarios found by GPT 5.6. Verified each one
manually. See each commit.

I'm going to keep searching so not going to merge this yet.
2026-07-09 20:03:10 -07:00
Mitchell Hashimoto
0aaedf4360 terminal: saturate origin cursor offsets
setCursorPos added origin-mode margins to requested row and column
values before clamping them to the scrolling region. A request near
maxInt(usize) overflowed during that addition and crashed instead of
landing on the region boundary.

Use saturating addition for the origin offsets. The existing clamp then
places oversized requests on the bottom-right margin without changing
normal cursor positioning.
2026-07-09 19:56:06 -07:00
Mitchell Hashimoto
0ff4e41b22 terminal: fix pin wrapping at row boundaries
Pin.leftWrap and rightWrap calculated the destination using the
remainder after consuming the current row. When that remainder was an
exact multiple of the column count, rightWrap subtracted one from zero
and leftWrap produced a column equal to the width. Dereferencing either
pin could panic. A maximum usize offset on a one-column page also
overflowed the row count.

Base the row and column calculations on the remainder minus one. This
maps exact multiples to the final cell of the correct row and keeps the
maximum offset calculation in range so traversal reports overflow
normally.
2026-07-09 19:56:06 -07:00
Mitchell Hashimoto
5bc6588e43 terminal/search: ignore empty search needles
Low-level search state accepted an empty needle even though the search
thread normally filters it out. SlidingWindow treated the empty string
as a zero-length match and underflowed while calculating its inclusive
end offset. Active and viewport overlap calculations could also
underflow while loading adjacent pages.

Treat an empty needle as an inactive search with no matches or history,
and saturate the viewport overlap length.
2026-07-09 19:55:59 -07:00
Mitchell Hashimoto
a23d90c89a terminal/search: reset cached results after resize (#13274)
Screen searches only reset cached dimensions while feeding more history.
Selecting or reloading a result immediately after a resize left
flattened highlights pointing at page nodes freed by reflow. The next
selection operation could dereference those stale pointers and crash.

Centralize dimension invalidation and run it before feed, reload, and
selection paths inspect cached state. Add regression coverage for
selecting a cached active match after a column resize.
2026-07-09 19:46:37 -07:00
Mitchell Hashimoto
7f073c4cf2 terminal: dispatch APC string bytes in bulk slices (#13270)
APC payloads such as Kitty graphics images can be megabytes of base64
data, but every byte was dispatched individually: through the VT state
machine table, an apc_put action, the stream handler, the APC protocol
handler, and finally a per-byte ArrayList append in the Kitty command
parser. Five layers of dispatch per byte made large image transfers far
slower than they needed to be.

Add a bulk fast path alongside the existing CSI fast paths in
consumeUntilGround: scan the longest run of apc_put bytes (stopping at
any byte the parse table doesn't treat as APC payload: CAN, SUB, ESC,
and most C1 bytes exit or abort the string state, and 0xA0-0xFF are
ignored by it) and dispatch the run as a single new apc_put_slice
action. The APC handler identifies the protocol from the first few bytes
as before, then passes the remainder of each slice to the protocol
parser in bulk; the Kitty parser appends payload data with a single
appendSlice. Ignored/unknown APC sequences now drop each slice in O(1)
instead of per-byte dispatch.

The fast path is guarded the same way as the CSI fast paths: handlers
with a vtRaw hook (the inspector) keep receiving per-byte apc_put
actions, and the scalar next() path is unchanged.

Also add benchmark support: a `ghostty-gen +kitty` synthetic generator
emitting well-formed Kitty graphics transmit commands with 4 KiB random
base64 payloads (not valid image data; the corpus exercises the parsing
paths, not image decoding), and a `ghostty-bench +apc-parser` benchmark
that measures the stream -> APC -> Kitty parse path without image
decode/storage.

Benchmarks on a 64 MiB corpus (hyperfine, ReleaseFast, x86_64 Linux,
baseline is identical source with only the fast path disabled):

  apc-parser:               1.061 s -> 43 ms  (~25x)
  terminal-stream (kitty):  1.163 s -> 72 ms  (~16x)
  terminal-stream (ascii):  no change

The ascii case was verified with retired instruction counts (perf stat,
pinned to one core) since wall time on the test machine has 4-7 ms of
noise: 988,030,458 vs 988,045,833 instructions (+0.0016%), a fixed
startup-size delta; the ground-state hot loop never reaches the new
branch.

AI Disclosure: This code was written with the assistance of Fable 5 via
Amp.
2026-07-09 19:38:39 -07:00
Mitchell Hashimoto
6275184473 terminal/search: reset cached results after resize
Screen searches only reset cached dimensions while feeding more
history. Selecting or reloading a result immediately after a resize
left flattened highlights pointing at page nodes freed by reflow. The
next selection operation could dereference those stale pointers and
crash.

Centralize dimension invalidation and run it before feed, reload, and
selection paths inspect cached state. Add regression coverage for
selecting a cached active match after a column resize.
2026-07-09 19:38:29 -07:00
dependabot[bot]
035ae8ddb6 build(deps): bump cachix/install-nix-action from 31.10.6 to 31.10.7
Bumps [cachix/install-nix-action](https://github.com/cachix/install-nix-action) from 31.10.6 to 31.10.7.
- [Release notes](https://github.com/cachix/install-nix-action/releases)
- [Changelog](https://github.com/cachix/install-nix-action/blob/master/RELEASE.md)
- [Commits](8aa03977d8...a49548c11d)

---
updated-dependencies:
- dependency-name: cachix/install-nix-action
  dependency-version: 31.10.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-07-10 00:12:41 +00:00
Tim Culverhouse
f6f79acce6 terminal: dispatch APC string bytes in bulk slices
APC payloads such as Kitty graphics images can be megabytes of base64
data, but every byte was dispatched individually: through the VT state
machine table, an apc_put action, the stream handler, the APC protocol
handler, and finally a per-byte ArrayList append in the Kitty command
parser. Five layers of dispatch per byte made large image transfers
far slower than they needed to be.

Add a bulk fast path alongside the existing CSI fast paths in
consumeUntilGround: scan the longest run of apc_put bytes (stopping
at any byte the parse table doesn't treat as APC payload: CAN, SUB,
ESC, and most C1 bytes exit or abort the string state, and 0xA0-0xFF
are ignored by it) and dispatch the run as a single new apc_put_slice
action. The APC handler identifies the protocol from the first few
bytes as before, then passes the remainder of each slice to the
protocol parser in bulk; the Kitty parser appends payload data with a
single appendSlice. Ignored/unknown APC sequences now drop each slice
in O(1) instead of per-byte dispatch.

The fast path is guarded the same way as the CSI fast paths: handlers
with a vtRaw hook (the inspector) keep receiving per-byte apc_put
actions, and the scalar next() path is unchanged.

Also add benchmark support: a `ghostty-gen +kitty` synthetic generator
emitting well-formed Kitty graphics transmit commands with 4 KiB
random base64 payloads (not valid image data; the corpus exercises
the parsing paths, not image decoding), and a `ghostty-bench
+apc-parser` benchmark that measures the stream -> APC -> Kitty parse
path without image decode/storage.

Benchmarks on a 64 MiB corpus (hyperfine, ReleaseFast, x86_64 Linux,
baseline is identical source with only the fast path disabled):

  apc-parser:               1.061 s -> 43 ms  (~25x)
  terminal-stream (kitty):  1.163 s -> 72 ms  (~16x)
  terminal-stream (ascii):  no change

The ascii case was verified with retired instruction counts (perf
stat, pinned to one core) since wall time on the test machine has
4-7 ms of noise: 988,030,458 vs 988,045,833 instructions (+0.0016%),
a fixed startup-size delta; the ground-state hot loop never reaches
the new branch.
2026-07-09 17:07:55 -05:00
Mitchell Hashimoto
7e02af8798 terminal: scrollback page compression (70 to 90% memory savings) (#13264)
This adds transparent compression for non-active/non-viewport scrollback
pages, reducing physical memory for compressed pages by anywhere from
70% to 90%. Compression is obviously highly dependent on the shape of
the data, but these are the numbers I got for normal scrollback.

Due to compression being available, I bumped the default scrollback
limit from 10MB to 50MB. On average, a full scrollback still uses less
memory than the prior limit due to the compression ratios.

## Demo

Here is a demo video showing me filling my scrollback and using the
inspector so you can see the live compression/decompression activity and
results:


https://github.com/user-attachments/assets/7b9d0383-42f7-47bf-8b3f-853e3f89549c

## Resident vs. Virtual Memory

This PR works by lowering _resident/physicalmemory, but doesn't touch
_virtual_ memory.

Practically what this means is that users need to make sure they're
looking at resident memory to see the change.

We use OS primitives like `MADV_DONTNEED` on Linux or
`MADV_FREE_REUSABLE` on Darwin to discard our physical memory, but
retain our virtual memory allocations. This is awesome because it means
our decompression is infallible: the OS has already given us the memory,
but it just remaps it at that point.

This is baked into the core implementation, so compression only works on
systems that support an OS primitive to retain virtual mappings while
discarding physical. Today, that is macOS and 64-bit Linux. Other
operating systems have support we just haven't coded it up yet.

## Implementation

A major refactor had to happen to PageList to represent nodes as either
resident or compressed. Pins typically accessed node content directly so
we had to add a bunch of helpers to read metadata without decompression
(but some access requires it).

Compression is relatively slow and its important we don't impact IO
performance. So we support incremental compression passes and they only
run when the terminal is idle (250ms timer on the render thread that
resets on any activity). Benchmarks show zero regression in IO
throughput on this change.

In order to maintain the no-libc invariant for libghostty-vt, we use a
hand-written (an AI assisted optimization) LZ4 compression
implementation. The performance and compression ratio is _okay_. Its a
good first step for this. I think in the future I want to look at other
implementations we can bring in based on build configuration.

## Performance

Measured with a saved 7.3 MB corpus made by repeating `gh --help` output
into a 120x80 terminal with a 50 MB scrollback limit on my machine:

| compression measurement | result |
|-------------------------|--------|
| pages compressed | 121 |
| raw page backing | 49.56 MB |
| encoded storage | 3.03 MB (6.11% of raw) |
| estimated physical memory savings | 46.53 MB (93.89%) |
| full compression | 30.3 ms total, 12.2 ms over the 18.1 ms no-op
baseline (~101 µs/page) |
| incremental drain | 29.7 ms total, 11.6 ms over baseline (~96 µs/page)
|
| compress and restore | 33.5 ms total, 3.2 ms over full compression
(~26 µs/page to restore) |

The workload above is especially repetitive, so its 6.11% encoded ratio
is better than the 10% to 30% expected for text-heavy terminal history
in general. Steady-state throughput is unchanged within noise
(`terminal-stream` benchmarks and manual `cat` timings).

## libghostty-vt

The same caller-driven compression controls are exposed to Zig and C. 

Note that compression _is not automatic_ for libghostty users. Callers
must determine their own definition of "idle" and when to compress and
call our incremental callback APIs to perform the compression.
Decompression is automatic and as-needed (and will trigger
recompression-required flags so callers are aware).

## LLM Notes

This work was done in concert with Codex. I reviewed and
rewrote/reshaped pretty much every change extensively, particularly
PageList/Terminal. This PR message is written by hand, commit messages
are LLM written but reviewed.
2026-07-09 13:19:21 -07:00
Mitchell Hashimoto
11b9a6ef17 renderer: hand off state mutex to avoid starving frames (#13265)
The mutex is unfair. From my understanding (after a brief convo with
@rockorager), it's a race between the parse thread and the renderer
thread.

The parse thread is unlocking it then locking it faster than the
renderer thread can get a lock/unlock. Under sustained pty output the
parse thread never sleeps between batches, so the renderer's frame
snapshot can starve for as long as the output lasts.

The fix here makes the parse thread voluntarily stay off the lock until
the renderer has had one turn by introducing two atomics:

- `demand`: a waiter count. The renderer increments it before locking
and decrements it after acquiring, so demand > 0 means "the renderer is
queued on the lock or about to be."
- `handoff_gen`: a generation counter. The renderer bumps it (with a
futex wake) after unlocking, meaning "a waiter completed one full turn."

At each batch boundary the parse thread checks demand (a single relaxed
load in the common case, so this **should** cost nothing when the
renderer isn't waiting). If a waiter exists, it futex-waits on
handoff_gen until the renderer has taken and released the lock, bounded
by a 1ms timeout, trying to ensure a lost wake can't stall IO.

This is inspired from `parking_lot` form rust land, which has eventual
fairness, but applied only at the one site that misbehaves.

## LLM Note

I used Fable 5 in Amp to write the code, but only after I identified the
problem myself from previous experience with mutex fairness (and the
convo above with rockorager).

The diagnosis — the parse thread barging the unfair mutex and starving
the renderer — came first.Fable then traced the exact loop in
`Exec.zig/generic.zig` and implemented the handoff protocol.

I've reviewed the changes.
2026-07-09 12:47:46 -07:00
Uzair Aftab
4f53b846bc renderer: move State declaration to top of file 2026-07-09 21:41:42 +02:00
Mitchell Hashimoto
9a4bd2120a terminal: optimize LZ4 decoding and add differential tests
The block decoder previously copied literals through variable-length
memcpy calls and expanded every match with word loops that carried an
overcopy fallback in each branch. Real page blocks decode as millions
of tiny sequences, so per-sequence overhead dominated restore time.

Decode short literal runs and in-token matches with blind fixed-width
copies whose margin checks subsume the exact bounds checks they
replace. Expand small repeating periods into pattern-word stores,
copy distant long matches with one exact memcpy, and propagate the
rare non-power-of-two short offsets bytewise. Page corpora restore
13% to 19% faster and text around twice as fast, while compressor
output stays byte-for-byte unchanged.

Replace the fuzz test with a differential property suite which
round-trips generated inputs, validates blocks with an independent
format walker, rejects wrong-size outputs, and decodes corrupted and
truncated blocks. A light version runs as a normal unit test; the
exhaustive version runs when GHOSTTY_LZ4_SLOW is set. An AGENTS.md
records the benchmarking, testing, and verification workflow for
this directory.
2026-07-09 12:38:59 -07:00
Uzair Aftab
d34b54e9b4 renderer: hand off state mutex to avoid starving frames
The renderer state mutex is unfair on all platforms (os_unfair_lock
on macOS, a futex based lock elsewhere). A thread that unlocks and
right away locks again wins over a sleeping waiter, because the
waiter first has to be woken up and scheduled. The termio parse
thread does exactly this under heavy pty output: it relocks the
mutex for every batch and never sleeps in between, so the renderer
can starve in updateFrame for as long as the output lasts.

Fix this by letting the parse thread stay off the lock until the
renderer had its turn. `renderer.State` gets two atomics: a waiter
count (`demand`) and a generation counter (`handoff_gen`). The renderer
takes the mutex through lockDemand/unlockDemand which update these,
and the parse thread calls yieldToDemand between batches. If a
waiter exists it sleeps on a futex until the renderer took and
released the lock, with a 1ms timeout so a lost wake can not stall
IO forever.

All the atomics are monotonic on purpose: they are only a hint for
scheduling, the mutex still protects the terminal state itself.
When the renderer is not waiting the cost for the parse loop is a
single relaxed atomic load per batch.
2026-07-09 20:29:09 +02:00
Mitchell Hashimoto
25e6245691 renderer: avoid starving scrollback compression
Inspector rendering can hold the terminal mutex while waking the
renderer. When the compression scheduler failed to acquire that mutex,
it treated every wake as possible terminal activity and restarted the
idle timer. Frequent inspector frames could therefore postpone
compression indefinitely until another interaction changed the timing.

Keep an existing compression deadline when a wake encounters lock
contention. The timer callback already rechecks both terminal activity
and lock availability, while the first contended wake still arms a
timer when none is active.
2026-07-09 10:47:31 -07:00
Mitchell Hashimoto
172f15da3b terminal: expose compression through libghostty-vt
Scrollback compression scheduling was only available to Zig callers that
used Terminal directly, leaving C embedders unable to drive the same idle
compression policy.

Define ABI-aware mode and result enums on Terminal and export activity
and compression operations through the C API. Keep scheduling
caller-owned, validate C inputs, and document the incremental contract
with a complete example.

Report unsupported reclamation consistently for full passes so callers
can disable compression on targets that cannot retain decommitted
mappings.
2026-07-09 10:11:07 -07:00
Mitchell Hashimoto
6d5dda40db inspector: clarify page compression memory
The PageList overview mixed structural state with a long list of exact
byte counters, making the compression result difficult to interpret.

Keep the overview focused on grid and scrollback structure, and add a
dedicated compression section before scrollbar details. Present page
states, uncompressed size, encoded ratio, resident estimate, and savings
using readable units and scoped help text.
2026-07-09 09:47:13 -07:00
Mitchell Hashimoto
0fb89f4ffe terminal: configure scrollback compression
Idle compression was always enabled on supported renderer-backed
surfaces and the default logical scrollback limit remained sized for
fully resident history.

Add a default-on scrollback-compression option and make renderer
scheduling honor it across config reloads. Existing compressed pages
remain encoded when the option is disabled, while reenabling it starts a
fresh idle pass.

Raise the default logical scrollback limit from 10 MB to 50 MB and
document typical physical-memory savings, content-dependent behavior,
and retained virtual address usage.
2026-07-09 09:47:13 -07:00