Chugging along towards #189
This adds significantly more internal work for searching. A long time
ago, I added #2885 which had a hint of what I was thinking of. This
simultaneously builds on this and changes direction.
The change of direction is that instead of making PageList fully
concurrency safe and having a search thread access it concurrently, I'm
now making an architectural shift where our search thread will grab the
big lock (blocking all IO/rendering), but with the bet that we can make
our critical areas small enough and time them well enough that the
performance hit while actively searching will be minimal. **Results yet
to be seen, but the path to implement this is much, much simpler.**
## Rearchitecting Search
To that end, this PR builds on #2885 by making `src/terminal/search` and
entire package (rather than a single file).
```mermaid
graph TB
subgraph Layer5 ["<b>Layer 5: Thread Orchestration</b>"]
Thread["<b>Thread</b><br/>━━━━━━━━━━━━━━━━━━━━━<br/>• MPSC queue management<br/>• libxev event loop<br/>• Message handling<br/>• Surface mailbox communication<br/>• Forward progress coordination"]
end
subgraph Layer4 ["<b>Layer 4: Screen Coordination</b>"]
ScreenSearch["<b>ScreenSearch</b><br/>━━━━━━━━━━━━━━━━━━━━━<br/>• State machine (tick + feed)<br/>• Result caching<br/>• Per-screen (alt/primary)<br/>• Composes Active + History search<br/>• Interrupt handling"]
end
subgraph Layer3 ["<b>Layer 3: Domain-Specific Search</b>"]
ActiveSearch["<b>ActiveSearch</b><br/>━━━━━━━━━━━━━━━━━━━━━<br/>• Active area only<br/>• Invalidate & re-search<br/>• Small, volatile data"]
PageListSearch["<b>PageListSearch</b><br/>━━━━━━━━━━━━━━━━━━━━━<br/>• History search (reverse order)<br/>• Separated tick/feed ops<br/>• Immutable PageList assumption<br/>• Garbage pin detection"]
end
subgraph Layer2 ["<b>Layer 1: Primitive Operations</b>"]
SlidingWindow["<b>SlidingWindow</b><br/>━━━━━━━━━━━━━━━━━━━━━<br/>• Manual linked list node management<br/>• Circular buffer maintenance<br/>• Zero-allocation search<br/>• Match yielding<br/>• Page boundary handling"]
end
Thread --> ScreenSearch
ScreenSearch --> ActiveSearch
ScreenSearch --> PageListSearch
ActiveSearch --> SlidingWindow
PageListSearch --> SlidingWindow
classDef layer5 fill:#0a0a0a,stroke:#ff0066,stroke-width:3px,color:#ffffff
classDef layer4 fill:#0f0f0f,stroke:#ff6600,stroke-width:3px,color:#ffffff
classDef layer3 fill:#141414,stroke:#ffaa00,stroke-width:3px,color:#ffffff
classDef layer2 fill:#1a1a1a,stroke:#00ff00,stroke-width:3px,color:#ffffff
class Thread layer5
class ScreenSearch layer4
class ActiveSearch,PageListSearch layer3
class SlidingWindow layer2
style Layer5 fill:#050505,stroke:#ff0066,stroke-width:2px,color:#ffffff
style Layer4 fill:#080808,stroke:#ff6600,stroke-width:2px,color:#ffffff
style Layer3 fill:#0c0c0c,stroke:#ffaa00,stroke-width:2px,color:#ffffff
style Layer2 fill:#101010,stroke:#00ff00,stroke-width:2px,color:#ffffff
```
Within the package, we have composable layers that let us test each
point:
- `SlidingWindow`: The lowest layer, the caller manually adds linked
list page nodes and it maintains a sliding window we search over,
yielding results without allocation (besides the circular buffers to
maintain the sliding window).
- `PageListSearch`: Searches a PageList structure in reverse order
(assumption: more recent matches are more valuable than older), but
separates out the `tick` (search, but no PageList access) and `feed`
(PageList access, prep data for search but don't search) operations.
This lets us `feed` in a critical area and `tick` outside. **This
assumes an immutable PageList, so this is for history.**
- `ActiveSearch`: Searches only the active area of a PageList. The
expectation is that the active area changes much more regularly, but it
is also very small (relative to scrollback). Throws away and re-searches
the active area as necessary.
- `ScreenSearch`: Composes the previous three components to coordinate
searching an active terminal screen. You'd have one of these per screen
(alt vs primary). This also caches results unlike the other components,
with the expectation that the caller will revisit the results as screens
change (so if you switch from neovim back to your shell and vice versa
with a search active, it won't start over).
- `Thread`: A dedicated search thread that will receive messages via
MPSC queues while managing the forward progress of a `ScreenSearch` and
sending matches back to the surface mailbox for apprt rendering. **The
thread component is not functional, just boilerplate, in this PR.**
ScreenSearch is a state machine that moves in an iterative `tick` +
`feed` fashion. This will let us "interrupt" the search with updates on
the search thread (read our mailbox via libxev loops for example) and
will let us minimize critical areas with locks (only `feed`).
Each component is significantly unit tested, especially around page
boundary cases. Given the complexity, there is no way this is perfect,
but the architecture is such that we can easily add regression tests as
we find issues.
## Other Changes, Notes
The only change to actually used code is that tracked pins in a
`PageList` can now be flagged as "garbage." A garbage tracked pin is one
that had to be moved in a non-sensical way because the previous location
it tracked has been deleted. This is used by the searcher to detect that
our history was pruned.
**If my assumption about the big lock is wrong** and this ends up being
godawful for performance, then it should still be okay because more
granular locking and reference counting such as that down by @dave-fl in
#8850 can be pushed into these components and reused. So this work is
still valuable on its own.
## Future
This PR is still just a bunch of internals, split out into its own PR so
I don't make one huge 10K diff PR. There are a number of future tasks:
- Flesh out `ScreenSearch` and hook it up to `Thread`
- Pull search thread management into `Surface` (or possibly the render
thread or shared render state since active area changes can be
synchronized with renderer frame rebuilds. Not sure yet.)
- Send updates back to the surface thread so that apprts can update UI.
- Apprt actions, input bindings, etc. to hook this all up (the easy
part, really).
The next step is to continue to flesh out the `ScreenSearch` as required
and hook it up to `Thread`.
**AI disclosure:** AI reviewed the code and assisted with some tests,
but didn't write any of the logic or design. This is beyond its ability
(or my ability to spec it out clearly enough for AI to succeed).
Fixes#9579
Protect against panics caused by integer overflows by using functions
that allow integer overflows to be caught instead of causing a panic.
Also protect against DOS from images that might not cause an
overflow but do consume an absurd amount of memory by limiting
images to a maximum size of 4GiB (which is the maximum size of
`image-storage-limit`).
This fixes the VS16 issues found in this test:
https://ucs-detect.readthedocs.io/sw_results/ghostty.html#ghostty
This is also a more robust way to handle VS15/16 in general.
This commit also changes our propeties to be a packed struct which
reduces its size from 4 bytes to 1 and likewise drops our unicode table
size 4x.
This updates uucode. As part of this, the wcwidth implementation was
updated (in uucode) to make emoji modifiers width ZERO. But if they're
standalone, we want them as width 2.
So this also contains a change to force them as width 2 for our width
calculation. This only matters for standalone emoji modifiers, because
when they form a valid grapheme we don't use this width calculation.
Fixes#9426
Since we can't set the meta charset tag since we emit partial HTML, we
use codepoint entities like `{` for non-ASCII characters to
ensure proper rendering.
This adds HTML formatting capabilities to the formatter package. HTML is
emitted as inline styles. For palette indexes, direct RGB is emitted if
we have access to a palette; otherwise, we fall back to CSS variables.
This isn't exposed to end users yet, but will enable copy as html
features. This is available in libghostty.
Fixes#9395
**AI disclosure:** I used AI (Amp) to help me write tests, but the
implementation was done manually. I reviewed everything.
This removes all existing functionality that I know of that encodes a
terminal, screen, pagelist, or page as plaintext and unifies all logic
onto the formatter system.
This replaces the logic of Screen.selectionString with calls to
ScreenFormatter.
This means that all our various selection-based features like copying to
clipboards now uses the new formatter. The formatter code is now
user-facing.
This forced us to pass all selectionString tests which revealed some
edge cases that were not handled correctly before in the formatter! The
formatter now handles:
- Plain text now emits `\n` instead of `\r\n`. VT emits `\r\n`
- Rectangular selections
- Various wide character edge cases
- Selection is now inclusive on the end, not exclusive
This adds the option `pin_map` or `point_map` (for pages) to formatter,
allowing callers to get a byte-by-byte mapping for where on the screen
each encoding maps to. This is used by search internals and hyperlinks.
I haven't hooked that all up yet. This diff was big enough I wanted this
as one.
Tests significantly cover the new feature.
Next up, we'll rip out `selectionString` and replace it with formatters!
This adds a new formatter that can be used with standard Zig `{f}`
formatting that emits any portion of the terminal screen as VT
sequences. In addition to simply styling, this can emit the entire
terminal/screen state such as cursor positions, active style, terminal
modes, etc.
To do this, I've extracted all formatting to a dedicated `formatter`
package within `terminal`. This handles all formatting types (currently
plaintext and VT formatting, but can imagine things like HTML in the
future). Presently, we have "formatting" split out across a variety of
places in Terminal, Screen, PageList, and Page. I didn't remove this
code yet but I intend to unify it all on formatter in the future.
This also doesn't expose this functionality in any user-facing way yet.
This PR just adds it to the ghostty-vt Zig module and unit tests it.
Ghostty app changes will come later.
**This also improves the readonly stream** to handle OSC color
operations for _setting_ but it doesn't emit any responses of course,
since its readonly.
This exposes the SGR parser to the C and Wasm APIs. An example is shown
in c-vt-sgr.
Compressed example:
```c
#include <assert.h>
#include <stdio.h>
#include <ghostty/vt.h>
int main() {
// Create parser
GhosttySgrParser parser;
assert(ghostty_sgr_new(NULL, &parser) == GHOSTTY_SUCCESS);
// Parse: ESC[1;31m (bold + red foreground)
uint16_t params[] = {1, 31};
assert(ghostty_sgr_set_params(parser, params, NULL, 2) == GHOSTTY_SUCCESS);
printf("Parsing: ESC[1;31m\n\n");
// Iterate through attributes
GhosttySgrAttribute attr;
while (ghostty_sgr_next(parser, &attr)) {
switch (attr.tag) {
case GHOSTTY_SGR_ATTR_BOLD:
printf("✓ Bold enabled\n");
break;
case GHOSTTY_SGR_ATTR_FG_8:
printf("✓ Foreground color: %d (red)\n", attr.value.fg_8);
break;
default:
break;
}
}
ghostty_sgr_free(parser);
return 0;
}
```
**AI disclosure:** Amp wrote most of the C headers, but I verified it
all. https://ampcode.com/threads/T-d9f145cb-e6ef-48a8-ad63-e5fc85c0d43e
This adds a new stream handler implementation that updates terminal
state in reaction to VT sequences, but doesn't perform any of the
actions that would require responses (e.g. queries).
This is exposed in two ways: first, as a standalone `ReadonlyStream` and
`ReadonlyHandler` type that contains all the implementation. Second, as
a convenience func on `Terminal` as `vtStream` and `vtHandler` which
return their respective types preconfigured to update the calling
terminal state.
This dramatically simplifies libghostty-vt usage from Zig (and will
eventually be exposed to C, too) since a Terminal on its own is ready to
go as a full VT parser and state machine without needing to build any
custom types!
There's a second big bonus here which is that our `stream_readonly.zig`
tests are true end-to-end tests for raw bytes to terminal state. This
will let us test a wider variety of situations more broadly. To start,
there are only a handful of tests implemented here.
**AI disclosure:** Amp wrote basically this whole thing, but I reviewed
it. https://ampcode.com/threads/T-3490efd2-1137-4112-96f6-4bf8a0141ff5
