Drop UTF-8 decoded C1 controls entirely. This matches xterm's default
behavior which is our standard policy (but note it diverges from libvte
which executes them). There isn't really any standard I could find
around this.
The ground state UTF-8 fast paths (both the scalar decoder and the
batched SIMD path) previously treated decoded codepoints C1 control
codepoints as normal UTF-8 text and routed them to print.
Drop UTF-8 decoded C1 controls entirely. This matches xterm's default
behavior which is our standard policy (but note it diverges from libvte
which executes them). There isn't really any standard I could find
around this.
The ground state UTF-8 fast paths (both the scalar decoder and the
batched SIMD path) previously treated decoded codepoints C1 control
codepoints as normal UTF-8 text and routed them to print.
Fixes#14021
The ground state UTF-8 fast paths only classified 0x00-0x0F plus 0x1B
(escape) as C0 controls. The remaining C0 bytes (0x10-0x1A, 0x1C-0x1F)
were decoded as ordinary codepoints and routed to print as if they were
text.
This resulted in incorrect grids but also very weird font fallback, e.g.
U+0014 would find CJK fonts.
This commit fixes this by routing every ground state C0 byte except ESC
to execute as it should be.
Fixes#14021
The ground state UTF-8 fast paths only classified 0x00-0x0F plus 0x1B (escape)
as C0 controls. The remaining C0 bytes (0x10-0x1A, 0x1C-0x1F) were decoded
as ordinary codepoints and routed to print as if they were text.
This resulted in incorrect grids but also very weird font fallback, e.g.
U+0014 would find CJK fonts.
This commit fixes this by routing every ground state C0 byte except ESC to
execute as it should be.
Ref #12034
This commit releases many GPU resources when a surface becomes invisible
and rebuilds it on the next draw. I don't say "all" because there are
still some things we can improve on (Kitty images).
We previously held onto all GPU resources for the lifetime of the
surface regardless of its visibility state. This is 3x (for
triple-buffering): screen render targets, uniform/cell/custom shader
buffers, font textures, and more.
Measured on macOS (Metal):
| Measurement (1 visible + 20 hidden tabs) | Before | After |
|---------------------------------------------|-----------|---------|
| Tracked GPU allocations (steady state) | 384.6 MiB | 18.3 MiB |
| `MTLDevice.currentAllocatedSize` | 393.3 MiB | 19.7 MiB |
| `footprint` IOSurface (dirty) | 309 MB | 15 MB |
| Swap chain rebuild on unhide (42 tab switches) | n/a | avg 0.43 ms,
max 0.55 ms |
As you can see, importantly, swap chain rebuild is fast: 0.43ms average.
That means that the rebuild is imperceptible and happens well within a
frame draw time.
This is macOS only, but most of the work was in the generic renderer.
GTK only needs to call `releaseGpuResources` when it becomes invisible
to get the same benefits. I didn't have my VM handy to test this yet so
I didn't include it.
Ref #12034
This commit releases many GPU resources when a surface becomes invisible and
rebuilds it on the next draw. I don't say "all" because there are still
some things we can improve on (Kitty images).
We previously held onto all GPU resources for the lifetime of the surface
regardless of its visibility state. This is 3x (for triple-buffering):
screen render targets, uniform/cell/custom shader buffers, font textures,
and more.
Measured on macOS (Metal):
| Measurement (1 visible + 20 hidden tabs) | Before | After |
|---------------------------------------------|-----------|---------|
| Tracked GPU allocations (steady state) | 384.6 MiB | 18.3 MiB |
| `MTLDevice.currentAllocatedSize` | 393.3 MiB | 19.7 MiB |
| `footprint` IOSurface (dirty) | 309 MB | 15 MB |
| Swap chain rebuild on unhide (42 switches) | n/a | avg 0.43 ms, max 0.55 ms |
As you can see, importantly, swap chain rebuild is fast: 0.43ms average.
That means that the rebuild is imperceptible and happens well within
a frame draw time.
This is macOS only, but most of the work was in the generic renderer.
GTK only needs to call `releaseGpuResources` when it becomes invisible
to get the same benefits. I didn't have my VM handy to test this yet so
I didn't include it.
Refs #11216
The dcs_passthrough state only forwarded bytes 0x00-0x7E to the DCS
handler. Bytes 0x80-0x9F hit the "anywhere" C1 transitions and exited
the string, while 0xA0-0xFF fell through to the default transition and
were silently dropped. **This breaks any DCS payload carrying UTF-8. **
A continuation byte in the C1 range terminates or corrupts the string:
"Ü" is 0xC3 0x9C, so the 0xC3 is dropped and the 0x9C acts as 8-bit ST,
ending the DCS mid-character.
Also, a payload byte such as 0x9B (second byte of "Û") transitions to
csi_entry, so the remainder of the payload executes as a live control
sequence. This is a prerequisite for tmux control mode (#1935), whose
%output notifications carry raw UTF-8 pane content.
Fix this in the parse table only: override 0x80-0xFF in dcs_passthrough
to put and in dcs_ignore to ignore, exactly how osc_string already
claims 0x20-0xFF (including 0x9C) as data. This deviates from the
vt100.net state machine
(https://vt100.net/emu/dec_ansi_parser) deliberately and includes 0x9C:
a raw 0x9C is indistinguishable from a UTF-8 continuation byte, and we
don't honor 8-bit C1 controls in the ground state either.
The demo code shown in the `+list-themes` theme preview was stale from
before the Zig 0.16 migration (context: #12228). It referenced
`std.Io.getStdOut().writer()`, which never existed in any Zig release,
and `pub fn main() !void`. This rewrites the rendered sample to valid
Zig 0.16 idioms:
```zig
const std = @import("std");
pub fn main(init: std.process.Init) !void {
var buf: [1024]u8 = undefined;
var stdout = std.Io.File.stdout().writer(init.io, &buf);
const w = &stdout.interface;
var i: usize = 1;
while (i <= 16) : (i += 1) {
if (i % 15 == 0) {
try w.writeAll("ZiggZagg\n");
} else if (i % 3 == 0) {
try w.writeAll("Zigg\n");
} else if (i % 5 == 0) {
try w.writeAll("Zagg\n");
} else {
try w.print("{d}\n", .{i});
}
}
try w.flush();
}
```
The gutter line numbers, row offsets, and child window height were
renumbered to match, and the zig version shown in the demo prompt line
was updated from v0.13.0 to v0.16.0.
Refs #11216
The dcs_passthrough state only forwarded bytes 0x00-0x7E to the DCS
handler. Bytes 0x80-0x9F hit the "anywhere" C1 transitions and exited
the string, while 0xA0-0xFF fell through to the default transition and
were silently dropped. **This breaks any DCS payload carrying UTF-8. **
A continuation byte in the C1 range terminates or corrupts the string:
"Ü" is 0xC3 0x9C, so the 0xC3 is dropped and the 0x9C acts as 8-bit ST,
ending the DCS mid-character.
Also, a payload byte such as 0x9B (second byte of "Û") transitions to
csi_entry, so the remainder of the payload executes as a live control sequence.
This is a prerequisite for tmux control mode (#1935), whose %output
notifications carry raw UTF-8 pane content.
Fix this in the parse table only: override 0x80-0xFF in
dcs_passthrough to put and in dcs_ignore to ignore, exactly how
osc_string already claims 0x20-0xFF (including 0x9C) as data. This
deviates from the vt100.net state machine
(https://vt100.net/emu/dec_ansi_parser) deliberately and includes
0x9C: a raw 0x9C is indistinguishable from a UTF-8 continuation byte,
and we don't honor 8-bit C1 controls in the ground state either.
The Kitty clipboard protocol now specifies base64 handling: All OSC 5522
payloads and the base64 metadata values (mime, name, pw) use strict RFC
4648 with the standard alphabet. Characters outside the alphabet
(including whitespace) and incorrect padding must be rejected, never
silently skipped.
For wdata payloads for one MIME type, the base64 stream can be split at
arbitrary packet boundaries and only the concatenation must be correctly
padded.
Simdutf has a strict mode for base64 so we got this for free. Benchmarks
to be safe:
| Decoder | Time | Throughput |
|-------------------------------|-------|------------|
| permissive (previous) | 99ms | 10.8 GB/s |
| strict | 97ms | 11.1 GB/s |
| strict, streaming 4KiB chunks | 102ms | 10.5 GB/s |
| strict w/ separate scan pass | 143ms | 7.5 GB/s |
| std.base64 scalar | 234ms | 4.6 GB/s |
Spec changes upstream:
479872838fhttps://sw.kovidgoyal.net/kitty/clipboard/#encoding-of-payloads
Update nixpkgs-unstable to pick up fontconfig 2.18. Currently we are
linking against 2.17 and you get errors like this on standard error when
using config files meant for fontconfig 2.18:
Fontconfig warning: "/etc/fonts/conf.d/48-guessfamily.conf", line 20:
invalid constant used :
Fontconfig warning: "/etc/fonts/conf.d/48-guessfamily.conf", line 23:
invalid constant used : monospace
Fontconfig warning: "/etc/fonts/conf.d/48-guessfamily.conf", line 42:
invalid attribute 'xsi:nil'
Fontconfig warning: "/etc/fonts/conf.d/48-guessfamily.conf", line 43:
invalid constant used :
Fontconfig warning: "/etc/fonts/conf.d/48-guessfamily.conf", line 46:
invalid constant used : sans-serif
Fontconfig warning: "/etc/fonts/conf.d/48-guessfamily.conf", line 68:
invalid attribute 'xsi:nil'
Fontconfig warning: "/etc/fonts/conf.d/48-guessfamily.conf", line 69:
invalid constant used :
The Kitty clipboard protocol now specifies base64 handling:
All OSC 5522 payloads and the base64 metadata values (mime, name, pw)
use strict RFC 4648 with the standard alphabet. Characters outside
the alphabet (including whitespace) and incorrect padding must be
rejected, never silently skipped.
For wdata payloads for one MIME type, the base64 stream can be split
at arbitrary packet boundaries and only the concatenation must be
correctly padded.
Simdutf has a strict mode for base64 so we got this for free.
Benchmarks to be safe:
| Decoder | Time | Throughput |
|-------------------------------|-------|------------|
| permissive (previous) | 99ms | 10.8 GB/s |
| strict | 97ms | 11.1 GB/s |
| strict, streaming 4KiB chunks | 102ms | 10.5 GB/s |
| strict w/ separate scan pass | 143ms | 7.5 GB/s |
| std.base64 scalar | 234ms | 4.6 GB/s |
Spec changes upstream:
479872838fhttps://sw.kovidgoyal.net/kitty/clipboard/#encoding-of-payloads
Update nixpkgs-unstable to pick up fontconfig 2.18. Currently we are linking against 2.17 and you get errors
like this on standard error when using config files meant for fontconfig 2.18:
Fontconfig warning: "/etc/fonts/conf.d/48-guessfamily.conf", line 20: invalid constant used :
Fontconfig warning: "/etc/fonts/conf.d/48-guessfamily.conf", line 23: invalid constant used : monospace
Fontconfig warning: "/etc/fonts/conf.d/48-guessfamily.conf", line 42: invalid attribute 'xsi:nil'
Fontconfig warning: "/etc/fonts/conf.d/48-guessfamily.conf", line 43: invalid constant used :
Fontconfig warning: "/etc/fonts/conf.d/48-guessfamily.conf", line 46: invalid constant used : sans-serif
Fontconfig warning: "/etc/fonts/conf.d/48-guessfamily.conf", line 68: invalid attribute 'xsi:nil'
Fontconfig warning: "/etc/fonts/conf.d/48-guessfamily.conf", line 69: invalid constant used :
This also removes an override for libfyaml on Darwin that was merged upstream into nixpkgs.
We now have large OSCs (e.g. Kitty clipboard protocol) on the order of
megabytes. OSC was still byte-at-a-time. This adds a vector-optimized
plus bulk storing path to OSC, similar to APC.
Throughput measured with the terminal-stream benchmark:
| Corpus | Before | After | Speedup |
|--------------------------|--------|-------|---------|
| OSC 52, 1MiB payloads | 446ms | 14ms | 32x |
| OSC 5522, 1MiB payloads | 446ms | 15ms | 30x |
| OSC 5522, 64KiB payloads | 448ms | 14ms | 32x |
| OSC 5522, 4KiB payloads | 448ms | 16ms | 29x |
| Tiny titles (~24B each) | 450ms | 73ms | 6.2x |
| Mixed OSCs (16MiB) | 595ms | 531ms | 1.13x |
Used Fable to help validate this with a barrage of differential tests.
The actual implementation was AI-written but was guided to basically
mimic the APC path and then I hand verified everything too.
We now have large OSCs (e.g. Kitty clipboard protocol) on the order
of megabytes. OSC was still byte-at-a-time. This adds a vector-optimized
plus bulk storing path to OSC, similar to APC.
Throughput measured with the terminal-stream benchmark:
| Corpus | Before | After | Speedup |
|--------------------------|--------|-------|---------|
| OSC 52, 1MiB payloads | 446ms | 14ms | 32x |
| OSC 5522, 1MiB payloads | 446ms | 15ms | 30x |
| OSC 5522, 64KiB payloads | 448ms | 14ms | 32x |
| OSC 5522, 4KiB payloads | 448ms | 16ms | 29x |
| Tiny titles (~24B each) | 450ms | 73ms | 6.2x |
| Mixed OSCs (16MiB) | 595ms | 531ms | 1.13x |
Used Fable to help validate this with a barrage of differential tests.
The actual implementation was AI-written but was guided to basically
mimic the APC path and then I hand verified everything too.
Fixes#13940
Various operations like scroll, line insert, erase, etc. operations
would clear cells but remain row metadata such as wrap flags and
semantic prompt state.
When we had fast paths in `grow` and other places like resize we would
adopt that without knowing it because we didn't properly clear.
Fix this by properly clearing row metadata too at the points where we
erase a row that might be reused.
Benchmarked with terminal-stream workloads for each affected path. The
added cost is 2-4 instructions per recycled row next to the existing
full-row cell clear. There was no measurable wall-clock change on any
workload.
AI helped run the benchmarks for me and analyze for missing places (it
found some!) but otherwise this was hand-designed.
Fixes#13940
Various operations like scroll, line insert, erase, etc. operations
would clear cells but remain row metadata such as wrap flags and
semantic prompt state.
When we had fast paths in `grow` and other places like resize
we would adopt that without knowing it because we didn't properly clear.
Fix this by properly clearing row metadata too at the points where we
erase a row that might be reused.
Benchmarked with terminal-stream workloads for each affected path.
The added cost is 2-4 instructions per recycled row next to the existing
full-row cell clear. There was no measurable wall-clock change on any
workload.
AI helped run the benchmarks for me and analyze for missing places (it found
some!) but otherwise this was hand-designed.
Partial changes for #13205, known issues are marked as warnings.
### AI Disclosure
Claude generated these tests from linked pr, I cherrypicked and reviewed
myself.
Follow up for #13888, and prepare for #13205.
The comments are copied from the history commit.
## AI Disclosure
The tests are updated by Claude, I cherrypicked them.
Validate decoded OSC 5522 metadata, read MIME lists, and alias lists as
UTF-8. Treat an alias without a target MIME type as an invalid write
packet.
Malformed write packets now return EINVAL and terminate the in-flight
transaction instead of leaving it active. Malformed reads are dropped
without disturbing an active write.
Latest changes upstream to spec:
458421af46
Update OSC 5522 writes to reject every transaction that exceeds the
configured decoded-data limit. The previous behavior truncated text
while rejecting only non-text data.
Programs now receive EFBIG as soon as a write crosses the limit. The
clipboard remains untouched, and remaining write packets are ignored
until a new transaction begins. Raise the default to the protocol
minimum of 64 MiB.
This applies the latest spec change:
32ea104192
Validate decoded OSC 5522 metadata, read MIME lists, and alias
lists as UTF-8. Treat an alias without a target MIME type as an
invalid write packet.
Malformed write packets now return EINVAL and terminate the in-flight
transaction instead of leaving it active. Malformed reads are dropped
without disturbing an active write.
Latest changes upstream to spec:
458421af46
Update OSC 5522 writes to reject every transaction that exceeds the
configured decoded-data limit. The previous behavior truncated text
while rejecting only non-text data.
Programs now receive EFBIG as soon as a write crosses the limit. The
clipboard remains untouched, and remaining write packets are ignored
until a new transaction begins. Raise the default to the protocol
minimum of 64 MiB.
This applies the latest spec change:
32ea104192
Add a new `clipboard-write-limit-bytes` option (similar to
`scrollback-limit-bytes`) to limit the maximum OSC 5522 write size.
Defaults to 32 MB.
This also adds a new `GHOSTTY_TERMINAL_OPT_CLIPBOARD_WRITE_MAX_BYTES`
option for libghostty-vt embedders to control the same.
Kitty has a limit too and it works by truncating all data. I decided on
purpose to diverge from this because I don't think truncated binary data
is useful. Instead, we reject it so the application knows the write
didn't work.
We truncate text data, and we try to do it at the nearest complete UTF-8
sequence (if possible).
For the future: Kitty spools any write data more than some size (can't
remember) to a temp file on disk. We might want to consider doing
something similar since we're all in-memory at the moment. This PR
doesn't change that.
A Kitty clipboard protocol (OSC 5522) read that only requests the
targets type ('.') is now served without a permission prompt and never
consults (or consumes) session password grants.
The spec requires this so that a client listing the available data types
before reading one doesn't present the user with a double permission
prompt.
Add a new `clipboard-write-limit-bytes` option (similar to
`scrollback-limit-bytes`) to limit the maximum OSC 5522 write size.
Defaults to 32 MB.
This also adds a new `GHOSTTY_TERMINAL_OPT_CLIPBOARD_WRITE_MAX_BYTES`
option for libghostty-vt embedders to control the same.
Kitty has a limit too and it works by truncating all data. I decided on
purpose to diverge from this because I don't think truncated binary data
is useful. Instead, we reject it so the application knows the write
didn't work.
We truncate text data, and we try to do it at the nearest complete UTF-8
sequence (if possible).
Discussion #13979
Dropped paths and text once again honor bracketed paste mode. IME,
dictation, emoji picker, and character viewer commits remain typed
input.
sendText calls ghostty_surface_text, which applies the clipboard paste
pipeline and bracketed paste framing when enabled. Separating the paths
at the drag-and-drop caller preserves the input-method behavior
introduced by #13817.
A Kitty clipboard protocol (OSC 5522) write transaction targeting
`loc=primary` replied `type=write:status=DONE` in the macOS app even
though macOS has no primary selection and the data was silently
discarded.
The spec requires ENOSYS when the requested location is not available on
the system, which the read path already answers correctly:
https://sw.kovidgoyal.net/kitty/clipboard/
A Kitty clipboard protocol (OSC 5522) read that only requests the
targets type ('.') is now served without a permission prompt and never
consults (or consumes) session password grants.
The spec requires this so that a client listing the available data types
before reading one doesn't present the user with a double permission prompt.
A Kitty clipboard protocol (OSC 5522) write transaction targeting
`loc=primary` replied `type=write:status=DONE` in the macOS app even
though macOS has no primary selection and the data was silently
discarded.
The spec requires ENOSYS when the requested location is not
available on the system, which the read path already answers correctly:
https://sw.kovidgoyal.net/kitty/clipboard/
Discussion #13979
Dropped paths and text once again honor bracketed paste mode.
IME, dictation, emoji picker, and character viewer commits remain typed input.
sendText calls ghostty_surface_text, which applies the clipboard paste
pipeline and bracketed paste framing when enabled. Separating the
paths at the drag-and-drop caller preserves the input-method behavior
introduced by #13817.
Programs can now write the system clipboard through the Kitty clipboard
protocol in the macOS app. This also does all the hard work plumbing
through core termio/apprt so GTK should be an easy follow.
This functionality lets clients copy arbitrary representations (images,
HTML, etc.) into the clipboard. Writes honor `clipboard-write`: allow
applies silently, deny answers EPERM up front before any data is used,
and ask shows the standard confirmation prompt.
After this, I believe the core and macOS have 100% Kitty clipboard
implementation but I'll double check after this.
## Demo
https://github.com/user-attachments/assets/71234fa0-f539-48eb-a633-8dea3addddd5
Programs can now write the system clipboard through the Kitty
clipboard protocol in the macOS app. This also does all the hard work
plumbing through core termio/apprt so GTK should be an easy follow.
This functionality lets clients copy arbitrary representations (images,
HTML, etc.) into the clipboard. Writes honor `clipboard-write`: allow
applies silently, deny answers EPERM up front before any data is used,
and ask shows the standard confirmation prompt.
Clearing the screen into scrollback and then printing could crash debug
builds with a page integrity violation, or silently corrupt
style/hyperlink reference counts in release builds. Found in #13991 via
fuzzing.
The cursor's style and hyperlink IDs are only valid on the page the
cursor is on. When the scroll clear moved the start of the fresh screen
onto a new page, the reset path in cursorReload updated the cursor's
position directly instead of going through cursorChangePin, so the
cursor kept IDs from its old page. On the new page those IDs pointed at
entries that were dead or belonged to something else, and the next print
used them.
Fix this by making the reset path go through `cursorChangePin` like
every other cross-page cursor move, which releases the style and
hyperlink from the old page and recreates them on the new one.
Clearing the screen into scrollback and then printing could crash debug
builds with a page integrity violation, or silently corrupt style/hyperlink
reference counts in release builds. Found in #13991 via fuzzing.
The cursor's style and hyperlink IDs are only valid on the page the
cursor is on. When the scroll clear moved the start of the fresh
screen onto a new page, the reset path in cursorReload updated the
cursor's position directly instead of going through cursorChangePin,
so the cursor kept IDs from its old page. On the new page those IDs
pointed at entries that were dead or belonged to something else, and
the next print used them.
Fix this by making the reset path go through `cursorChangePin` like
every other cross-page cursor move, which releases the style and
hyperlink from the old page and recreates them on the new one.
Profiling `mpv --vo=kitty --vo-kitty-use-shm <video>` shows up a copy
and then swizzle in `prepImage`
This comes from the renderer copying the raw image data for ownership,
then doing an rgb to rgba conversion to replace the copied data.
This change optimize this case by letting the format conversion read
from the data source instead of a copy of it.
<img width="3825" height="1579" alt="image"
src="https://github.com/user-attachments/assets/25851c04-b582-4bad-8f8d-930448d89fa2"
/>
<img width="3825" height="1579" alt="image"
src="https://github.com/user-attachments/assets/fbb1ca73-a86f-421b-992d-a764ce08c83e"
/>
> Above-Before: prepImage profiles a memcpy + swizzle
> Below-After: prepImage profiles just a swizzle
The existing data path for kitty images is:
```
Read tty for Kitty image transmission (srgb, srgba, or PNG bytes)
-> copy into Kitty graphics ImageStorage (cpu-owned)
Renderer updateFrame creates Image.Pending in renderer-owned CPU storage
-> sync Kitty ImageStorage with renderer-owned ImageMap
-> copy bytes into renderer.ImageMap (renderer-owned)
-> convert ImageMap bytes to preferred GPU upload format
Renderer drawFrame uploads image data to the GPU
-> iterate through renderer.ImageMap
-> for Pending image uploads
-> convert the pixel format (no-op if already done), create the GPU-side texture and upload the data
```
# Note
Kitty graphics only supports RGB and RGBA data
The image file decode path uses a wuffs png and jpeg decode function
configured to return RGBA 8-bit, so I think in practice we only ever
upload RGB8 or RGBA8. And the gray-alpha and gray pixel formats aren't
ever used.
# AI Disclosure
I didn't use any LLM assistance for this.
