d351d9ce07
Reworks the terminal PAGE grid wire format and optimize both
encode/decode. Example improvements for 1MB of VT input w/ full
scrollback: ~30x smaller wire size, ~45x faster encoding and decoding.
> [!IMPORTANT]
>
> **Snapshot version 1 is still explicitly a work-in-progress format, so
this breaks wire compatibility**.
The original snapshot version I merged favored simplicity over
optimization. This was the format used a proof-of-concept in my own
projects, but I knew it wasn't what I wanted to ship. This PR looks at
the record formats and trades simplicity for performance, a fair trade
for a performance-sensitive binary format.
Overview of changes:
- **8-byte grid cells.** Cells are now one 64-bit word whose layout
deliberately coincides with the native cell. Previously, cells were 16
bytes each and in our 1MB corpus 97% of the data was `0`. Lol.
- **Blank trailing cells are not written.** Rows declare an encoded cell
count so trailing blank cells cost nothing.
- **Hardware CRC32C.** Added `src/crc32c.zig` that uses inline-asm on
aarch64/x86_64 to get hardware speeds for CRC32. Zig's stdlib is 0.56
GB/s, aarch64 hardware is 10 GB/s on my computer.
- **Variable-width cells.** Each row declares how many bytes transport
each cell word: 1, 2, 4, or 8 depending on the widest row cell.
## Format
Grid layout, per PAGE record:
```
old new
+--------------------------+ +--------------------------+
| row 0 | | row 0 |
| flags (1) | | flags + width (1) |
| cols * 16B cells with | | encoded cell count (2) |
| inline suffixes | | count * width cells |
+--------------------------+ +--------------------------+
| ... | | ... |
+--------------------------+ +--------------------------+
| row (rows - 1) | | row (rows - 1) |
+--------------------------+ +--------------------------+
| grapheme suffix section |
+--------------------------+
```
Every row previously carried exactly `cols` cells; now it carries cells
only through its last non-default cell, and the cells past the count are
implicitly zero. The row flag byte gains the encoded cell width in its
previously reserved bits:
```
bit 0 wrap bit 2-3 semantic prompt
bit 1 wrap continuation bit 4-5 encoded cell width (log2 bytes)
```
The cell itself, old fixed 16-byte header versus the new single word:
```
old (16 bytes + inline suffixes) new (one u64 word)
+--------+---------+--------+ bit 0 +------------------+
| kind 1 | width 1 | flags 1| | content kind 2b |
+--------+---------+--------+ bit 2 +------------------+
| zero 1 | style id 2 | | content 24b |
+--------+------------------+ bit 26 +------------------+
| hyperlink id 2 | | style ID 16b |
+---------------------------+ bit 42 +------------------+
| value 4 | | width kind 2b |
+---------------------------+ bit 44 +------------------+
| grapheme count 4 | | protected 1b |
+---------------------------+ bit 45 +------------------+
| grapheme cps 4 * count | | hyperlink 1b |
+---------------------------+ bit 46 +------------------+
| semantic 2b |
bit 48 +------------------+
| hyperlink ID 16b |
bit 64 +------------------+
```
The word's bit layout intentionally matches the native cell (with the
wire hyperlink ID in the native padding), so full-width rows are a
straight copy of page memory. The row's encoded width then transports
each word truncated, and decode is the matching zero-extension:
```
width | bytes | admitted cells
------+-------+------------------------------------------------
0 | 1 | codepoint <= U+00FF, nothing else set
1 | 2 | codepoint <= U+FFFF, nothing else set
2 | 4 | any content kind/codepoint, style IDs 1-63,
| | narrow, no flags, no hyperlink
3 | 8 | everything
```
Grapheme suffixes were inline after each cell, which forced per-cell
framing decisions; they are now one section after the rows, so a
grapheme-free page (the overwhelming case) pays 4 bytes total:
```
old: ... | cell | cp cp | cell | ... (inline, per cell)
new: +----------------+----------------------------------+
| entry count 4 | entries: row 2, col 2, count 2, |
| | count * codepoint 4 |
+----------------+----------------------------------+
```
## Performance
Setup: `ghostty-bench +terminal-snapshot`, 80x24 terminal with unlimited
scrollback fed 1 MB of VT input.
Per-commit improvements:
| change | wire size | encode | decode |
|-------------------------------|-----------|---------|----------|
| baseline (v1 before this PR) | 34.16 MB | 92.8 ms | 119.8 ms |
| 8-byte cells + blank elision | 7.66 MB | 18.2 ms | 28.0 ms |
| hardware CRC32C | 7.66 MB | 5.8 ms | 15.5 ms |
| gate page verification | 7.66 MB | 5.8 ms | 12.2 ms |
| staged PAGE payload decoding | 7.66 MB | 5.8 ms | 8.1 ms |
| variable-width cells | 1.03 MB | 2.0 ms | 2.7 ms |
Final result across various inputs:
| corpus | wire size | encode | decode |
|----------------------------|------------------------|----------------------|-----------------------|
| ascii lines 1-70 | 34.16 -> 1.03 MB (33x) | 92.8 -> 2.0 ms (46x) |
119.8 -> 2.7 ms (44x) |
| ascii full-width wrap | 16.01 -> 1.04 MB (15x) | 43.6 -> 1.3 ms (34x)
| 56.2 -> 1.8 ms (31x) |
| utf8 (wide/grapheme heavy) | 4.33 -> 1.89 MB (2.3x) | 12.4 -> 1.7 ms
(7x) | 19.0 -> 2.2 ms (9x) |
### Relationship with Compression
I expect that users of this will wrap everything in compression, so I
also benchmarked all my changes against a caller-owned zstd compressor
to ensure we're making the write tradeoffs. Less bytes means less time
in a compressor, even if a ton of 0s compresses really well.
My results: `zstd -1` over the `lines` snapshot drops from 12.7 ms to
0.8 ms, and the compressed artifact shrinks from 1.35 MB to 0.86 MB. So
the end state is a win-win.
Ghostty
Fast, native, feature-rich terminal emulator pushing modern features.
A native GUI or embeddable library via libghostty.
About
·
Download
·
Documentation
·
Contributing
·
Developing
About
Ghostty is a terminal emulator that differentiates itself by being fast, feature-rich, and native. While there are many excellent terminal emulators available, they all force you to choose between speed, features, or native UIs. Ghostty provides all three.
libghostty is a cross-platform, zero-dependency C and Zig library
for building terminal emulators or utilizing terminal functionality
(such as style parsing). Anyone can use libghostty to build a terminal
emulator or embed a terminal into their own applications. See
Ghostling for a minimal complete project
example or the examples directory
for smaller examples of using libghostty in C and Zig.
For more details, see About Ghostty.
Download
See the download page on the Ghostty website.
Documentation
See the documentation on the Ghostty website.
Contributing and Developing
If you have any ideas, issues, etc. regarding Ghostty, or would like to contribute to Ghostty through pull requests, please check out our "Contributing to Ghostty" document. Those who would like to get involved with Ghostty's development as well should also read the "Developing Ghostty" document for more technical details.
Roadmap and Status
Ghostty is stable and in use by millions of people and machines daily.
The high-level ambitious plan for the project, in order:
| # | Step | Status |
|---|---|---|
| 1 | Standards-compliant terminal emulation | ✅ |
| 2 | Competitive performance | ✅ |
| 3 | Rich windowing features -- multi-window, tabbing, panes | ✅ |
| 4 | Native Platform Experiences | ✅ |
| 5 | Cross-platform libghostty for Embeddable Terminals |
✅ |
| 6 | Ghostty-only Terminal Control Sequences | ❌ |
Additional details for each step in the big roadmap below:
Standards-Compliant Terminal Emulation
Ghostty implements all of the regularly used control sequences and can run every mainstream terminal program without issue. For legacy sequences, we've done a comprehensive xterm audit comparing Ghostty's behavior to xterm and building a set of conformance test cases.
In addition to legacy sequences (what you'd call real "terminal" emulation), Ghostty also supports more modern sequences than almost any other terminal emulator. These features include things like the Kitty graphics protocol, Kitty image protocol, clipboard sequences, synchronized rendering, light/dark mode notifications, and many, many more.
We believe Ghostty is one of the most compliant and feature-rich terminal emulators available.
Terminal behavior is partially a de jure standard (i.e. ECMA-48) but mostly a de facto standard as defined by popular terminal emulators worldwide. Ghostty takes the approach that our behavior is defined by (1) standards, if available, (2) xterm, if the feature exists, (3) other popular terminals, in that order. This defines what the Ghostty project views as a "standard."
Competitive Performance
Ghostty is generally in the same performance category as the other highest performing terminal emulators.
"The same performance category" means that Ghostty is much faster than traditional or "slow" terminals and is within an unnoticeable margin of the well-known "fast" terminals. For example, Ghostty and Alacritty are usually within a few percentage points of each other on various benchmarks, but are both something like 100x faster than Terminal.app and iTerm. However, Ghostty is much more feature rich than Alacritty and has a much more native app experience.
This performance is achieved through high-level architectural decisions and low-level optimizations. At a high-level, Ghostty has a multi-threaded architecture with a dedicated read thread, write thread, and render thread per terminal. Our renderer uses OpenGL on Linux and Metal on macOS. Our read thread has a heavily optimized terminal parser that leverages CPU-specific SIMD instructions. Etc.
Rich Windowing Features
The Mac and Linux (build with GTK) apps support multi-window, tabbing, and splits with additional features such as tab renaming, coloring, etc. These features allow for a higher degree of organization and customization than single-window terminals.
Native Platform Experiences
Ghostty is a cross-platform terminal emulator but we don't aim for a least-common-denominator experience. There is a large, shared core written in Zig but we do a lot of platform-native things:
- The macOS app is a true SwiftUI-based application with all the things you would expect such as real windowing, menu bars, a settings GUI, etc.
- macOS uses a true Metal renderer with CoreText for font discovery.
- macOS supports AppleScript, Apple Shortcuts (AppIntents), etc.
- The Linux app is built with GTK.
- The Linux app integrates deeply with systemd if available for things like always-on, new windows in a single instance, cgroup isolation, etc.
Our goal with Ghostty is for users of whatever platform they run Ghostty on to think that Ghostty was built for their platform first and maybe even exclusively. We want Ghostty to feel like a native app on every platform, for the best definition of "native" on each platform.
Cross-platform libghostty for Embeddable Terminals
In addition to being a standalone terminal emulator, Ghostty is a
C-compatible library for embedding a fast, feature-rich terminal emulator
in any 3rd party project. This library is called libghostty.
Due to the scope of this project, we're breaking libghostty down into
separate libraries, starting with libghostty-vt. The goal of
this project is to focus on parsing terminal sequences and maintaining
terminal state. This is covered in more detail in this
blog post.
libghostty-vt is already available and usable today for Zig and C and
is compatible for macOS, Linux, Windows, and WebAssembly. The functionality
is extremely stable (since its been proven in Ghostty GUI for a long time),
but the API signatures are still in flux.
libghostty is already heavily in use. See examples
for small examples of using libghostty in C and Zig or the
Ghostling project for a
complete example. See awesome-libghostty
for a list of projects and resources related to libghostty.
We haven't tagged libghostty with a version yet and we're still working on a better docs experience, but our Doxygen website is a good resource for the C API.
Ghostty-only Terminal Control Sequences
We want and believe that terminal applications can and should be able to do so much more. We've worked hard to support a wide variety of modern sequences created by other terminal emulators towards this end, but we also want to fill the gaps by creating our own sequences.
We've been hesitant to do this up until now because we don't want to create more fragmentation in the terminal ecosystem by creating sequences that only work in Ghostty. But, we do want to balance that with the desire to push the terminal forward with stagnant standards and the slow pace of change in the terminal ecosystem.
We haven't done any of this yet.
Crash Reports
Ghostty has a built-in crash reporter that will generate and save crash
reports to disk. The crash reports are saved to the $XDG_STATE_HOME/ghostty/crash
directory. If $XDG_STATE_HOME is not set, the default is ~/.local/state.
Crash reports are not automatically sent anywhere off your machine.
Crash reports are only generated the next time Ghostty is started after a crash. If Ghostty crashes and you want to generate a crash report, you must restart Ghostty at least once. You should see a message in the log that a crash report was generated.
Note
Use the
ghostty +crash-reportCLI command to get a list of available crash reports. A future version of Ghostty will make the contents of the crash reports more easily viewable through the CLI and GUI.
Crash reports end in the .ghosttycrash extension. The crash reports are in
Sentry envelope format. You can
upload these to your own Sentry account to view their contents, but the format
is also publicly documented so any other available tools can also be used.
The ghostty +crash-report CLI command can be used to list any crash reports.
A future version of Ghostty will show you the contents of the crash report
directly in the terminal.
To send the crash report to the Ghostty project, you can use the following CLI command using the Sentry CLI:
SENTRY_DSN=https://e914ee84fd895c4fe324afa3e53dac76@o4507352570920960.ingest.us.sentry.io/4507850923638784 sentry-cli send-envelope --raw <path to ghostty crash>
Warning
The crash report can contain sensitive information. The report doesn't purposely contain sensitive information, but it does contain the full stack memory of each thread at the time of the crash. This information is used to rebuild the stack trace but can also contain sensitive data depending on when the crash occurred.