This PR builds on https://github.com/ghostty-org/ghostty/pull/9678 ~so
the diff from there is included here (it's not possible to stack PRs
unless it's a PR against my own fork)--review that one first!~
This PR updates the `graphemeBreak` calculation to use `uucode`'s
`computeGraphemeBreakNoControl`, which has [tests in
uucode](215ff09730/src/x/grapheme.zig (L753))
that confirm it passes the `GraphemeBreakTest.txt` (minus some
exceptions).
Note that the `grapheme_break` (and `grapheme_break_no_control`)
property in `uucode` incorporates `emoji_modifier` and
`emoji_modifier_base`, diverging from UAX #29 but matching UTS #51. See
[this comment in
uucode](215ff09730/src/grapheme.zig (L420-L434))
for details.
The `grapheme_break_no_control` property and
`computeGraphemeBreakNoControl` both assume `control`, `cr`, and `lf`
have been filtered out, matching the current grapheme break logic in
Ghostty.
This PR keeps the `Precompute.data` logic mostly equivalent, since the
`uucode` `precomputedGraphemeBreak` lacks benchmarks in the `uucode`
repository (it was benchmarked in [the original PR adding `uucode` to
Ghostty](https://github.com/ghostty-org/ghostty/pull/8757)). Note
however, that due to `grapheme_break` being one bit larger than
`grapheme_boundary_class` and the new `BreakState` also being one bit
larger, the state jumps up by a factor of 8 (u10 -> u13), to 8KB.
## Benchmarks
~I benchmarked the old `main` version versus this PR for
`+grapheme-break` and surprisingly this PR is 2% faster (?). Looking at
the assembly though, I'm thinking something else might be causing that.
Once I get to the bottom of that I'll remove the below TODO and include
the benchmark results here.~
When seeing the speedup with `data.txt` and maybe a tiny speedup on
English wiki, I was surprised given the 1KB -> 8KB tables. Here's what
AI said when I asked it to inspect the assembly:
https://ampcode.com/threads/T-979b1743-19e7-47c9-8074-9778b4b2a61e, and
here's what it said when I asked it to predict the faster version:
https://ampcode.com/threads/T-3291dcd3-7a21-4d24-a192-7b3f6e18cd31
It looks like two loads got reordered and that put the load that
depended on stage1 -> stage2 -> stage3 second, "hiding memory latency".
So that makes the new one faster when looking up the `grapheme_break`
property. These gains go away with the Japanese and Arabic benchmarks,
which spend more time processing utf8, and may even have more grapheme
clusters too.
### with data.txt (200 MB ghostty-gen random utf8)
<img width="1822" height="464" alt="CleanShot 2025-11-26 at 08 42 03@2x"
src="https://github.com/user-attachments/assets/56d4ee98-21db-4eab-93ab-a0463a653883"
/>
### with English wiki dump
<img width="2012" height="506" alt="CleanShot 2025-11-26 at 08 43 15@2x"
src="https://github.com/user-attachments/assets/230fbfb7-272d-4a2a-93e7-7268962a9814"
/>
### with Japanese wiki dump
<img width="2008" height="518" alt="CleanShot 2025-11-26 at 08 43 49@2x"
src="https://github.com/user-attachments/assets/edb408c8-a604-4a8f-bd5b-80f19e3d65ee"
/>
### with Arabic wiki dump
<img width="2010" height="512" alt="CleanShot 2025-11-26 at 08 44 25@2x"
src="https://github.com/user-attachments/assets/81a29ac8-0586-4e82-8276-d7fa90c31c90"
/>
TODO:
* [x] Take a closer look at the assembly and understand why this PR (8
KB vs 1 KB table) is faster on my machine.
* [x] _(**edit**: checking this off because it seems unnecessary)_ If
this turns out to actually be unacceptably slower, one possibility is to
switch to `uucode`'s `precomputedGraphemeBreak` which uses a 1445 byte
table since it uses a dense table (indexed using multiplication instead
of bitCast, though, which did show up in the initial benchmarks from
https://github.com/ghostty-org/ghostty/pull/8757 a small amount.)
AI was used in some of the uucode changes in
https://github.com/ghostty-org/ghostty/pull/9678 (Amp--primarily for
tests), but everything was carefully vetted and much of it done by hand.
This PR was made without AI with the exception of consulting AI about
whether the "Prepend + ASCII" scenario is common (hopefully it's right
about that being uncommon).
Add test coverage for bitmap font rendering using the Spleen 8x16 font
in three formats: BDF (Bitmap Distribution Format), PCF (Portable
Compiled Format), and OTB (OpenType Bitmap). Tests validate glyph
rendering against expected pixel patterns.
Addresses #8524.
These now properly match the FreeType API- compared directly in the unit
tests against the values provided by the FreeType header itself.
This was ridiculously wrong before, like... wow.
This change should give more consistent results between high and low DPI
displays, and generally approximate the authorial intent of the metrics
a little better.
Also changed the cell height adjustment to prioritize the top or bottom
when adjusting by an odd number depending on whether the face is higher
or lower in the cell than it "should" be. This should make it easier for
users who have an issue with a glyph protruding from the cell to adjust
the height and resolve it.
The code that re-creates the font descriptor from scratch using the same
attributes also rubs off the magic dust that makes CoreText not throw a
fit at us for using a "hidden" system font (name prefixed with a dot) by
name when we use the descriptor. This means that a small subset of chars
that only have glyphs in these fallback system fonts like ".CJK Symbols
Fallback HK Regular" and ".DecoType Nastaleeq Urdu UI" would not be able
to be rendered, since when we requested the font with the non-magical
descriptor CoreText would complain in the console and give us Times New
Roman instead.
Using `CTFontDescriptorCreateCopyWithAttributes` to clear the charset
attribute instead of recreating from scratch makes the copy come out
magical, and CoreText lets us instantiate the font from it, yippee!
### ℹ️ For anyone who came to this PR from a search engine rabbit hole
trying to figure this out for your own code:
It seems like if you want to use one of these fonts, you have to avoid
creating a new descriptor from scratch with the name. To create a
descriptor with one of these names that CoreText won't get upset about,
you have to get it in one of these ways (there may be others but these
are ones that definitely work at time of writing):
- Get it from `CTFontCreateForString`(`WithLanguage`) with a character
only provided by that font.
- Get it from `CTFontCreateUIFontForLanguage` or any of the mechanisms
for generally receiving a system font that you don't necessarily have
full control over the exact choice of font with.
- Get it from a collection created with
`CTFontCollectionCreateMatchingFontDescriptors`
- Get it from `CTFontCreateWithGraphicsFont` ... and it seems that for
now, `CGFontCreateWithFontName` lets you use these "hidden" names
without complaining :)
I wouldn't trust any of these methods to necessarily *continue* working
in to the future, since Apple seems to be trying to dissuade
*intentional* use of any particular named hidden system font.
Realistically, the `CreateForString` option is the most likely to stick
around.
If all else fails, check FireFox or Chrome's source code to see how
they're solving it in current year.
And to help it get indexed for those looking for it...
<details>
<summary>CoreText's angry message</summary>
```
CoreText note: Client requested name ".CJKSymbolsFallbackHK-Regular", it will get TimesNewRomanPSMT rather than the intended font. All system UI font access should be through proper APIs such as CTFontCreateUIFontForLanguage() or +[NSFont systemFontOfSize:].
CoreText note: Set a breakpoint on CTFontLogSystemFontNameRequest to debug.
```
</details>
Draw proper anti-aliased dots now instead of rectangles, thanks to z2d
this is very easy to do, and the results are very nice, no more weird
gaps in dotted underlines if your cell is the wrong number of pixels
across.
Use z2d to draw the undercurl instead of the manual raster code we had
before- the code was cool but unnecessarily complicated. Plus z2d lets
us have rounded caps on the undercurl which is neat.
Also make sure we won't draw off the canvas with our underlines-- the
canvas has padding but it's not infinite.
This change should give more consistent results between high and low DPI
displays, and generally approximate the authorial intent of the metrics
a little better.
Also changed the cell height adjustment to prioritize the top or bottom
when adjusting by an odd number depending on whether the face is higher
or lower in the cell than it "should" be. This should make it easier for
users who have an issue with a glyph protruding from the cell to adjust
the height and resolve it.
The code that re-creates the font descriptor from scratch using the same
attributes also rubs off the magic dust that makes CoreText not throw a
fit at us for using a "hidden" system font (name prefixed with a dot) by
name when we use the descriptor. This means that a small subset of chars
that only have glyphs in these fallback system fonts like ".CJK Symbols
Fallback HK Regular" and ".DecoType Nastaleeq Urdu UI" would not be able
to be rendered, since when we requested the font with the non-magical
descriptor CoreText would complain in the console and give us Times New
Roman instead.
Using `CTFontDescriptorCreateCopyWithAttributes` to clear the charset
attribute instead of recreating from scratch makes the copy come out
magical, and CoreText lets us instantiate the font from it, yippee!
A whole bunch of optimizations in hot paths in the IO processing areas
of our code (well, one of them covers everything). I validated that each
commit either improved one or more of our vtebench results, or improved
the time it takes to process 2 years worth (2.4GB) of data from
asciinema.
## vtebench
<img width="1278" height="903" alt="image"
src="https://github.com/user-attachments/assets/bad46777-4606-4870-b7d7-8df0c4bb3b39"
/>
(I decided to patch vtebench to report in nanoseconds instead of
milliseconds since clearly it was not designed for a machine as fast as
mine. Nanoseconds gives much more useful results when the numbers are
this low.)
Do note the *slight* regression in the "unicode" test, this is probably
because I added a branch hint in `Terminal.print` in order to optimize
for printing narrow characters, since they make up the vast majority of
characters typically printed in the terminal, but the vtebench "unicode"
test is pretty much all wide characters.
This shouldn't have a negative effect on users of CJK languages since
it's a *very* slight reduction in speed and they will still be printing
many narrow characters, especially in TUIs; spaces, box drawing
characters, symbols, punctuation, etc.
## asciinema processing
I wrote a program that uses libghostty to push 2 years worth (2.4GB) of
data from publicly uploaded asciinema recordings in to the terminal as
fast as possible- since it's just libghostty, there's no renderer
overhead happening, it's just the core terminal emulation, effectively
everything that io-reader thread does if it didn't have wait for the
renderer ever.
On main, this took roughly 26.1–26.7 seconds to process, on this branch
it takes just 18.4–18.6 seconds, that's a ~30% improvement in raw IO
processing speed when processing real world data!
## Summary of changes
In order of commits:
- Fixed a bug that I hit when trying to have Ghostty process all that
asciinema data, in certain bad cases it was possible to accidentally
insert the `0` hyperlink ID in to a page, which would then cause a
lockup in ReleaseFast mode when trying to clone that page since the
string alloc would try to iterate `1..0` to allocate 0 chunks.
- I noticed in profiling Ghostty that `std.debug.assert` was showing up
in the profile, which it should not have been since its doc comment
promises that it will be optimized out in ReleaseFast- but evidently
something is wrong with Zig, or that comment's promise is based on an
expectation from LLVM that it fails to meet - but either way, by
replacing all uses of `assert` with a version that is explicitly marked
`inline`, that function call overhead in tight loops and hotpaths is
avoided. This change alone accounts for like a third of the IO
processing time improvement, though it had minimal impact on vtebench
scores.
- I optimized the SGR parser somewhat by adding branch hints and
removing the `.reset_underline` action, replacing it with `.{ .underline
= .none }`.
- Gated a somewhat expensive assert in RefCountedSet behind a runtime
safety check.
- Improved the performance of `Style.eql` and `Style.hash` since these
are hot functions, called extremely frequently since adding styles to
the style set is a very common operation. Achieved this by making `eql`
less generic - explicitly comparing each part of the style rather than
looping over fields - and ordering checks from most likely to differ to
least likely to differ so that differences can be found as soon as
possible; and changed the hash from xxhash to simply folding the packed
struct down to 64 bits and then using `std.hash.int`. Also manually
inlined the code from `std.meta.activeTag` in `Packed.fromStyle`, since
profiling showed it in the callstack and it's a single cast so it really
should not have the function call overhead.
- Explicitly marked some trivial functions as inline, the optimizer
would already have been doing this (probably) but doing it explicitly
gives the optimizer more time to spend on other things. Added cold
branch hints to "should be impossible" and error-returning paths that
should be very rare, and unlikely branch hints to a lot of "invalid"
paths- to optimize for receiving valid data.
- Removed a branch in the parser csi param action, just unconditionally
multiply by 10 before adding digit value, even if it's the first digit.
This codepath is rarely hit since we have a fast path for this in the
stream code, but the stream code already has this optimization so I just
copied it over.
- `CharsetState.charsets` used to be an `EnumArray`, but the
layout/access logic for that was less-than-ideal, and the access
functions were not inlining-- and these are very hot since we access
this for every single print, so I wrote a bespoke struct to hold that
info instead, gained a couple percent of IO perf with that.
- Added branch hints based on the data I derived from the asciinema
dump, which gave big boost to vtebench results, especially for the
cursor movement and dense cells tests (which makes sense, since cursor
movement and setting attributes both got `likely` hints :p) -- data at
https://github.com/qwerasd205/asciinema-stats
- This is probably the most invasive change in this PR: I removed the
dirty bitset from `Page` and replaced it with a dirty flag on each row,
for the majority of operations this is faster to write, since the row
being dirtied is probably already loaded and probably will be written to
for other changes as well. This gave a couple percent IO processing
improvement. The only exception is scrolling-type operations, which are
extremely efficient by just moving rows around with a single memmov, so
looping through the rows to mark each dirty slows them down, and indeed
after this change the scrolling benchmarks in vtebench regressed,
*however*...
- Added a "full page dirty" flag on `Page`, which is set when an
operation is performed that dirties most or all the rows in the page,
which is used for scrolling-type operations. This *does* make the dirty
tracking slightly less precise for these operations, but with the
caching and stuff we do in the renderer, I don't think `rebuildCells` is
a bottleneck, so rebuilding a few extra rows shouldn't hurt. After this
change, all the scrolling benchmarks in vtebench improved drastically.
- Tiny micro-improvements to RefCountedSet; streamlined the control flow
in `lookup`, added an unlikely branch hint in `insert` for the branch
that resurrects dead items since dead items aren't that common.
- Improve SGR parser performance again by using `@call(.always_inline`
to explicitly inline calls to `StaticBitSet.isSet` (for the separator
list), since I noticed they weren't being inlined, causing function call
overhead in a hotpath.
- I noticed that `clearGrapheme` and `clearHyperlink` would check every
cell in the row after they were done in order to update the
`grapheme`/`hyperlink` flag on the row if there were none left, which
isn't great since `clearCells` called these functions for multiple cells
in the same row back-to-back, which leads to a ton of excess work. I
separated the flag updating parts of these functions out and called them
only if necessary (if the cells being cleared were the full row then the
flag could unconditionally be set to false) and only after all the cells
were cleared. This gave a nice improvement to IO processing since
clearCells is evidently a very hot function.
- Removed inline annotations on `Page.clearGrapheme` and
`Page.clearHyperlink` in favor of inlining directly at the one callsite
that benefited from inlining, this improved IO processing speed.
- Inlined trivial function `Charset.table`.
- Inlined `size.getOffset` and `size.intFromBase` as they are both
trivial pointer math that often benefits from surrounding context.
---
If you'd like me to separate out the trivial improvements (branch hints,
inline annotations, 1-line changes) from the functionality-changing ones
(pretty much just the changes to dirty tracking), just let me know!
As pointed out in #9156, an unintended consequence of all the work to
get icon sizing right is that `adjust-icon-height` now only applies to
the small icons you get when the next cell is not whitespace. Large
icons are unaffected.
With this PR, `adjust-icon-height` affects the maximum height of every
symbol specifying the `.icon` constraint height, regardless of
constraint width. This includes most Nerd Font icons, but excludes emoji
and other unicode symbols, and also excludes terminal graphics-oriented
Nerd Font symbols such as Powerline symbols.
In the following screenshots, **Baseline** is without
`adjust-icon-height`, while **Before** and **After** are with
`adjust-icon-height = -25%`.
**Baseline**
<img width="711" height="95" alt="Screenshot 2025-10-11 at 23 28 20"
src="https://github.com/user-attachments/assets/7499db4d-75a4-4dbd-b107-8cb5849e31a3"
/>
**Before** (only small icons affected)
<img width="711" height="95" alt="Screenshot 2025-10-11 at 23 20 12"
src="https://github.com/user-attachments/assets/9afd9fbf-ef25-44cc-9d8e-c39a69875163"
/>
**After** (both small and large icons affected, but not emoji)
<img width="711" height="95" alt="Screenshot 2025-10-11 at 23 21 05"
src="https://github.com/user-attachments/assets/90999f59-3b43-4684-9c8e-2c3c1edd6d18"
/>
Of course #9142 would require a minor follow-up!
* Scale groups can cut across patch sets, but not across fonts. We had
some scale group mixing between Font Awesome and the weather symbols,
which is removed by this PR.[^cp_table_full]
* There's one case where a scale group includes a glyph that's not part
of any patch sets, just for padding out the group bounding box.
Previously, an unrelated glyph from a different font would be pulled in.
Now we use an appropriate stand-in. (See code comment for details.)
* I noticed overlaps weren't being split between each side of the
bounding box, they were added to both sides, resulting in twice as much
padding as specified.
Screenshots showing the extra vertical padding for progress bar elements
due to the second bullet point:
**Before**
<img width="191" height="42" alt="Screenshot 2025-10-11 at 15 33 54"
src="https://github.com/user-attachments/assets/cf288cce-86d3-46fd-ae86-18e5c274b0e4"
/>
**After**
<img width="191" height="42" alt="Screenshot 2025-10-11 at 15 33 20"
src="https://github.com/user-attachments/assets/7ac799c7-bf50-4e65-a74a-f8a2c42d2441"
/>
[^cp_table_full]: Forming and using the merged `cp_table_full` table
should have been a red flag. Such a table doesn't make sense, it would
be a one-to-many map. You need the names of the original fonts to
disambiguate.
