Commit Graph

25 Commits

Author SHA1 Message Date
Feoramund
4b08e8cfff Support getting page sizes at runtime 2025-05-11 14:29:58 -04:00
Feoramund
78006adf67 Use AddressSanitizer with new heap allocator 2025-05-09 17:56:55 -04:00
Feoramund
b056b9e130 Rewrite the heap allocator
This is a total rewrite after having experimented with a variety of
techniques including a dual-allocator strategy for small and large size
classes where the large class allocator had a coalescing mechanism.

This allocator design is much closer to `mimalloc` in spirit and
performs almost twice as fast as the original bitmap-based design.

Additionally, several bugs have been fixed. The most important one being
the lock-free synchronization method used for remote frees. This design
uses a single atomic pointer that is doubly-tagged to pass remote frees
either to the heap or to the slab, depending on the ownership status.
2025-05-09 17:56:36 -04:00
Feoramund
2a33e1c1bb Forbid freeing of superpages that were not fully used once 2025-05-08 11:06:03 -04:00
Feoramund
94bfdd489f Use cache-friendly array to update next free slab index 2025-05-08 11:06:03 -04:00
Feoramund
920487be04 Zero only needed part of memory when changing size categories 2025-05-08 11:06:03 -04:00
Feoramund
92fdb4d79a Cache contiguous free slabs per heap superpage 2025-05-08 11:06:03 -04:00
Feoramund
e038c3cf4e Make superpage caching linear time instead of quadratic 2025-05-08 11:06:03 -04:00
Feoramund
dcd9bd7888 Make slab map caching linear time instead of quadratic 2025-05-08 11:06:03 -04:00
Feoramund
fa1f6abe78 Don't atomically subtract zero from remote_free_count 2025-05-08 11:06:03 -04:00
Feoramund
a0425ca220 Do not return superpages that had remote frees to the OS 2025-05-08 11:06:03 -04:00
Feoramund
acd5d0d40b Fix slab iteration bug in the remote free collection phase
Previously, a wide slab could be freed during this phase which could
trigger the superpage itself to be freed, resulting in iteration across
potentially invalid memory if the superpage was returned to the host and
not placed in the orphanage.

`heap_free_wide_slab` has had its responsibilities reduced to just
freeing the wide slab, and `heap_free_superpage_if_empty_and_unused` has
been added to better clarify what is happening where.
2025-05-08 11:06:02 -04:00
Feoramund
2d0547dcaf Remove unneeded call to min
At this point, we know we're expanding the wide slab, so `old_size` will
always be the right size.
2025-05-08 11:06:02 -04:00
Feoramund
44a7015609 Fix in-place expansion of wide slabs 2025-05-08 11:06:02 -04:00
Feoramund
8638bd3b89 Fix slab iteration bug in heap_cache_register_superpage
Increment of `i` is handled near the start of this loop.
2025-05-08 11:06:02 -04:00
Feoramund
ee3d859c6b Remove HEAP_PANIC_ON_FREE_NIL 2025-05-08 11:06:02 -04:00
Feoramund
247ce3cc97 Favor rescheduling superpage cache entry instead
This should be more robust during merges that happen simultaneously with
remote frees.

Previously, we would preserve the cache entry, but there is an edge case
where a slab of a lower index could be remotely freed and missed during
a merge.
2025-05-08 11:06:02 -04:00
Feoramund
c52c29856c Unify heap allocators
- Simplify the malloc-based allocator
- Correct the type signatures on malloc/calloc/realloc
- Make `heap_alloc/heap_free/heap_resize` API consistent across platforms
2025-05-08 11:06:02 -04:00
Feoramund
30e600699d Strengthen order to prevent reordering 2025-05-08 11:06:01 -04:00
Feoramund
541443fe33 Don't build feoramalloc on web platforms
Their allocation pathways are already tuned for their respective
platforms.
2025-05-08 11:06:01 -04:00
Feoramund
e9bf350cca Add heap allocator exception for Orca
This is a copy of the libc malloc-based allocator that we previously
had. It's preferable to use the Orca runtime allocator over our own.
2025-05-08 11:06:01 -04:00
Feoramund
1e62f2c8c4 Rearrange slab iteration conditionals 2025-05-08 11:06:01 -04:00
Feoramund
76a2877d7d Hoist reused calculations out into variables 2025-05-08 11:06:01 -04:00
Feoramund
17b38c1bca Strengthen Consume to Acquire in heap allocator 2025-05-08 11:06:01 -04:00
Feoramund
a4a776634c Add native heap allocator
- Add the test bench for the allocator
- Move old allocator code to the test bench
- Fix `heap_resize` usage in `os2/env_linux.odin` to fit new API
  requiring `old_size`
2025-05-08 11:06:00 -04:00