Skip to content

feat: add owned typed arrays - #68

Open
GrapeBaBa wants to merge 9 commits into
mainfrom
fix/external-typed-array-lifecycle
Open

feat: add owned typed arrays#68
GrapeBaBa wants to merge 9 commits into
mainfrom
fix/external-typed-array-lifecycle

Conversation

@GrapeBaBa

@GrapeBaBa GrapeBaBa commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add js.OwnedTypedArray(Element, array_type) and concrete Owned*Array aliases
  • transfer allocator-owned element storage to an external ArrayBuffer without copying
  • support owned typed arrays as return values from DSL functions and class methods
  • make ownership transfer transactional: pre-transfer failures preserve the Zig owner; successful or potentially transferred values leave the source empty
  • align OwnedBuffer.intoValue with the same pointer-based transactional ownership model while retaining its existing unsupported-runtime copy fallback
  • validate each Element against its N-API TypedArray kind at compile time
  • keep TypedArray.fromExternal unchanged from main
  • propagate NoExternalBuffersAllowed unchanged for owned typed arrays; no copy fallback

API

pub fn serialize() !js.OwnedUint8Array {
    const allocator = js.allocator();
    const data = try allocator.alloc(u8, 32);
    // Fill data...
    return js.OwnedUint8Array.fromOwnedSlice(allocator, data);
}

fromOwnedSlice takes an existing allocator-owned mutable slice without copying. fromSlice creates an owned copy when that is what the caller needs.

intoValue takes *OwnedTypedArray. Once ownership transfers, it empties the source, so a normal deferred deinit is safe. The allocator must remain valid until the ArrayBuffer finalizer runs.

OwnedBuffer.intoValue now follows the same pointer-based transfer contract. It retains its existing NoExternalBuffersAllowed copy fallback and consumes the source only after that copy succeeds.

Lodestar-Z use case

In the Lodestar-Z BLS bindings, compress and serialize currently produce fixed-size stack arrays and pass them to TypedArray.fromExternal, which copies them into allocator-owned native storage before creating the external ArrayBuffer. With this API, the bindings can allocate the output slice on the native heap, let blst write into it directly, and return it with OwnedUint8Array.fromOwnedSlice. This removes one stack-to-native-heap copy per result while preserving the external ArrayBuffer GC lifecycle.

Ownership boundary

  • zero length: create a normal zero-length V8 ArrayBuffer and empty the source only after TypedArray creation succeeds
  • owner heap allocation failure: leave the allocation owned by the caller
  • NoExternalBuffersAllowed, PendingException, or CannotRunJS: Node returns before finalizer registration, so restore ownership to the caller and propagate the original error
  • other external ArrayBuffer errors: leave the source empty because Node may already have installed or invoked the finalizer
  • typed-array view creation failure or success: leave the source empty; the external ArrayBuffer finalizer owns the allocation
  • DSL return conversion: defer deinit on the local owner, which frees restored/untransferred data and is a no-op after transfer

The heap finalizer context is the owning value itself, matching the boxed-owner model rather than duplicating ownership fields in a second context type.

Node implementation reference for the ambiguous failure boundary:

Verification

  • zig build test:zapi — 37/37
  • zig build
  • pnpm test:js — 121/121
  • pnpm lint:js
  • zig fmt --check src/OwnedBuffer.zig src/to_from_value.zig src/js/typed_arrays.zig src/js/wrap_function.zig
  • git diff --check
  • independent Zig ownership audit — PASS, 0 MUST / 0 SHOULD / 0 NIT

Spec tests were intentionally not run.

The repository-wide zig fmt --check src still reports the pre-existing formatting issue in src/create_callback.zig; this PR does not modify that file.

Coverage limitation

The JS tests exercise non-empty and zero-length OwnedUint8Array values and a non-empty OwnedBuffer through the real Node addon boundary. They do not deterministically force GC to count typed-array finalizer invocations or inject NoExternalBuffersAllowed, pending/cannot-run, generic external-creation, typed-array-view creation, or fallback-copy failures. No production test seam or compatibility branch was added for those cases.

@GrapeBaBa GrapeBaBa changed the title fix: preserve external typed array ownership feat: add owned typed arrays Jul 30, 2026
@matthewkeil matthewkeil moved this to In Progress in Lodestar Team Coordination Jul 30, 2026
@GrapeBaBa
GrapeBaBa force-pushed the fix/external-typed-array-lifecycle branch from cf8a1fd to f1b7b72 Compare August 10, 2026 06:35
@GrapeBaBa
GrapeBaBa marked this pull request as ready for review August 10, 2026 10:01
@GrapeBaBa
GrapeBaBa requested review from nazarhussain, spiral-ladder and wemeetagain and removed request for spiral-ladder August 10, 2026 10:03

@spiral-ladder spiral-ladder left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

with this API, isn't this strictly better than fromExternal? Is there any reason to keep the original API?

@nazarhussain nazarhussain left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Over all looks good. One API asymmetry.

We have OwnedBuffer.intoValue(self: OwnedBuffer, env) vs OwnedTypedArray.intoValue(self: *Self, env) .

The later pattern introduced in this PR seems better approach. Suggest to update the the existing OwnedBuffer to follow the same.

nazarhussain
nazarhussain previously approved these changes Aug 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

4 participants