Skip to content

fix(outputs): size shared-memory wires from promoted dtype - #507

Open
harsh839 wants to merge 1 commit into
JdeRobot:masterfrom
harsh839:fix/outputs-share-dtype-bugs
Open

harsh839 wants to merge 1 commit into
JdeRobot:masterfrom
harsh839:fix/outputs-share-dtype-bugs

Conversation

@harsh839

Copy link
Copy Markdown

Fixes #483 and #484.

Outputs.share() decided how much shared memory the shape/dim/data wires
need before deciding what dtype it would store there, and the two did
not agree:

  • In Outputs.share() fails on Windows: shape/dim wires are sized as int32 but always read as int64 #483 (Windows): shape and dim are built from
    platform-default integers (np.array(data.shape)int32 on
    Windows
    ), but the buffers are always read back as np.int64 in
    outputs.py and inputs.py. The wire gets 4 bytes and numpy is asked
    for an 8-byte view, so every share() call fails with
    TypeError: buffer is too small for requested array.
  • In Outputs.share() under-allocates the data wire whenever check_type() promotes the dtype #484 (.py lines 46-48): the data wire is allocated from the
    un-promoted data.nbytes, but the buffer is then viewed with the
    check_type()-promoted dtype (int32→int64, float32→float64,
    U/SU64). Whenever promotion increases the itemsize past both the
    input nbytes and the 256-byte floor, the view is larger than the
    buffer (e.g. a float32 array straight out of a model, or a list of
    class labels), and numpy refuses it.
  • Scalar payloads (share("Out", 5), share("Out", "hello")) produced a
    0-sized shape wire, which SharedMemory rejects, and then a 0-d
    write through [:], which numpy rejects.

Changes

Verification

Tested with a real two-process round-trip (writer block process →
reader block process, the architecture the runtime uses) covering
float32, float16, int32, uint8, float64, 2-D arrays, lists of
strings (<U64 promotion), and both scalar int/string payloads. All
share/read round-trips now pass on Linux; the shape/dim fixes are the
exact dtypes the reader already expects, which is what makes Windows
work. Previously #484 fails deterministically on Linux for any payload
larger than 256 bytes whose dtype gets promoted.

Note: this branch intentionally leaves formatting untouched so the PR
stays a focused logic fix; the formatting-only PR (#444) on top of
master may need a one-line rebase once this merges.

Outputs.share() decided how much shared memory the shape/dim/data wires
need before deciding what dtype it would store there, and the two did
not agree:

- shape and dim wires were built from platform-default ints (int32 on
  Windows) but read back as np.int64, so every share() call failed on
  Windows with 'buffer is too small'.
- the data wire was allocated from the un-promoted nbytes, but the
  buffer is later viewed with the check_type()-promoted dtype, so any
  payload whose promoted itemsize exceeds both the input nbytes and the
  256-byte floor failed on every platform (e.g. float32/model outputs,
  lists of strings).
- scalar payloads (np.array(5), 'hello') produced a 0-sized shape wire,
  which SharedMemory rejects, and a 0-d write via [:] which numpy
  rejects.

Compute the wire sizes from the promoted dtype with an int64-explicit
shape/dim, floor the shape wire at one int64 so scalars are creatable,
and write via ... indexing so 0-d (scalar) and N-d payloads both marshal
correctly.
@harsh839

Copy link
Copy Markdown
Author

@jmplaza could you take a look when you get a chance? This sizes the shared-memory output wires using the promoted dtype so promoted types (e.g. float32->float64) don't overflow their buffers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Outputs.share() fails on Windows: shape/dim wires are sized as int32 but always read as int64

1 participant