AVRO-4329: [Java] Bound bytes/string allocation on non-seekable streams

[iemejia]

What is the purpose of the change

Java SDK implementation of AVRO-4303 (parent). The available-bytes guard added
under AVRO-4292 rejects a declared bytes/string length that exceeds the data
remaining only when the decoder can report the number of bytes remaining
(memory-backed or seekable sources). On a non-seekable stream (socket, pipe,
decompression stream) the check is a no-op, so a huge declared length still
drives a single large up-front allocation before any payload is read. A tiny
truncated input can therefore force a multi-hundred-MB allocation.

When the remaining byte count is unknown, this reads the bytes/string value into
a buffer that grows in bounded chunks rather than allocating the full
attacker-declared length up front. A truncated or hostile stream then fails with
a bounded EOFException after a bounded allocation instead of an
OutOfMemoryError. The existing single-allocation fast path is kept when the
remaining byte count is known. The same fix is applied to DirectBinaryDecoder,
which reads straight from a stream.

Verifying this change

This change added tests and can be verified as follows:

• Added TestBinaryDecoderBoundedRead: a near-2GB declared bytes/string length
  on a truncated non-seekable stream fails with a bounded EOFException (not an
  OutOfMemoryError) for both the buffered BinaryDecoder and
  DirectBinaryDecoder; a legitimately large value on a non-seekable stream
  still round-trips; small values and seekable sources keep the direct path.
• Existing TestBinaryDecoder (68 tests) continues to pass.

Documentation

• Does this pull request introduce a new feature? (no — DoS hardening)
• If yes, how is the feature documented? (JavaDocs on the new bounded-read
  helper in BinaryDecoder)

[Copilot]

Pull request overview

Hardens the Java Avro binary decoders against DoS-by-allocation when reading length‑prefixed bytes/string values from non-seekable streams by avoiding single up-front allocations based on attacker-declared lengths, while keeping the existing fast path for sources with known remaining-byte counts.

Changes:

• Add bounded, chunk-growing read path in BinaryDecoder for large declared bytes/string lengths when remaining bytes are unknown.
• Add equivalent bounded allocation behavior in DirectBinaryDecoder for large bytes reads from streaming sources.
• Add regression tests covering huge declared lengths on non-seekable streams and round-trip behavior for legitimately large values.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
lang/java/avro/src/main/java/org/apache/avro/io/BinaryDecoder.java	Adds bounded growing-buffer helper and routes large bytes/string reads on unknown-length sources through it.
lang/java/avro/src/main/java/org/apache/avro/io/DirectBinaryDecoder.java	Avoids large up-front allocations for large declared bytes lengths when reading directly from streams.
lang/java/avro/src/test/java/org/apache/avro/io/TestBinaryDecoderBoundedRead.java	Adds regression coverage for huge declared lengths on non-seekable streams plus round-trip tests for large payloads.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (2)

lang/java/avro/src/test/java/org/apache/avro/io/TestBinaryDecoderBoundedRead.java:75

• This comment claims that seeing an EOFException “proves” the decoder didn’t attempt the full up-front allocation, but the same EOFException could still occur even if a large allocation succeeded (e.g., on a large-heap CI runner). This makes the intent of the test misleading; it’s really relying on typical heap limits to make an up-front allocation fail.

  /**
   * A near-2GB declared length that a single {@code new byte[len]} could not
   * satisfy on a normal test heap, so reaching an {@link EOFException} proves the
   * decoder never attempted the full up-front allocation.
   */

lang/java/avro/src/main/java/org/apache/avro/io/BinaryDecoder.java:659

• The Javadoc says the length “must be positive”, but the implementation correctly handles a zero length (it returns an empty array without reading). Either validate and reject 0, or update the contract to match the actual behavior.

   * @param length the number of bytes to read; must be positive