Skip to content

fix(parquet): reuse RowGroupPageIndexReader in FileReaderWrapper layer - #166

Open
zhf999 wants to merge 9 commits into
apache:mainfrom
zhf999:reuse-rowgroup
Open

fix(parquet): reuse RowGroupPageIndexReader in FileReaderWrapper layer#166
zhf999 wants to merge 9 commits into
apache:mainfrom
zhf999:reuse-rowgroup

Conversation

@zhf999

@zhf999 zhf999 commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Purpose

Arrow caches the file-level PageIndexReader (ParquetFileReader::GetPageIndexReader()), but
PageIndexReader::RowGroup(i) builds a brand-new RowGroupPageIndexReaderImpl on every call, and
the OffsetIndex/ColumnIndex byte buffer (offset_index_buffer_) is cached inside that instance.
So every new RowGroup(i) call re-issues a real ReadAt for the whole page-index region of the row
group. These reads are not covered by the data-page pre-buffer/range cache, so they are genuine
small I/Os against the underlying file.

Previously each stage of a page-filtered read created its own row-group reader, so for one
partially-matched row group the page index region was read up to three times:

  1. FileReaderWrapper::CollectPreBufferRanges() -> ComputePageRanges()
  2. FileReaderWrapper::NextPageFiltered() -> ComputePageRanges()
  3. PageFilteredRowGroupReader::ReadFilteredRowGroup() (for the per-field OffsetIndex lookups)

This PR makes the row-group page index reader an explicit, reusable input:

  • FileReaderWrapper gains GetRowGroupPageIndexReader(row_group_index) plus a
    row_group_page_index_readers_ map that memoizes one reader per row group (including the
    "no page index" negative result), so all stages share the same page-index buffers.
  • PageFilteredRowGroupReader::ComputePageRanges() and
    PageFilteredRowGroupReader::ReadFilteredRowGroup() now take the
    std::shared_ptr<::parquet::RowGroupPageIndexReader> from the caller instead of deriving it
    internally from ParquetFileReader::GetPageIndexReader().

Behavior is unchanged: files without a page index still pass a null reader and fall back to the
existing whole-column-chunk path; read results and computed ranges are identical. The only
difference is that the OffsetIndex region of each row group is read and deserialized once per file
reader instead of once per stage.

Tests

No.

API and Format

No changes.

Documentation

No changes.

Generative AI tooling

Generated-by: No.

std::shared_ptr<::arrow::MemoryPool> pool, ::parquet::arrow::FileReader* arrow_file_reader);
std::shared_ptr<::arrow::MemoryPool> pool,
std::shared_ptr<::parquet::RowGroupPageIndexReader> row_group_page_index_reader,
::parquet::arrow::FileReader* arrow_file_reader);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’m a bit curious whether row_group_page_index_reader and arrow_file_reader are both intended to be output parameters. If so, should we make them consistent and use shared_ptr for both before putting them into the pool?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since arrow_file_reader is a unique_ptr hold by FileReaderWrapper, it cannot be transfer into shared_ptr, we use raw pointer here.

The order of those parameter has been resorted, output parameter pool and arrow_file_reader have been just at the end of this function.

/// Calculate row ranges based on predicate and column indices.
/// @param predicate The predicate to evaluate.
/// @param page_index_reader The page index reader for the file.
/// @param column_name_to_index Map from column name to column index.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please adjust comments.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@zhf999
zhf999 requested a review from lxy-9602 August 3, 2026 09:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants