fix(parquet): reuse RowGroupPageIndexReader in FileReaderWrapper layer - #166
Open
zhf999 wants to merge 9 commits into
Open
fix(parquet): reuse RowGroupPageIndexReader in FileReaderWrapper layer#166zhf999 wants to merge 9 commits into
zhf999 wants to merge 9 commits into
Conversation
lxy-9602
reviewed
Aug 3, 2026
| std::shared_ptr<::arrow::MemoryPool> pool, ::parquet::arrow::FileReader* arrow_file_reader); | ||
| std::shared_ptr<::arrow::MemoryPool> pool, | ||
| std::shared_ptr<::parquet::RowGroupPageIndexReader> row_group_page_index_reader, | ||
| ::parquet::arrow::FileReader* arrow_file_reader); |
Contributor
There was a problem hiding this comment.
I’m a bit curious whether row_group_page_index_reader and arrow_file_reader are both intended to be output parameters. If so, should we make them consistent and use shared_ptr for both before putting them into the pool?
Contributor
Author
There was a problem hiding this comment.
Since arrow_file_reader is a unique_ptr hold by FileReaderWrapper, it cannot be transfer into shared_ptr, we use raw pointer here.
The order of those parameter has been resorted, output parameter pool and arrow_file_reader have been just at the end of this function.
| /// Calculate row ranges based on predicate and column indices. | ||
| /// @param predicate The predicate to evaluate. | ||
| /// @param page_index_reader The page index reader for the file. | ||
| /// @param column_name_to_index Map from column name to column index. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Arrow caches the file-level
PageIndexReader(ParquetFileReader::GetPageIndexReader()), butPageIndexReader::RowGroup(i)builds a brand-newRowGroupPageIndexReaderImplon every call, andthe OffsetIndex/ColumnIndex byte buffer (
offset_index_buffer_) is cached inside that instance.So every new
RowGroup(i)call re-issues a realReadAtfor the whole page-index region of the rowgroup. These reads are not covered by the data-page pre-buffer/range cache, so they are genuine
small I/Os against the underlying file.
Previously each stage of a page-filtered read created its own row-group reader, so for one
partially-matched row group the page index region was read up to three times:
FileReaderWrapper::CollectPreBufferRanges()->ComputePageRanges()FileReaderWrapper::NextPageFiltered()->ComputePageRanges()PageFilteredRowGroupReader::ReadFilteredRowGroup()(for the per-fieldOffsetIndexlookups)This PR makes the row-group page index reader an explicit, reusable input:
FileReaderWrappergainsGetRowGroupPageIndexReader(row_group_index)plus arow_group_page_index_readers_map that memoizes one reader per row group (including the"no page index" negative result), so all stages share the same page-index buffers.
PageFilteredRowGroupReader::ComputePageRanges()andPageFilteredRowGroupReader::ReadFilteredRowGroup()now take thestd::shared_ptr<::parquet::RowGroupPageIndexReader>from the caller instead of deriving itinternally from
ParquetFileReader::GetPageIndexReader().Behavior is unchanged: files without a page index still pass a null reader and fall back to the
existing whole-column-chunk path; read results and computed ranges are identical. The only
difference is that the OffsetIndex region of each row group is read and deserialized once per file
reader instead of once per stage.
Tests
No.
API and Format
No changes.
Documentation
No changes.
Generative AI tooling
Generated-by: No.