Skip to content

fix(python): make dataframe index opt-in - #919

Open
ColinLeeo wants to merge 2 commits into
developfrom
fix/dataframe-index-opt-in
Open

fix(python): make dataframe index opt-in#919
ColinLeeo wants to merge 2 commits into
developfrom
fix/dataframe-index-opt-in

Conversation

@ColinLeeo

Copy link
Copy Markdown
Contributor

Description

This PR makes the Python TsFileDataFrame persistent dataset index opt-in.

By default, TsFileDataFrame(..., use_index=False) now keeps the previous direct metadata loading behavior and does not create or reuse any .tsidx file. Users can explicitly enable persistent indexing with use_index=True; when enabled, the dataframe will reuse a matching index if present, or build one when the matching index is missing or stale.

The persistent index file name is now scoped by the expanded canonical TsFile set, so different subsets under the same directory no longer contend for the same .tsidx file.

Changes

  • Add use_index: bool = False to TsFileDataFrame.
  • Preserve direct scan behavior when indexing is disabled.
  • Use a hash of the expanded absolute file set in the persistent index file name.
  • Keep index validation based on file fingerprints before reuse.
  • Add tests for default no-index behavior.
  • Add tests for same-directory different-file-subset index isolation.
  • Update runtime/index-specific tests to pass use_index=True.

Validation

  • ./mvnw -P with-cpp -pl cpp -DskipTests package
  • ./mvnw -P with-python -pl python test
    • 200 passed

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 61.26%. Comparing base (b4b40c0) to head (7b9aa9a).

Additional details and impacted files
@@           Coverage Diff            @@
##           develop     #919   +/-   ##
========================================
  Coverage    61.26%   61.26%           
========================================
  Files          746      746           
  Lines        51560    51560           
  Branches      8353     8353           
========================================
  Hits         31586    31586           
  Misses       18359    18359           
  Partials      1615     1615           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants