[Feature] Support deletion vectors for data-evolution tables

[SteNicholas]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Java Paimon supports deletion vectors on data-evolution tables since [core] Introducing DeletionVector mechanism for DataEvolution tables (apache/paimon#8380): the data-evolution.enabled × deletion-vectors.enabled mutual exclusion was dropped from SchemaValidation.validateRowTracking (data evolution now only conflicts with clustering.incremental), and the scan, read and compaction paths learned to apply deletion files to data-evolution splits.

paimon-cpp still carries the old constraint, both in schema validation and in the read path:

• SchemaValidation::ValidateRowTracking rejects the combination outright (src/paimon/core/schema/schema_validation.cpp:477-483).
• The same function requires data-evolution.enabled for any table holding a BLOB column (src/paimon/core/schema/schema_validation.cpp:501-504), so for BLOB tables the two checks form an unsatisfiable pair — a BLOB table can never enable deletion vectors.
• Even with the schema check relaxed, reading would still fail: DataEvolutionSplitRead::ApplyIndexAndDvReaderIfNeeded returns Invalid: DataEvolutionSplitRead do not support deletion vector (src/paimon/core/operation/data_evolution_split_read.cpp:334-336), as documented on src/paimon/core/operation/data_evolution_split_read.h:68.

Creating a table with an INT column and a BLOB column plus row-tracking.enabled=true, data-evolution.enabled=true and deletion-vectors.enabled=true therefore fails at create time with:

Invalid: Data evolution config must disabled with deletion-vectors.enabled

The current behavior is pinned by SchemaValidationTest.TestRowTracking (src/paimon/core/schema/schema_validation_test.cpp:773-782).

This blocks cross-language use of BLOB + deletion vector tables: a table created and written by a Java engine that includes apache/paimon#8380 can neither be created by nor read through paimon-cpp. It is also the follow-up already noted in #452 ("Java applies deletion vectors to placeholder gap readers; the C++ data-evolution blob read path does not wire deletion vectors yet (pre-existing)").

Solution

Port the Java design of apache/paimon#8380:

1. Schema validation — drop the data-evolution.enabled × deletion-vectors.enabled check in SchemaValidation::ValidateRowTracking and update SchemaValidationTest.TestRowTracking accordingly, keeping the other data-evolution constraints (data evolution requires row tracking, a BLOB table must have other normal columns, a BLOB column cannot be a partition key).
2. Scan and split plumbing — populate the deletion files of data-evolution splits so every field bunch knows which deletion file applies to which file. DataSplitImpl already carries DeletionFiles(); DataEvolutionFileStoreScan and the data-evolution split generation need to fill and align them, mirroring the Java changes in DataEvolutionFileStoreScan, DataSplit and the new DataEvolutionUtils.
3. Read path — wire the existing ApplyDeletionVectorBatchReader (src/paimon/core/deletionvectors/apply_deletion_vector_batch_reader.h) into DataEvolutionSplitRead::ApplyIndexAndDvReaderIfNeeded instead of returning Invalid, composing it with the row-range selection and with the blob layers of BlobFallbackBatchReader, mirroring Java's ApplyDeletionFileRecordIterator, BlobFallbackRecordReader and the new AllPlaceholdersRecordReader.
4. Write path — data-evolution tables are unaware-bucket (bucket = -1) append tables, while paimon-cpp only has BucketedDvMaintainer (src/paimon/core/deletionvectors/bucketed_dv_maintainer.h); producing and merging deletion files for these tables needs the equivalent of Java's AppendDeleteFileMaintainer. This step is only required for deleting from C++ — reading a Java-written DV table needs steps 1–3 only.

Coverage: schema validation tests for the accepted combination, DataEvolutionSplitRead tests for a data-evolution split whose files carry deletion files (including a blob bunch with multiple sequence layers), and a cross-language check that Java and C++ agree on the rows surviving the deletion vector of a BLOB table with row tracking, data evolution and deletion vectors enabled.

Anything else?

apache/paimon#8380 also updated data-evolution compaction (DataEvolutionCompactCoordinator, DataEvolutionCompactTask) to account for deletion files. paimon-cpp has no data-evolution compaction yet, so that part maps to a follow-up rather than to this issue.

Are you willing to submit a PR?

• I'm willing to submit a PR!