deps: upgrade to DataFusion 55.0 and Arrow/Parquet 59.2

[mbutrovich]

Which issue does this PR close?

• Closes #.

What changes are included in this PR?

• Update cargo dependencies to datafusion 55.0, arrow/parquet 59.2.
• API adjustments based on the upgrade guide https://datafusion.apache.org/library-user-guide/upgrading/55.0.0.html

Are these changes tested?

Existing tests.

AI Disclosure

Claude helped with some mechanical stuff as this was in development (bumping WIP branches and commits, API changes) but I understand and support the changes.

[mbutrovich]

I will get this cleaned up and ready for review today now that crates are available.

[gabotechs]

👍 Everything looks good here, just the expected mechanical changes from the datfusion@55 upgrade.

[shanielh]

@mbutrovich Care to rebase this so this can be merged? :)

[dannycjones]

@CTTY would like to see the 0.11 release branch be cut ahead of this version bump, with the reasoning being that DataFusion 54.1 can be considered stable at this point and consumers may benefit from that.
#3034 (comment)

Please can you confirm you're aligned, and we can go ahead with cutting that branch before merging this?

[dannycjones]

That doesn't stop the rebase of course :)

[CTTY]

Yes, I'm leaning toward releasing 0.11 before bumping DataFusion to v55

[mbutrovich]

@CTTY would like to see the 0.11 release branch be cut ahead of this version bump, with the reasoning being that DataFusion 54.1 can be considered stable at this point and consumers may benefit from that. #3034 (comment)

Please can you confirm you're aligned, and we can go ahead with cutting that branch before merging this?

No concerns from me. Seems like 0.11 will happen pretty quickly. Note that I'm about to head on vacation until after Labor Day (US) so would need a committer to keep resolving the merge conflicts as they arise.

[dannycjones]

Given we've now cut the 0.11.x branch, I think we're good to move forward here towards merging into main.

[kevinjqliu]

LGTM!

[kevinjqliu]

@dannycjones should we wait until RC passes before merging?

[dannycjones]

@dannycjones should we wait until RC passes before merging?

I think we're OK to go ahead merging. I am planning to backport anything to branch 0.11.x if there are changes required.

[kevinjqliu]

thanks @mbutrovich and everyone for the review!