[SPARK-58500][PYTHON] Accept tuple in DataFrame.describe and selectExpr - #57702
Draft
Spenserrrr wants to merge 1 commit into
Draft
[SPARK-58500][PYTHON] Accept tuple in DataFrame.describe and selectExpr#57702Spenserrrr wants to merge 1 commit into
Spenserrrr wants to merge 1 commit into
Conversation
Widen the ``describe`` and ``selectExpr`` type annotations from ``Union[str, List[str]]`` to ``Union[str, Sequence[str]]`` (overloads + impl) across the base, classic, and connect layers, and change the runtime unwrap check from ``isinstance(x[0], list)`` to a Sequence check so a single tuple of column names / expressions is unpacked like a list. Normalizes the overload shape (describe had none; connect selectExpr had none) and drops the ``# type: ignore[assignment]`` on the unwrap.
uros-b
approved these changes
Aug 2, 2026
Member
|
Thank you @Spenserrrr! cc @Yicong-Huang |
Spenserrrr
marked this pull request as draft
August 2, 2026 22:01
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
DataFrame.describeandDataFrame.selectExpraccept their column names / SQL expressions as varargs, and also allow a single sequence to be passed in place of the varargs (e.g.df.describe(["a", "b"])). Today that single-sequence form is unwrapped only when it is alist— the runtime check isisinstance(x[0], list)— so passing a tuple such asdf.describe(("a", "b"))is not unwrapped and is instead treated as a single, invalid column argument.This PR:
describeandselectExprtype annotations fromUnion[str, List[str]]toUnion[str, Sequence[str]](overloads + implementation) across the base (sql/dataframe.py), classic (sql/classic/dataframe.py), and connect (sql/connect/dataframe.py) layers.isinstance(x[0], list)tonot isinstance(x[0], str) and isinstance(x[0], Sequence), and unwraps withx = tuple(x[0]), so a single tuple is unpacked the same way a list is.describehad no overloads in any layer (added the 2-overload shape*x: str/__x: Sequence[str]);selectExprhad a*expr: List[str]overload in base + classic (fixed to a single-arg__expr: Sequence[str]) and none in connect (added). Drops the now-unnecessary# type: ignore[assignment]on the unwrap.This is a follow-up to the SPARK-58488 cleanup (#57693). Unlike the writer methods (
partitionBy/clusterBy/bucketBy/sortBy), which already accepted a list or a tuple,describe/selectExpraccepted alistonly, so this adds tuple/sequence support.Why are the changes needed?
The runtime already intends to accept "a name or a sequence of names", but the check and the annotation only admitted
list. Passing a tuple silently did the wrong thing (treated the whole tuple as one column/expression) rather than being unpacked. Accepting any non-strSequencemakes the behavior consistent with the writer methods and with the general varargs-or-sequence convention in the DataFrame API, and makes the type annotation honest about what is accepted.Does this PR introduce any user-facing change?
Yes. Previously
df.describe(("a", "b"))/df.selectExpr(("e1", "e2"))(a single tuple) was not unpacked and did not behave like the list form; now a single tuple of column names / expressions is accepted and unpacked just like a list. This is a backward-compatible addition — the existing str-varargs and single-list forms are unchanged.How was this patch tested?
test_connect_stat.py::test_describeandtest_connect_basic.py::test_select_expr(these assert the same call on the classic and connect sessions produce equal results, so they cover both layers). Both pass.mypyoverpython/pysparkis clean.ruff format --checkandruff checkpass on the changed files.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)