Skip to content

[SPARK-58500][PYTHON] Accept tuple in DataFrame.describe and selectExpr - #57702

Draft
Spenserrrr wants to merge 1 commit into
apache:masterfrom
Spenserrrr:list-widening
Draft

[SPARK-58500][PYTHON] Accept tuple in DataFrame.describe and selectExpr#57702
Spenserrrr wants to merge 1 commit into
apache:masterfrom
Spenserrrr:list-widening

Conversation

@Spenserrrr

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

DataFrame.describe and DataFrame.selectExpr accept their column names / SQL expressions as varargs, and also allow a single sequence to be passed in place of the varargs (e.g. df.describe(["a", "b"])). Today that single-sequence form is unwrapped only when it is a list — the runtime check is isinstance(x[0], list) — so passing a tuple such as df.describe(("a", "b")) is not unwrapped and is instead treated as a single, invalid column argument.

This PR:

  • Widens the describe and selectExpr type annotations from Union[str, List[str]] to Union[str, Sequence[str]] (overloads + implementation) across the base (sql/dataframe.py), classic (sql/classic/dataframe.py), and connect (sql/connect/dataframe.py) layers.
  • Changes the runtime unwrap check from isinstance(x[0], list) to not isinstance(x[0], str) and isinstance(x[0], Sequence), and unwraps with x = tuple(x[0]), so a single tuple is unpacked the same way a list is.
  • Normalizes the overload shape: describe had no overloads in any layer (added the 2-overload shape *x: str / __x: Sequence[str]); selectExpr had a *expr: List[str] overload in base + classic (fixed to a single-arg __expr: Sequence[str]) and none in connect (added). Drops the now-unnecessary # type: ignore[assignment] on the unwrap.

This is a follow-up to the SPARK-58488 cleanup (#57693). Unlike the writer methods (partitionBy/clusterBy/bucketBy/sortBy), which already accepted a list or a tuple, describe/selectExpr accepted a list only, so this adds tuple/sequence support.

Why are the changes needed?

The runtime already intends to accept "a name or a sequence of names", but the check and the annotation only admitted list. Passing a tuple silently did the wrong thing (treated the whole tuple as one column/expression) rather than being unpacked. Accepting any non-str Sequence makes the behavior consistent with the writer methods and with the general varargs-or-sequence convention in the DataFrame API, and makes the type annotation honest about what is accepted.

Does this PR introduce any user-facing change?

Yes. Previously df.describe(("a", "b")) / df.selectExpr(("e1", "e2")) (a single tuple) was not unpacked and did not behave like the list form; now a single tuple of column names / expressions is accepted and unpacked just like a list. This is a backward-compatible addition — the existing str-varargs and single-list forms are unchanged.

How was this patch tested?

  • Added tuple-form cases to the existing connect-vs-classic parity tests test_connect_stat.py::test_describe and test_connect_basic.py::test_select_expr (these assert the same call on the classic and connect sessions produce equal results, so they cover both layers). Both pass.
  • Full-scope mypy over python/pyspark is clean.
  • ruff format --check and ruff check pass on the changed files.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 4.8)

Widen the ``describe`` and ``selectExpr`` type annotations from
``Union[str, List[str]]`` to ``Union[str, Sequence[str]]`` (overloads + impl)
across the base, classic, and connect layers, and change the runtime unwrap
check from ``isinstance(x[0], list)`` to a Sequence check so a single tuple of
column names / expressions is unpacked like a list. Normalizes the overload
shape (describe had none; connect selectExpr had none) and drops the
``# type: ignore[assignment]`` on the unwrap.
@uros-b

uros-b commented Aug 2, 2026

Copy link
Copy Markdown
Member

Thank you @Spenserrrr! cc @Yicong-Huang

@uros-b
uros-b requested a review from Yicong-Huang August 2, 2026 16:38
@Spenserrrr
Spenserrrr marked this pull request as draft August 2, 2026 22:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants