Add deprecation warnings for Expr passed to confirmed literal-only function arguments#1605
Open
kosiew wants to merge 2 commits into
Open
Add deprecation warnings for Expr passed to confirmed literal-only function arguments#1605kosiew wants to merge 2 commits into
kosiew wants to merge 2 commits into
Conversation
- Introduced shared `_warn_if_expr_for_literal_arg` in `functions/__init__.py` - Added `DeprecationWarning` for the following methods when `Expr` is passed as argument: - `encode(..., encoding=Expr)` - `decode(..., encoding=Expr)` - `digest(..., method=Expr)` - `arrow_cast(..., data_type=Expr)` - `arrow_try_cast(..., data_type=Expr)` - `arrow_metadata(..., key=Expr)` test: update tests to check for warnings - Implemented tests in `test_functions.py` to ensure: - Warning is raised for `Expr` form - No warning for native literal form
nuno-faria
approved these changes
Jun 19, 2026
nuno-faria
left a comment
nuno-faria
left a comment
Contributor
There was a problem hiding this comment.
Thanks @kosiew. In addition to the comments below, it's better to update the arrow_cast calls at user-guide/common-operations/functions.md which are still using expressions.
Comment on lines
67
to
+82
| def _warn_expr_for_literal_arg(function_name: str, arg_name: str) -> None: | ||
| warnings.warn( | ||
| f"Passing Expr for {function_name}() argument {arg_name!r} is deprecated; " | ||
| "pass a Python literal instead.", | ||
| DeprecationWarning, | ||
| stacklevel=4, | ||
| ) | ||
|
|
||
|
|
||
| def _warn_if_expr_for_literal_arg( | ||
| value: Any, function_name: str, arg_name: str | ||
| ) -> None: | ||
| if isinstance(value, Expr): | ||
| _warn_expr_for_literal_arg(function_name, arg_name) | ||
|
|
||
|
|
Contributor
There was a problem hiding this comment.
Wouldn't it be better to combine these two into a single function?
Comment on lines
1603
to
1619
| def test_binary_string_functions(df): | ||
| df = df.select( | ||
| f.encode(column("a").cast(pa.string()), literal("base64").cast(pa.string())), | ||
| f.decode( | ||
| f.encode( | ||
| column("a").cast(pa.string()), literal("base64").cast(pa.string()) | ||
| ), | ||
| literal("base64").cast(pa.string()), | ||
| ), | ||
| ) | ||
| result = df.collect() | ||
| assert len(result) == 1 | ||
| result = result[0] | ||
| assert result.column(0) == pa.array(["SGVsbG8", "V29ybGQ", "IQ"]) | ||
| assert pa.array(result.column(1)).cast(pa.string()) == pa.array( | ||
| ["Hello", "World", "!"] | ||
| ) |
Contributor
There was a problem hiding this comment.
The calls to encode and decode need to be updated to not raise warnings.
| assert result.column(0) == expected_result | ||
|
|
||
|
|
||
| def test_hash_functions(df): |
Contributor
There was a problem hiding this comment.
The calls here also need to be updated.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
As part of the audit of
Expr | literalfunction arguments, this change targets a set of arguments that act as control/configuration parameters rather than row-varying data expressions. These arguments are intended to be provided as Python literals (for example, encoding names, digest methods, data type specifiers, and metadata keys).To prepare for future API cleanup while preserving compatibility, passing an
Exprto these arguments now emits aDeprecationWarningbut continues to work.What changes are included in this PR?
Added a reusable helper,
_warn_if_expr_for_literal_arg, that emits a deprecation warning when anExpris supplied to a confirmed literal-only argument.Added deprecation warnings for the following function arguments:
encode(..., encoding=...)decode(..., encoding=...)digest(..., method=...)arrow_cast(..., data_type=...)arrow_try_cast(..., data_type=...)arrow_metadata(..., key=...)Preserved existing behavior by continuing to coerce these values to expressions after warning.
Added tests covering both warning and non-warning cases.
Are these changes tested?
Yes.
Added tests in
python/tests/test_functions.pythat verify:Passing
literal(...)/Exprvalues to the audited literal-only arguments emits aDeprecationWarning.Passing native Python literal values does not emit a
DeprecationWarning.Coverage includes:
encodedecodedigestarrow_castarrow_try_castarrow_metadataAre there any user-facing changes?
Yes.
Users will now receive a
DeprecationWarningwhen passing anExprto the following literal-only arguments:encode(..., encoding=...)decode(..., encoding=...)digest(..., method=...)arrow_cast(..., data_type=...)arrow_try_cast(..., data_type=...)arrow_metadata(..., key=...)Existing behavior is otherwise unchanged, and Python literal inputs remain fully supported without warnings.
LLM-generated code disclosure
This PR includes code, comments generated with assistance from LLM. All LLM-generated content has been manually reviewed.