Skip to content

Fix TimeDeltaParameter serialization of negative and sub second values - #3474

Open
sebxt wants to merge 1 commit into
spotify:masterfrom
sebxt:fix/timedeltaparameter-negative-subsecond-serialize
Open

sebxt wants to merge 1 commit into
spotify:masterfrom
sebxt:fix/timedeltaparameter-negative-subsecond-serialize

Conversation

@sebxt

@sebxt sebxt commented Oct 18, 2026

Copy link
Copy Markdown

Description

TimeDeltaParameter.serialize now falls back to a bare number of seconds (str(x.total_seconds())) when the value is negative or has a microsecond part. parse already accepts that form through its float() branch. Non negative whole second values keep the existing "n w n d n h n m n s" output, so their task ids are unchanged.

Motivation and Context

The "w d h m s" form cannot express these values:

  • timedelta(hours=-1) serializes to -1 w 6 d 23 h 0 m 0 s, which parse rejects. A task with such a parameter yielded as a dynamic dependency makes luigi.build raise ParameterException: Invalid time delta - could not parse -1 w 6 d 23 h 0 m 0 s, because the worker rebuilds the task from to_str_params().
  • timedelta(milliseconds=200) and timedelta(milliseconds=700) both serialize to 0 w 0 d 0 h 0 m 0 s, so tasks that differ only in that parameter share a task_id.

This is the part of #1966 that #1968 did not cover.

The check reads microseconds with getattr so duck typed values, as used in test_warn_on_non_timedelta_param, keep working.

Have you tested this? If so, how?

I have included unit tests. testSerializeRoundTripNegativeAndSubSecond in test/parameter_test.py fails on master and passes with this change. test/parameter_test.py, test/task_test.py, test/date_interval_test.py, test/date_parameter_test.py and test/task_register_test.py pass locally (246 passed). I also ran the dynamic dependency example from the issue: on master it raises the ParameterException, with this change both tasks run.

Fixes #3473

Signed-off-by: Sebastian Turner <sebastiant3484@gmail.com>

@Manishram-ai Manishram-ai left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested this locally. On master, timedelta(hours=-1) serializes to -1 w 6 d 23 h 0 m 0 s, and parse can't read it back. With your change the round trip works and all 162 tests in parameter_test.py pass. Your new test also fails on master, so it catches the bug.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TimeDeltaParameter.serialize cannot round trip negative or sub second durations

2 participants