HistoryPubnetParallelCatchupV2 Mission Rewrite - #423

Closed

Jonathan-Eid wants to merge 120 commits into

stellar:mainfrom

Jonathan-Eid:jonathan/catchup-k8s-job

Commits on Jul 30, 2026

Rewrite pubnet parallel catchup v2 as Job-per-range

Jonathan-Eid
and
claude
committed
b5e6ed6
Sum txApply and seconds across the resumed attempt chain

Jonathan-Eid
and
claude
committed
0280156
Drop the dead Prometheus test helper, pin the retry taxonomy

Jonathan-Eid
and
claude
committed
d1b4ada
Remove dead cpu-sizing code; close mutation-sweep coverage gaps

Jonathan-Eid
and
claude
committed
cfc69d4
Record attempt duration in the collector, as a fallback for reaped pods

Jonathan-Eid
and
claude
committed
4ecc93e
Stop a worker's own output from killing its log stream

Jonathan-Eid
and
claude
committed
43547e8
Gate the retry-path reap; cut the per-stream line buffer

Jonathan-Eid
and
claude
committed
08980ca
Collect logs by polling instead of one follow stream per pod

Jonathan-Eid
and
claude
committed
c0bd891
Poll only pods whose log endpoint can answer

Jonathan-Eid
and
claude
committed
2dffc45
Do not reap a succeeded Job until its record is actually complete

Jonathan-Eid
and
claude
committed
1635311
Gate the reap on an explicit done marker from the collector

Jonathan-Eid
and
claude
committed
fd8b099
Wake a poller the moment its pod goes terminal or vanishes

Jonathan-Eid
and
claude
committed
5bd41f6
Keep a ceiling-hit attempt's peak across a fresh start

Jonathan-Eid
and
claude
committed
e95f104
Put the attempt number on the worker pod, not just the Job

Jonathan-Eid
and
claude
committed
16d9067
Never cpu-throttle a worker; escalate memory per OOM, not per attempt

Jonathan-Eid
and
claude
committed
38ad01c
Treat a range already at its target as done, not as work to redo

Jonathan-Eid
and
claude
committed
83e9b61
Fix the offline-info LCL extraction; it was silently reading nothing

Jonathan-Eid
and
claude
committed
c17296c
Do not report a duration for a pod the collector never watched run

Jonathan-Eid
and
claude
committed
07469cf
Never let a peak on disk be lowered by a later write

Jonathan-Eid
and
claude
committed
c15689f
Read a pod's own start->finish instead of how long the poller watched

Jonathan-Eid
and
claude
committed
2b475ee
Raise the OOM retry budget to 10

Jonathan-Eid
and
claude
committed
4e08897
Keep the OOM budget at 5, and say why

Jonathan-Eid
and
claude
committed
fca8180
Never condemn a range on exit 3; it means "interrupted" as often as "broken"

Jonathan-Eid
and
claude
committed
945efdc
Project peakAnonBytes and wallSeconds into the mission's profile artifact

Jonathan-Eid
and
claude
committed
dcbce73
Drain the run before failing on a condemned range

Jonathan-Eid
and
claude
committed
7038644
A condemned range must not freeze dispatch

Jonathan-Eid
and
claude
committed
b09e5f6
Behavioral test suite: a fake Kubernetes cluster, and fixes for 8 races

Jonathan-Eid
and
claude
committed
eb3c677
Statelessness suite, and six defects it found

Jonathan-Eid
and
claude
committed
e30ee14
Move the suite into tests/, grouped by what it exercises

Jonathan-Eid
and
claude
committed
dded200
Delete the monotonic-progress guard rather than persist it

Jonathan-Eid
and
claude
committed
0b42421
Re-read measurements from the volume when the mirror is all that is left

Jonathan-Eid
and
claude
committed
d4fd259
Wire avoidNodeLabels through to the worker pods again

Jonathan-Eid
and
claude
committed
93cae0e
Retire test_job_monitor.py for tests that run the code

Jonathan-Eid
and
claude
committed
1422824
Add longest-first dispatch, ordered by the profile rather than by position

Jonathan-Eid
and
claude
committed
15ca968
Size the cpu request from a range's slack, not from its demand

Jonathan-Eid
and
claude
committed
b840544
Rename profile_floor to longest_range_seconds

Jonathan-Eid
and
claude
committed
e17cfc6
Key the cpu tier on rank, not on absolute seconds

Jonathan-Eid
and
claude
committed
d72e219
Collapse the cpu ladder into one setting

Jonathan-Eid
and
claude
committed
9856687
Improve worker liveness metrics

Jonathan-Eid
and
Copilot
committed
afd80ac
Pass mission namespace to Helm

Jonathan-Eid
and
Copilot
committed
0885850
Add runtime-weighted memory insurance

Jonathan-Eid
and
Copilot
committed
e68a0f9
Fix retry counter reconstruction

Jonathan-Eid
and
Copilot
committed
56b4463
Count unique disruption-retried ranges

Jonathan-Eid
and
Copilot
committed
7967edd
Extend parallel catchup attempt deadline

Jonathan-Eid
and
Copilot
committed
acfcde8
Pass --namespace to every helm and kubectl call

Jonathan-Eid
and
claude
committed
3500dac
Bound an attempt by its own profiled cost, not by one global number

Jonathan-Eid
and
claude
committed
76e1a40
Fix durable resume detection in collector

Jonathan-Eid
and
Copilot
committed
f68df05
Reconstruct resumed catchup profiles

Jonathan-Eid
and
Copilot
committed
6e74f10
Harden restart-safe profile finalization

Jonathan-Eid
and
Copilot
committed
01b0a42
Omit unverifiable chain aggregates

Jonathan-Eid
and
Copilot
committed
71dc1ae
A range that reaches its deadline fails the mission

Jonathan-Eid
and
claude
committed
450729a
Add synthetic resume restart harness

Jonathan-Eid
and
Copilot
committed
6305366
Port the live monitor's liveness sampler and runtime memory insurance
Jonathan-Eid
committed
106a563
Harden synthetic harness startup

Jonathan-Eid
and
Copilot
committed
483001d
Capture synthetic monitor startup failures

Jonathan-Eid
and
Copilot
committed
e3ebbdd
Collapse the eight tmp+rename blocks into one _write_atomic
Jonathan-Eid
committed
b57c70b
Stop condemning a range whose pod terminated with no exit code
Jonathan-Eid
committed
1ed7e82

Commits on Jul 31, 2026

Commits on Aug 1, 2026

Widen the cpu ladder to 8 bands topping at 3.0 cores
Jonathan-Eid
committed
55fe6f2

Commits on Aug 7, 2026

Restructure the monitor and make pooled on-demand runs schedulable

Jonathan-Eid
and
claude
committed
8553e77
catchup: size spot nebula for 1800m/9Gi on half-sized nodes

Jonathan-Eid
and
claude
committed
97fc01a

Commits on Aug 8, 2026

Move the driver onto the monitor's HTTP surface

Jonathan-Eid
and
claude
committed
8be7d87
Validate the whole config at /start, not at import

Jonathan-Eid
and
claude
committed
1913623
Move the range into /start and pull logs per file

Jonathan-Eid
and
claude
committed
c809215
Fix two faults the first HTTP run surfaced

Jonathan-Eid
and
claude
committed
40434b1
Keep the collector's resume cursor out of the manifest

Jonathan-Eid
and
claude
committed
f6b5e3e

Commits on Aug 9, 2026

Drop the ConfigMap plumbing the HTTP surface replaced

Jonathan-Eid
and
claude
committed
63056dc
Open the reconcile gate where the run is installed, not in the POST handler

Jonathan-Eid
and
claude
committed
f192eb2
Say in /status whether the run has started

Jonathan-Eid
and
claude
committed
967f0a4

Commits on Aug 10, 2026

catchup: make the job monitor image settable per run

Jonathan-Eid
and
claude
committed
ed82235
Pin the dev image to 2026-08-09a

Jonathan-Eid
and
claude
committed
abfc682
Delete the logarithmic range generator
Jonathan-Eid
committed
f55038d
Keep the parallel-catchup test suite local
Jonathan-Eid
committed
21a0dfd
catchup: gate all chart RBAC behind monitor.createRbac

Jonathan-Eid
and
claude
committed
277632f
Add a worker memory-request override and delete dead PCv2 code
Jonathan-Eid
committed
a0cec3c
Decide cache-bump rungs from a block list, not from node vCPU
Jonathan-Eid
committed
39a9f01
Delete the orphaned cpu-tiering prose from values.yaml
Jonathan-Eid
committed
ad2a9fd
Delete the volume-spread constraint
Jonathan-Eid
committed
908c47b
Take capacity from a node-label list, not a values overlay
Jonathan-Eid
committed
e4bae21
Move the ParallelCatchupV2 tests into CatchupV2Tests.fs
Jonathan-Eid
committed
df7a200
Cut eight redundant V2 tests, add --pubnet-parallel-catchup-create-rbac
Jonathan-Eid
committed
0fa75f4
Remove HANDOFF.md
Jonathan-Eid
committed
d3e191e
Cut the field-projection test
Jonathan-Eid
committed
7e12140
Cut the volume-peak test
Jonathan-Eid
committed
f496adc
Stop sizing ephemeral storage at all in pvc mode
Jonathan-Eid
committed
daacac9
Drop the two size-only assertions from the profile test
Jonathan-Eid
committed
f59bd49
Name the profile guard test after what it actually prevents
Jonathan-Eid
committed
15f9c93
Cover the mixed case: some ranges measured, some not
Jonathan-Eid
committed
8ef9814
Tighten the comments in CatchupV2Tests
Jonathan-Eid
committed
54ac412
Fold the V2 tests back into Tests.fs
Jonathan-Eid
committed
7a633af
Stop passing the chart its own values file
Jonathan-Eid
committed
d3d4997
Make the values file overridable with SUPERCLUSTER_VALUES_PATH
Jonathan-Eid
committed
f106b00
Accept an http profile URL, and cover every resolution path
Jonathan-Eid
committed
bde2160
Drop the profile-resolution tests
Jonathan-Eid
committed
5031d1a

Commits on Aug 11, 2026

Log the monitor process to one file, named for the process
Jonathan-Eid
committed
530644f
Sort collected artifacts into folders, and summarise pods by phase
Jonathan-Eid
committed
b79a883
Merge branch 'main' into jonathan/catchup-k8s-job
Jonathan-Eid
authored
b1a6e18
Print status and pod lines every ten minutes
Jonathan-Eid
committed
746b2be
Refetch rewritten artifacts whole instead of resuming them
Jonathan-Eid
committed
8c8d5b8
Always pull the job monitor image
Jonathan-Eid
committed
32e8ed1
Delete the leftover CAPACITY_TYPE env
Jonathan-Eid
committed
9a07a28
Let a run choose where the job monitor lands
Jonathan-Eid
committed
07aa5a5
Stop values.yaml re-deriving what lib/config.py already establishes
Jonathan-Eid
committed
fd49f9c
Run fantomas, and cap values.yaml comments at three sentences
Jonathan-Eid
committed
3029b80
Merge branch 'main' into jonathan/catchup-k8s-job
Jonathan-Eid
authored
950cc59
Put main's new tests back inside the Tests class
Jonathan-Eid
committed
07616f7

Commits on Aug 14, 2026

refactor(parallel-catchup): file lib/ by owner, split config three ways

Jonathan-Eid
and
claude
committed
3b8ec9c
refactor(parallel-catchup): rewrite the collector as one cycle over the pod list

Jonathan-Eid
and
claude
committed
a453350

Commits on Aug 17, 2026

Replace the parallel-catchup job monitor with the v2 rewrite

Jonathan-Eid
and
claude
committed
c0f88c4
Settle a failed attempt the collector never opened

Jonathan-Eid
and
claude
committed
3c78b25
Split monitor config out of the shared module, and de-duplicate the path contract

Jonathan-Eid
and
claude
committed
ef3e8f3
Step the resumed chain over an attempt that ran nothing

Jonathan-Eid
and
claude
committed
ccfaa20

Commits on Aug 18, 2026

Claim an attempt while its pod is still Pending

Jonathan-Eid
and
claude
committed
26e7e66