UI stream in-process Luigi workflow output live - #1073
Merged
Merged
Conversation
Two single-path files replace the INPROCESS_LUIGI branching in the child. WorkflowRunner now picks the target function. Delete workflow_child.py.
Only the in-process child is used now. Delete run_workflow_child_subprocess.py, drop the INPROCESS_LUIGI flag and the branching in WorkflowRunner.
The child streams captured output as tagged messages on the result queue instead of a second pipe. One channel, so per-task grouping stays exact and arrives live. New workflow_log_builder builds the flat log + chunks on the parent; PipeTee removed. Split out task_pointer, luigi_task_events, run_status, time_utils. Rewrote the runner tests to the in-process contract. Client unchanged: poll() still returns the same output and chunks.
Group the runner and its helpers under ui/server/workflow_runner/ (filenames unchanged, intra-package imports made relative). __init__ re-exports WorkflowRunner and RunStatus, so `from workflow_runner import ...` still works. Update the conftest fake path.
Type dispatch_id Optional[str] on WorkflowDone and WorkflowRunResult (it's None when a run uses no dispatch id; caught by pyright). Add tests for the Luigi event handlers moving the task pointer and WorkflowLogBuilder grouping.
The exec_code endpoint returns WARMING_UP until the off-thread hera warmup finishes. There is no real hera in tests, so warmup never completed and the 4 exec tests failed on the WARMING_UP guard. Add a `warmed` fixture that marks warmup ready via monkeypatch and apply it to the exec tests, so they exercise the exec path itself.
Move the pre-execution steps (rebuild the hermes workflow, build the Luigi module, write it to disk, clear old target files) into a new prepareWorkflowRunFromDoc method. It returns the module path, workflow name, and target-files directory. executeWorkflowFromDB now calls it.
Drop the duplicated build steps; call the toolkit's prepareWorkflowRunFromDoc instead. Add type hints and a print of the target-files directory. Remove the now-unused shutil and hermes.workflow imports.
Fold executeWorkflowFromDB_inprocess into run_workflow_child_inprocess and delete its module. Drop the loop (the UI runs one workflow, so require exactly one doc) and the always-None dispatch id. Merge the workers logging into one line and trim comments.
Inline executeWorkflowFromDB_inprocess into WorkflowChildInProcess and run it via the WorkflowChildInProcess.start_child entry point. Drop the loop (one workflow per run) and the always-None dispatch id. Add add_python_path for sys.path + PYTHONPATH, and make OutputRouter.stop idempotent via a stopped flag.
One shape for output everywhere: an ordered list of {name, text}
segments. The client reads chunks live and on done; the server no
longer builds or sends the flat log.
- runWorkflow: drop output from PollWorkflowResult; add chunksToText
- store: WorkflowRun holds chunks; setRunOutput -> setRunChunks
- poller/button/dialog render from chunks (running view joins texts)
- WorkflowLogBuilder: drop _flat and output(); keep chunks()
- WorkflowRunResult, poll(), RunWorkflowResponse: drop output
- tests assert on chunks
- WorkflowChunkLog: one chunk as a self-contained card - WorkflowChunkedLog/WorkflowLogView: pure renderers, no toolbar - dialog: filter + copy-all toolbar now in the bottom action bar - logMetrics: shared counts/copy-text helpers
The client posts the whole workflow document in the start payload. The
server wraps it and builds the run straight from it, so it no longer
looks the document up in the DB.
Server:
- RunWorkflowPayload carries projectName + doc (drop workflowName).
- New SentWorkflowDoc wraps the sent dict the way prepareWorkflowRunFromDoc
reads a saved doc: .desc (attribute) and ['resource'] (item).
- Thread doc through start_workflow, WorkflowRunner, and the child; drop
getWorkflowListDocumentFromDB here. The doc's own resource is honored.
Client:
- startWorkflow sends { projectName, doc }.
- RunWorkflowButton takes the doc and fills desc.workflowName from the
resolved name; DetailsViewDocumentContent passes shownDoc.
Tests:
- test_sent_workflow_doc covers each field the run reads.
- Update client and server tests for the new doc payload.
- luigi_task_events: FAILURE/BROKEN handlers record task errors; the child
raises them when luigi.build returns False, not a bare "False".
- conftest: fake toolkit now writes a real Luigi task module, matching the
child's luigi.build path.
- test_workflow_runner: in-process run has no dispatch id; assert None.
The log vanished on failure and only the error showed. Kept the output at all three layers: the runner returns the captured chunks on error, the poller stores them, and the dialog shows the log with the error.
The view showed a flat log during the run, then reflowed into cards at the end. Chunks carry their task name live, so render cards throughout.
The run uses the shown workflow, so a plain run no longer needs a save. Stop disabling the button when auto-save is off, and reword the tooltip and menu item to "run as shown".
Seed each parameter under input_parameters as expanded, so nested parameter values show without a click.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.