Project-agent pause and continuation options

Design

Pause the run, then continue safely

Autonomous runs need explicit continuation. Studio chat already has a safer first response: keep the old pod and stream alive for a bounded drain, then cancel and settle.

FIMO-2088 Full recap Files 15 evidence files Diff Design only

Outcome

The recommendation splits by product path. Autonomous project-agent runs should stop at a safe boundary, retain their task Branch, settle the old run, and continue through a fresh planned invocation. QStash remains a wake-up transport, not a model checkpoint.

Studio chat should first get a bounded natural drain. The current engine already registers chat turns, keeps the initiating HTTP stream open, supports cross-replica replay while the old producer lives, persists partial assistant parts, and fences a dead producer with a 30-second Postgres lease.

With a 15-minute Kubernetes grace period, reserving 10 minutes for natural chat completion is reasonable. The remaining time must be an explicit shutdown budget, not hope: cancel and settle stragglers, drain requests and Branch cleanup, then leave headroom before SIGKILL.

Failure boundary

A deploy can end the run before Kubernetes' hard deadline

SIGTERM starts Fimo cleanup. It is not a five-minute delay. The project-agent pickup is an active HTTP request, but the autonomous run is absent from the live-turn registry. That registry resolves immediately, the request drain waits 30 seconds, and Fimo exits while generateText is still in flight.

The first repair is lifecycle ownership. The shutdown path needs a cooperative stop and a settled promise for autonomous runs, with enough time to preserve the task Branch before the container exits.

flowchart LR
  Deploy[Deploy] --> Term[SIGTERM]
  Term --> Handler[Fimo shutdown]
  Handler --> Registry[Chat turns only]
  Registry --> Drain[HTTP drain 30s]
  Drain --> Exit[process.exit]
  Exit --> Lost[In-flight model call lost]

Source inspection of signal-handlers.ts, live-turns.ts, and jobs.ts. The exact pod signal remains a production check.

apps/api/src/signal-handlers.tsShutdown exits after a bounded drain

401const naturalFinish = await waitForLiveTurns(CHAT_NATURAL_FINISH_TIMEOUT_MS);
622const result = await waitForRequestsDrained(REQUEST_DRAIN_TIMEOUT_MS);
65logger.warn(`[shutdown] Proceeding with ${result.remaining} requests still active`);
1103await cleanup();
120process.exit(process.exitCode ?? exitCode);
1264process.on('SIGTERM', () => shutdown('SIGTERM', null, 0));

1
Wrong registry. This wait knows only about registered chat turns, so an autonomous project-agent run contributes nothing here.
2
Bounded request drain. The active pickup remains in the request set, but timeout does not cancel its model loop.
3
Process-owned exit. Once cleanup returns, Fimo exits even if the request was still active.
4
Trigger. Kubernetes starts this code with SIGTERM. SIGKILL is the later, uncapturable fallback.

Retry contract

A QStash 5xx retry cannot resume the same running invocation

QStash is the wake-up mechanism. The durable invocation row owns admission. The first pickup changes the row from planned to running; a later delivery of the same message calls loadPlanned, sees that it is no longer planned, and returns not_planned.

A 5xx can redeliver a lost wake-up while the row is still planned. It cannot carry the model transcript, restore an in-flight provider call, or safely turn a running row back into a new worker. Making that work would require a new state machine and a second duplicate-worker protocol.

› POST /jobs/agent-pickup Admits one planned invocation

The route delegates to agentsService.pickup. A missing planned row is a successful no-op, not a continuation signal.

QStash deliveryplanned rowCAS to runninginline run

QStashWakes a durable row; it does not own agent state.
One invocationMust never be both the dead worker and the resumed worker.
ContinuationNeeds a new planned root, a lineage link, and a retained Branch.

Recommended shape

Preserve the Branch first, then start a new continuation

Step one is deploy-safe interruption. At a model-step boundary, stop the loop, push the task Branch, and settle the old session as cancelled with the internal reason shutdown. If a provider call never returns before the grace window, the current call cannot be recovered, but all previously committed work can survive.

Step two is explicit continuation. Create a new planned invocation with lineage and a handoff note, adopt the retained Branch instead of forking a new one, then publish a fresh QStash wake-up. Existing CAS and lease logic fence duplicate workers.

flowchart LR
  Run[Inline run] --> Stop[Safe boundary stop]
  Stop --> Push[Push task Branch]
  Push --> Settle[Cancel old run]
  Settle --> NewRoot[New planned root]
  NewRoot --> Adopt[Adopt retained Branch]
  Adopt --> Wake[Fresh QStash wake-up]

Proposed lifecycle. The Branch and the new invocation are separate owners, so a retry cannot revive the dead worker.

Safe stop + fresh continuationRecommended

Work: Pushes the retained Branch and keeps a handoff.
Cost: Medium now, with full transcript checkpoints deferred.

Same-message QStash retryReject

Work: Cannot restore an in-flight model call.
Risk: Running rows become no-op retries or need a second state machine.

Wrap-up promptFallback

Work: May save a handoff only at a safe boundary.
Risk: The model can be the thing that is stuck.

Separate durable workerLater

Work: Improves isolation but still needs stop and resume state.
Risk: A worker pod also receives SIGTERM during deploy.

OrderStop loop, push Branch, settle old run, then expose continuation.
Durable handoffBranch id, pushed SHA, trigger payload, capability snapshot, reason, and prior cost.
Full checkpoint laterTranscript, tool parts, step count, and prompt version belong to a larger follow-up.

Studio chat

Keep the current reply alive for 10 minutes, then stop it cleanly

Studio chat does not need a second continuation system for the normal deploy case. On SIGTERM, Fimo marks the old pod as shutting down but deliberately leaves its server and existing streams open while every registered chat receipt gets time to finish. A connected user keeps seeing the reply from the old pod.

If the browser refreshes, Studio asks an authenticated API replica for the current invocation. That replica uses Redis pub/sub to ask the old producer for its in-memory replay buffer and then relays the same stream. Studio must never connect to Redis directly. Redis carries replay requests, chunk fan-out, the stream sentinel, and abort delivery; Postgres owns the invocation, transcript, lease, settlement, and billing.

At 10 minutes, Fimo should abort any remaining chat and allow 90 seconds for the engine to settle it. The engine can spend about 31 seconds in a child barrier before Branch cleanup and a second barrier, so the current 30-second abort window is too tight. A clean abort can end the same stream with a cancelled status. The current UI renders that as Stopped; the better product copy is a deploy-specific explanation with a user-triggered Retry. Do not retry automatically because earlier tool calls or content writes may already have happened.

sequenceDiagram
  participant K as Kubernetes
  participant O as Old API pod
  participant R as Redis
  participant N as New API pod
  participant S as Studio
  K->>O: SIGTERM
  O->>O: Drain chats for 10m
  O-->>S: Existing stream continues
  S->>N: Refresh and reconnect
  N->>R: Request invocation replay
  R->>O: Ask live producer
  O-->>N: Buffered and new chunks
  N-->>S: Same reply continues
  O->>O: Abort straggler at cutoff

Recommended rolling-deploy path. Cross-replica replay works only while the old producer process still owns the model request and its memory buffer.

0 to 10 minutesDrain

Server: Keep the provider call, receipt, and existing SSE response alive.
Studio: Continue directly or reconnect through the API replay endpoint.

10 to 11.5 minutesSettle

Server: Abort the remaining root controller and await terminal settlement.
Studio: Receive a cancelled terminal result when the abort finishes.

11.5 to 12.5 minutesCleanup

Server: Close the listener, drain requests, finish deferred Branch cleanup, and flush telemetry.
Budget: Use a 13-minute Fimo hard stop and leave about 2 minutes before the 15-minute Kubernetes deadline.

Producer dies earlyFallback

Redis: Cannot recreate the dead process or its in-memory replay buffer.
Postgres: Stops reporting the turn live after the 30-second lease expires; the next send repairs the stale row.

Current codeWaits 8 minutes, allows 30 seconds for abort, and has a 9.5-minute process hard stop. All three must change before a 10-minute drain is real.
No direct Redis clientStudio stays behind project, environment, thread, invocation, and organization authorization in the API.
No automatic retryA fresh model request can repeat non-idempotent tool effects and spend credits twice.

Proof and limits

The design is clear, but the deploy receipt is still missing

Fable's Strapi review and the parent source check agree on the core boundary: autonomous runs need a new continuation contract, while Studio chat should first use its existing live-turn drain, replay, persistence, and lease machinery. The two paths should share cancellation vocabulary but not pretend they have the same recovery guarantees.

The incident still needs the old pod's termination record, the final lease renewal, and the QStash delivery history. Those checks decide whether this exact run followed the deploy-kill path; they do not change the design recommendation.

Evidence-bearing files15 files

⌄▱apps/api/src

⌄▱apps/ui/src/features/chat

⌄▱packages/agents/src

VerifiedSource contract

Current origin/develop confirms shutdown, pickup, cancellation, Branch, and continuation boundaries.

PartialStrapi Fable review

Returned a concrete recommendation from an isolated checkout. The parent checked the current shutdown, stream, reconnect, persistence, and lease paths.

UnknownExact production event

Pod termination state, final lease renewal, and QStash delivery history were not collected in this recap.