- Action
- Stop admission and set the shared shutdown flag.
The recovery architecture
From Kubernetes SIGTERM to a visually continuous Studio reply, with the durable and transient state kept separate.
Outcome
The recovery protocol is technically compatible with the cleaner transcript Marc described. The database should keep two assistant messages because they belong to two invocations, but the renderer can group them into one visual reply, remove the intermediate action toolbar, and omit “Continued after a Fimo update.”
The reconnect indicator is a different state. It appears only after an ambiguous transport loss and currently lives as a separate scroller row. Moving it into the active assistant message, immediately after the latest thinking or tool row, fixes the spacing without changing any server contract.
Nothing here proves the production grace period. The code assumes at least 900 seconds and derives a 10-minute natural-finish window from it; infrastructure must still confirm the same lower bound for API, jobs, and chat.
System map
One deploy, three ownership layers
The API process owns shutdown and durable invocation state. The agent runtime owns the safe place to stop. Studio owns transport recovery and presentation. The design works because each layer communicates through explicit state instead of treating a closed HTTP stream as proof of failure.
Process
SIGTERMReject new work and request a safe stop
Live-work registryWait for chat turns and autonomous sessions
↓
Agent runtime
Model-step boundaryFinish the current response and tool outputs
Durable handoffPush retained work and cancel with a typed reason
↓
Studio
ReconnectReattach when the same invocation is alive
ContinueFollow one successor when deploy interruption is durable
Timing
The 15-minute grace becomes 10 minutes for natural work
SHUTDOWN_GRACE_MS=900000 is an asserted platform contract, not a timer Fimo controls. Fimo reserves five minutes of that window for forced settlement, request draining, deferred cleanup, telemetry headroom, and exit margin. The current worker therefore gives registered work up to 600 seconds to reach a safe boundary naturally.
- Action
- Let registered turns finish their current model step and tools.
- Action
- Abort remaining work, drain requests, and finish branch cleanup.
- Action
- Leave two minutes before the platform may send SIGKILL.
apps/api/src/shutdown.tsProduction budget at 900 secondshard stop 780s grace minus 2-minute exit margin
natural finish 600s after abort, drain, cleanup, and telemetry reserves
forced abort 90s persist the typed terminal and retained work
request drain 30s
branch cleanup 30sSafe boundary
Stop after a completed step, before another model request
requestShutdown() flips process-wide state. Every registered runtime reads that state through shouldStop. The Studio agent does not interrupt a tool halfway through its output: it observes the flag while streaming, persists the settled tool output, cancels the reader, publishes changed workspace state when necessary, and lets the engine record deploy_interrupted.
If work refuses to reach that boundary within 10 minutes, the shutdown owner aborts it with the same typed reason and gives settlement another 90 seconds. This is the emergency path, not the normal path.
sequenceDiagram participant K as Kubernetes participant S as shutdown.ts participant R as live-work.ts participant A as Studio agent K->>S: SIGTERM S->>R: requestShutdown() R-->>A: shouldStop() = true A->>A: finish model response A->>A: persist tool outputs A->>A: push retained branch if changed A-->>R: receipt settles as deploy_interrupted R-->>S: live work drained
Durable handoff
The next worker continues from database truth
The old invocation remains cancelled with a machine reason and its partial assistant message. A new worker reads that terminal state and may admit one successor within 15 minutes. The successor reuses the original user message, adopts the retained task branch, and receives a compact reminder containing the previous invocation and partial reply.
An idempotency key derived from the interrupted invocation ensures that several tabs or workers racing to recover create only one successor. A chain is capped at three deploy successors before the product falls back to manual Retry.
- invocationId
- The live producer a client can attach to.
- step
- Preparing, working, compacting, or merging.
- continuesInvocationId
- The interrupted invocation this live turn succeeds.
- Window
- 15 minutes from terminal settlement.
- Attempts
- At most three successors in one deploy chain.
- Identity
deploy-successor:<invocationId>.
API surface
Three small endpoints expose the recovery state machine
The browser does not recreate an invocation from local guesses. It asks the server what is live, requests a continuation mutation when appropriate, and attaches to the replayable stream for whichever invocation owns the thread now.
/api/chat/project/live-turn
- Purpose
- Read authoritative live ownership and successor linkage.
- Result
{ threadId, liveTurn }, whereliveTurnmay be null.- Not proven
- Reading state does not itself create or attach to a successor.
/api/chat/project/live-turn/continue
- Purpose
- Idempotently admit or discover a deploy successor on the new worker.
- Result
- The same live-turn state shape after the mutation.
- Not proven
- Admission does not guarantee the browser has attached to the new stream.
/api/chat/project/stream
- Purpose
- Attach to the current invocation's replayable NDJSON stream.
- Result
- A turn head followed by AI SDK message chunks.
- Not proven
- An attached stream does not prove the invocation will finish successfully.
Browser state
A dropped stream is evidence of disconnection, not evidence of failure
recoverableTurnStream wraps the AI SDK stream. If the stream disappears without a terminal finish, it tries five times after 1, 3, 7, 15, and 30 seconds. Each attempt asks the server whether the original invocation is still live, whether a successor now owns the thread, or whether persisted history has settled.
Only this ambiguous transport path displays “Reconnecting 1/5.” A graceful deploy terminal does not need that indicator while the connection remains open. If the retries exhaust, the existing Retry surface appears.
flowchart TD Drop[Stream ends without finish] --> Read[Read live-turn state] Read -->|same invocation| Attach[Attach to replay and skip delivered chunks] Read -->|successor continues it| Follow[Finish old stream and follow successor] Read -->|nothing live| History[Refetch persisted history] Read -->|unreachable| Backoff[1s · 3s · 7s · 15s · 30s] Backoff --> Read Backoff -->|five attempts exhausted| Retry[Show manual Retry]
Presentation seam
Keep two stored messages; render one continuous reply
The current screen exposes the internal handoff. TurnStatus renders “Continued after a Fimo update” inside the interrupted assistant message, and AssistantMessage gives that partial row its own copy, regenerate, and more actions. The successor then starts as another full assistant row.
The cleaner presentation is a display-only reply group: when an interrupted assistant message is immediately followed by the successor named in its continuation metadata, omit the boundary label and intermediate toolbar, collapse the spacing, and keep actions only on the completed successor. Storage, billing lineage, model history, and source message IDs remain unchanged.
The reconnect indicator has the same kind of presentation leak. Messages.tsx appends it after every message as an independent scroller item. Passing recovery state to the active assistant row and rendering it after the latest visible part places it directly below “Thinking…” or the current tool call.
- Append-only stays intactNever merge or rewrite the two persisted assistant messages.
- One visible replyGroup only an explicit interrupted-message → named-successor pair.
- No middle actionsThe partial row is not a complete answer; actions belong to the final row.
- Reconnect stays transientRender it beside active work and clear it on recovery or terminal settlement.
Proof and boundary
The mechanism is covered; the infrastructure contract is not yet proven
The feature branch passed the four targeted Dev Remote paths with zero retries: reconnecting to the same live invocation, continuing after a deploy interruption, rendering that continuation on mobile, and exhausting reconnects into manual Retry. The screenshots above explain the remaining presentation refinement, not a failed recovery protocol.
Do not merge or deploy based on this evidence alone. The platform must confirm at least 900 seconds of termination grace for every service that can host these runs, then a real rollout canary must show SIGTERM, safe settlement, successor admission, and no SIGKILL before settlement.
⌄apps/api
⌄packages
⌄apps/ui
4/4 deploy and reconnect paths passed, with zero retries in 143.6 seconds.
Append-only messages, idempotent successor admission, bounded retries, and typed terminal state are explicit in code.
Infrastructure still needs to confirm ≥900 seconds for API, jobs, and chat.
No deployed SIGTERM-to-successor trace has been accepted yet.