Atlas recap

Updated 2026-09-04

Agent work across deploys

The implemented shutdown and recovery contract: finish the current safe step, retain productive Branch work, continue through a fresh invocation, and show reconnect UI only when the browser actually loses its stream.

Scope FIMO-2425 · PR #2396 Base develop · 452daf8 → 8d49cdf Depth Normal Evidence Source + 4-path Dev Remote E2E Draft implementation

Draft implementation

Pause the run, then continue safely

The old worker settles at a safe model-step boundary. Studio either reattaches to that live producer after a real cutoff or silently adopts one durable successor after a deployment interruption.

FIMO-2425 Draft PR #2396 Quick recap Files 20 evidence files Diff 45 files · 2 commits

Outcome

SIGTERM now marks every in-process Studio, Slack, and autonomous execution as shutting down. Each agent checks that flag after a completed model response, tool calls, billing, and step bookkeeping, so it can stop before beginning another model request.

With a 900-second provider grace, Fimo gives safe-boundary work up to 10 minutes, forced settlement up to 90 seconds, request drain and deferred Branch cleanup 30 seconds each, and keeps two minutes before SIGKILL. Those timings are implemented, but the provider setting still needs infrastructure proof.

A deployment interruption becomes a durable boundary, not a replay of the dead request. Fimo retains productive Branch work, settles the old invocation as cancelled with internal reason deploy_interrupted, and creates one idempotent successor. Studio keeps the same user turn and silently follows that successor. Only an ambiguous connection cutoff shows Reconnecting 1/5 through 5/5.

Failure boundary

Before this draft, Fimo exited long before Kubernetes' hard deadline

In the develop baseline, SIGTERM starts Fimo cleanup immediately. The project-agent pickup is an active HTTP request, but the autonomous run is absent from the live-turn registry. That registry resolves immediately, the request drain waits 30 seconds, and Fimo exits while generateText is still in flight.

The draft repairs lifecycle ownership. Studio, Slack, and autonomous work now register one cooperative stop and settled promise, giving the engine time to preserve the task Branch before the container exits.

flowchart LR
  Deploy[Deploy] --> Term[SIGTERM]
  Term --> Handler[Fimo shutdown]
  Handler --> Registry[Chat turns only]
  Registry --> Drain[HTTP drain 30s]
  Drain --> Exit[process.exit]
  Exit --> Lost[In-flight model call lost]
Develop-baseline failure path from signal-handlers.ts, live-turns.ts, and jobs.ts. PR #2396 replaces the chat-only registry; the provider signal remains a production check.
apps/api/src/signal-handlers.tsShutdown exits after a bounded drain
401const naturalFinish = await waitForLiveTurns(CHAT_NATURAL_FINISH_TIMEOUT_MS);
622const result = await waitForRequestsDrained(REQUEST_DRAIN_TIMEOUT_MS);
65logger.warn(`[shutdown] Proceeding with ${result.remaining} requests still active`);
1103await cleanup();
120process.exit(process.exitCode ?? exitCode);
1264process.on('SIGTERM', () => shutdown('SIGTERM', null, 0));
  1. 1

    Wrong registry. This wait knows only about registered chat turns, so an autonomous project-agent run contributes nothing here.

  2. 2

    Bounded request drain. The active pickup remains in the request set, but timeout does not cancel its model loop.

  3. 3

    Process-owned exit. Once cleanup returns, Fimo exits even if the request was still active.

  4. 4

    Trigger. Kubernetes starts this code with SIGTERM. SIGKILL is the later, uncapturable fallback.

Timing budget

The 15-minute grace period has five separate jobs

Kubernetes sends SIGTERM at the beginning of termination. The 15-minute grace period starts then. Fimo must finish every cooperative phase before Kubernetes reaches the deadline because SIGKILL cannot be handled, delayed, or used to save state.

The draft's 13-minute application deadline leaves two minutes for scheduler delay, event-loop stalls, telemetry flushes, and the gap between Fimo's own timer and Kubernetes' clock. The slack is intentional. Setting both deadlines to 15 minutes would turn a small timing difference into lost work.

flowchart TB
  Term[SIGTERM<br/>0:00] --> Natural[Natural drain<br/>to 10:00]
  Natural --> Abort[Abort and settle<br/>to 11:30]
  Abort --> Requests[HTTP drain<br/>to 12:00]
  Requests --> Cleanup[Branch cleanup<br/>to 12:30]
  Cleanup --> AppExit[Fimo hard stop<br/>13:00]
  AppExit --> Kill[Kubernetes SIGKILL<br/>15:00]
Draft production budget. Work may finish earlier at every phase; these are ceilings, not mandatory waits.
Natural finish10:00
Waits for
Registered Studio, Slack, and autonomous agent work.
Stops early
As soon as every live receipt settles.
Abort settlement1:30
Starts at
Only when a turn remains after the natural window.
Pays for
Root cancellation, child barriers, Branch handling, persistence, and billing.
HTTP drain0:30
Starts after
The listener closes and chat settlement has had its chance.
Purpose
Lets remaining accepted requests return before exit.
Deferred cleanup0:30
Waits for
Task-Branch cleanup already moved out of the response path.
Purpose
Avoids exiting while a retained or discarded Branch is unresolved.
Safety margin2:00
Fimo exits by
13 minutes after SIGTERM.
Kubernetes kills at
15 minutes after SIGTERM.
  • Develop baselineUses 8 minutes for natural finish, 30 seconds for abort, and a 9.5-minute hard stop.
  • Draft PRUses 10 minutes, 90 seconds, and a 13-minute hard stop. Kubernetes must move to 15 minutes before this code deploys.
  • Why abort needs 90 secondsThe engine may spend about 31 seconds in one child-settlement barrier, perform Branch work, and encounter a second barrier before terminal persistence completes.

Retry contract

A QStash 5xx retry cannot resume the same running invocation

QStash is the wake-up mechanism. The durable invocation row owns admission. The first pickup changes the row from planned to running; a later delivery of the same message calls loadPlanned, sees that it is no longer planned, and returns not_planned.

A 5xx can redeliver a lost wake-up while the row is still planned. It cannot carry the model transcript, restore an in-flight provider call, or safely turn a running row back into a new worker. Making that work would require a new state machine and a second duplicate-worker protocol.

› POST /jobs/agent-pickup Admits one planned invocation

The route delegates to agentsService.pickup. A missing planned row is a successful no-op, not a continuation signal.

QStash deliveryplanned rowCAS to runninginline run
  • QStashWakes a durable row; it does not own agent state.
  • One invocationMust never be both the dead worker and the resumed worker.
  • ContinuationNeeds a new planned root, a lineage link, and a retained Branch.

Studio chat

Drain to a safe boundary, then continue without a second user message

On SIGTERM, Fimo marks the old pod as shutting down but leaves accepted streams open while registered work reaches its next safe boundary. If the turn finishes naturally, nothing changes for the user.

If the HTTP stream disappears without a semantic terminal event, Studio preserves the partial reply and makes five bounded attempts to reattach to the same invocation. This is the only path that renders the temporary offline icon and Reconnecting n/5. A graceful deploy_interrupted terminal does not show that row.

After the old invocation settles, Studio requests one durable successor. The server authorizes and rate-limits the mutation, adopts the retained Branch, appends continuation context, and wakes the planned invocation through QStash. The successor completes the same user turn; the old partial assistant row remains as a historical Continued after a Fimo update boundary.

sequenceDiagram
  participant K as Kubernetes
  participant O as Old API pod
  participant R as Redis
  participant N as New API pod
  participant S as Studio
  K->>O: SIGTERM
  O->>O: Finish current safe step
  O-->>S: Semantic deploy_interrupted terminal
  O->>O: Publish Branch and settle old invocation
  S->>N: Request continuation
  N->>N: Create or reuse one planned successor
  N->>R: Publish fresh QStash wake-up
  N-->>S: Successor continues same user turn
Implemented rolling-deploy path. Reconnect to the old producer remains a separate fallback for an ambiguous stream cutoff.
0 to 10 minutesDrain
Server
Keep the provider call, receipt, and existing SSE response alive.
Studio
Continue directly or reconnect through the API replay endpoint.
10 to 11.5 minutesSettle
Server
Abort the remaining root controller and await terminal settlement.
Studio
Silently create or reuse one successor and attach it to the same user turn.
11.5 to 12.5 minutesCleanup
Server
Close the listener, drain requests, finish deferred Branch cleanup, and flush telemetry.
Budget
Use a 13-minute Fimo hard stop and leave about 2 minutes before the 15-minute Kubernetes deadline.
Producer dies earlyFallback
Studio
Shows five bounded reconnect attempts only because no semantic terminal event arrived.
Fallback
Preserves partial text and exposes Retry if no producer or successor can be recovered.
  • Current draftDerives a 10-minute natural window and 13-minute hard stop from the declared 900-second provider grace.
  • No direct Redis clientStudio stays behind project, environment, thread, invocation, and organization authorization in the API.
  • No blind replayThe successor receives explicit continuation context and the retained Branch; it does not resend the dead HTTP request.

State ownership

Redis reconnects the browser, but the old pod still owns the model call

The stream has three different owners. The old API pod owns the live Gemini request and the in-memory replay buffer. Redis carries a stream sentinel, replay requests, chunk fan-out, and abort delivery between API replicas. Postgres owns the durable invocation, assistant parts, lease, terminal state, and billing.

This is why a refresh can recover the same reply while the producer lives, but a dead producer cannot be reconstructed from Redis. The durable terminal row and retained Branch now become an explicit recovery boundary: Studio can create one fresh planned successor without claiming that the provider request itself resumed.

Browser

Studio transportCalls authenticated API endpoints
Live-turn probePolls active state and reconciles history

API replicas

New API podAuthorizes resume and relays chunks
Old API podOwns provider call and replay memory

Shared state

RedisPub/sub bridge while producer lives
PostgresInvocation, parts, lease, settlement, billing
  • Browser boundaryStudio never connects to Redis. The API keeps project, environment, thread, invocation, and organization authorization in one place.
  • Replay boundaryRedis can reach a live producer; it cannot move the provider request or recreate its memory after process death.
  • Durable boundaryPostgres decides whether a turn is live, records the parts that survive process loss, and fences exactly-one successor creation.

User experience

Only a real connection cutoff shows reconnect progress

A rolling deploy remains invisible when the turn reaches a natural finish. When it stops at the safe boundary instead, Studio silently adopts the successor and keeps the same right-aligned user message. It does not show a generic update banner or a reconnect counter.

The reconnect counter belongs only to uncertainty: the browser stream vanished and no semantic terminal event explained why. Studio tries the same invocation five times with backoff. If that cannot recover a producer or discover the durable successor, it preserves the partial reply and offers Retry.

01
Studio stays connected
Starts with
An assistant reply is streaming when a deploy sends SIGTERM.
You do
Keep the conversation open.
You see
The same reply continues from the old pod and normally finishes, with no status row.
Check
The live-turn receipt remains registered until terminal settlement.
Not proven
No rolling-deploy browser canary has exercised the full 10-minute window.
02
The browser stream is cut off
Starts with
No semantic terminal event arrived before the HTTP response disappeared.
You do
Nothing.
You see
The partial reply plus an offline icon and Reconnecting 1/5 through 5/5.
Check
The transport reattaches to the same invocation and never sends the user message again.
Verified
Dev Remote E2E captured the temporary reconnect state and successful same-producer reattach.
Not proven
A real browser or proxy cutoff against production infrastructure has not run.
03
Studio reaches the deploy boundary
Starts with
The reply remains live after 10 minutes.
You do
Nothing. Fimo aborts the root invocation.
You see
No temporary reconnect row. The successor continues below the preserved partial boundary.
Check
Settlement records cancelled with internal reason deploy_interrupted, then one successor adopts the same Branch and user turn.
Verified
Dev Remote E2E asserted exactly one successor, one user row, and zero reconnect attempts on this path.
Not proven
A real rolling deployment has not yet exercised the full old-pod to new-pod handoff.
04
Slack finishes inside the window
Starts with
A Slack agent turn uses the same registered chat-run path.
You do
Wait in Slack while the deploy rolls.
You see
The final Slack reply when the turn settles before cutoff.
Check
The shared live-turn registry includes the Slack execution receipt.
Not proven
Slack has no equivalent replayed browser stream and needs its own cutoff copy.
05
Autonomous work continues later
Starts with
A project agent has useful changes on its task Branch when shutdown begins.
You do
Open the run after the deployment.
You see
The old run is interrupted and a linked continuation adopts the retained Branch.
Check
The old invocation settles before a fresh planned invocation receives a QStash wake-up.
Implemented
The shared shutdown registry now includes autonomous sessions; a real rolling-deploy canary remains outstanding.
Not proven
A production autonomous run has not yet been interrupted and recovered across a deploy.

Delivery state

The code and deterministic UX proof are ready; infrastructure proof is not

The draft implements the application budget, shared shutdown registry, safe-boundary stop, Branch retention, typed cancellation, idempotent Studio successor, and cutoff-only reconnect state machine together.

The rollout order is still strict. Infrastructure must first prove at least 900 seconds of termination grace for API, jobs, and chat. A deployment canary must then show the old worker reaches the boundary and the new worker adopts the successor before this can be treated as production acceptance.

1. Application contractImplemented
Configuration
SHUTDOWN_GRACE_MS=900000 is declared.
Budget
10-minute boundary window, 90-second settlement, 13-minute hard stop.
2. Recover StudioImplemented
Machine state
Carry deploy_interrupted through cancellation settlement.
User state
Automatically continue the same turn; keep Retry only as the exhausted fallback.
3. Distinguish cutoffImplemented
Ambiguous cutoff
Show five reconnect attempts against the same invocation.
Graceful deploy
Show no reconnect row; adopt the durable successor.
4. Prove provider graceBlocked
Infrastructure
Confirm 900 seconds for API, jobs, and chat.
Canary
Exercise real SIGTERM, old-worker boundary, successor pickup, and later SIGKILL.
  • Rollout orderNever deploy a 13-minute application budget while Kubernetes still kills the pod at five minutes.
  • Retry ownershipStudio automatically recovers known lifecycle states; the user Retry remains only after bounded recovery is exhausted.
  • Continuation ownershipThe new autonomous invocation adopts durable work only after the old invocation has settled.

Proof and limits

The deterministic contract passes; the real deploy receipt is still missing

The draft preserves the core boundary from the Fable review: an in-flight provider request is never resumable, but its durable transcript, invocation lineage, and retained Branch can seed a fresh planned continuation. Studio treats a semantic deploy terminal differently from an unexplained network cutoff.

The Dev Remote matrix passed all four paths with zero retries in 143.6 seconds: same-producer reattach, exactly-one deployment successor, unchanged provider error, and exhausted reconnect fallback. Desktop and mobile screenshots were inspected. This proves the deterministic product contract, not Kubernetes timing.

Actual cutoffTemporary Reconnecting 1/5 · Dev Remote
Graceful deploy handoffNo reconnect row · exactly one successor
Same handoff on mobileNormal user bubble · preserved history
Recovery exhaustedPartial reply preserved · Retry remains
Evidence-bearing files20 files
⌄▱apps/api/src
⌄▱apps/ui/src/features/chat
⌄▱packages/agents/src
VerifiedSource contract

Draft PR #2396 implements shutdown, settlement, retained Branch, continuation, and transport-state boundaries.

VerifiedDev Remote E2E

Four recovery paths passed once each with no test retries, including exact successor and user-row counts.

VerifiedRecap rendering

The local and published normal recap render at desktop and mobile widths without horizontal overflow.

UnknownProvider grace

The repository declares 900 seconds; infrastructure has not yet confirmed the effective API, jobs, and chat termination settings.

UnknownRolling-deploy canary

No acceptance run has yet kept a real provider stream open across a two-pod rollout and observed the old worker settle before SIGKILL.

Source peek

Open at head
Path copied