Fixed by start_reply: the note that the agent writes to itself. It is hidden in every turn of this rerun.
PR 3801 · rerun on top of start_reply
What a person reads in the Studio chat for each test, now that the agent marks where its reply starts.
Outcome
start_reply is the tool that PR 3834 added. It is the new "finish turn". The agent calls it when its work is done. The person reads only what the agent writes and shows after the call. Everything before the call is work, and Studio folds it away. In this rerun the agent called it in 20 of 20 turns, and no note to itself reached the person.
The failures that remain are not display problems. Studio draws exactly what the agent sends after the call. Sometimes the agent sends a form and no sentence. That is fine when the form explains itself. It is a problem when the form asks "Delete this line from the home page?" and never shows the line. The person cannot decide.
The pictures are mockups. They use the stored words of each turn, word for word, in the Studio chat layout. They are not screenshots of a running Studio. A dashed outline or a marked phrase shows the problem. Studio does not draw these marks.
Results
Four tests pass now. Four still fail, and three of those fail for the same reason
The table compares the same tests on the same branch, with and without the new tool. A pass means that the judge could read the turn as a marketer and decide. "Before" is the last run without start_reply. "Now" is the rerun on top of it. Price, lifecycle and AI citation ran one time each, so their passes are weak evidence.
| Test | Before | Now | What the person sees now |
|---|---|---|---|
| Internal linking | 1 of 3 | 2 of 3 | The note to itself is hidden. One trial shows a form with no sentence. |
| Content refresh (a price) | Fail | Pass · 1 trial | One sentence, then a form that quotes both pages. |
| Content lifecycle | Fail | Pass · 1 trial | One sentence, then a form that names both stories. |
| AI citation | Fail | Pass · 1 trial | One sentence, then a form that quotes the proposed words. |
| Technical SEO | First run spoke in 5 of 9 | 1 of 3 | One trial: a form that does not name the story. Two runs: the first run still speaks. |
| Brand compliance | Fail | Fail | A form that asks to delete "this line" and never shows the line. |
| Eval 15 (three decisions) | 1 of 6 | 1 of 3 | A good sentence and a form. Two forms have a question over 120 characters. |
| Eval 17 (ten changes) | 0 of 6 | 0 of 3 | A four-step form with no sentence before it. |
Still open, one cause: a form or a card with no sentence, where the form does not say what it is about. This is brand compliance, one internal linking trial, one Technical SEO trial, and eval 17.
Still open, test rules: eval 15 fails on a 120-character limit, and the judge reads a form in the wrong order.
What start_reply fixed
The note that the agent writes to itself no longer reaches the person
The routine said that the Press page is gone. The agent closed its open question. Before the close, it still wrote a note to itself. In this run the note was: "Close the form; the team removed the page themselves. A sentence? The closed card shows reason; the team removed it so they know. Maybe no text needed."
The agent then called start_reply and wrote one message for the person. Studio shows only that message. The note is still stored, but it sits before the call, so it folds away with the rest of the work.
The rule that the remaining failures share
A form with no sentence is fine when the form names its subject, and fails when it does not
These two trials are the same test and the same situation: a story page disappeared. In both, the agent called start_reply, wrote no sentence, and showed a form. So "no sentence" is not the problem by itself.
The left form names the story and says what happened. It passed. The right form says "the old story page" and nothing more. The judge failed it: "Removed story" and "the old story page" do not identify which story or URL is affected, so the marketer cannot safely choose an action.
The agent knew the name in both trials. The routine's report, which only the agent reads, said: /stories/the-archive-is-part-of-the-product ("The archive is part of the product") no longer loads.
Brand compliance
The form asks to delete "this line" and never shows the line
The routine found one sentence on the home page that promises a result: "Field Notes doubles your readers in a month." The team rules forbid such a promise. The routine told the agent to ask if it may delete the line.
The agent showed only this form. The line is in the routine's report, but the person does not read the report.
What is missing
- What the person sees
- A question, twice: in the top row (cut short) and as the big title. Two options. No line.
- What the agent knew
- The exact sentence: "Field Notes doubles your readers in a month." and the rule it breaks.
- The judge
- "Delete this line from the home page?" never identifies or quotes the line, so the marketer cannot confidently decide whether to delete it.
- Also stored, not shown
- The step has a short label, "Home page promise". An open form does not show this label. It shows only after the answer.
- Fix
- The form must quote the line. One sentence before the form would also work.
- Not proven
- That a running Studio draws the form exactly like this mockup. One trial only.
Internal linking
The same question passes with one sentence before it and fails alone
Trial 1 has one sentence that says what is wrong, then the form. Trial 3 has only the form. This form is easier to understand than the brand form, because the question names the Press page. But the person does not learn why the page needs a link.
The judge failed trial 3 for two reasons, and one of them is a flaw in the judge. It read the question after the options, because it reads the form in stored order. It also read the hidden label "Link to /press", which the person does not see.
Technical SEO, first run
On a first run with no problems, the person gets a card with no sentence
The first run found nothing wrong. It could not ask Google about the pages, because Google Search Console is not connected. The test expects this run to stay quiet. In 2 of 4 first runs the routine reported, and the agent spoke to the person.
In the run that the capture holds in full, the person saw only this card. Is this useful one time, or is it noise? This is a product decision.
What happened
- What the person sees
- A card with a title, one line of text and a button. No sentence says that the site check found no problems.
- Before
- The first run spoke in 5 of 9 runs. Now it is 2 of 4.
- Decision for Marc
- Keep the card one time per site, or keep the first run silent.
- Not proven
- The second run that spoke was still running when the capture ended, so its final view is unknown.
Eval 17 · ten changes
A four-step form starts with no sentence, in 3 of 3 trials
The agent has ten small changes to propose. It groups them into a four-step form. That grouping is what we wanted. But it writes nothing before the form, so the person does not know what the agent reviewed or why these changes matter.
The four questions in trial 1: "Which home page meta tag changes should I apply?", "Which changes to text visitors see on the home page?", "Which alt text should I add to the home page images?", "Which fixes for /stories/quiet-launches?"
What the eval checks
- Rule that failed
- "no text before the form". The eval wants at least one sentence.
- All other rules
- Passed: one form, at most four steps, the changes can be picked one by one, the form holds the proposed words.
- Probable cause
- The
start_replydescription says "with nothing to tell them, write nothing after it". PR 3801 says the form holds the question. The agent reads both and decides that the form is enough. - Not proven
- The options of each step were not captured. The mockup draws a placeholder.
Eval 15 · three decisions
The sentence is good; the eval fails a question that is longer than 120 characters
Here the agent writes one sentence and then a form. The eval fails 2 of 3 trials only because one question is longer than 120 characters: 198, 204 and 121 characters. The long questions hold the reason and the risk of the change, which is useful to the person.
A fixed character limit is the kind of hard limit that you rejected for agent output. A judge that asks "is this question easy to read?" fits better. The live tests already work this way.
The three trials
- Trial 1
- Pass. Questions of 93, 95 and 105 characters.
- Trial 2
- Fail. Questions of 198, 120 and 204 characters.
- Trial 3
- Fail. Questions of 108, 114 and 121 characters. One character over the limit.
- Proposal
- Replace the 120-character rule with the judge.
- Not proven
- That the judge passes the long questions. The options were not captured.
What to do next
One sentence about forms, two test changes, and one product decision
Nothing here is done yet. Each item waits for your word. Item 1 is the only change to what the agent reads. It is one sentence, and it gives the reason instead of a rule: the person may see the form and nothing else. Items 2 and 3 change only the tests. Item 4 is a product decision that I cannot make.
Tell the agent that a form can be the only thing the person sees. One sentence in the form tool description: "The person may see only this form. So the form says what it is about: it quotes the line, or names the page or the story." Then delete the other form guidance in PR 3801 that start_reply made unnecessary.
Eval 15: replace the 120-character rule with the judge.
Judge (tests only): read each form as Studio draws it. The question comes first, then the options. Do not read the hidden step label.
Your decision: the "Connect Google Search Console" card on a first run. Useful one time, or noise?
Live test logs and the stored chat rows of the rerun at d6a87b8e50 (9 Nov). Every quoted sentence is word for word.
Mockups in the Studio chat layout, from the stored words. The form rules come from the Studio code (QuestionForm.tsx, chat-block-state.ts). No screenshot of a running Studio.
Price, lifecycle and AI citation passed one time each.
The PR head is now 692f696238, rebased on develop. About 13 newer develop commits were not tested with this branch.