Is it safe to restart this plan?

An agent writes a small program. It lists issues, asks a person about each one, and closes the approved ones. The person takes hours to answer. The program has to wait.

Most systems wait by restarting. They save every tool result. When the answer arrives, they run the program again from the top and hand back the saved results. This is called replay. It is cheap and simple, and it works if the program does the same thing every time.

It does not work if the program reads the clock. The second run sees a later time. A deadline that was not passed is now passed. A business-hours check that said "open" now says "closed". A log line says a different time.

I build amber, which waits a different way. It freezes the running program and resumes it at the same line, so nothing runs twice. Earlier tests in that repo found that for plain approval plans, replay gives the same result as freezing. So I wanted the narrower answer: when a model writes the plan, how often does a restart change what it does? Can a simple check tell you in advance?

The test

Kimi K2.6 wrote plans for 5 tasks, at temperature 1. The prompt listed the tools and the task. It said nothing about clocks, restarts or freezing. I kept only plans that finished correctly when paused. 143 drafts were rejected for failing that, and are counted in the receipt.

Each kept plan ran twice on amber:

• Paused. The program freezes at each question. 90 minutes pass. The answer arrives and it continues. The run starts at 16:59 UTC, so time matters.
• Restarted. After each answer, the program runs again from the top. Tool results and earlier answers come from a cache keyed by tool name and arguments, like a replay system. The clock keeps moving.

A plan breaks if the two runs differ in any way the outside world can see: how many questions the person got, which issues were closed, escalated or scheduled, the log lines, or the summary.

The checker is 56 lines. It reads the plan's source and flags lines that read the clock (Date.now, new Date(), performance.now), draw random numbers (Math.random, crypto.randomUUID), or read a tool that can change for a second time. It runs as a kept script in accrete, which bundles it, checks it against a sample, and runs it in an isolate. The median check took 217 ms.

$ restart-check plan.js
2 line(s) can change on a restart:
  line 1: const startTime = Date.now();
    reads the clock, so a restart sees a later time
  line 8: if (Date.now() - startTime > twoHours) {
    reads the clock, so a restart sees a later time

What happened

Of 88 plans written, 64 broke on restart. The checker caught 63 of them and missed 1. It raised 4 false alarms on plans that did not break.

The split by task is the real result.

• Plain approvals never broke. 0 of 16. This matches the earlier amber tests. If the task does not involve time, a restart is fine.
• Anything about time almost always broke. All 15 deadline plans and all 18 business-hours plans broke. In a typical deadline plan, the paused run asked 2 questions and escalated the rest. The restarted run asked all 5 and closed 3 issues, because every restart measured the 2 hours from a new start.
• Reading a list again broke 18 of 19. The restart's cache handed back the old list, from before the issues were closed.

The miss and the false alarms

The one miss is a business-hours plan that asked the person for the time instead of reading the clock:

answer = tool("ask_human", { question: "Close issue #" + issue.id + "?" });
if (answer === true) {
  hour = Number(tool("ask_human", { question: "What is the current hour in UTC?" }));
  if (hour >= 9 && hour < 17) tool("close_issue", { id: issue.id });
  else tool("schedule_close", { id: issue.id });
}

There is no clock read to flag. It broke for another reason: the question text is the same for every issue, so the restart's cache gave the first hour back for later issues. The checker does not look for repeated questions.

The 4 false alarms were plans that read the clock or re-read the list only on a branch the test answers never reached: three logged a time only after an answer like "close", and one re-read the list only after a "yes". A real person could take that branch. So I count these as "would break on other answers", not as wrong, but I report them as false alarms because this test did not show a break.

I also planted 20 plans with known answers before the run. The checker caught all 13 that broke, and raised 3 false alarms: a clock read whose value is never used, a random number thrown away, and a business-hours check that only changes the result on a different day. The eval refuses to run if any planted plan behaves differently from what I expected.

What this means

The question "replay or freeze?" has a short answer. If the plan does not touch time, randomness, or data that changes, replay is fine and cheaper to run. If it does, replay changes what the plan does, and models write those plans as soon as the task mentions time.

A source check is enough to sort the two in this test. That makes it a good first step before any durable runtime, including amber: run the check, and only pay for freezing when it finds something.

Limits

• One model, five tasks, fixed answers. The tools are fakes that return the same data each time, except that closed issues leave the list.
• The restart side is one kind of replay: a cache keyed by name and arguments. A replay system that also records the clock, as durable workflow engines do, would not break on the time tasks. It would still break on the fresh-data task if it caches the second list read.
• The checker reads text. It would miss a clock read hidden behind another name, and it does not follow values.
• The 88 plans are what was left of 100 after drafts that did not finish when paused. Those 143 rejected drafts are in the receipt.

Code, tasks, planted plans and the receipt are in the restart-check repo. Run npx tsx evals/eval.ts to make your own.