feat(codemode): add durable execution retries

Author: Matt Carey

.changeset/codemode-runtime-retries.md

@@ -0,0 +1,5 @@
+---
+"@cloudflare/codemode": patch
+---
+
+Add durable execution retries to `runtime.tool()`. Connectors can throw `RetryableError` with an optional delay; by default the runtime makes three total attempts, honors that delay or uses bounded exponential backoff, and can be customized or disabled with `retry`. Failed passes restart under the same execution id, replaying applied calls from the log and re-executing only the failure boundary. Dynamic-worker timeouts are surfaced as structured failures but are not retried by default, so applications can conservatively decide which executions are safe to retry. Attempt fencing prevents calls or results from a superseded timed-out sandbox from mutating the replay log.

docs/codemode/runtime.md

@@ -23,7 +23,7 @@ const runtime = createCodemodeRuntime({
 
 | Handle method                                        | Purpose                                                                                                                                    |
 | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
-| `runtime.tool(options?)`                             | The single model-facing AI SDK tool, `codemode({ code })`                                                                                  |
+| `runtime.tool(options?)`                             | The framework-independent model-facing tool, `codemode({ code })`                                                                          |
 | `runtime.pending(executionId?)`                      | Actions awaiting approval — drives approval UIs; no id aggregates all paused runs                                                          |
 | `runtime.approve({ executionId })`                   | Approve the pending action and continue via replay                                                                                         |
 | `runtime.reject({ seq, executionId })`               | Reject a pending action; ends the execution. Returns `false` if it was a no-op (action no longer pending — approved or rejected elsewhere) |
@@ -149,6 +149,32 @@ A tool can opt out of result recording with [`replay: "reexecute"`](./connectors
 
 Any single recorded value (a call's arguments, a recorded result, the final result) is capped at 1 MB serialized (`MAX_DURABLE_VALUE_BYTES`). Truncating a logged value is never an option — replay would feed resumed code corrupted data — so an oversized argument or call result **fails the run** with a model-actionable error suggesting the data be written to a file/workspace and passed by reference. An oversized **final** result does not fail the run (replay never needs it): the run completes, the model receives the real value, and the audit trail stores a placeholder note.
 
+## Retrying failed passes
+
+A connector requests a retry by throwing `RetryableError`. By default the runtime makes three total attempts, honoring `retryAfterMs` when present and otherwise using bounded exponential backoff (500ms, 1s, up to 10s). A retry keeps the same execution id and re-runs the stored code: applied calls replay from the durable log, while the call left `executing` at the failure boundary executes again. No configuration is needed for this default.
+
+```ts
+const runtime = createCodemodeRuntime({ ctx, executor, connectors });
+```
+
+Customize the policy when needed, or pass `retry: false` to disable automatic retries:
+
+```ts
+const runtime = createCodemodeRuntime({
+  ctx,
+  executor,
+  connectors,
+  retry: {
+    maxAttempts: 4,
+    shouldRetry: ({ failure }) => failure.kind === "retryable"
+  }
+});
+```
+
+The error message and optional `retryAfterMs` cross the connector RPC and sandbox boundaries as structured metadata. Dynamic-worker timeouts also arrive as `failure.kind === "timeout"`, but are not retried by default: an operation may have succeeded remotely before its response was lost, so the application must decide whether that boundary is safe to repeat.
+
+Before retry policy or delay callbacks run, the runtime advances a durable attempt fence. Late calls and results from the superseded sandbox are inert and cannot overwrite the newer pass.
+
 ## Rollback
 
 Rollback walks the log backward and calls the `revert` of **every** applied action that has one — independent of `requiresApproval`. A non-approval write with a `revert` is still undone; an approval-gated action without a `revert` is not. `revert` (via `revertAction`) returns whether it actually reverted, and the runtime marks only those entries `reverted`:

packages/codemode/README.md

@@ -380,6 +380,49 @@ The `tool(name, t)` decoration hook adjusts tools you didn't author inline (used
 
 The agent drives approvals through the runtime: `runtime.pending()`, `runtime.approve({ executionId })`, `runtime.reject({ seq, executionId })`, `runtime.rollback({ executionId })` (see [docs/codemode/approvals.md](../../docs/codemode/approvals.md)).
 
+### Durable retries
+
+A runtime automatically retries `RetryableError` under the same execution id, up to three total attempts. Applied connector calls replay from the durable log; the call at the failure boundary executes again. `retryAfterMs` is honored when present, otherwise retries use bounded exponential backoff (500ms, 1s, up to 10s).
+
+```ts
+import { RetryableError } from "@cloudflare/codemode";
+
+// Connector code can signal a transient, safe-to-repeat failure.
+throw new RetryableError("Rate limited", { retryAfterMs: 2_000 });
+
+const runtime = createCodemodeRuntime({
+  ctx,
+  executor,
+  connectors
+  // No retry configuration needed for RetryableError.
+});
+
+// Customize the policy when needed — for example, to opt safe reads into
+// timeout retries. Timeout retries are off by default because a timed-out
+// mutation may already have succeeded remotely.
+const customized = createCodemodeRuntime({
+  ctx,
+  executor,
+  connectors,
+  retry: {
+    maxAttempts: 4,
+    shouldRetry: ({ failure, execution }) =>
+      failure.kind === "retryable" ||
+      (failure.kind === "timeout" && timeoutIsSafe(execution))
+  }
+});
+
+// Or disable automatic retries entirely.
+const noRetries = createCodemodeRuntime({
+  ctx,
+  executor,
+  connectors,
+  retry: false
+});
+```
+
+Each pass has a durable attempt fence. If a timed-out old sandbox finishes after its retry started, its later calls and results are ignored rather than corrupting the replay log.
+
 ### Snippets
 
 Snippets are durable, addressable saved scripts. The model writes and runs scripts; the developer promotes the ones worth keeping (`runtime.saveSnippet`), and the model reuses them (`codemode.run`). No authoring step, no skill-source interface.

packages/codemode/src/connectors/types.ts

@@ -57,7 +57,7 @@ export type ExecutionEndStatus =
  * resources (an open socket, a lease) should be released even though
  * per-execution resources (a session) must survive.
  */
-export type PassEndStatus = ExecutionEndStatus | "paused";
+export type PassEndStatus = ExecutionEndStatus | "paused" | "retrying";
 
 // ---------------------------------------------------------------------------
 // Connector description — returned by describe() RPC.

packages/codemode/src/executor-types.ts

@@ -3,9 +3,13 @@
  * code in a sandbox (DynamicWorkerExecutor, IframeSandboxExecutor, ...).
  */
 
+import type { ExecuteFailure } from "./retry";
+
 export interface ExecuteResult {
   result: unknown;
   error?: string;
+  /** Structured failure metadata. Executors should set this when available. */
+  failure?: ExecuteFailure;
   logs?: string[];
 }
 

packages/codemode/src/executor.ts

@@ -372,6 +372,11 @@ export class DynamicWorkerExecutor implements Executor {
         `          if (__r && typeof __r === "object") {\n` +
         `            if (__r.${CONNECTOR_CONTROL_KEY} === "pause") throw new Error("${PAUSE_SENTINEL_LITERAL}");\n` +
         `            if (__r.${CONNECTOR_CONTROL_KEY} === "error") throw new Error(String(__r.message));\n` +
+        `            if (__r.${CONNECTOR_CONTROL_KEY} === "retryable") {\n` +
+        `              const __e = new Error(String(__r.message));\n` +
+        `              __e.__codemode_failure__ = { kind: "retryable", message: String(__r.message), retryAfterMs: __r.retryAfterMs };\n` +
+        `              throw __e;\n` +
+        `            }\n` +
         `          }\n` +
         `          return __r;\n` +
         `        };\n` +
@@ -400,13 +405,13 @@ export class DynamicWorkerExecutor implements Executor {
       .concat([normalized])
       .concat([
         ")(),",
-        '        new Promise((_, reject) => setTimeout(() => reject(new Error("Execution timed out")), ' +
+        '        new Promise((_, reject) => setTimeout(() => { const e = new Error("Execution timed out"); e.__codemode_failure__ = { kind: "timeout", message: "Execution timed out" }; reject(e); }, ' +
           timeoutMs +
           "))",
         "      ]);",
         "      return { result, logs: __logs };",
         "    } catch (err) {",
-        "      return { result: undefined, error: err.message, logs: __logs };",
+        "      return { result: undefined, error: err.message, failure: err.__codemode_failure__, logs: __logs };",
         "    }",
         "  }",
         "}"
@@ -472,13 +477,19 @@ export class DynamicWorkerExecutor implements Executor {
       ): Promise<{
         result: unknown;
         error?: string;
+        failure?: import("./retry").ExecuteFailure;
         logs?: string[];
       }>;
     };
     const response = await entrypoint.evaluate(dispatchers, connectorBindings);
 
-    if (response.error) {
-      return { result: undefined, error: response.error, logs: response.logs };
+    if (response.error || response.failure) {
+      return {
+        result: undefined,
+        error: response.error,
+        failure: response.failure,
+        logs: response.logs
+      };
     }
 
     return { result: response.result, logs: response.logs };

packages/codemode/src/index.ts

@@ -37,6 +37,17 @@ export {
   type PendingAction
 } from "./runtime";
 export { type Snippet, type SaveSnippetOptions } from "./snippet";
+export {
+  RetryableError,
+  DEFAULT_RETRY_MAX_ATTEMPTS,
+  DEFAULT_RETRY_BASE_DELAY_MS,
+  DEFAULT_RETRY_MAX_DELAY_MS,
+  defaultRetryDelayMs,
+  type ExecuteFailure,
+  type CodemodeRetryContext,
+  type CodemodeRetryOptions,
+  type CodemodeRetryPolicy
+} from "./retry";
 export {
   createCodemodeRuntime,
   type CreateCodemodeRuntimeOptions,