In short
- Model the outcome, not the chat. Store a fixed goal, success criteria, a disposable plan, a budget and a wait state in Postgres before you write any agent code.
- Buy the plumbing: integrations, sandboxes, browser, durable jobs, memory, messaging, phone and meetings. Spend your own time on the outcome engine, permissions, replanning, memory rules and verification.
- Use the API when one exists, a browser when it does not, and a sandbox for anything the agent has to compute or generate. Put a hard spend cap on every outcome.
- Park long waits in the database, not in a running process. A reply that arrives three days later should resume the outcome from stored state.
- Add a separate verifier that checks the result against the success criteria. A tool saying "sent" is not done.
Who it is for: Engineers and technical founders who want to build an agent that does real work across tools, not a chatbot. You should be comfortable with Postgres, background jobs and calling APIs from code.
Overview
What you will have at the end
In late September, Walid saw a consumer outcome agent announce a $1B raise. He was already busy, but he got curious about the best way to build something like it. So he tested it. He asked indirect questions, gave it odd jobs, ran parallel tasks, changed instructions while it worked, made it generate files and code, and inspected what came back.
What he found is a short list of parts. Most of them you can buy. A few you have to build, and those decide whether your agent finishes real work or only looks busy. This playbook turns that list into a build plan with the schema, the prompts and the code for the awkward parts.
Diagram
The whole system, from goal to approved result
User sets a goal
one fixed sentence plus a budget
Outcome engine
state in Postgres, Supabase or Convex
Planner
success criteria, parallel tracks, replans
Tool layer
Integrations, data, browser, sandbox
Composio, looot, Browserbase and Stagehand, Daytona or E2B
Channels
Email, chat, phone, meetings
Unipile, Vapi, Recall.ai
Durable runner
Trigger.dev or Inngest: retries, waits, wake-ups
Verifier
checks the result against the success criteria
Person approves in the agent UI
CopilotKit and AG-UI
The outcome row in your database is the center. Everything else reads from it and writes back to it.
This is an advanced build. You will write a database schema, background job code and evaluation tests. If you want to automate a single workflow, the n8n production playbook is a better start. Come back here when one workflow is no longer enough.
Background
Who this is for and what you need
This guide is for people building the agent itself: a product team adding an agent to an app, an internal platform team, or a founder who wants a personal agent that can act across email, files and websites. It assumes you have shipped backend code before. It does not teach you to call a model API.
A note on why tools matter so much. Walid wrote in November that the gap between an agent with tools and one without feels closer to 100x, and that the bottleneck is becoming access to tools. That is why most of this stack is about reaching tools safely, and why the model is one box among many.
Before you start
- A Postgres database. Supabase, Convex or plain Postgres all work. You need transactions and row locks for the outcome state.
- A durable job runner account: Trigger.dev or Inngest. Check each vendor's docs for current plans and limits.
- A sandbox provider for code execution: Daytona or E2B. Check their docs for pricing and session limits.
- A browser provider for sites without APIs: Browserbase with Stagehand.
- An integrations layer: Composio for auth and tool discovery. Add looot if the agent needs data endpoints (one key or MCP connection, priced per call before it runs).
- Model access for a planner and an executor. Walid's own pairing, posted in November: Fable 5.2 for planning and Opus 5.5 for execution. Pick what holds up on your tasks.
- A test set of 10 to 20 real goals with known correct results. You will use it for verification in step 7.
- A named person who owns permissions and approves risky actions during testing.
Step 1
Model the outcome before the agent
Walid's first finding was "outcomes, not agents." You give the system a goal. It can split that goal into parallel tracks, throw away a bad plan and try another one. The destination stays fixed. The plan is disposable. Build that into your data model first, because it is hard to retrofit.
In the schema below, goal is written by a person and the agent never edits it. plan and plan_version change as often as needed. waiting_on and resume_at let an outcome sleep. budget_usd and spent_usd give you a hard cap. locked_by stops two workers from acting on the same outcome at once. The Postgres, Supabase or Convex choice does not change this shape, only the syntax.
Schema
Outcomes, events and permissions
create table outcomes (
id uuid primary key default gen_random_uuid(),
created_at timestamptz not null default now(),
created_by text not null,
goal text not null,
success_criteria jsonb not null,
status text not null default 'planning'
check (status in ('planning','running','waiting','needs_approval','verifying','done','failed','cancelled')),
plan_version int not null default 1,
plan jsonb,
waiting_on text,
resume_at timestamptz,
budget_usd numeric(10,2) not null,
spent_usd numeric(10,2) not null default 0,
locked_by text,
locked_until timestamptz
);
create table outcome_events (
id bigint generated always as identity primary key,
outcome_id uuid not null references outcomes(id),
created_at timestamptz not null default now(),
kind text not null
check (kind in ('plan','replan','tool_call','wait','resume','approval_asked','approval_result','verify')),
tool text,
proposed jsonb,
executed jsonb,
previous jsonb,
cost_usd numeric(10,4) not null default 0,
note text
);
create table permissions (
id bigint generated always as identity primary key,
pattern text not null,
scope jsonb not null,
granted_by text not null check (granted_by <> 'agent'),
granted_at timestamptz not null default now(),
expires_at timestamptz,
revoked_at timestamptz
);The previous column on events is your undo. Log the value before every write, and you can reverse a bad run. scope holds JSON such as the allowed tools, recipients and a daily maximum. success_criteria holds a list of checks, each with an id and an evidence type (file, number, message or state).
- Never let the plan overwrite the goal. If the plan changes, bump
plan_versionand log areplanevent with the reason. - Write the event before you call the tool, then update it with the result. A crash then leaves a record of what was attempted.
- Lock the outcome row while a worker acts on it, with a short
locked_untilso a dead worker releases it. - Set `budget_usd` when the goal is created, by a person. The agent can ask for more, but it cannot raise its own cap.
Step 2
Buy the plumbing, build the five hard parts
Walid's advice for building something similar today: do not rebuild the supporting systems. Get these, and spend engineering time on the parts that are specific to your product. The table lists each component, what it gives you and what you still have to build around it. Check each vendor's docs for current features and pricing before you commit. We only state the role each one plays.
| Component | Role | What you still build |
|---|---|---|
| Composio | Integrations, auth and tool discovery | Which tools each outcome may use, and the permission check in front of them |
| CopilotKit and AG-UI | Agent UI, streaming, approvals and app to agent communication | The approval cards, what they show, and what happens on yes or no |
| Daytona or E2B | Isolated computers where agents write and run code | What goes in and out of the sandbox, and cleanup so sessions do not pile up |
| Browserbase and Stagehand | Browser execution when there is no good API | The rule for when the browser is allowed, and checks on what the page returned |
| Trigger.dev or Inngest | Durable jobs, retries, waiting and wake-ups | The outcome state machine that the jobs read and write |
| supermemory | Long-term user memory and retrieval | The rules for what memory may change (step 6) |
| Supabase, Convex or Postgres | Outcomes, permissions, state, locks and audit logs | The schema from step 1 and the queries that enforce it |
| Unipile | Email, WhatsApp, LinkedIn and messaging | Recipient checks, rate caps and the approval for first contact |
| Vapi | Phone calls | Scripts, consent rules and how a call result feeds the verifier |
| Recall.ai | Meetings turned into context | Which meeting content may become a goal or a memory |
Composio
- Role
- Integrations, auth and tool discovery
- What you still build
- Which tools each outcome may use, and the permission check in front of them
CopilotKit and AG-UI
- Role
- Agent UI, streaming, approvals and app to agent communication
- What you still build
- The approval cards, what they show, and what happens on yes or no
Daytona or E2B
- Role
- Isolated computers where agents write and run code
- What you still build
- What goes in and out of the sandbox, and cleanup so sessions do not pile up
Browserbase and Stagehand
- Role
- Browser execution when there is no good API
- What you still build
- The rule for when the browser is allowed, and checks on what the page returned
Trigger.dev or Inngest
- Role
- Durable jobs, retries, waiting and wake-ups
- What you still build
- The outcome state machine that the jobs read and write
supermemory
- Role
- Long-term user memory and retrieval
- What you still build
- The rules for what memory may change (step 6)
Supabase, Convex or Postgres
- Role
- Outcomes, permissions, state, locks and audit logs
- What you still build
- The schema from step 1 and the queries that enforce it
Unipile
- Role
- Email, WhatsApp, LinkedIn and messaging
- What you still build
- Recipient checks, rate caps and the approval for first contact
Vapi
- Role
- Phone calls
- What you still build
- Scripts, consent rules and how a call result feeds the verifier
Recall.ai
- Role
- Meetings turned into context
- What you still build
- Which meeting content may become a goal or a memory
Add looot to this list if the agent needs outside data. It gives the agent one key or MCP connection to thousands of data endpoints, and each call is priced before it runs. That price check is useful for the budget cap in step 4.
Walid's list of what is left over is short, and it is where the hard engineering sits. These five are not in any vendor's box.
- Hard part 1
Outcome engine
Goes wrong: the agent forgets the goal after a long wait or a replan. Build: the outcome table, the state machine, locks and the budget.
- Hard part 2
Permissions
Goes wrong: the agent uses a broad token for a narrow job. Build: scoped credentials, a permission table, approval for risky writes and a daily cap.
- Hard part 3
Replanning
Goes wrong: the agent keeps retrying a dead plan, or abandons a good one. Build: abandon conditions in the plan and a log of every replan with the reason.
- Hard part 4
Memory rules
Goes wrong: an observed habit becomes a standing instruction nobody agreed to. Build: a proposal flow where only a person grants.
- Hard part 5
Verification
Goes wrong: a tool returns success and the job is wrong. Build: a separate verifier and a test set with known answers.
Step 3
Write the planner: success criteria first
The planner has three jobs. It writes success criteria that a third party could check. It splits the work into tracks that can run in parallel. And it states, up front, what would make it abandon the plan. The third one is what makes a plan disposable: you decide the exit conditions before the agent gets attached to its own approach.
Success criteria are the contract with the verifier in step 7. "Send the report" is a task. "The report PDF reaches the named recipient and its totals match the source sheet" is a criterion. Write the second kind.
Prompt
Outcome planner
You turn a goal into a plan that can be thrown away. The goal stays fixed.
Goal (written by a person, do not rewrite it):
{{goal}}
Context you may use:
{{context_json}}
Tools available (name and what each one can do):
{{tool_list}}
Budget in USD: {{budget_usd}}
Return JSON only:
{
"success_criteria": [
{"id": "c1", "check": "a statement a third party could verify", "evidence_type": "file | number | message | state"}
],
"tracks": [
{"id": "t1", "purpose": "...", "steps": ["..."], "tools": ["..."], "can_run_in_parallel_with": ["t2"], "estimated_cost_usd": 0.0}
],
"abandon_plan_if": [
"a condition that tells us this plan is wrong, with a number or a visible signal"
],
"needs_permission_for": [
{"action": "...", "why": "..."}
],
"open_questions": ["only questions that block the first step"]
}
Rules:
- Every success criterion must be checkable from an artefact: a file, a number, a message, or a state in a system. "Done well" is not a criterion.
- Name the exact recipient, the exact file type and the exact figures wherever the goal implies them.
- Split into parallel tracks only when the tracks do not depend on each other.
- If an API exists for a step, plan the API. Plan the browser only when no API covers it.
- Total estimated cost must stay under the budget. If it cannot, say so in open_questions. Do not shrink the criteria to fit.
- Anything that sends a message to a person, spends money, or changes data you did not create goes in needs_permission_for.Run the planner on a stronger model and execution on whatever holds quality for the cheaper steps. Walid plans with Fable 5.2 and executes with Opus 5.5.
| Signal | Example | What the engine does |
|---|---|---|
| A tool failed twice with the same error | The API rejects the request format | Mark the track failed, ask the planner for a different path, bump plan_version |
A condition in abandon_plan_if became true | The vendor site no longer has the form | Switch the track to another tool, keep the goal |
| New instruction from the person | "Use the second supplier instead" | Keep the goal, replan the affected tracks, log the reason |
| Spend passed a set share of the budget with the criteria unmet | Many retries on a hard step | Stop, ask the person, do not silently continue |
A tool failed twice with the same error
- Example
- The API rejects the request format
- What the engine does
- Mark the track failed, ask the planner for a different path, bump
plan_version
A condition in
abandon_plan_ifbecame true- Example
- The vendor site no longer has the form
- What the engine does
- Switch the track to another tool, keep the goal
New instruction from the person
- Example
- "Use the second supplier instead"
- What the engine does
- Keep the goal, replan the affected tracks, log the reason
Spend passed a set share of the budget with the criteria unmet
- Example
- Many retries on a hard step
- What the engine does
- Stop, ask the person, do not silently continue
Pick the spend threshold yourself and write it into the engine, not into the prompt.
Step 4
Build the tool layer: API first, browser second, sandbox for the weird jobs
Walid tested three behaviors that matter here. When he asked it to generate a video, it wrote Python with Pillow, NumPy and ffmpeg. The spreadsheet it produced was built with Python and openpyxl. For odd jobs, it can create the tooling it needs while doing the job. That is the case for a sandbox: an isolated computer where the agent can write and run code without touching your real systems. Daytona and E2B fill that role.
The second behavior is the order of attempts. If an API exists, use it. If not, use the website. Do not memorize clicks. Remember the goal and work out the path again when the site changes. In stack terms: Composio for integrations, Browserbase with Stagehand for the browser, and a rule in your own code that the browser is the fallback.
| Order | Use | When | What to log |
|---|---|---|---|
| 1 | API through your integrations layer | An API covers the step | The call, the response and the previous value on writes |
| 2 | Data endpoint (for example looot) | The step needs outside data | The price shown before the call and the cost after |
| 3 | Browser session | No usable API | What the agent saw on the page and which element it acted on |
| 4 | Sandbox | The agent must compute, convert or generate a file | The code it ran, the inputs and the output files |
1
- Use
- API through your integrations layer
- When
- An API covers the step
- What to log
- The call, the response and the previous value on writes
2
- Use
- Data endpoint (for example looot)
- When
- The step needs outside data
- What to log
- The price shown before the call and the cost after
3
- Use
- Browser session
- When
- No usable API
- What to log
- What the agent saw on the page and which element it acted on
4
- Use
- Sandbox
- When
- The agent must compute, convert or generate a file
- What to log
- The code it ran, the inputs and the output files
Keep the browser rule in code, not in the prompt. The planner proposes a track, and the tool router checks whether an API path exists before it allows a browser session. That keeps the agent from clicking through a site when a clean call would do.
- Give every sandbox a time and cost limit, and close it when the track ends.
- Treat files from the sandbox as untrusted input until the verifier has opened them.
- Give the agent its own credentials with the smallest permissions. Never reuse a person's admin token.
- Log every proposed and executed action with the previous value, so you can undo.
- Cap daily volume for anything that touches other people, such as messages or calls.
Step 5
Make long waits cheap: the outcome stays alive, the compute does not
Walid's example: send an email, wait three days, the reply arrives, continue from exactly where you stopped. The compute does not need to stay alive for three days. The outcome does. This is the main reason to use a durable job runner such as Trigger.dev or Inngest, which handle retries, waiting and wake-ups. Check their docs for the exact functions, limits and pricing.
Your part is the state. When the agent starts waiting, write status = waiting, waiting_on and resume_at to the outcome row. When anything wakes the outcome, read that row and continue from it. Never rely on variables in a process that may have restarted. The pseudocode below shows the whole pattern, including the permission gate and a send that cannot run twice.
Code
Send, wait three days, resume on reply (pseudocode)
// PSEUDOCODE. Map each call to your durable job runner (Trigger.dev, Inngest or similar).
// The point: the process can exit. The outcome row keeps the state.
async function sendAndWaitForReply(outcomeId: string, draft: Draft) {
const outcome = await db.outcomes.get(outcomeId)
// 1. Permission gate before anything leaves the building
const allowed = await permissions.allows("email.send", draft)
if (!allowed) {
await db.outcomes.update(outcomeId, { status: "needs_approval", waiting_on: "approval:email" })
await ui.askPerson(outcomeId, draft) // approval card in your agent UI
return waitFor.approval(outcomeId) // durable: nothing runs while we wait
}
// 2. Send once. The idempotency key stops a retry from sending twice.
const sent = await tools.email.send(draft, { idempotencyKey: outcomeId + ":send:" + outcome.plan_version })
await db.events.insert({ outcome_id: outcomeId, kind: "tool_call", tool: "email", executed: sent })
// 3. Park the outcome. No compute is held for 3 days.
await db.outcomes.update(outcomeId, {
status: "waiting",
waiting_on: "reply:" + sent.threadId,
resume_at: daysFromNow(3),
})
// 4. Wake on whichever comes first: the reply event or the timer.
const wake = await waitFor.firstOf([
waitFor.event("email.reply", { threadId: sent.threadId }),
waitFor.time(daysFromNow(3)),
])
// 5. Resume from the stored state, not from memory in a process.
await db.outcomes.update(outcomeId, { status: "running", waiting_on: null })
await db.events.insert({ outcome_id: outcomeId, kind: "resume", note: wake.kind })
if (wake.kind === "timer") return followUpOrReplan(outcomeId) // no reply: nudge, or change plan
return continueWithReply(outcomeId, wake.payload) // reply arrived: carry on
}This is pseudocode, not a Trigger.dev or Inngest function. Map each wait and event call to your runner's own API.
- Idempotency on every send. A retry after a timeout must not send a second email.
- Two wake-up reasons. The reply, or the timer, whichever comes first. A missing reply is information too.
- Resume writes an event. You can see in the log how long the outcome slept and what woke it.
- Lock on resume. If a reply and a timer arrive together, only one worker continues.
Step 6
Permissions and memory: observed patterns never grant themselves
Walid's fourth finding: the agent can notice patterns in how you work, but it does not turn a pattern into permission automatically. It asks first. One yes can turn an observed pattern into a standing instruction. Copy that rule exactly. Memory is useful for relevance and drafts. It must not be the thing that decides what the agent may do.
Supermemory gives you long-term user memory and retrieval. The rules around it are yours: what counts as a repeated pattern, what the request to the user says, what scope the permission has and when it expires. Store the answer in the permissions table from step 1 with granted_by set to a person. Nothing in the code path should be able to write agent there.
Prompt
Memory rule proposal
You review notes about how a user works and decide whether to ask for a standing instruction. You never grant yourself anything.
Observations (each with dates and the outcome it came from):
{{observations_json}}
Existing permissions (do not propose a duplicate):
{{permissions_json}}
If the same pattern shows up at least {{min_repeats}} times and the user did not object, draft a request. Otherwise return {"ask": false}.
The request is written to the user in plain words. It must contain:
- what you noticed, with the dates
- the exact standing instruction you propose, in one sentence
- the scope: which tools, which recipients or records, how many per day
- an expiry date
- how to revoke it
Return JSON only:
{
"ask": true,
"message_to_user": "I noticed ... May I make this a standing instruction? ...",
"proposed_permission": {"pattern": "...", "scope": {...}, "expires_at": "YYYY-MM-DD"}
}
Rules:
- Observing a pattern gives no permission. Only a "yes" from the user does, and the app records that, not you.
- Never widen the scope beyond what you observed.
- Never ask about anything involving money, deleting data or messaging someone for the first time.The model drafts the question. Your app shows it, records the answer and writes the permission row. The model never does.
| Rule | Why |
|---|---|
| Every standing instruction has a scope and an expiry | Old habits stop applying without anyone remembering to revoke them |
| Money, deletes and first contact always ask | These are the actions that hurt most when wrong |
| A risky write needs a person's approval each time | The agent proposes, a person approves |
| Revoking is one click and takes effect on the next call | People trust what they can switch off |
| Log approvals asked and granted | The ratio tells you whether the agent asks too often or too rarely |
Every standing instruction has a scope and an expiry
- Why
- Old habits stop applying without anyone remembering to revoke them
Money, deletes and first contact always ask
- Why
- These are the actions that hurt most when wrong
A risky write needs a person's approval each time
- Why
- The agent proposes, a person approves
Revoking is one click and takes effect on the next call
- Why
- People trust what they can switch off
Log approvals asked and granted
- Why
- The ratio tells you whether the agent asks too often or too rarely
Approvals need a place to happen. That is what CopilotKit with AG-UI is for: streaming agent progress into your app and showing approval cards where the person already works. The card should show the exact action, the recipient or record, and the evidence.
Step 7
Verify the outcome, not the tool call
Walid's sixth finding: tool success is not job success. "Email sent" is not done. Done means the outcome is correct: exact numbers, the right file, the exact recipient and the exact result. A tool can return success while the email went to the wrong person or carried last month's figures.
So add a verifier that runs after the agent says it is finished. It does not trust the agent's summary. It reads artefacts and values read back from the systems, and compares them with the criteria from the planner. Use a separate prompt, and where you can, a separate model call, so the checker has no stake in the answer.
Prompt
Outcome verifier
You decide whether an outcome is done. You are not the agent that did the work. Assume nothing.
Goal:
{{goal}}
Success criteria:
{{success_criteria_json}}
Artefacts and system state the agent produced (files, message logs, tool results, current values read back from the systems):
{{artefacts_json}}
For each criterion, decide "met", "not_met" or "cannot_tell", and quote the evidence.
Rules:
- A tool returning success is not evidence. "Email sent" does not mean the email went to the right person with the right numbers. Check the recipient, the body, the attachment and the numbers against the criterion.
- Evidence must come from the artefacts or from values read back from the system after the action. Never from the agent's own summary.
- Recompute any figure you can from the source data. If it does not match, the criterion is "not_met".
- If a file is part of the outcome, check that it opens, has the right name and type, and contains the requested content.
- If you cannot check a criterion with what you were given, say "cannot_tell" and list the artefact you would need.
Return JSON only:
{
"criteria": [{"id": "c1", "verdict": "met | not_met | cannot_tell", "evidence": "quote or value", "gap": "what is missing, or null"}],
"overall": "done | not_done | needs_more_evidence",
"next_action": "one concrete action, or null if done"
}Read back state after the action. Check the sent message in the mail system, not the draft. Open the file, do not trust its name.
Diagram
What happens at the end of an outcome
Agent claims done
tool calls all returned success
Collect artefacts
files, sent messages, current values read back
Verifier checks each criterion
met, not met or cannot tell, with evidence
All criteria met
Mark done, notify the person
keep the evidence in the log
Any criterion not met
Replan the gap
same goal, new plan version
Cannot tell
Ask a person
show what evidence is missing
Then test the verifier itself. Take the 10 to 20 goals with known correct results from your checklist. Seed some wrong results: a wrong recipient, a stale number, a file with the wrong type. The verifier should catch every one. Run this set each time you change the planner, the verifier or the model.
Watch out
Failure modes we see most
The goal drifts
After two replans the agent is solving a different problem. Fix: the goal is immutable, stored apart from the plan, and every replan event references it.
Spend goes past the budget
A soft limit in the prompt gets ignored, as in the $6.49 render on a $6 budget. Fix: check the cap in the tool router before every paid call.
A wait loses state
The process restarts and the agent forgets what it was waiting for. Fix: store
waiting_onandresume_atin the outcome row and resume from them.A pattern becomes a permission
Memory turns a habit into a rule nobody agreed to. Fix: only a person can write to the permissions table, and every permission has a scope and an expiry.
Success without a result
The tool returns ok, the email goes to the wrong person. Fix: a verifier that reads back state and checks recipient, file and numbers.
The browser used by default
The agent clicks through a site when an API call exists, and breaks when the page changes. Fix: the router allows the browser only after it finds no API path.
Time and money
Time and cost
Our planning estimate is a first version in about a month for one experienced builder. Scope decides it. A first version means one tool family, one kind of outcome and a person approving everything. It does not mean a general agent.
Timeline
A first version in about 30 days
Outcome model and one tool, day 1 to 7. Schema from step 1, planner prompt, one integration working end to end, spend cap in the router.
Sandbox and browser, day 8 to 14. Sandbox sessions with limits, browser fallback behind the router rule, event logging for both.
Durable waits and approvals UI, day 15 to 21. Send, wait and resume on the job runner, permission table, approval cards in the agent UI.
Verification and evals, day 22 to 30. Verifier prompt, 10 to 20 goals with known answers, seeded wrong results, baseline metrics.
- Outcome model and one toolDay 1-7
Schema from step 1, planner prompt, one integration working end to end, spend cap in the router.
- Sandbox and browserDay 8-14
Sandbox sessions with limits, browser fallback behind the router rule, event logging for both.
- Durable waits and approvals UIDay 15-21
Send, wait and resume on the job runner, permission table, approval cards in the agent UI.
- Verification and evalsDay 22-30
Verifier prompt, 10 to 20 goals with known answers, seeded wrong results, baseline metrics.
Planning estimate for one experienced builder, not a promise. Scope decides it.
Running cost has six drivers. We do not quote vendor prices here, because they change. Check each vendor's pricing page and put the numbers in your own sheet. The one real figure we have is Walid's: a $6 budget for the GPU render, and $6.49 spent.
| Cost | Scales with | Keep it down by |
|---|---|---|
| Model tokens | Plan length, replans, verification passes | A stronger model for planning, a cheaper one for routine execution. See the LLM cost control playbook |
| Sandbox minutes | How long each session runs | Time limits, close on track end, reuse results instead of recomputing |
| Browser sessions | How often there is no API | API first, browser only through the router rule |
| Durable job runs | Steps, retries and wake-ups | Park long waits, avoid polling loops, set retry limits |
| Data calls | Outside data per outcome | Check the price shown before each call and cap total spend |
| Approval time | Approvals asked per day | Scoped standing permissions, a daily cap, batching requests |
Model tokens
- Scales with
- Plan length, replans, verification passes
- Keep it down by
- A stronger model for planning, a cheaper one for routine execution. See the LLM cost control playbook
Sandbox minutes
- Scales with
- How long each session runs
- Keep it down by
- Time limits, close on track end, reuse results instead of recomputing
Browser sessions
- Scales with
- How often there is no API
- Keep it down by
- API first, browser only through the router rule
Durable job runs
- Scales with
- Steps, retries and wake-ups
- Keep it down by
- Park long waits, avoid polling loops, set retry limits
Data calls
- Scales with
- Outside data per outcome
- Keep it down by
- Check the price shown before each call and cap total spend
Approval time
- Scales with
- Approvals asked per day
- Keep it down by
- Scoped standing permissions, a daily cap, batching requests
Results
How to measure it
| Metric | Where it comes from | Which way you want it |
|---|---|---|
| Outcomes completed and verified | outcomes with status done and a passing verify event | Up |
| False done rate | Agent claimed done, verifier overturned it | Down, and never hidden |
| Replans per outcome | replan events grouped by outcome | Stable or down, with the reasons reviewed |
| Approvals asked versus granted | approval_asked and approval_result events | Asked falls as scoped permissions grow, granted stays high |
| Cost per outcome | spent_usd per outcome and the sum of event costs | Down for the same kind of goal |
| Time to resume after a wait | Reply or timer time vs. the resume event | Down, in minutes |
Outcomes completed and verified
- Where it comes from
outcomeswith status done and a passing verify event- Which way you want it
- Up
False done rate
- Where it comes from
- Agent claimed done, verifier overturned it
- Which way you want it
- Down, and never hidden
Replans per outcome
- Where it comes from
replanevents grouped by outcome- Which way you want it
- Stable or down, with the reasons reviewed
Approvals asked versus granted
- Where it comes from
approval_askedandapproval_resultevents- Which way you want it
- Asked falls as scoped permissions grow, granted stays high
Cost per outcome
- Where it comes from
spent_usdper outcome and the sum of event costs- Which way you want it
- Down for the same kind of goal
Time to resume after a wait
- Where it comes from
- Reply or timer time vs. the
resumeevent - Which way you want it
- Down, in minutes
Take a baseline on your 10 to 20 test goals before you let anyone use the agent: false done rate, replans and cost per outcome. Re-measure after each change to the planner, the verifier or the model, and again two and six weeks after real users start. A rising false done rate is the first sign something is wrong, so look at it before the others.
Questions
FAQ
How do I build an AI agent that works toward a goal instead of a single task?
Store the goal as a fixed outcome in your database, with success criteria, a disposable plan, a budget and a wait state. Let the agent split the goal into parallel tracks and replan when a track fails, while the goal itself stays unchanged.
What is an AI agent sandbox and why do I need one?
A sandbox is an isolated computer where the agent can write and run code without touching your real systems. Walid saw an agent write Python with Pillow, NumPy and ffmpeg to make a video, and use openpyxl to build a spreadsheet. Daytona and E2B are two providers for this role.
What is in an AI agent stack?
Walid's list: Composio for integrations, CopilotKit and AG-UI for the agent UI, Daytona or E2B for sandboxes, Browserbase and Stagehand for the browser, Trigger.dev or Inngest for durable jobs, supermemory for memory, a database for state, Unipile for messaging, Vapi for calls and Recall.ai for meetings. You build the outcome engine, permissions, replanning, memory rules and verification.
How can an agent wait days for a reply without running the whole time?
Write the waiting state to the outcome row and let a durable job runner wake it when the reply arrives or a timer fires. The compute does not need to stay alive for three days. The outcome does, because its state is stored.
How do I know the agent really finished the job?
Use a separate verifier that checks the result against the success criteria. It reads back the sent message, opens the file and recomputes the numbers. Tool success is not job success, so a call returning ok is never enough.