How to build your own AI agent that works toward outcomes: the stack and the hard parts

A plan for building an agent that takes a goal, works on it for days, asks before it gets new permissions, and only calls a job done when the result is right. You buy the plumbing and spend your engineering time on five hard parts.

By the AY Automate team · Updated Nov 22, 2026 · 16 min read

Build time
About a month for a first version, one builder
Level
Advanced
Stack
Bought plumbing plus five parts you build
Main risk
Calling a job done when it is wrong

Inside this playbook

  • 5prompts, templates and code to copy
  • 3diagrams and real screenshots
  • 7steps, start to finish

Diagram

An outcome agent, from goal to verified result

  1. Goal

    a fixed outcome, not a task list

  2. Plan

    parallel tracks, disposable

  3. Act

    API first, browser when needed, sandbox for code

  4. Wait and resume

    durable jobs, the outcome stays alive

  5. Ask before new permissions

    memory proposes, a person says yes

  6. Verify the outcome

    right numbers, right file, right recipient

Trigger or inputModel stepYour systemsPerson decidesResult

The six parts Walid found when he tested an outcome agent, as one loop. The goal stays fixed; the plan can be thrown away.

In short

  • Model the outcome, not the chat. Store a fixed goal, success criteria, a disposable plan, a budget and a wait state in Postgres before you write any agent code.
  • Buy the plumbing: integrations, sandboxes, browser, durable jobs, memory, messaging, phone and meetings. Spend your own time on the outcome engine, permissions, replanning, memory rules and verification.
  • Use the API when one exists, a browser when it does not, and a sandbox for anything the agent has to compute or generate. Put a hard spend cap on every outcome.
  • Park long waits in the database, not in a running process. A reply that arrives three days later should resume the outcome from stored state.
  • Add a separate verifier that checks the result against the success criteria. A tool saying "sent" is not done.

Who it is for: Engineers and technical founders who want to build an agent that does real work across tools, not a chatbot. You should be comfortable with Postgres, background jobs and calling APIs from code.

Overview

What you will have at the end

In late September, Walid saw a consumer outcome agent announce a $1B raise. He was already busy, but he got curious about the best way to build something like it. So he tested it. He asked indirect questions, gave it odd jobs, ran parallel tasks, changed instructions while it worked, made it generate files and code, and inspected what came back.

What he found is a short list of parts. Most of them you can buy. A few you have to build, and those decide whether your agent finishes real work or only looks busy. This playbook turns that list into a build plan with the schema, the prompts and the code for the awkward parts.

Diagram

The whole system, from goal to approved result

  1. User sets a goal

    one fixed sentence plus a budget

  2. Outcome engine

    state in Postgres, Supabase or Convex

  3. Planner

    success criteria, parallel tracks, replans

  4. Tool layer

    Integrations, data, browser, sandbox

    Composio, looot, Browserbase and Stagehand, Daytona or E2B

    Channels

    Email, chat, phone, meetings

    Unipile, Vapi, Recall.ai

  5. Durable runner

    Trigger.dev or Inngest: retries, waits, wake-ups

  6. Verifier

    checks the result against the success criteria

  7. Person approves in the agent UI

    CopilotKit and AG-UI

Trigger or inputYour systemsModel stepPerson decides

The outcome row in your database is the center. Everything else reads from it and writes back to it.

This is an advanced build. You will write a database schema, background job code and evaluation tests. If you want to automate a single workflow, the n8n production playbook is a better start. Come back here when one workflow is no longer enough.

Background

Who this is for and what you need

This guide is for people building the agent itself: a product team adding an agent to an app, an internal platform team, or a founder who wants a personal agent that can act across email, files and websites. It assumes you have shipped backend code before. It does not teach you to call a model API.

A note on why tools matter so much. Walid wrote in November that the gap between an agent with tools and one without feels closer to 100x, and that the bottleneck is becoming access to tools. That is why most of this stack is about reaching tools safely, and why the model is one box among many.

Before you start

  • A Postgres database. Supabase, Convex or plain Postgres all work. You need transactions and row locks for the outcome state.
  • A durable job runner account: Trigger.dev or Inngest. Check each vendor's docs for current plans and limits.
  • A sandbox provider for code execution: Daytona or E2B. Check their docs for pricing and session limits.
  • A browser provider for sites without APIs: Browserbase with Stagehand.
  • An integrations layer: Composio for auth and tool discovery. Add looot if the agent needs data endpoints (one key or MCP connection, priced per call before it runs).
  • Model access for a planner and an executor. Walid's own pairing, posted in November: Fable 5.2 for planning and Opus 5.5 for execution. Pick what holds up on your tasks.
  • A test set of 10 to 20 real goals with known correct results. You will use it for verification in step 7.
  • A named person who owns permissions and approves risky actions during testing.

Step 1

Model the outcome before the agent

Walid's first finding was "outcomes, not agents." You give the system a goal. It can split that goal into parallel tracks, throw away a bad plan and try another one. The destination stays fixed. The plan is disposable. Build that into your data model first, because it is hard to retrofit.

In the schema below, goal is written by a person and the agent never edits it. plan and plan_version change as often as needed. waiting_on and resume_at let an outcome sleep. budget_usd and spent_usd give you a hard cap. locked_by stops two workers from acting on the same outcome at once. The Postgres, Supabase or Convex choice does not change this shape, only the syntax.

Schema

Outcomes, events and permissions

create table outcomes (
  id               uuid primary key default gen_random_uuid(),
  created_at       timestamptz not null default now(),
  created_by       text not null,
  goal             text not null,
  success_criteria jsonb not null,
  status           text not null default 'planning'
    check (status in ('planning','running','waiting','needs_approval','verifying','done','failed','cancelled')),
  plan_version     int  not null default 1,
  plan             jsonb,
  waiting_on       text,
  resume_at        timestamptz,
  budget_usd       numeric(10,2) not null,
  spent_usd        numeric(10,2) not null default 0,
  locked_by        text,
  locked_until     timestamptz
);

create table outcome_events (
  id          bigint generated always as identity primary key,
  outcome_id  uuid not null references outcomes(id),
  created_at  timestamptz not null default now(),
  kind        text not null
    check (kind in ('plan','replan','tool_call','wait','resume','approval_asked','approval_result','verify')),
  tool        text,
  proposed    jsonb,
  executed    jsonb,
  previous    jsonb,
  cost_usd    numeric(10,4) not null default 0,
  note        text
);

create table permissions (
  id          bigint generated always as identity primary key,
  pattern     text not null,
  scope       jsonb not null,
  granted_by  text not null check (granted_by <> 'agent'),
  granted_at  timestamptz not null default now(),
  expires_at  timestamptz,
  revoked_at  timestamptz
);

The previous column on events is your undo. Log the value before every write, and you can reverse a bad run. scope holds JSON such as the allowed tools, recipients and a daily maximum. success_criteria holds a list of checks, each with an id and an evidence type (file, number, message or state).

  • Never let the plan overwrite the goal. If the plan changes, bump plan_version and log a replan event with the reason.
  • Write the event before you call the tool, then update it with the result. A crash then leaves a record of what was attempted.
  • Lock the outcome row while a worker acts on it, with a short locked_until so a dead worker releases it.
  • Set `budget_usd` when the goal is created, by a person. The agent can ask for more, but it cannot raise its own cap.

Step 2

Buy the plumbing, build the five hard parts

Walid's advice for building something similar today: do not rebuild the supporting systems. Get these, and spend engineering time on the parts that are specific to your product. The table lists each component, what it gives you and what you still have to build around it. Check each vendor's docs for current features and pricing before you commit. We only state the role each one plays.

Buy versus build, component by component
  • Composio

    Role
    Integrations, auth and tool discovery
    What you still build
    Which tools each outcome may use, and the permission check in front of them
  • CopilotKit and AG-UI

    Role
    Agent UI, streaming, approvals and app to agent communication
    What you still build
    The approval cards, what they show, and what happens on yes or no
  • Daytona or E2B

    Role
    Isolated computers where agents write and run code
    What you still build
    What goes in and out of the sandbox, and cleanup so sessions do not pile up
  • Browserbase and Stagehand

    Role
    Browser execution when there is no good API
    What you still build
    The rule for when the browser is allowed, and checks on what the page returned
  • Trigger.dev or Inngest

    Role
    Durable jobs, retries, waiting and wake-ups
    What you still build
    The outcome state machine that the jobs read and write
  • supermemory

    Role
    Long-term user memory and retrieval
    What you still build
    The rules for what memory may change (step 6)
  • Supabase, Convex or Postgres

    Role
    Outcomes, permissions, state, locks and audit logs
    What you still build
    The schema from step 1 and the queries that enforce it
  • Unipile

    Role
    Email, WhatsApp, LinkedIn and messaging
    What you still build
    Recipient checks, rate caps and the approval for first contact
  • Vapi

    Role
    Phone calls
    What you still build
    Scripts, consent rules and how a call result feeds the verifier
  • Recall.ai

    Role
    Meetings turned into context
    What you still build
    Which meeting content may become a goal or a memory

Add looot to this list if the agent needs outside data. It gives the agent one key or MCP connection to thousands of data endpoints, and each call is priced before it runs. That price check is useful for the budget cap in step 4.

Walid's list of what is left over is short, and it is where the hard engineering sits. These five are not in any vendor's box.

  • Hard part 1

    Outcome engine

    Goes wrong: the agent forgets the goal after a long wait or a replan. Build: the outcome table, the state machine, locks and the budget.

  • Hard part 2

    Permissions

    Goes wrong: the agent uses a broad token for a narrow job. Build: scoped credentials, a permission table, approval for risky writes and a daily cap.

  • Hard part 3

    Replanning

    Goes wrong: the agent keeps retrying a dead plan, or abandons a good one. Build: abandon conditions in the plan and a log of every replan with the reason.

  • Hard part 4

    Memory rules

    Goes wrong: an observed habit becomes a standing instruction nobody agreed to. Build: a proposal flow where only a person grants.

  • Hard part 5

    Verification

    Goes wrong: a tool returns success and the job is wrong. Build: a separate verifier and a test set with known answers.

Step 3

Write the planner: success criteria first

The planner has three jobs. It writes success criteria that a third party could check. It splits the work into tracks that can run in parallel. And it states, up front, what would make it abandon the plan. The third one is what makes a plan disposable: you decide the exit conditions before the agent gets attached to its own approach.

Success criteria are the contract with the verifier in step 7. "Send the report" is a task. "The report PDF reaches the named recipient and its totals match the source sheet" is a criterion. Write the second kind.

Prompt

Outcome planner

You turn a goal into a plan that can be thrown away. The goal stays fixed.

Goal (written by a person, do not rewrite it):
{{goal}}

Context you may use:
{{context_json}}

Tools available (name and what each one can do):
{{tool_list}}

Budget in USD: {{budget_usd}}

Return JSON only:
{
  "success_criteria": [
    {"id": "c1", "check": "a statement a third party could verify", "evidence_type": "file | number | message | state"}
  ],
  "tracks": [
    {"id": "t1", "purpose": "...", "steps": ["..."], "tools": ["..."], "can_run_in_parallel_with": ["t2"], "estimated_cost_usd": 0.0}
  ],
  "abandon_plan_if": [
    "a condition that tells us this plan is wrong, with a number or a visible signal"
  ],
  "needs_permission_for": [
    {"action": "...", "why": "..."}
  ],
  "open_questions": ["only questions that block the first step"]
}

Rules:
- Every success criterion must be checkable from an artefact: a file, a number, a message, or a state in a system. "Done well" is not a criterion.
- Name the exact recipient, the exact file type and the exact figures wherever the goal implies them.
- Split into parallel tracks only when the tracks do not depend on each other.
- If an API exists for a step, plan the API. Plan the browser only when no API covers it.
- Total estimated cost must stay under the budget. If it cannot, say so in open_questions. Do not shrink the criteria to fit.
- Anything that sends a message to a person, spends money, or changes data you did not create goes in needs_permission_for.

Run the planner on a stronger model and execution on whatever holds quality for the cheaper steps. Walid plans with Fable 5.2 and executes with Opus 5.5.

Reasons to replan, and what each one triggers
  • A tool failed twice with the same error

    Example
    The API rejects the request format
    What the engine does
    Mark the track failed, ask the planner for a different path, bump plan_version
  • A condition in abandon_plan_if became true

    Example
    The vendor site no longer has the form
    What the engine does
    Switch the track to another tool, keep the goal
  • New instruction from the person

    Example
    "Use the second supplier instead"
    What the engine does
    Keep the goal, replan the affected tracks, log the reason
  • Spend passed a set share of the budget with the criteria unmet

    Example
    Many retries on a hard step
    What the engine does
    Stop, ask the person, do not silently continue

Pick the spend threshold yourself and write it into the engine, not into the prompt.

Step 4

Build the tool layer: API first, browser second, sandbox for the weird jobs

Walid tested three behaviors that matter here. When he asked it to generate a video, it wrote Python with Pillow, NumPy and ffmpeg. The spreadsheet it produced was built with Python and openpyxl. For odd jobs, it can create the tooling it needs while doing the job. That is the case for a sandbox: an isolated computer where the agent can write and run code without touching your real systems. Daytona and E2B fill that role.

The second behavior is the order of attempts. If an API exists, use it. If not, use the website. Do not memorize clicks. Remember the goal and work out the path again when the site changes. In stack terms: Composio for integrations, Browserbase with Stagehand for the browser, and a rule in your own code that the browser is the fallback.

Which tool to use, in this order
  • 1

    Use
    API through your integrations layer
    When
    An API covers the step
    What to log
    The call, the response and the previous value on writes
  • 2

    Use
    Data endpoint (for example looot)
    When
    The step needs outside data
    What to log
    The price shown before the call and the cost after
  • 3

    Use
    Browser session
    When
    No usable API
    What to log
    What the agent saw on the page and which element it acted on
  • 4

    Use
    Sandbox
    When
    The agent must compute, convert or generate a file
    What to log
    The code it ran, the inputs and the output files

Keep the browser rule in code, not in the prompt. The planner proposes a track, and the tool router checks whether an API path exists before it allows a browser session. That keeps the agent from clicking through a site when a clean call would do.

  • Give every sandbox a time and cost limit, and close it when the track ends.
  • Treat files from the sandbox as untrusted input until the verifier has opened them.
  • Give the agent its own credentials with the smallest permissions. Never reuse a person's admin token.
  • Log every proposed and executed action with the previous value, so you can undo.
  • Cap daily volume for anything that touches other people, such as messages or calls.

Step 5

Make long waits cheap: the outcome stays alive, the compute does not

Walid's example: send an email, wait three days, the reply arrives, continue from exactly where you stopped. The compute does not need to stay alive for three days. The outcome does. This is the main reason to use a durable job runner such as Trigger.dev or Inngest, which handle retries, waiting and wake-ups. Check their docs for the exact functions, limits and pricing.

Your part is the state. When the agent starts waiting, write status = waiting, waiting_on and resume_at to the outcome row. When anything wakes the outcome, read that row and continue from it. Never rely on variables in a process that may have restarted. The pseudocode below shows the whole pattern, including the permission gate and a send that cannot run twice.

Code

Send, wait three days, resume on reply (pseudocode)

// PSEUDOCODE. Map each call to your durable job runner (Trigger.dev, Inngest or similar).
// The point: the process can exit. The outcome row keeps the state.

async function sendAndWaitForReply(outcomeId: string, draft: Draft) {
  const outcome = await db.outcomes.get(outcomeId)

  // 1. Permission gate before anything leaves the building
  const allowed = await permissions.allows("email.send", draft)
  if (!allowed) {
    await db.outcomes.update(outcomeId, { status: "needs_approval", waiting_on: "approval:email" })
    await ui.askPerson(outcomeId, draft)        // approval card in your agent UI
    return waitFor.approval(outcomeId)          // durable: nothing runs while we wait
  }

  // 2. Send once. The idempotency key stops a retry from sending twice.
  const sent = await tools.email.send(draft, { idempotencyKey: outcomeId + ":send:" + outcome.plan_version })
  await db.events.insert({ outcome_id: outcomeId, kind: "tool_call", tool: "email", executed: sent })

  // 3. Park the outcome. No compute is held for 3 days.
  await db.outcomes.update(outcomeId, {
    status: "waiting",
    waiting_on: "reply:" + sent.threadId,
    resume_at: daysFromNow(3),
  })

  // 4. Wake on whichever comes first: the reply event or the timer.
  const wake = await waitFor.firstOf([
    waitFor.event("email.reply", { threadId: sent.threadId }),
    waitFor.time(daysFromNow(3)),
  ])

  // 5. Resume from the stored state, not from memory in a process.
  await db.outcomes.update(outcomeId, { status: "running", waiting_on: null })
  await db.events.insert({ outcome_id: outcomeId, kind: "resume", note: wake.kind })

  if (wake.kind === "timer") return followUpOrReplan(outcomeId)   // no reply: nudge, or change plan
  return continueWithReply(outcomeId, wake.payload)                // reply arrived: carry on
}

This is pseudocode, not a Trigger.dev or Inngest function. Map each wait and event call to your runner's own API.

  • Idempotency on every send. A retry after a timeout must not send a second email.
  • Two wake-up reasons. The reply, or the timer, whichever comes first. A missing reply is information too.
  • Resume writes an event. You can see in the log how long the outcome slept and what woke it.
  • Lock on resume. If a reply and a timer arrive together, only one worker continues.

Step 6

Permissions and memory: observed patterns never grant themselves

Walid's fourth finding: the agent can notice patterns in how you work, but it does not turn a pattern into permission automatically. It asks first. One yes can turn an observed pattern into a standing instruction. Copy that rule exactly. Memory is useful for relevance and drafts. It must not be the thing that decides what the agent may do.

Supermemory gives you long-term user memory and retrieval. The rules around it are yours: what counts as a repeated pattern, what the request to the user says, what scope the permission has and when it expires. Store the answer in the permissions table from step 1 with granted_by set to a person. Nothing in the code path should be able to write agent there.

Prompt

Memory rule proposal

You review notes about how a user works and decide whether to ask for a standing instruction. You never grant yourself anything.

Observations (each with dates and the outcome it came from):
{{observations_json}}

Existing permissions (do not propose a duplicate):
{{permissions_json}}

If the same pattern shows up at least {{min_repeats}} times and the user did not object, draft a request. Otherwise return {"ask": false}.

The request is written to the user in plain words. It must contain:
- what you noticed, with the dates
- the exact standing instruction you propose, in one sentence
- the scope: which tools, which recipients or records, how many per day
- an expiry date
- how to revoke it

Return JSON only:
{
  "ask": true,
  "message_to_user": "I noticed ... May I make this a standing instruction? ...",
  "proposed_permission": {"pattern": "...", "scope": {...}, "expires_at": "YYYY-MM-DD"}
}

Rules:
- Observing a pattern gives no permission. Only a "yes" from the user does, and the app records that, not you.
- Never widen the scope beyond what you observed.
- Never ask about anything involving money, deleting data or messaging someone for the first time.

The model drafts the question. Your app shows it, records the answer and writes the permission row. The model never does.

Permission rules that hold up in practice
  • Every standing instruction has a scope and an expiry

    Why
    Old habits stop applying without anyone remembering to revoke them
  • Money, deletes and first contact always ask

    Why
    These are the actions that hurt most when wrong
  • A risky write needs a person's approval each time

    Why
    The agent proposes, a person approves
  • Revoking is one click and takes effect on the next call

    Why
    People trust what they can switch off
  • Log approvals asked and granted

    Why
    The ratio tells you whether the agent asks too often or too rarely

Approvals need a place to happen. That is what CopilotKit with AG-UI is for: streaming agent progress into your app and showing approval cards where the person already works. The card should show the exact action, the recipient or record, and the evidence.

Step 7

Verify the outcome, not the tool call

Walid's sixth finding: tool success is not job success. "Email sent" is not done. Done means the outcome is correct: exact numbers, the right file, the exact recipient and the exact result. A tool can return success while the email went to the wrong person or carried last month's figures.

So add a verifier that runs after the agent says it is finished. It does not trust the agent's summary. It reads artefacts and values read back from the systems, and compares them with the criteria from the planner. Use a separate prompt, and where you can, a separate model call, so the checker has no stake in the answer.

Prompt

Outcome verifier

You decide whether an outcome is done. You are not the agent that did the work. Assume nothing.

Goal:
{{goal}}

Success criteria:
{{success_criteria_json}}

Artefacts and system state the agent produced (files, message logs, tool results, current values read back from the systems):
{{artefacts_json}}

For each criterion, decide "met", "not_met" or "cannot_tell", and quote the evidence.

Rules:
- A tool returning success is not evidence. "Email sent" does not mean the email went to the right person with the right numbers. Check the recipient, the body, the attachment and the numbers against the criterion.
- Evidence must come from the artefacts or from values read back from the system after the action. Never from the agent's own summary.
- Recompute any figure you can from the source data. If it does not match, the criterion is "not_met".
- If a file is part of the outcome, check that it opens, has the right name and type, and contains the requested content.
- If you cannot check a criterion with what you were given, say "cannot_tell" and list the artefact you would need.

Return JSON only:
{
  "criteria": [{"id": "c1", "verdict": "met | not_met | cannot_tell", "evidence": "quote or value", "gap": "what is missing, or null"}],
  "overall": "done | not_done | needs_more_evidence",
  "next_action": "one concrete action, or null if done"
}

Read back state after the action. Check the sent message in the mail system, not the draft. Open the file, do not trust its name.

Diagram

What happens at the end of an outcome

  1. Agent claims done

    tool calls all returned success

  2. Collect artefacts

    files, sent messages, current values read back

  3. Verifier checks each criterion

    met, not met or cannot tell, with evidence

  4. All criteria met

    Mark done, notify the person

    keep the evidence in the log

    Any criterion not met

    Replan the gap

    same goal, new plan version

    Cannot tell

    Ask a person

    show what evidence is missing

Model stepYour systemsResultPerson decides

Then test the verifier itself. Take the 10 to 20 goals with known correct results from your checklist. Seed some wrong results: a wrong recipient, a stale number, a file with the wrong type. The verifier should catch every one. Run this set each time you change the planner, the verifier or the model.

Watch out

Failure modes we see most

  • The goal drifts

    After two replans the agent is solving a different problem. Fix: the goal is immutable, stored apart from the plan, and every replan event references it.

  • Spend goes past the budget

    A soft limit in the prompt gets ignored, as in the $6.49 render on a $6 budget. Fix: check the cap in the tool router before every paid call.

  • A wait loses state

    The process restarts and the agent forgets what it was waiting for. Fix: store waiting_on and resume_at in the outcome row and resume from them.

  • A pattern becomes a permission

    Memory turns a habit into a rule nobody agreed to. Fix: only a person can write to the permissions table, and every permission has a scope and an expiry.

  • Success without a result

    The tool returns ok, the email goes to the wrong person. Fix: a verifier that reads back state and checks recipient, file and numbers.

  • The browser used by default

    The agent clicks through a site when an API call exists, and breaks when the page changes. Fix: the router allows the browser only after it finds no API path.

Time and money

Time and cost

Our planning estimate is a first version in about a month for one experienced builder. Scope decides it. A first version means one tool family, one kind of outcome and a person approving everything. It does not mean a general agent.

Timeline

A first version in about 30 days

  1. Outcome model and one toolDay 1-7

    Schema from step 1, planner prompt, one integration working end to end, spend cap in the router.

  2. Sandbox and browserDay 8-14

    Sandbox sessions with limits, browser fallback behind the router rule, event logging for both.

  3. Durable waits and approvals UIDay 15-21

    Send, wait and resume on the job runner, permission table, approval cards in the agent UI.

  4. Verification and evalsDay 22-30

    Verifier prompt, 10 to 20 goals with known answers, seeded wrong results, baseline metrics.

Planning estimate for one experienced builder, not a promise. Scope decides it.

Running cost has six drivers. We do not quote vendor prices here, because they change. Check each vendor's pricing page and put the numbers in your own sheet. The one real figure we have is Walid's: a $6 budget for the GPU render, and $6.49 spent.

Cost drivers per outcome
  • Model tokens

    Scales with
    Plan length, replans, verification passes
    Keep it down by
    A stronger model for planning, a cheaper one for routine execution. See the LLM cost control playbook
  • Sandbox minutes

    Scales with
    How long each session runs
    Keep it down by
    Time limits, close on track end, reuse results instead of recomputing
  • Browser sessions

    Scales with
    How often there is no API
    Keep it down by
    API first, browser only through the router rule
  • Durable job runs

    Scales with
    Steps, retries and wake-ups
    Keep it down by
    Park long waits, avoid polling loops, set retry limits
  • Data calls

    Scales with
    Outside data per outcome
    Keep it down by
    Check the price shown before each call and cap total spend
  • Approval time

    Scales with
    Approvals asked per day
    Keep it down by
    Scoped standing permissions, a daily cap, batching requests

Results

How to measure it

Scorecard for an outcome agent
  • Outcomes completed and verified

    Where it comes from
    outcomes with status done and a passing verify event
    Which way you want it
    Up
  • False done rate

    Where it comes from
    Agent claimed done, verifier overturned it
    Which way you want it
    Down, and never hidden
  • Replans per outcome

    Where it comes from
    replan events grouped by outcome
    Which way you want it
    Stable or down, with the reasons reviewed
  • Approvals asked versus granted

    Where it comes from
    approval_asked and approval_result events
    Which way you want it
    Asked falls as scoped permissions grow, granted stays high
  • Cost per outcome

    Where it comes from
    spent_usd per outcome and the sum of event costs
    Which way you want it
    Down for the same kind of goal
  • Time to resume after a wait

    Where it comes from
    Reply or timer time vs. the resume event
    Which way you want it
    Down, in minutes

Take a baseline on your 10 to 20 test goals before you let anyone use the agent: false done rate, replans and cost per outcome. Re-measure after each change to the planner, the verifier or the model, and again two and six weeks after real users start. A rising false done rate is the first sign something is wrong, so look at it before the others.

Questions

FAQ

How do I build an AI agent that works toward a goal instead of a single task?

Store the goal as a fixed outcome in your database, with success criteria, a disposable plan, a budget and a wait state. Let the agent split the goal into parallel tracks and replan when a track fails, while the goal itself stays unchanged.

What is an AI agent sandbox and why do I need one?

A sandbox is an isolated computer where the agent can write and run code without touching your real systems. Walid saw an agent write Python with Pillow, NumPy and ffmpeg to make a video, and use openpyxl to build a spreadsheet. Daytona and E2B are two providers for this role.

What is in an AI agent stack?

Walid's list: Composio for integrations, CopilotKit and AG-UI for the agent UI, Daytona or E2B for sandboxes, Browserbase and Stagehand for the browser, Trigger.dev or Inngest for durable jobs, supermemory for memory, a database for state, Unipile for messaging, Vapi for calls and Recall.ai for meetings. You build the outcome engine, permissions, replanning, memory rules and verification.

How can an agent wait days for a reply without running the whole time?

Write the waiting state to the outcome row and let a durable job runner wake it when the reply arrives or a timer fires. The compute does not need to stay alive for three days. The outcome does, because its state is stored.

How do I know the agent really finished the job?

Use a separate verifier that checks the result against the success criteria. It reads back the sent message, opens the file and recomputes the numbers. Tool success is not job success, so a call returning ok is never enough.

Sources

AI Engineers

Want an outcome agent built with your engineers?

An AY engineer joins your team, sets up the outcome model, the permissions and the verifier with you, and wires the bought parts of the stack into your product. You keep the code and the data.

From €5,000 a month per engineer. Starting price, your quote may be higher. First month money-back.