How to Build AI Agents From Scratch (Without a Framework)

Most people building their first AI agent reach for a framework, get something working in an afternoon, and then spend three weeks fighting abstractions when it doesn't behave in production. The framework was hiding about forty lines of logic, and those forty lines were exactly the part that needed to be different for their use case.

Building from scratch is worth doing at least once — not out of purism, but because an agent is a much simpler thing than the ecosystem implies, and understanding the loop makes every subsequent decision obvious.

What an Agent Actually Is

Strip everything away and an AI agent is a loop:

That's the whole thing. Everything else — memory, planning, multi-agent handoffs, reflection — is a variation on what you put in the history and what tools you expose.

The reason this matters: once you see it as a loop over a growing message list, the failure modes stop being mysterious. Agent gets stuck? Your loop has no iteration cap. Agent forgets earlier context? Your history got truncated. Agent calls the wrong tool? Your tool descriptions are ambiguous. Every problem maps to a specific line you control.

Step 1: The Loop and Its Guardrails

Write the loop first, before any interesting logic. It needs four guardrails from the start, because all four failure modes will happen:

Do these on day one. Retrofitting them after your first runaway session is a bad afternoon.

Step 2: Design Tools Like Public APIs

This is where agent quality actually lives, and it's the part frameworks let you skip badly.

A tool definition has a name, a description, and a typed parameter schema. The model reads all three and decides whether and how to call it. That means the description is not documentation — it's the instruction that drives behavior.

Rules that matter in practice:

Be explicit about when to use it, not just what it does. "Checks calendar availability" is weak. "Checks available appointment slots for a specific service type and date range. Call this before offering any specific time to a customer. Do not guess availability." is strong. The second version prevents the most common bug in booking agents: confidently offering a slot that doesn't exist.

Constrain parameters as tightly as reality allows. Enums instead of free-form strings. Explicit formats for dates. Required fields marked required. Every degree of freedom you leave is a degree the model will eventually explore.

Keep tools coarse-grained. One tool that books an appointment beats four tools that check availability, create a contact, create an event, and send a confirmation. Fewer round trips, fewer chances to break the chain halfway. Each loop iteration costs latency and tokens.

Return structured, informative errors. Not "error." Instead: "No availability on 2026-08-04. Nearest available: 2026-08-05 at 9:00am, 11:30am." Now the model can recover in the same turn instead of apologizing and stopping.

Aim for under ten tools. Beyond that, selection accuracy degrades noticeably. If you need more, split into specialized agents with a router.

Ninety percent of "the model is dumb" complaints are actually tool descriptions that didn't say what the model was supposed to do.

Step 3: Manage Context Deliberately

Your message history grows every iteration. Left alone, it eventually exceeds the context window, and more importantly, it gets expensive — you're paying for the entire history on every single call.

Three techniques, in order of how often you'll need them:

The system prompt itself deserves scrutiny. Most production prompts accumulate instructions that fixed a bug six months ago and now just cost tokens on every call. Audit it quarterly. Cutting a bloated prompt in half is one of the easiest cost wins available.

Step 4: Use Less AI Than You Think

The most valuable instinct in agent building: anything with a correct, computable answer should be code, not a model call.

Every deterministic thing you hand to a model is a place it can be creative when you needed it to be correct. Good agent architecture is a thin layer of language understanding wrapped around a lot of ordinary, testable business logic.

Step 5: Validate Outputs Before Acting

Never let a model's output flow directly into a system that changes state. Between the model and the action, put a validation layer:

This is the difference between an agent that occasionally embarrasses you and one you can leave running unattended.

Step 6: Instrument From the First Line

Log every iteration: the messages sent, the model's response, which tool was called with which arguments, what came back, latency, and token cost. Store it against a session ID you can look up in one query.

The reason is simple. When a customer says "your agent told me something wrong," you need the exact transcript in under a minute. And the systematic improvements — the ones that move qualification rate several points — come from reading fifty real transcripts a month, not from staring at a dashboard.

Step 7: Build the Escalation Path

Every agent needs a defined exit to a human. Triggers worth hard-coding:

And the handoff has to carry context: transcript, customer record, what the agent already attempted. Escalating a frustrated person to someone with no context is worse than not escalating.

What Production Actually Adds

An agent that works on your laptop is maybe 30% of a production agent. The rest:

That last piece is what makes the system improvable rather than fragile. Fifty to two hundred real conversations with expected outcomes, run before you ship any change. Without it, every prompt edit is a coin flip.

In the lead systems we build, the agent loop is a small fraction of the work. The bulk is making sure the outcome reaches the pipeline within seconds, the salesperson sees full context, and the ad platform learns which leads qualified — that last loop is what took accounts to a 9.2× peak ROAS.

When to Use a Framework Anyway

Build from scratch to learn, and for anything with unusual requirements or hard latency constraints. Reach for a framework when you need multi-agent orchestration with complex handoffs, or when your team needs a shared vocabulary and moving fast matters more than control.

But build one by hand first. The loop is forty lines. Understanding those forty lines will save you three weeks the first time something behaves strangely in production.

If you'd rather deploy a working agent than debug your own for a quarter, [book a free strategy call](/book) and we'll scope what it would take to put one in front of your leads.