Run AI Agents Locally: Models, Hardware, and the Token Bill
Running AI agents locally means the model itself runs on your hardware instead of a cloud API (a paid service you call over the internet). The pitch is real: no per-request bill, no data leaving your machine. So is the fine print: local is not free, and small models fail at agent work in specific, predictable ways. Here is the honest picture.
Why the token bill is the reason
Cloud models charge per token, the unit AI usage is billed in, with input and output priced separately. Current published rates run from $1 per million input tokens for Anthropic's cheapest model to several dollars for frontier ones, and prices move. The trap for agents specifically: an agent loop re-sends its context on every step, so unattended agents multiply input tokens in a way chat never does. Our own build log records the moment that bill passed $3 a day and climbing in early operations, and the rebuild that cut it three to four times. Local models flip the cost shape: you pay upfront in hardware and a little electricity, and each request after that has no marginal price.
What do you actually install?
Two free tools cover this on-ramp, and both speak the same protocol.
- Ollama: an open-source model runner for Mac, Windows, and Linux. It serves an OpenAI-compatible API on your machine, which means any agent tool that can talk to OpenAI can be pointed at your local model by changing one URL.
- LM Studio: a free desktop app for running open models, with the same OpenAI-compatible local server and explicit tool-use support. Its interface is friendlier if you have never done this before.
That compatibility detail is the whole unlock: your agents do not need to know or care that the model became local.
Which models hold up in agent loops?
Size matters more here than in chat. Our course guidance, written from running these systems, is blunt: models of 9 billion parameters or smaller (parameters are a model's internal settings; the count is the standard rough measure of its size) miss details and make errors in agent work; around 30B they work but require vigilance; at 70B and up they handle complexity reliably. The broader 2026 reporting agrees with the shape: local models hold up well for structured extraction, classification, summarization, and well-defined templated tasks, and fall behind frontier cloud models on complex tool chains, error recovery, and long multi-step plans. One more trap: aggressive quantization (shrinking a model to fit smaller hardware) has been reported to degrade tool calling harder than anything else the model does.
The pattern that works, and the one we teach: a frontier cloud model does the planning and judgment, local models grind through the high-volume routine steps. That split is where the token savings live without the reliability cost.
What hardware do you need?
Commonly cited ballparks, not promises, since it depends on the model format. Note these numbers are for local model inference only; the agent layer itself needs almost nothing, a split the hardware guide walks through in full.
| Model class | GPU memory ballpark | In agent work |
|---|---|---|
| 7B class | Around 8GB | Misses details, makes errors |
| 14B to 32B class | 16 to 24GB | Works, requires vigilance |
| 70B class | 40GB+, or a Mac with high unified memory (Apple Silicon's shared RAM pool that the GPU can also use) | Handles complexity reliably |
If that reads as expensive, note the order of operations: start with cloud models, measure where your volume actually is, and buy hardware for the workload you proved. Not the other way around.
When local is not worth it
Honestly: when your volume is low. If your agents run occasionally, if you need frontier reasoning on every step, or if you have no appetite for managing your own stack, the cloud API is the right answer and cheaper in practice. Local earns its keep when volume is high, tasks are routine, or the data must not leave your machine.
Where this fits in a factory
Level 2 is where we teach this end to end: a multi-agent system with an orchestrator (the agent that assigns work to the others), a task board, shared memory, and local-model monitoring so you can watch the spend and route work to the cheapest model that can do it. New to agents entirely? Start with the free primer, or with Level 1, which builds the single-agent foundation all of this runs on.
The free primer covers the thinking. Level 2 covers the factory, token costs included: $129 one-time. New to the term? What a dark factory is, explained. Not sure it fits: the compare page covers what Dark Factory is not.