Why the rate card misleads
Providers price per million tokens, but tasks are not tokens. A naive pipeline that stuffs full documents into a frontier model can cost 50× more per task than a designed pipeline producing the same output. The differences come from prompt size discipline, output length control, caching, and - biggest of all - which model handles which step.
When we rebuilt the scoring pipeline in LeadMine AI around routed models, per-lead cost fell by roughly an order of magnitude with no measurable quality loss. Nothing about the providers changed. The architecture did.
The routing principle
Every multi-step AI workflow has steps of wildly different difficulty. Extracting a company name from a webpage is trivial; judging whether that company fits a nuanced customer profile is not. Routing means matching each step to the cheapest model that clears the quality bar:
Frontier APIs (GPT, Claude, Gemini Pro)
Reasoning-heavy steps: qualification judgments, complex drafting, multi-document synthesis. Highest cost, use surgically.
Fast tiers (mini/flash/haiku-class)
Extraction, classification, formatting, summaries. The workhorse tier - most production volume belongs here.
Hosted open models (Groq and similar)
Very high-volume, latency-sensitive steps. Extremely cheap per token with excellent speed.
Self-hosted (Ollama, vLLM)
Wins when volume is enormous and steady, data cannot leave your infrastructure, or per-call latency budgets are tiny. You trade API fees for ops burden.
Caching: the cheapest tokens are the ones you never send
Three layers matter in practice. Prompt caching (supported by major providers) makes repeated system prompts nearly free. Response caching - hashing inputs and reusing outputs - eliminates whole calls for repeated work. Semantic caching, reusing answers for near-identical inputs, is powerful but needs care to avoid stale or wrongly-matched results. In back-office automation, response caching alone often removes 30–50% of calls.
When open models actually win
The break-even is further out than most teams think. Self-hosting carries GPU cost, ops time, and model-quality lag. Our rule: open models win on privacy requirements first, latency second, and raw cost only at sustained high volume. Below that, a routed mix of API tiers is cheaper once engineering time is priced in.
The strategic reason to keep an open-model path anyway: negotiating leverage and resilience. Systems we build abstract the model layer so providers can be swapped per step - a week of repricing, not a rewrite.
Budgeting a project: the numbers to demand
Before committing to any AI system, insist on per-unit economics: cost per lead processed, per document parsed, per conversation handled - at your projected volume, with the routing plan shown. Any builder who cannot produce that table has not designed the system yet. We provide it in every AI proposal because inference cost is a design output, not a surprise on the first invoice.
Questions we hear
Which LLM is cheapest in 2026?
Per token, hosted open models are cheapest. Per task, whichever architecture routes most of the volume to fast tiers and caches aggressively. A well-routed pipeline on premium APIs regularly beats a naive pipeline on cheap models.
Should we self-host to save money?
Only with sustained high volume, a privacy mandate, or hard latency limits. Below that, GPU and ops costs usually exceed the API bill you are trying to avoid.
How do you prevent runaway API bills?
Hard budget guardrails in code: per-task token ceilings, daily spend caps with alerts, and kill switches on anomalous loops. Agents especially need loop limits - a stuck agent retrying is the classic bill spike.