Apple Spends 15x Less on AI — and Read the Future Right
AI compute is splitting in two: a thin, costly cloud frontier and a fast-collapsing commodity tier that runs free on your own hardware — and Apple is betting the edge wins.
Executive summary
The most expensive bet in the history of technology is being placed against a workload that is busy becoming free. In 2026, the world's largest companies will spend more than a trillion dollars building AI compute — the first trillion-dollar year of capital expenditure on a single class of infrastructure ever recorded — and the overwhelming majority of that money assumes intelligence will live in giant cloud data centres that everyone rents by the token. At the very same moment, the cost of running a model of any given capability is falling by roughly ten times a year, small models have grown competent enough to do the bulk of real work, and the most interesting AI products of the year are quietly pushing that work off the cloud and onto the laptop, the phone, the device already in your hand. The compute is not consolidating. It is splitting in two.
What is emerging is a two-tier world. A thin, genuinely expensive frontier — the hardest reasoning, the longest chains of thought, the few problems that still need the biggest model — stays in the cloud and stays worth paying for. Underneath it, a fast-collapsing commodity tier — classification, extraction, summarisation, routing, the thousand small tool-calls that make an agent run — is sliding toward something close to free, and increasingly runs on hardware you already own, with no API key, no per-token meter, and no data leaving the machine. The strategic question for a company or an engineer is no longer "which model is best." It is "where does each piece of this work belong, and who pays for it" — and getting that split right now cuts inference bills by sixty to eighty percent without a meaningful loss in quality.
The deepest implication is a re-scoring of the entire race. For three years the industry has measured itself by one number: whose model sits at the top of the benchmarks. But if most AI work is commoditising and draining to the edge, the frontier becomes a smaller and smaller slice of where the value actually accrues — and the durable advantage shifts from owning the best model to owning the distribution of cheap intelligence to where people already are. That is precisely the bet Apple is making in plain sight: spending roughly fifteen times less than any single one of its rivals, refusing to build a frontier model at all, and wiring efficient models directly into a billion devices. The catch — the reason this future is arriving unevenly rather than all at once — is that the hard part of going local was never the model. It is the harness: the unglamorous engineering that makes an unreliable, probabilistic component behave like dependable software. Whoever solves that, at the edge, owns the next decade.
Context: this is a fight over where intelligence physically lives
For most of the modern AI era there has been one obvious answer to the question of where artificial intelligence runs: somewhere else. In a hyperscale data centre, on someone else's GPUs, reached over the internet, paid for by the token. That arrangement felt permanent because it was load-bearing for everyone's business model — the labs sell access to models too large to run anywhere but the cloud, and the cloud providers sell the GPUs the labs run on. The entire industry was organised around the assumption that intelligence is a remote utility.
That assumption is now under genuine pressure, and the pressure is the most important under-told story in AI right now. It matters because the geography of compute — where the work physically happens — determines almost everything downstream: who captures the economic value, what a product can cost, how private your data can be, how fast a response can come back, and which companies hold structural power as AI becomes ambient. A revolution that runs in a handful of data centres owned by five companies is a very different revolution from one that runs, mostly, on the billions of capable devices people already carry. We are, right now, watching the field choose between those two worlds — and the honest answer is that it is choosing both at once, in a specific split that is worth understanding precisely. What follows maps that split by following the money and the engineering: how cheap the commodity tier has become, what small models can now actually do, where the combined local-plus-cloud stack already ships, why the harness is the real obstacle, how to decide what to run where, and why the company everyone wrote off as the AI laggard may have read the board most clearly.
The trillion-dollar paradox
Start with the number, because it is almost too large to feel. In 2026, total capital expenditure on compute across the industry — the big cloud providers, the specialist "neoclouds," sovereign programmes, China — reaches roughly $1.04 trillion, by the Futurum Group's accounting the first trillion-dollar year of compute capex in human history. The four largest American hyperscalers alone plan to spend on the order of $725 billion between them, with Amazon guiding to around $200 billion, Microsoft tracking near $190 billion, Alphabet targeting $175–185 billion, and Meta in the $115–135 billion range; roughly three-quarters of that, something like $450 billion, is pointed directly at AI infrastructure. This is one of the largest concentrated capital deployments any industry has ever attempted, and every dollar of it is a bet that demand for centralised, cloud-based AI compute will keep climbing steeply enough to justify the build
.
Now hold that against a second fact, drawn from the same year. The price of running a model at any fixed level of capability is, by Andreessen Horowitz's reckoning, falling by about ten times every year — and by Epoch AI's measurement across many capability thresholds, at a median closer to fifty times a year, accelerating to two hundred times for some tasks since early 2024. The work itself is getting radically cheaper to perform even as the spending to perform it explodes. Both things are true. The reconciliation is volume: the bull case says demand is growing fast enough that a hundredfold drop in unit cost is swamped by a thousandfold rise in usage, so the absolute spend still climbs. That may well hold. But it papers over a structural shift hiding inside the aggregate — because not all of that exploding demand needs to be served from a multi-billion-dollar data centre.
Here is the paradox stated plainly. The industry is pouring its trillion dollars into the cloud half of AI compute at precisely the moment a large and growing share of the actual workload is becoming cheap enough, and small enough, to run somewhere else entirely — on the edge, on devices, off the metered cloud. The capex is building cathedrals for a congregation that is quietly learning to pray at home. This does not mean the cathedrals were a mistake; the hardest workloads, and the training of the frontier models themselves, genuinely need them. It means the assumption that all AI compute consolidates upward is wrong, and the people building only for that future are exposed to a second one arriving underneath them. To see why, you have to look at how far the floor has already fallen.
LLMflation: the floor is falling at ten times a year
The single most important economic fact in AI is not how good the best model is. It is how fast the cost of "good enough" is collapsing. Andreessen Horowitz gave the phenomenon a name — LLMflation, the inverse of inflation: the same dollar buys you dramatically more intelligence every year. Their headline finding is stark. GPT-3, when it launched in November 2021, cost about $60 per million tokens. The cheapest model of broadly equivalent capability today — a small open model like Llama 3.2 3B served through a commodity inference provider — runs around $0.06 per million tokens. That is a thousandfold drop in three years, an annualised rate of roughly ten times a year for a constant level of capability
.
Epoch AI, measuring more carefully across many capability milestones, found the decline is real but uneven — "unequally across tasks" — ranging from nine times a year at the slow end to nine hundred times a year at the fast end, with a median around fifty times annually that jumped to two hundred times for some benchmarks after January 2024. The price to reach GPT-4's level on hard, PhD-grade science questions, for instance, fell about forty times a year. The texture matters: the cheapest, most commoditisable tasks fall fastest, while the genuinely frontier capabilities hold their price longest. That is the engine of the two-tier world — the bottom collapsing toward free while the top stays expensive.
What drives a curve this steep? Andreessen Horowitz identifies six forces stacking on top of each other: better price-performance from each new generation of GPU; aggressive quantisation, squeezing models from sixteen-bit math down toward four-bit with little quality loss; software and serving optimisations; smaller models reaching capability that used to require large ones; better instruction-tuning getting more out of each parameter; and brutal open-source competition compressing everyone's margins. None of these is a one-off. They compound. And crucially, several of them — quantisation, smaller models, open weights — are exactly the forces that make a model small and cheap enough to leave the data centre. LLMflation is not just lowering the price of cloud tokens; it is manufacturing the very models that no longer need the cloud at all. The clearest place to see that is in what a genuinely small model can now do.
The small model grew up
Two years ago, "small model" meant "toy." A few-billion-parameter model could autocomplete a sentence and little else; anything resembling reasoning, reliable tool use, or multi-step work demanded a large frontier model reached over an API. That gap has closed faster than almost anyone expected, and the reason is architectural as much as it is scale.
The breakthrough is sparsity — mixture-of-experts designs that decouple a model's total knowledge from the amount of computation any single token requires. Qwen2.5-Coder, a popular open coding model, carries 30.5 billion total parameters but activates only about 3.3 billion per token: it runs at roughly the speed of a 3-billion-parameter model while reasoning closer to a 30-billion one, with a context window large enough to hold a real codebase. Its 2026 successors push the same trick further — a 35-billion-parameter model that fires only about 3.5 billion parameters at a time. The dense small models have matured too: Microsoft's Phi-4 at 14 billion parameters scores about 84.8 percent on the MMLU knowledge benchmark and fits comfortably on a consumer 12-gigabyte GPU; the 3-to-4-billion class — Phi-4-mini, Gemma 3 4B, Qwen3 4B — now handles production coding assistance and document work on a laptop most developers already own, with Phi-4-mini generating fifteen to twenty tokens a second on a MacBook Air
.
The capability that changed everything, though, is not raw knowledge — it is agency. The single biggest shift of 2026, as the open-model community itself frames it, is that open and small models now reliably support the agentic patterns that were until recently the exclusive preserve of proprietary APIs: function calling, structured tool use, and the model-context-protocol plumbing that lets a model act on the world. Open coding models now score in the mid-eighties on τ²-bench, the benchmark built specifically to test whether a model can call tools, execute steps, and recover from errors across a multi-step workflow. That is the threshold that turns a small model from a clever autocomplete into a worker.
NVIDIA's research arm put the conclusion in a title that reads like a thesis statement: Small Language Models Are the Future of Agentic AI. The paper, by Peter Belcak and colleagues, argues that small models — defined as compact enough to run on an everyday device while answering a single user with low latency — are not a budget compromise but the correct tool for agentic systems on three grounds at once: they are sufficiently powerful for the narrow, repetitive tasks agents actually perform, they are inherently more suitable because those tasks have low variation, and they are necessarily more economical, running ten to thirty times cheaper to serve than a frontier model and fine-tunable in hours rather than weeks. The key observation is about the shape of agentic work, and it is the hinge of this whole argument.
Why builders are the leading edge
If you want to know where the local-compute future arrives first, watch the people building with AI rather than the people chatting with it — because agentic workloads have an economic structure that makes local irresistible, and they have it now.
A conversational chatbot makes one expensive call and returns a paragraph a human reads. An agent does something entirely different: it makes thousands of small calls to complete a single complex task, most of them narrow and repetitive — parse this command, extract these fields, classify this diff, format this output, decide which tool to call next, summarise what just happened. As NVIDIA's researchers put it, agentic systems are "a mass of applications in which language models perform a small number of specialised tasks repetitively and with little variation." Each of those calls, individually, is easy. None of them needs a frontier model. But because there are thousands of them per task, the cost of routing every one to a premium cloud API is enormous — and the cost of not doing so is the single largest lever in agent economics. Even a partial shift of those calls from a large model to a small one, the paper shows, cuts operating costs by orders of magnitude while leaving the result essentially unchanged.
This is the same economic wound, seen from the other side, that makes frontier inference so punishing to sell. We covered before, in the episode on what an AI subscription really costs the maker, just how unforgiving the unit economics of premium reasoning models are: heavy users running long chains of thought can consume far more compute than their subscription covers. Push that workload onto a small local model and the equation inverts — the marginal cost of a token drops to the electricity to compute it, the per-token meter disappears, and the code never leaves the laptop. For a builder running an agent that hammers a model all day, "free, private, and on my own hardware" is not a privacy nicety; it is a structural cost advantage.
The cultural shift among builders is already visible. Andrej Karpathy, who has a sharper feel for where this is heading than almost anyone, coined the term agentic engineering in early 2026 to describe his own working life: "You are not writing the code directly 99 percent of the time. You are orchestrating agents who do, and acting as oversight." His own experiments point the same way — an autonomous research agent he released runs machine-learning experiments unattended on a single GPU, looping through propose-train-evaluate cycles seven hundred times over two days, and a personal knowledge system he built on plain text files that he reports is seventy times more efficient than the vector-database approach it replaced. The pattern underneath all of it is the same: take the expensive, general-purpose frontier model out of the inner loop, and put cheap, specialised, often-local compute in the hot path where the volume is. Builders are not waiting for permission. The tooling has arrived to meet them.
The combined stack is already shipping
The reason this is a 2026 story and not a 2028 prediction is that the combined local-plus-cloud stack stopped being an architecture diagram and started being a product you can download. The defining design pattern has a name now — hybrid inference — and a clear shape: a small model running locally acts as a router, deciding for each request, in real time, whether it can be answered on-device or must be escalated to a frontier model in the cloud
.
The most striking demonstration came from Perplexity, which unveiled a hybrid local-server inference orchestrator at Computex 2026, built with Intel and rolling out to its desktop app in July. What makes it notable is the framing of the routing decision itself: the local model treats where to run each subtask as an inference problem to be reasoned about in real time, mid-task, invisibly to the user. Sensitive data — financial records, health files — stays on the device by default; genuinely hard reasoning is sent to the cloud, and the system asks permission before anything sensitive leaves the machine. Perplexity's chief executive, Aravind Srinivas, was unusually candid about the motive. He does not want, he said, "all your compute centralised in servers"; he noted that "some people are spending half a billion dollars per month" on inference, and that pushing work onto users' own devices is a way to cut that bill. His company's own numbers make the point — revenue grew from $100 million to $500 million while headcount rose just thirty-four percent — and the post announcing the system carried a title that doubles as the thesis of this whole story: The data centre moves to your machine.
Perplexity is not alone; it is simply the most legible example of an industry-wide move. Microsoft shipped Foundry Local, a roughly twenty-megabyte runtime that runs models on a Windows machine's CPU, GPU, or neural processing unit with no API key, no per-token cost, and no cloud dependency — and at its Build conference introduced Aion 1.0 Plan, a 14-billion-parameter model with a 32,000-token context window and fully local agentic tool-calling and multi-step planning. NVIDIA pushes the same direction with its desktop AI hardware; Google has wired a small Gemini model directly into Chrome. The largest companies in computing are, simultaneously and independently, building the on-ramp for AI work to leave their own data centres. When the incumbents whose revenue depends on cloud inference are themselves shipping the tools to move that inference to the edge, the trend has stopped being speculative. The only thing standing between this stack and ubiquity is that, the moment you go local, it gets genuinely hard to make it work.
The harness is the hard part
Here is the counterintuitive truth that the cheerful "run it on your laptop for free" tutorials tend to skip: the model was never the hard part of going local. The harness is.
The evidence comes from the engineering of agents in general. Across real-world deployments, roughly sixty-five percent of enterprise AI agent failures trace not to the model's reasoning at all but to "harness defects" — context drift, schema misalignment, state degradation: the surrounding system losing track of what the model knows, mangling the format of a tool call, or corrupting the agent's memory of its own task. The blunt conclusion the field has reached is that, in many deployments, improving the system around the model produces better results than upgrading to a larger model. The reason is fundamental: a language model's compliance with instructions is probabilistic, not deterministic, and the only way to make a probabilistic component behave like dependable software is to wrap it in deterministic constraints — linters, type checks, CI gates, a plan-execute-verify loop that catches the model's mistakes before they propagate. The harness is the machinery that converts an unreliable genius into a reliable worker
.
That tax is steepest exactly where you would hope it would be lightest: on your own machine. Running a capable coding agent against a local model is now genuinely possible — since early 2026, the open-source Ollama runtime speaks the same API as the major coding agents, so tools like Claude Code, Cline, and OpenCode work with local models out of the box. But the lived experience is a cautionary tale about how much scaffolding the cloud was quietly providing. Get the local model's configuration subtly wrong — and the defaults are wrong — and the agent "loses track of file contents mid-edit, forgets earlier instructions, and produces fragmented changes," silently breaking a refactor in ways that look like success until you read the code. The single most common failure is mundane and brutal: the runtime's default context window is just four thousand tokens, which an agentic coding session exhausts in seconds, so the model simply forgets the first half of its own task unless you know to raise it to thirty-two thousand or more. None of this is the model's fault. It is the harness — the part that, in a polished cloud product, an army of engineers has already built and tuned for you, and that you inherit raw and unassembled the moment you go local.
This is the real gate on the edge future, and it is why the transition will be uneven rather than sudden. The economic case for local compute is overwhelming and getting stronger every quarter. But economics do not deploy themselves; engineering does. The winners of the local-AI era will not be whoever has the best small model — those are converging and increasingly free. They will be whoever builds the best harness around a small model: the orchestration, the verification, the routing, the recovery from error, all the boring deterministic plumbing that makes a probabilistic component trustworthy in production. That is a software problem, not a model problem, and it is wide open.
So what should you actually run where
Strip away the noise and the practical question facing any company or engineer is concrete: of all the AI work you do, how much should run on your own hardware versus a cloud API, and what should you actually pay for? The research now supports a usefully sharp answer, and it has two axes — task type and cost crossover.
By task type, the dividing line is reliability under variation. The work that belongs local is the high-volume, well-specified, low-variation work that needs competence but not brilliance: classification, extraction, summarisation, formatting, routing, the routine tool-calls inside an agent, and — to take the cases a builder feels most — running test suites, triaging and labelling, routine repository chores, first-pass dependency and security scanning, the deterministic-with-a-little-reasoning tasks that recur thousands of times and rarely surprise you. Industry routing data is consistent and striking: somewhere between seventy and eighty percent of production queries never need a frontier model, and a well-tuned router that sends that bulk to small, cheap, often-local models while reserving a frontier model for the genuinely hard reasoning cuts total inference cost by sixty to eighty percent with negligible loss in quality. What stays in the cloud is the minority that earns it: dynamic planning, complex multi-step reasoning, novel problems, sophisticated error recovery — the work where a frontier model's extra capability is the difference between success and failure.
The second axis is the cost crossover, and it answers "what do you actually pay for." Cloud APIs are pure operating expense — you pay per token, forever, and you pay a margin on top of the provider's own costs. Self-hosting is capital expense — a meaningful upfront cost (an eight-GPU server runs $400,000 to $500,000, or nothing beyond hardware you already own for laptop-scale local models) amortised across everything you run on it. The crossover is a function of utilisation: once a workload runs at sustained load above roughly sixty to seventy percent, self-hosted inference comes in eight to eighteen times cheaper per token than the equivalent cloud API, and the break-even against cloud arrives in as little as a few months for a genuinely steady workload. The decision rule that falls out of this is clean. Run it local when the load is high, steady, and predictable, when data residency matters, or when token volume is large enough that the per-token meter dominates. Reach for the cloud when the work is bursty or experimental, when you cannot or should not commit capital up front, when you need multi-region reach, or when you need the very newest frontier capability the instant it ships and before any small model has caught up. Pay, in other words, for the frontier and for elasticity — and stop paying, by the token, for the commodity. It is no accident that roughly seventy percent of enterprises are expected to run hybrid architectures in 2026; the hybrid is not a hedge, it is the cost-optimal answer
.
Apple's heresy
Which brings us to the company that read this board earliest, and was mocked for it. For two years the consensus on Apple has been that it lost the AI race — no frontier model, no viral chatbot, an assistant that lagged. That consensus mistakes the game Apple chose not to play for a game it lost. Apple is making a deliberate, structural bet that the model is becoming the commodity and the edge is the moat — and in 2026 the shape of that bet became undeniable.
Start with what Apple actually built, because its architecture is the two-tier world rendered in silicon. Apple's third-generation Foundation Models, detailed by its own machine-learning researchers in June 2026, are led on-device by a model called AFM 3 Core Advanced: twenty billion total parameters, but built sparse so that it activates just one to four billion of them at a time depending on the request, using a pruning technique that keeps the full model in flash storage and loads only the needed experts into memory. That is the mixture-of-experts efficiency trick from the earlier chapters, productised and shipped to run privately in your pocket. Above it sits a server tier on Apple's Private Cloud Compute for the hardest work — and here is the tell. Apple's most capable cloud model, AFM 3 Cloud Pro, is optimised for NVIDIA GPUs inside Google Cloud, and all five of Apple's foundation models were, in Apple's own words, "custom-built in collaboration with Google." The world's most valuable company, asked to field a frontier model, chose to build one with a rival and run it on that rival's infrastructure rather than win the model race itself. It put its own engineering into the on-device tier and the privacy architecture, and it leaned on Google for the cloud frontier. That is not the behaviour of a company that thinks the biggest model is where the value is.
The financials make the bet legible. Apple's 2026 capital expenditure is roughly $13 billion — against Amazon's roughly $200 billion, Microsoft's roughly $190 billion, and Alphabet's roughly $175 to $185 billion. Apple is spending more than ten times less than any single one of its hyperscaler rivals, and around fifty times less than the four of them combined, on the AI infrastructure build everyone else treats as existential
. A company that believed the future belonged to whoever owns the most cloud compute could not possibly spend this little. Apple spends this little because it does not believe that. As Bank of America's Wamsi Mohan frames it, Apple's real asset is an "agentic AI moat" — its control over the device, the user's identity, their payments, and their trust — and that moat becomes more critical, not less, "as AI agents proliferate" and the underlying model commoditises beneath them. Apple's logic is that when intelligence is everywhere and roughly free, the scarce, defensible thing is not the model but the trusted surface it runs on — and Apple owns the most valuable such surface on Earth: a billion-plus devices it can upgrade with AI overnight through a software update, in homes and pockets that no rival AI company can reach without first solving the hardware, distribution, and ecosystem problems Apple solved decades ago.
The bet is not guaranteed. Apple's on-device models are genuinely behind the frontier on hard reasoning, its assistant has disappointed before, and leaning on Google for the cloud tier is a real dependency on a competitor. But the strategic premise is the most clear-eyed read of the trend in this whole story: if most AI interaction becomes ambient, private, and edge-resident, then the company that wins is not the one with the cleverest model in a data centre but the one whose efficient-enough models are already running, trusted and integrated, on the device in your hand. Apple is not failing to play the AI race. It is betting that the race everyone is watching is the wrong one.
Bottom line
The trillion-dollar capex race is real, and the cloud frontier it is building is real and worth paying for — but it is not the whole map, and treating it as the whole map is the strategic error of the moment. Underneath the frontier, the cost of competent intelligence is collapsing at roughly ten times a year, small models have crossed the threshold into reliable agentic work, and the combined local-plus-cloud stack — a small model on your own hardware deciding what it can handle and what to escalate — has shipped from Perplexity, Microsoft, Apple, and Google in the same twelve months. For a company or an engineer, the move is no longer to send everything to a frontier API; it is to route the seventy to eighty percent of work that is repetitive, well-specified, and high-volume to small, cheap, increasingly local models, and pay frontier prices only for the genuine reasoning that earns it — a split that cuts cost by sixty to eighty percent today and tilts further toward the edge every quarter. The thing that gates this future is not the model, which is converging toward free, but the harness — the deterministic engineering that makes a probabilistic component dependable — and that is where the next decade's advantage is hiding in plain sight. Apple has already placed its chips on the edge, spending more than ten times less than its rivals and refusing to build a frontier model, because it has concluded that owning the trusted device matters more than owning the biggest model. You do not have to agree with Apple to take the operational lesson: the question that matters is no longer "which model is best." It is "where should this run, and who pays for it" — and the centre of gravity is moving, steadily and measurably, off the cloud and into your hands.
Sources
- LLMflation: LLM inference cost is going down fast — Andreessen Horowitz — the 10x-per-year thesis and the GPT-3 $60 to $0.06 per million tokens, 1,000x-in-3-years figure.
- LLM inference prices have fallen rapidly but unequally across tasks — Epoch AI — the median ~50x/year (200x post-2024) decline and the 9x-to-900x range across capability milestones.
- Small Language Models Are the Future of Agentic AI — Belcak et al., NVIDIA Research (arXiv:2506.02153) — SLMs as sufficiently powerful, more suitable, and 10-30x cheaper for agentic work.
- Introducing the Third Generation of Apple's Foundation Models — Apple Machine Learning Research — AFM 3 Core Advanced (20B sparse, 1-4B active) and the Google-built, NVIDIA-on-Google-Cloud server tier.
- Perplexity unveils hybrid local-cloud inference at Computex 2026 — VentureBeat — the local-model-as-router orchestrator shipping July 2026.
- Perplexity wants your laptop to do part of the AI work — Decrypt — Srinivas on not centralising compute, the half-a-billion-a-month inference reality, and what moves to the device.
- Accelerate Edge AI Development with Foundry Local — Microsoft Foundry Blog — the ~20MB no-API-key local runtime and the fully-local 14B Aion 1.0 Plan agentic model.
- On-Premise vs Cloud: Generative AI Total Cost of Ownership (2026) — Lenovo Press — the 8-to-18x self-host cost advantage above ~60-70% utilisation and the break-even maths.
- Hybrid cloud-edge LLM inference routing — TianPan.co — the "70-80% of queries never need a frontier model" and 60-80% cost-cut routing data.
- AI capex 2026: the infrastructure sprint — Futurum Group and Big Tech AI capex — Value Add VC — the ~$1.04T total / ~$725B Big-Four / ~$13B Apple capex figures.
- Apple's AI strategy and the agentic-AI moat — Sahm Capital — Bank of America's Wamsi Mohan on devices, identity, payments, and trust as the durable advantage.
- From Failed Trajectories to Reliable LLM Agents (arXiv:2606.06324) and Harness Engineering for AI Coding Agents — Augment Code — the 65%-of-failures-are-harness finding and the probabilistic-versus-deterministic constraint.
- Pairing Claude Code with local models — KDnuggets and Claude Code with Ollama — Ollama — the lived harness tax: mid-edit context loss, the 4K-default-context trap, and the 32K fix.
- Why everyone is talking about Andrej Karpathy's autonomous AI agent — Fortune — "agentic engineering" and the single-GPU autonomous research loop.
Transcript
Sam: Okay. The single most expensive thing humanity has ever built — and we're building it for a job that's busy becoming free.
Alex: That's the whole story in one sentence. This year the biggest companies on Earth will spend more than a trillion dollars building AI compute. The first trillion-dollar year of capital spending on one kind of infrastructure, ever. And the actual work that compute does is getting about ten times cheaper every single year.
Sam: So we're pouring concrete for the most expensive cathedral in history, and the congregation is quietly learning to pray at home.
Alex: Press play. Let's get into it. Welcome back to Dan's AI Intel — the show where we try to make sense of the fastest, most consequential shift any of us is likely to live through. I'm Alex, here with Sam, and today we're getting into something I think is the most under-told story in AI right now: where intelligence actually lives. Not which model is smartest — where the work physically happens. In a giant data centre somewhere else, or on the device already in your hand.
Sam: And that sounds almost academic until you realise the money riding on the answer. We're going to keep coming back to that opening tension all episode, because it really is the spine of the whole thing: a trillion dollars going one direction, and the actual workload sliding the other way.
Alex: The reason this matters — and the reason it's worth a whole episode — is that where the compute lives decides almost everything downstream. Who captures the money. What a product can cost. How private your data is. How fast an answer comes back. Which handful of companies hold real power as AI becomes ambient, just part of the air. A revolution that runs in five companies' data centres is a completely different revolution from one that runs on the billions of devices people already carry around.
Sam: So here's the map for today, and I want to lay it out because the shape of it is the argument. We start with the paradox — the trillion-dollar number, sitting right next to the fact that the work is collapsing in price. Then we go to why it's collapsing — there's a name for it, and it's a beautiful one. Then: can the small models actually do the job now, or is that hype? Then the people for whom this is already real — and it's not who you'd guess. Then the products that already shipped this year. Then the catch — because there's a brutal catch nobody puts in the tutorials. Then a genuinely useful decision rule for what to run where. And we finish on the company everyone says lost the AI race, which I'm told read the whole board more clearly than anyone.
Alex: And I'll just plant a flag on that last one, without giving it away: the company we end on is spending more than ten times less than its rivals on this, and being mocked for it. By the end you'll have a view on whether that's a blunder or the sharpest call in the industry.
Sam: A trillion dollars of "obviously" against one company's "obviously not." I know which way my gut leans, and I suspect by the end it'll have flipped. If you've been enjoying the show, follow us on Spotify or Apple Podcasts so you don't miss the next one — we do one of these properly, every week.
Alex: Let's start with the number, because honestly it's almost too large to feel. In 2026, total capital spending on compute across the whole industry — the big cloud providers, the specialist "neoclouds," national sovereign programmes, China, all of it — comes to roughly one-point-zero-four trillion dollars. By the Futurum Group's accounting, that's the first trillion-dollar year of compute capex in human history.
Sam: Give me a sense of who's actually writing those cheques, because "the industry" is doing a lot of work in that sentence.
Alex: Fair. Just the four biggest American hyperscalers between them are planning to spend on the order of seven hundred and twenty-five billion dollars. Amazon's guiding to around two hundred billion. Microsoft near a hundred and ninety. Alphabet's targeting a hundred and seventy-five to a hundred and eighty-five. Meta's in the hundred-and-fifteen to a hundred-and-thirty-five range. And of that whole pile, about three-quarters — call it four hundred and fifty billion dollars — points directly at AI infrastructure.
Sam: Four hundred and fifty billion. In one year. Aimed at one thing.
Alex: One of the largest concentrated deployments of capital any industry has ever attempted. And here's what every dollar of it is really saying: it's a bet that demand for centralised, cloud-based AI compute keeps climbing steeply enough to justify the build. That's the assumption baked into the concrete.
Sam: Okay, so that's bet number one. You said there's a second fact that sits awkwardly next to it.
Alex: From the same year. The price of running a model at any fixed level of capability is falling by about ten times every year. That's Andreessen Horowitz's figure. And by Epoch AI's measurement — and they look across a lot of different capability thresholds — the median is closer to fifty times a year, accelerating to two hundred times for some tasks since early 2024.
Sam: Wait. So the spending is exploding, and the cost of the work is collapsing — at the same time. Those feel like they can't both be true.
Alex: They're both true. And the way you reconcile them is one word: volume. The bull case says demand is growing so fast that even if the cost per unit drops a hundredfold, usage rises a thousandfold, so the absolute spend still climbs. And that might genuinely hold.
Sam: But.
Alex: But it papers over something structural hiding inside the aggregate. Because not all of that exploding demand needs to be served from a multi-billion-dollar data centre.
Sam: Right — "demand is up" and "all of that demand belongs in the cloud" are two completely different claims, and the trillion dollars only makes sense if the second one is true.
Alex: That's exactly the move. Here's the paradox stated plainly. The industry is pouring its trillion dollars into the cloud half of AI compute at precisely the moment a large and growing share of the actual workload is becoming cheap enough, and small enough, to run somewhere else entirely. On the edge. On your devices. Off the metered cloud.
Sam: The cathedrals are going up for a congregation that's learning to pray at home.
Alex: That's the line, and I want to be careful with it, because it's not "the cathedrals were a mistake." The hardest workloads, and training the frontier models in the first place, genuinely need them. The point is narrower and sharper: the assumption that all AI compute consolidates upward, into the cloud, is wrong. And the people building only for that future are exposed to a second future arriving underneath them.
Sam: So to see whether that second future is real, we have to go look at how far the floor has actually fallen. Which has a name, you said.
Alex: It has a wonderful name. The single most important economic fact in AI is not how good the best model is. It's how fast the cost of "good enough" is collapsing. And Andreessen Horowitz gave it a name — LLMflation. The inverse of inflation: the same dollar buys you dramatically more intelligence every year.
Sam: I love that, because inflation is the thing we all feel as our money getting weaker. This is your money getting stronger, but only when it's spent on intelligence.
Alex: And the headline number is stark. GPT-3, when it launched back in November 2021, cost about sixty dollars per million tokens. The cheapest model of broadly equivalent capability today — a small open model, something like Llama 3.2 3B, served through a commodity provider — runs around six cents per million tokens.
Sam: Sixty dollars to six cents. Let me just sit with that. That's a thousand times cheaper.
Alex: A thousandfold drop in three years. Which annualises to roughly ten times a year, for a constant level of capability.
Sam: And that's the part I want to nail, because it's easy to wave past. We're not saying the models got ten times better. We're saying for the same capability — the thing GPT-3 could do — the price fell by ten times a year.
Alex: Exactly. Capability held fixed, price in freefall. That's the measurement.
Sam: You mentioned Epoch had a more careful version of this.
Alex: They did, and the nuance is the good part. Epoch measured across many capability milestones and found the decline is real but uneven — their phrase is "unequally across tasks." It ranges from about nine times a year at the slow end, all the way to nine hundred times a year at the fast end. Median around fifty times a year, which jumped to two hundred times for some benchmarks after January 2024.
Sam: Nine times to nine hundred times. That's a massive spread. What decides where a given task lands in that range?
Alex: This is the bit that matters most. The cheapest, most commoditisable tasks fall fastest. The genuinely frontier capabilities hold their price the longest. So for instance, the price to reach GPT-4's level on hard, PhD-grade science questions fell about forty times a year — fast, but not nine-hundred-times fast.
Sam: So the floor is dropping out from under the easy stuff, and the ceiling — the really hard reasoning — is staying expensive.
Alex: And that, right there, is the engine of the whole two-tier world we keep gesturing at. The bottom collapsing toward free while the top stays costly. It's not one market anymore. It's splitting into two.
Sam: Can I just test that, though? Why would the easy stuff fall fastest? Intuitively I'd expect everything to get cheaper at roughly the same rate.
Alex: It's a great question, and the answer is competition. The cheap, commoditisable tasks — classification, extraction, summarising — lots of small open models can do those. So the moment one provider drops the price, three others undercut them, and the floor just caves in. The genuinely hard frontier reasoning? Almost nobody can do it. There's no competitive pressure dragging that price down, so it holds.
Sam: So scarcity protects the top, and a price war demolishes the bottom. The cheap stuff falls fast precisely because it's not special.
Alex: That's the whole texture of it. And it means the gap between "free intelligence" and "expensive intelligence" isn't closing — it's widening into two genuinely different markets, with different physics.
Sam: A curve that steep doesn't just happen. What's actually driving it?
Alex: Andreessen Horowitz pulls out six forces, and the important thing is they stack on top of each other. One: better price-performance from each new generation of GPU. Two: aggressive quantisation — and I'll explain that one. Three: software and serving optimisations. Four: smaller models reaching capability that used to need big ones. Five: better instruction-tuning, getting more out of each parameter. Six: brutal open-source competition compressing everyone's margins.
Sam: Hold on — quantisation. Translate.
Alex: So a model is, under the hood, a giant pile of numbers — the weights. Normally each of those numbers is stored at sixteen bits of precision. Quantisation is squeezing them down toward four bits — much coarser numbers — with surprisingly little loss in quality.
Sam: So it's like... taking a huge, ultra-high-resolution photo and saving it as a smaller file. Same picture, a fraction of the size, and your eye can barely tell.
Alex: That's a genuinely good way to hold it. Same model, far less memory and compute to run it, quality basically intact. And here's why I flagged it. None of those six forces is a one-off — they compound, year on year. But several of them — quantisation, smaller models, open weights — are exactly the forces that make a model small and cheap enough to leave the data centre in the first place.
Sam: Oh, that's the sleight of hand. LLMflation isn't just lowering the price of cloud tokens.
Alex: It's manufacturing the very models that no longer need the cloud at all. The same process that makes intelligence cheaper is also making it portable. And the clearest place to actually see that is in what a genuinely small model can now do — which two years ago, the answer was: not much. Two years ago, "small model" basically meant "toy." A few-billion-parameter model could autocomplete a sentence and not much else. Anything that looked like reasoning, or reliable tool use, or multi-step work, demanded a big frontier model reached over an API.
Sam: And now?
Alex: That gap has closed faster than almost anyone expected. And the reason is as much architectural as it is just scale. The breakthrough is something called sparsity — mixture-of-experts designs.
Sam: Okay, "mixture of experts." That's one of those phrases that sounds like it means something and I'm not sure it does. What is it actually doing?
Alex: It decouples how much a model knows from how much computation any single word costs it. Take Qwen2.5-Coder, a popular open coding model. It carries thirty-point-five billion total parameters — that's its total knowledge. But it only activates about three-point-three billion of them per token.
Sam: So it knows like a thirty-billion model but runs like a three-billion one.
Alex: That's it exactly. It runs at roughly the speed of a three-billion-parameter model while reasoning closer to a thirty-billion one — with a context window big enough to hold a real codebase. And the 2026 successors push the trick further: a thirty-five-billion-parameter model that fires only about three-and-a-half billion at a time.
Sam: Give me the analogy, because I think there's a clean one here.
Alex: Think of a huge hospital. It might have a thousand specialists on staff — that's the total knowledge. But you, walking in with one specific problem, don't get seen by all thousand. You get routed to the two or three who handle exactly your case. The hospital is enormous; your visit is small and fast. Mixture-of-experts is that: a giant model, but each question only ever wakes up the handful of "experts" inside it that it actually needs.
Sam: That's lovely, because it explains how you get both at once — the breadth of a big model and the speed of a small one. It's not a compromise between them.
Alex: And the dense small models matured too — the ones that aren't using that trick, that fire all their parameters every time. Microsoft's Phi-4, at fourteen billion parameters, scores about eighty-four-point-eight percent on the MMLU knowledge benchmark and fits comfortably on a consumer twelve-gigabyte graphics card. And the three-to-four-billion class — Phi-4-mini, Gemma 3 4B, Qwen3 4B — now handles real production coding help and document work on a laptop most developers already own. Phi-4-mini generates fifteen to twenty tokens a second on a MacBook Air.
Sam: A MacBook Air. Not a server rack. The thing in someone's bag.
Alex: But here's the capability that actually changed everything, and it's not raw knowledge. It's agency.
Sam: Meaning the model can do things, not just know things.
Alex: Right. The single biggest shift of 2026, the way the open-model community itself frames it, is that open and small models now reliably support the agentic patterns that until recently were the exclusive preserve of the big proprietary APIs. Function calling. Structured tool use. The plumbing — the model-context-protocol stuff — that lets a model actually act on the world rather than just talk about it.
Sam: How do you even measure "can it act," though? That sounds soft.
Alex: There's a benchmark built specifically for it — τ²-bench — and it tests exactly that: can the model call tools, execute steps, and recover when something goes wrong, across a multi-step workflow. Open coding models now score in the mid-eighties on it.
Sam: And that's the line between a clever autocomplete and an actual worker.
Alex: That's the threshold. And the part of that benchmark that matters most is the last bit — "recover from errors." Because calling a tool once when everything goes right is easy. The thing that separates a worker from a toy is what happens when the tool call fails, or returns garbage, and the model has to notice, back up, and try something else. That's the difference between something you babysit and something you can actually leave alone.
Sam: So it's not "can it do the step." It's "can it survive the step going wrong."
Alex: And once a small model can do that reliably, the economics get almost silly. Because remember — it's ten to thirty times cheaper to serve than a frontier model. But there's a second number that's just as important: you can fine-tune one of these in hours, not weeks. So you can take a small open model, train it on your specific narrow task overnight, and have a specialist that's cheap, fast, and better at your exact job than the giant generalist ever was.
Sam: Overnight. So this isn't some research-lab luxury. A normal team could actually do that.
Alex: A normal team with one decent graphics card could do that. Which is exactly why this stopped being theoretical. And NVIDIA's research arm put the whole conclusion into a title that reads like a thesis statement: "Small Language Models Are the Future of Agentic AI."
Sam: NVIDIA said that? The company whose entire business is selling the giant chips for the giant data centres?
Alex: Hold onto that, because it gets even sharper later. The paper — Peter Belcak and colleagues — argues small models aren't a budget compromise for agents. They're the correct tool, on three grounds at once. They're powerful enough for the narrow, repetitive tasks agents actually do. They're inherently more suitable, because those tasks have low variation. And they're necessarily more economical — ten to thirty times cheaper to serve than a frontier model, and fine-tunable in hours instead of weeks.
Sam: And the key word in there is "repetitive." The whole case rests on what agentic work actually looks like.
Alex: It's the hinge of the entire argument. So let's go look at it — at the people for whom this is already real. If you want to know where the local-compute future arrives first, here's the rule: watch the people building with AI, not the people chatting with it.
Sam: Why them specifically?
Alex: Because agentic workloads have an economic structure that makes local irresistible — and they have it right now, not in some projected future. Picture a chatbot. You ask it something, it makes one expensive call, it returns a paragraph a human reads. That's the chat shape.
Sam: One question, one answer.
Alex: An agent does something completely different. To complete a single complex task, it makes thousands of small calls. Most of them narrow and repetitive — parse this command, extract these fields, classify this code change, format this output, decide which tool to call next, summarise what just happened.
Sam: So the agent is having this frantic little internal monologue, thousands of tiny thoughts, just to do one thing I asked.
Alex: And NVIDIA's researchers describe it exactly that way — agentic systems are, in their words, "a mass of applications in which language models perform a small number of specialised tasks repetitively and with little variation." Each of those calls, on its own, is easy. None of them needs a frontier model.
Sam: But there are thousands of them.
Alex: There are thousands of them per task. So if you route every single one to a premium cloud API, the cost is enormous. And the flip side — not doing that — is the single largest lever in the whole economics of agents. Even a partial shift of those calls from a big model to a small one cuts operating costs by orders of magnitude, while leaving the result essentially unchanged.
Sam: Orders of magnitude. For basically the same output. That's not an optimisation, that's a different business.
Alex: And this is the same economic wound, seen from the other side, that makes selling frontier inference so brutal. We've talked before on the show about what an AI subscription really costs the company making it — how unforgiving the unit economics of premium reasoning models are. A heavy user, running long chains of thought all day, can burn far more compute than their subscription ever covers.
Sam: Right, the all-you-can-eat buffet where a few customers eat the whole kitchen.
Alex: Now invert it. Push that workload onto a small local model and the whole equation flips. The marginal cost of a token drops to basically the electricity it takes to compute it. The per-token meter just... disappears. And the code never leaves the laptop.
Sam: So for a builder running an agent that's hammering a model all day long, "free, private, on my own hardware" isn't a nice-to-have.
Alex: It's a structural cost advantage. It's a different cost base than your competitor who's renting every token. And the cultural shift is already visible. Andrej Karpathy — who has a sharper feel for where this is going than almost anyone — coined a term in early 2026 for his own working life: agentic engineering. His description of it: "You are not writing the code directly ninety-nine percent of the time. You are orchestrating agents who do, and acting as oversight."
Sam: So the human's job moved up a level. You're not the bricklayer, you're the foreman watching a crew of agents.
Alex: And step back and notice what that does to the whole race, because this is the deeper implication. For three years the industry has measured itself by one number — whose model sits at the top of the benchmarks. But if most of the actual work is commoditising and draining to the edge, then the frontier becomes a smaller and smaller slice of where the value actually lands.
Sam: So being king of the benchmarks matters less if the benchmark-grade work is the rare part, and the bulk has moved to cheap models on people's own machines.
Alex: And the durable advantage shifts. It stops being "who owns the best model" and becomes "who owns the distribution of cheap intelligence to where people already are." Which — keep that sentence in your pocket — is exactly the bet we end the episode on. But you can already feel it forming here, in how builders work: the prize is moving from the model to the place the model runs.
Sam: Okay, I'm officially holding that thought.
Alex: And his own experiments point the same direction. He released an autonomous research agent that runs machine-learning experiments unattended, on a single GPU — looping through propose, train, evaluate, seven hundred times over two days, no human in the loop. And a personal knowledge system he built on plain text files that he says is seventy times more efficient than the vector-database approach it replaced.
Sam: Seventy times — by going simpler. By using less machinery, not more.
Alex: And that's the pattern underneath all of it. Take the expensive, general-purpose frontier model out of the inner loop. Put cheap, specialised, often-local compute in the hot path — where the volume actually is. Builders aren't waiting for anyone's permission here. The tooling showed up to meet them. Which is the next question: what tooling? Because this stops being a prediction the moment you can download it. So here's why this is a 2026 story and not a 2028 prediction. The combined local-plus-cloud stack stopped being an architecture diagram and started being a product you can actually download. The design pattern even has a name now — hybrid inference. And the shape is clean: a small model running locally acts as a router. For each request, in real time, it decides — can I answer this on the device, or does this one need to go up to a frontier model in the cloud?
Sam: So the little local model is basically the receptionist. It takes every request, and most of the time it just handles it, and only the genuinely hard cases get sent through to the specialist.
Alex: That's the architecture, and it's exactly right. And the most striking demonstration came from Perplexity. At Computex 2026 they unveiled a hybrid local-server inference orchestrator — built with Intel, rolling out to their desktop app in July.
Sam: What makes it different from just "sometimes use a small model"?
Alex: The framing of the routing decision itself. The local model treats where to run each subtask as an inference problem — something to reason about in real time, mid-task, invisibly to you. Sensitive data — your financial records, health files — stays on the device by default. Genuinely hard reasoning goes to the cloud. And the system asks permission before anything sensitive ever leaves the machine.
Sam: So privacy isn't a setting you toggle. It's the default behaviour of the router.
Alex: And Perplexity's chief executive, Aravind Srinivas, was unusually blunt about why. He said he does not want "all your compute centralised in servers." He pointed out that "some people are spending half a billion dollars per month" on inference — and that pushing work onto users' own devices is a way to cut that bill.
Sam: Half a billion a month. That's the problem they're trying to make disappear.
Alex: And his company's own numbers make the case. Revenue grew from a hundred million dollars to five hundred million while headcount rose just thirty-four percent. And the blog post announcing the whole thing carried a title that honestly doubles as the thesis of this entire episode: "The data centre moves to your machine."
Sam: That's the cathedral line again, isn't it. Said by the people building it.
Alex: And I want to pull on that routing idea once more, because it's the genuinely clever part. The old way you'd build this is a dumb rule — "anything under N words, do it locally; otherwise, cloud." Brittle, and usually wrong. What Perplexity is doing is treating the question "where should this run" as itself a thing the model reasons about, fresh, for every single request, in the middle of the task.
Sam: So the decision isn't hard-coded. The system is genuinely thinking, per request, "can I handle this one, or is this one over my head?"
Alex: And it does it invisibly — you never see the handoff. And the privacy piece falls out of that for free, which is the part I think people will end up caring about most. Your financial records, your health files — those don't go anywhere by default. They're handled on the machine. The only time anything sensitive leaves is when the system stops and asks you first.
Sam: Which is the opposite of how the cloud era worked, where the default was "everything goes to the server and you hope they're careful with it."
Alex: Complete inversion. The default flips from "send it all up" to "keep it all here unless there's a reason not to." And that's not a privacy feature bolted on the side. It's just what the architecture does when the smart thing lives on the device.
Sam: But Perplexity's a challenger. They've got every reason to want to route around the giants. Is anyone with skin in the cloud game doing this?
Alex: That's exactly the right question, and the answer is the tell. Perplexity is just the most legible example of an industry-wide move. Microsoft shipped Foundry Local — a runtime of roughly twenty megabytes that runs models on a Windows machine's processor, graphics card, or neural chip, with no API key, no per-token cost, no cloud dependency at all. And at their Build conference they introduced Aion 1.0 Plan — a fourteen-billion-parameter model with a thirty-two-thousand-token context window and fully local agentic tool-calling and multi-step planning.
Sam: Microsoft. Who sells Azure. Who makes money when you run things in their cloud.
Alex: And NVIDIA pushes the same direction with its desktop AI hardware. And Google has wired a small Gemini model directly into Chrome. So sit with the shape of that. The largest companies in computing are, simultaneously and independently, building the on-ramp for AI work to leave their own data centres.
Sam: That's the part that actually convinces me. Because a challenger talking their book, fine. But when the companies whose revenue literally depends on cloud inference are themselves shipping the tools to move that inference off the cloud — they're not doing that for fun.
Alex: They're doing it because they can see where it's going, and they would rather sell you the shovel than miss the dig. When the incumbents build the off-ramp, the trend has stopped being speculative. There's only one thing standing between this stack and being everywhere. And it's a big one. The moment you go local — it gets genuinely, painfully hard to make it work. Here's the counterintuitive truth all the cheerful "run it on your laptop for free" tutorials tend to skip. The model was never the hard part of going local. The harness is.
Sam: Define "harness," because I don't think most people have a picture for that word.
Alex: The harness is everything around the model. All the surrounding software that takes this clever-but-unreliable component and makes it behave like dependable software you can actually ship. And here's the evidence for why it's the real problem. Across real-world deployments, roughly sixty-five percent of enterprise AI agent failures trace not to the model's reasoning at all — but to "harness defects."
Sam: Sixty-five percent. So most of the time the agent breaks, it's not because the model was dumb.
Alex: It's not the brain. It's the wiring. Three failure modes specifically: context drift, schema misalignment, state degradation. In plain terms — the system around the model loses track of what the model knows, or it mangles the format of a tool call, or it corrupts the agent's memory of its own task.
Sam: So the model says the right thing, and the plumbing fumbles it on the way to actually doing it.
Alex: And the blunt conclusion the field has landed on is striking: in a lot of deployments, improving the system around the model gets you better results than upgrading to a bigger model.
Sam: That's almost heretical, right? The whole industry's reflex is "throw a smarter model at it."
Alex: And the reason it's true is fundamental. A language model's compliance with your instructions is probabilistic, not deterministic. It usually does what you ask. Not always.
Sam: Which is fine for a chatbot and a disaster for software.
Alex: Right — and the only way to make a probabilistic component behave like dependable software is to wrap it in deterministic constraints. Linters. Type checks. CI gates. A plan-execute-verify loop that catches the model's mistakes before they spread.
Sam: Give me the analogy. What is the harness, really?
Alex: Think of an unbelievably talented intern. Brilliant, fast, occasionally just... confidently wrong. You would never let that intern push straight to production. You put checks around them — a senior reviews the work, tests have to pass, there's a checklist before anything ships. The harness is that whole apparatus of checks. It's the machinery that converts an unreliable genius into a reliable worker.
Sam: And the genius is free now. It's the apparatus that's expensive.
Alex: And here's the cruel twist. That tax — building all that scaffolding — is steepest exactly where you'd hope it would be lightest. On your own machine.
Sam: Why there? I'd assume my laptop is the simple case.
Alex: Because in the cloud, an army of engineers already built and tuned that scaffolding for you, invisibly. You inherit it polished. Go local, and you get it raw and unassembled. Now — running a capable coding agent against a local model is genuinely possible today. Since early 2026, the open-source Ollama runtime speaks the same API as the major coding agents, so tools like Claude Code, Cline, OpenCode work with local models right out of the box.
Sam: So the front door works. What goes wrong?
Alex: The lived experience is a cautionary tale about how much the cloud was quietly doing for you. Get the local model's configuration subtly wrong — and the defaults are wrong — and the agent "loses track of file contents mid-edit, forgets earlier instructions, and produces fragmented changes." It silently breaks a refactor in a way that looks like success until you actually read the code.
Sam: Looks like success. That's the nightmare, isn't it — not the error you can see, the one you can't.
Alex: And the single most common failure is so mundane it's almost funny, and it's brutal. The runtime's default context window is just four thousand tokens.
Sam: Meaning the model can only hold about four thousand tokens of the conversation in its head at once.
Alex: And an agentic coding session blows through four thousand tokens in seconds. So the model just... forgets the first half of its own task. Unless you happen to know to go in and raise that setting to thirty-two thousand or more.
Sam: So there's this invisible cliff, and if nobody told you the setting exists, you walk straight off it and the thing quietly falls apart — and you blame the model.
Alex: And none of that is the model's fault. It's the harness — the part that, in a polished cloud product, a whole team already built and tuned for you, and that you inherit raw the second you go local.
Sam: So this is the real gatekeeper. Not "is the model good enough."
Alex: This is the gate on the whole edge future, and it's why the transition is uneven rather than sudden. The economics of local compute are overwhelming and getting stronger every quarter. But economics don't deploy themselves. Engineering does.
Sam: So who actually wins the local-AI era? Because it sounds like it's not whoever has the best small model.
Alex: It's not — those are converging and increasingly free. The winners will be whoever builds the best harness around a small model. The orchestration, the verification, the routing, the error recovery — all the boring, deterministic plumbing that makes a probabilistic component trustworthy in production.
Sam: And that's a software problem. Not a model problem.
Alex: It's a software problem, and it is wide open. Nobody's locked it down yet.
Sam: Let me make sure I've actually got why the harness beats the bigger model, though, because it still feels backwards. If my model is smarter, surely it makes fewer mistakes, so I need less scaffolding?
Alex: You'd think so. But here's the thing — a smarter model is still probabilistic. It's wrong less often, but it's still wrong sometimes, and "sometimes" is fatal for software. Whereas the scaffolding — the linter, the type check, the verify step — is deterministic. It catches the mistake every single time, no matter how rare. So a slightly-dumber model wrapped in a good harness beats a slightly-smarter model running naked, because the harness closes the gap the model can never fully close on its own.
Sam: So you're not trying to make the model perfect. You're building a net that catches it whenever it slips.
Alex: And the shape of that net has a name — a plan-execute-verify loop. The model proposes a plan. It executes a step. And then something deterministic checks the result before anything moves forward. Propose, do, verify, repeat. That loop is most of what separates a demo from a product. Which, if you're trying to decide where to place your own effort as a builder, is maybe the most useful sentence in this whole episode: the edge isn't the model, it's the net. So let's get concrete — what should you actually run where? Strip away all the noise, and the practical question facing any company or any engineer is concrete. Of all the AI work you do, how much should run on your own hardware versus a cloud API? And what should you actually pay for? And the good news is the answer has gotten usefully sharp. It has two axes. The first is task type.
Sam: Okay, so how do I look at a piece of work and know which bucket it's in?
Alex: The dividing line is reliability under variation. The work that belongs local is the high-volume, well-specified, low-variation work that needs competence but not brilliance. Classification. Extraction. Summarisation. Formatting. Routing. The routine tool-calls inside an agent. And — to take the cases a builder feels most — running test suites, triaging and labelling, routine repository chores, first-pass dependency and security scanning. The deterministic-with-a-little-reasoning tasks that recur thousands of times and basically never surprise you.
Sam: So the stuff that's high-volume and boring and predictable goes on your own hardware. What earns the trip to the cloud?
Alex: The minority that genuinely earns it. Dynamic planning. Complex multi-step reasoning. Novel problems. Sophisticated error recovery. The work where a frontier model's extra capability is the actual difference between success and failure.
Sam: And do we have a number for how that splits in practice? Because "most of it is routine" is the kind of thing people assert.
Alex: We do, and it's consistent and striking. Industry routing data says somewhere between seventy and eighty percent of production queries never need a frontier model at all.
Sam: Seventy to eighty percent. So the giant expensive model is genuinely overkill for the large majority of what's actually being asked of it.
Alex: And a well-tuned router — that sends that bulk to small, cheap, often-local models and reserves the frontier model for the genuinely hard reasoning — cuts total inference cost by sixty to eighty percent, with negligible loss in quality.
Sam: That number's wild. You're telling me there's a sixty-to-eighty-percent saving sitting on the table, and the only thing standing between a company and it is being willing to route the easy stuff to the cheap model.
Alex: That's the lever. That's the whole game in one decision.
Sam: You said two axes. What's the second?
Alex: The cost crossover. This is the one that answers "what do you actually pay for," in money terms. Cloud APIs are pure operating expense. You pay per token, forever, and you pay a margin on top of the provider's own costs.
Sam: And that word "forever" is doing some quiet damage there, isn't it. Because operating expense never ends. It's not that you bought the thing — you're renting it every single month, for as long as you run it, and the bill only goes up as you use it more.
Alex: And you're paying the provider's profit margin on top of their own costs, every token, the whole time. Whereas self-hosting is capital expense — a meaningful cost up front, but then amortised across everything you run on it. You buy the machine once; after that, the marginal cost is mostly just power.
Sam: Give me the real numbers, because "it depends" is doing a lot of hiding.
Alex: An eight-GPU server runs four hundred to five hundred thousand dollars. Or — and this is the part people miss — nothing beyond hardware you already own, for laptop-scale local models. And the crossover between those two worlds is a function of one thing: utilisation. How busy you keep the machine.
Sam: So it's like buying a car versus taking taxis. The car's a fortune up front, but if you drive constantly it's far cheaper per mile. Take two trips a year, and the taxi wins easily.
Alex: That is exactly the shape, and the numbers are concrete. Once a workload runs at sustained load above roughly sixty to seventy percent, self-hosted inference comes in eight to eighteen times cheaper per token than the equivalent cloud API. And the break-even against the cloud arrives in as little as a few months for a genuinely steady workload.
Sam: Eight to eighteen times cheaper. So for the heavy, constant stuff, owning the machine isn't a little cheaper. It's an order of magnitude.
Alex: And out of all of this falls a clean decision rule. Run it local when the load is high, steady, and predictable. When data residency matters. Or when the token volume is so large the per-token meter dominates everything. Reach for the cloud when the work is bursty or experimental. When you can't or shouldn't commit capital up front. When you need multi-region reach. Or when you need the very newest frontier capability the instant it ships, before any small model has caught up.
Sam: So the one-liner is: pay for the frontier, and pay for elasticity. And stop paying, by the token, for the commodity.
Alex: That's the whole thing distilled. Pay for the two things that are genuinely scarce — the frontier capability, and the ability to scale up and down on demand. And stop renting, by the token, the thing that's becoming a commodity.
Sam: And the lovely part is that's not a compromise you settle for. The hybrid isn't "well, we couldn't decide, so we did half and half."
Alex: No — and that's the reframe I want to land. It's the actual optimum. It is no accident that roughly seventy percent of enterprises are expected to run hybrid architectures in 2026. They're not hedging their bets. They've worked out that the split — frontier on top for the hard minority, cheap and local underneath for the volume — is simply the lowest-cost way to get the same quality. The hybrid isn't a hedge. It's the right answer.
Sam: Which sets up the company we've been circling all episode. Because if the model is becoming the commodity, somebody big must be betting their whole strategy on it.
Alex: One of the biggest. And they got mocked for it for two straight years. For two years, the consensus on Apple has been simple: they lost the AI race. No frontier model. No viral chatbot. An assistant that visibly lagged. And here's my claim — that consensus mistakes a game Apple chose not to play for a game it lost.
Sam: That's a big distinction. Unpack it, because from the outside "we don't have a frontier model" sure looks like losing.
Alex: Apple is making a deliberate, structural bet that the model is becoming the commodity and the edge is the moat. And in 2026 the shape of that bet became undeniable. Start with what they actually built — because their architecture is the whole two-tier world we've been describing, rendered in silicon.
Sam: Okay, walk me through it.
Alex: On the device, Apple's third-generation Foundation Models — detailed by their own researchers in June 2026 — are led by a model called AFM 3 Core Advanced. Twenty billion total parameters. But built sparse, so it activates just one to four billion of them at a time, depending on the request. It uses a pruning technique that keeps the full model sitting in flash storage and loads only the experts it needs into memory.
Sam: That's the hospital again. The mixture-of-experts trick from earlier — only it's shipped, in your pocket, running privately.
Alex: That's exactly what it is — that efficiency trick, productised, running privately on the phone. And above it sits a server tier, on Apple's Private Cloud Compute, for the hardest work. And Sam — here is the tell. The single most revealing fact in the whole story.
Sam: Go on.
Alex: Apple's most capable cloud model, AFM 3 Cloud Pro, is optimised for NVIDIA GPUs inside Google Cloud. And all five of Apple's foundation models were — in Apple's own words — "custom-built in collaboration with Google."
Sam: Wait, wait. Apple — the most valuable company on Earth, the company that controls everything down to the screws — asked to field a frontier model, went and built it with Google? A rival? And runs it on the rival's computers?
Alex: They put their own engineering into the on-device tier and the privacy architecture, and they leaned on Google for the cloud frontier. And just think about what that decision reveals. That is not the behaviour of a company that believes the biggest model is where the value is.
Sam: Because if you thought the frontier model was the prize, you would never, ever outsource it to a competitor. You'd die before you did that.
Alex: You'd spend anything to own it. Which brings us to the money — because the financials make the bet legible in a way nothing else does. Apple's 2026 capital expenditure is roughly thirteen billion dollars.
Sam: Okay, hold that next to the others for me again. Amazon was —
Alex: Amazon, roughly two hundred billion. Microsoft, roughly a hundred and ninety. Alphabet, a hundred and seventy-five to a hundred and eighty-five. And Apple — thirteen.
Sam: So Apple is spending more than ten times less than any single one of its hyperscaler rivals.
Alex: More than ten times less than any one of them. And around fifty times less than the four of them combined — on the AI infrastructure build that everyone else is treating as existential, as life-or-death.
Sam: A company that believed the future belonged to whoever owns the most cloud compute could not possibly spend that little.
Alex: That's the whole argument in one line. Apple spends this little because it does not believe that. The thirteen billion isn't Apple failing to keep up. It's Apple telling you, in the only language that can't lie — where it puts its money — that it thinks everyone else is building the wrong thing.
Sam: So if the moat isn't the model, what does Apple think the moat is?
Alex: Bank of America's Wamsi Mohan has the sharpest framing of it. He calls Apple's real asset an "agentic AI moat" — its control over the device, the user's identity, their payments, their trust. And — this is the key move — that moat becomes more critical, not less, "as AI agents proliferate" and the underlying model commoditises beneath them.
Sam: So Apple's logic is: when intelligence is everywhere and basically free, the model stops being the scarce thing.
Alex: And the scarce, defensible thing becomes the trusted surface it runs on. And Apple owns the most valuable such surface on Earth — a billion-plus devices it can upgrade with AI overnight, through a software update. In homes and pockets that no rival AI company can reach without first solving the hardware, the distribution, the ecosystem problems Apple already solved decades ago.
Sam: Everyone else has to fight their way to the device. Apple's already there. It's already in the hand.
Alex: And sit with the asymmetry of that for a second, because it's the part I find genuinely hard to argue with. For a rival AI company to get where Apple already is, they'd have to build a phone, build an operating system, earn a billion people's trust with their payments and their identity and their photos, and win the retail and the carrier deals — decades of work, the exact problems Apple ground through twenty years ago. For Apple to get an efficient model onto a billion devices, it ships a software update on a Tuesday.
Sam: That's the whole moat in one image. One side has a twenty-year head start it can't be sped past; the other side just has a really good model — and the model is the thing turning into a commodity.
Alex: And that's why the thirteen-billion number isn't timidity. It's a company that looked at where the value is actually going to pool, and decided to spend its money on the part nobody can copy — the trusted surface — instead of racing for the part that's becoming free.
Sam: You're making it sound airtight. It can't be airtight. What's the case against?
Alex: It's genuinely not guaranteed, and I want to be honest about that. Apple's on-device models are genuinely behind the frontier on hard reasoning. Its assistant has disappointed before — more than once. And leaning on Google for the cloud tier is a real dependency on a competitor that could bite.
Sam: So the bet could lose.
Alex: The bet could lose. But the strategic premise underneath it is, I think, the most clear-eyed read of this entire trend. If most AI interaction becomes ambient, private, and edge-resident — running quietly on the device — then the company that wins is not the one with the cleverest model sitting in a data centre. It's the one whose efficient-enough models are already running. Trusted, integrated, on the device in your hand.
Sam: So the line that sticks for me is: Apple isn't failing to play the AI race. It's betting the race everyone's watching is the wrong race.
Alex: That's it exactly. And you don't have to agree with them to take the lesson. You just have to notice that the company everyone wrote off looked at the same board as everyone else — and read it completely differently. And put thirteen billion dollars where its read was.
Sam: Okay. Let's bring it home. If someone's been walking and listening to all of this, what are the few things they actually carry out the door?
Alex: Three things. First — the trillion-dollar capex race is real, and the cloud frontier it's building is real and worth paying for. But it is not the whole map. And treating it as the whole map is the strategic error of the moment.
Sam: Because underneath that frontier, the floor is collapsing.
Alex: Second thing. The cost of competent intelligence is falling at roughly ten times a year. Small models have crossed the line into reliable agentic work. And the combined local-plus-cloud stack — a small model on your own hardware deciding what it can handle and what to escalate — has shipped, this year, from Perplexity, Microsoft, Apple, and Google. All in the same twelve months. That's not a forecast. It happened.
Sam: And the third?
Alex: The third is the operational one — the bit you can actually use on Monday. The move is no longer to send everything to a frontier API. It's to route the seventy to eighty percent of work that's repetitive and well-specified and high-volume to small, cheap, increasingly local models — and pay frontier prices only for the genuine reasoning that earns it. A split that cuts cost by sixty to eighty percent today, and tilts further toward the edge every quarter.
Sam: And the catch — so nobody walks away thinking it's free.
Alex: The catch is the harness. The thing that gates this future isn't the model, which is converging toward free. It's the deterministic engineering that makes a probabilistic component dependable — the orchestration, …