The Model Is a Commodity Now: A Year of Building Around It
A processor die. Hold this image; it'll matter later. (Public domain, via Wikimedia Commons.)
Something funny happened this June. Anthropic shipped Claude Fable 5, the best model anyone had released, and then, a few days later, it was gone, pulled offline under a U.S. government export directive. The strongest model in the world, available on a Monday and unavailable by the weekend.
I keep coming back to that, because it crystallized a hunch I'd been chewing on all year. We spend an enormous amount of attention on which model is best (I do too; I swapped my daily driver maybe six times in twelve months). But if the best model on earth can vanish by directive in under a week, then the model was never really the thing you owned. It's rented. The thing you own is whatever you built around it.
That sounds abstract, so let me give you the mental model that actually worked for me. Think of the LLM as a CPU. A strange, fuzzy, expensive CPU, but a CPU. You do not win a race by buying a slightly faster CPU than your competitor, because you both buy the same one, off the same shelf, at the same price. You win with the program you run on it. And this year the CPU got dramatically cheaper and more abundant, which is historically the exact moment the interesting work moves up, into the software.
So I spent June 2025 to June 2026 writing that software, in the open, and treating each repo as a little experiment: read a paper, not quite believe it, build it, see what actually happens. None of the underlying ideas are mine. They came from the labs and from three papers I'll link as we go. What's mine is the building. Here is what I learned, loosely in three buckets: cheap, local, and verifiable.
Local: the frontier fits on a laptop now
The first thing I got obsessed with, around last October, was running real models on my own machine. The motivation is boring: cost, privacy, latency, and the fact that a model on your own hardware does not silently change behavior because a vendor shipped an update on a Tuesday. For a surprising amount of production work (classification, extraction, routing, scoring) you do not need a frontier model at all. You need a small, predictable one that you control.
Fine-tuning the small model is the easy part (mlx-finetuning is a little LoRA framework I wrote for Apple's MLX to do exactly that). The hard part, it turns out, is inference, and the wall you hit is the KV cache.
Quick intuition, because this is the thing nobody tells you: when a transformer generates text, it stores the "key" and "value" vectors for every token it has already seen so it doesn't recompute them on the next step. That store is the KV cache, and it grows with the length of your context. Past a certain point it, not the model's weights, is what fills up your memory. It's one of the first walls you hit when you push a local model past its comfort zone.
There's a 2026 paper from Google called TurboQuant (arXiv:2504.19874) that claims you can compress that cache 3 to 5x with almost no calibration and almost no quality loss. I did not believe the "almost no quality loss" part (I rarely do), so I implemented it: mlx-turboquant, a drop-in replacement for mlx-lm's cache. The repo's own benchmarks: on Llama 3.2-3B, 4-bit keeps a 0.997 cosine similarity to full precision, and 3-bit cuts cache memory by about 4.6x. (A caveat I'd want if I were you: cosine similarity is a fidelity proxy, not end-task accuracy, so I read it as a strong signal rather than a promise. But the memory math is real, and that's three to five times the context in the same RAM.)
Same laptop, about 4.6x more context. (Source: mlx-turboquant benchmarks.)
I went a little further down this hole than was strictly reasonable, porting NVIDIA's Nemotron (nemotron-mlx) and self-speculative decoding (ssd-macos) along the way.
The thing I actually took away: local inference is a systems problem, not a model problem. The bottleneck is not intelligence, it's memory bandwidth and cache. Get that right and a 4B model you own is genuinely enough for the long tail of narrow, repeated work, which is most of production and the part nobody puts in a demo.
Verifiable: I stopped trusting a single call
If there's one habit I'd transplant into every team, it's this one. Models are confident and wrong in ways that are systematic, not random, and a single model call taken at face value is a liability you've quietly decided not to look at.
So I started looking. phoenix-claude-code routes every call a coding agent makes through a proxy into Arize Phoenix, captures the trace, and runs automated judges over the sessions. The whole loop is just: trace every call, judge it, read the errors, fold them into a golden dataset, gate on it, repeat.
And here is the part that matters more than any tool: the failures live in the individual outputs, never in the aggregate dashboard. When you actually sit and read traces (which almost nobody does), the same shapes keep showing up. The model invents a tool argument. It silently drops a constraint you gave it three turns ago. It answers a slightly different question than the one you asked. It pads a thin answer into a confident-sounding essay. None of those light up as a red number on a chart. You only catch them by reading, and reading is the step everyone skips.
For the calls that really matter, I wrote llmcouncil, which asks several models from different families and makes them vote, debate, and red-team each other. This is not magic. If all the models share a blind spot (and models trained on similar data often do) the council will confidently agree and still be wrong. But it cancels the idiosyncratic errors, the ones a single model makes alone, and that turns out to be most of them.
The cheapest version of all this, if you want to feel it yourself tomorrow with no infrastructure: keep a small set of hand-labeled examples, run every comparison twice with the two answers swapped (only count a win if it wins in both positions), and judge with a different model family than the one that generated. That alone kills position bias and self-preference, which quietly invalidate most eval setups. (If you want the long version of why your judge is fooling you, I wrote that up separately.)
Cheap: stop writing prompts, compile them
This is the one that genuinely changed how I work. There's a paper called GEPA (arXiv:2507.19457, reflective prompt evolution) whose premise is that natural language is a much richer training signal than a single scalar reward. So instead of nudging a prompt by hand, you let a model read its own failure traces, write down in words what went wrong, and propose a fix, keeping a spread of good candidates so it doesn't overfit to one. On the paper's benchmarks it beats a standard RL method (GRPO) with a small fraction of the rollouts.
I wanted to see this on my own work rather than a benchmark, so I built optimize-anything, a little universal optimizer where the model proposes and judges its own improvements. The run I keep thinking about: it took one of my own Claude Code skills from 91% to 100% on its eval set, by itself, by reading why it was failing and rewriting itself. (Honest caveat: that's one skill on its own eval, so treat it as a demo, not a benchmark result.) But you watch that happen once and something shifts. "Prompt engineering" stops feeling like a craft and starts feeling like a compile step. If you have a good eval, this is mostly automatable. If you don't have a good eval, well, that's a bigger problem than your prompt.
The model read its own failures and closed the gap. One skill, in-sample, so read it as a demo. (Source: optimize-anything.)
Cheap: long context is a programming problem
Here's the result that made me sit up. There's a paper from MIT, Recursive Language Models (arXiv:2512.24601), and on a 1,000-document benchmark the plain frontier models score zero, because if you stuff all the documents into the prompt they just hit the context limit and fall over. Wrap the same model in a recursive loop and it scores 91.3%, at about a dollar a query, on inputs past ten million tokens.
Same base model. The only thing that changes is the system wrapped around it. (Source: Recursive Language Models, arXiv:2512.24601.)
The trick is lovely and obvious in hindsight. Instead of pouring a giant prompt into the window and hoping, you put the prompt in a variable and let the model program over it: examine it, break it up, and recursively call itself on the pieces that matter. The model writes code to navigate its own context. (Go back to the CPU analogy: this is the model using the prompt as memory instead of as registers.) I rebuilt this to run on any coding-agent harness as harness-rlm.
The lesson I'd tattoo somewhere if I were the tattoo type: "the context window is too small" is almost never the real problem. You're stuffing when you should be decomposing, and the cheapest token is the one you never send.
The plumbing: agents needed a memory
Underneath all of this, the quiet shift of the year is that the coding agent stopped being autocomplete and became general-purpose infrastructure, a thing you build other things on. I paired a planning model with an implementing one (cc-codex), and then immediately ran into the obvious wall: agents are amnesiacs. They forget everything between sessions, and the context they do keep slowly rots. So I built memex, a filesystem-native memory layer with an actual benchmark for memory quality (because otherwise you're just guessing), and deep-research, which fans out across agents and checks each claim before it writes a word. The newest one, from this month, is the direction I'm most excited about: artificer, which is basically make and ccache for AI computation, so you never pay to recompute an answer you already have.
What I actually think this all means
Step back and squint and the three buckets are one idea. Compress the cache so it runs on hardware you own. Decompose long context so you send fewer tokens. Compile your prompts against an eval. Cross-check instead of trusting one confident call. None of that is a model capability. All of it is engineering, and engineering is the part that doesn't get pulled offline by a memo.
Which brings us back to Fable 5, and to that processor die at the top of the page. The chip is gorgeous, but you'd never build a company on the assumption that you alone get to use it. The model is the same now: an extraordinary commodity component, the same one your competitor rents, on a schedule neither of you controls. The teams that had a rough week when Fable 5 disappeared were the ones whose product was the model. The teams that shrugged and changed one line were the ones who treated it as a component. That is the whole argument, and current events made it for me.
So the boring takeaway I believe more than ever: stop optimizing the one variable everyone else also has. The model is rented. The system is yours. Build the system.
(This is also, more or less, what I do for companies at Transfrm, but that isn't why I wrote this. I wrote it because I find it genuinely interesting, and because writing it down is how I figure out what I think.)
The repos, if you want to poke at them: mlx-turboquant, mlx-finetuning, nemotron-mlx, ssd-macos, phoenix-claude-code, llmcouncil, optimize-anything, harness-rlm, cc-codex, memex, deep-research, artificer.
The papers: GEPA (ICLR 2026), Recursive Language Models (MIT), TurboQuant (ICLR 2026).