Defaults are the policy

An organization watches its token bill grow faster than its payroll, and the reflex is a spending cap. The cap targets the heaviest users. Most of the spend sits somewhere else: in the defaults of the average engineer. Willpower does not bend an AI cost curve at fleet scale, and neither does a quota.

A layer sits between your engineers and the model. That layer carries a default model, a default context budget, a default tool set, and a default fallback. Most engineers never touch any of it. So the default runs thousands of times a day, applied to people who never chose it.

One public data point makes the case. Brian Armstrong reported that his engineering org cut AI spend nearly in half while token usage kept climbing. The team used better defaults, routing, and caching instead of caps. Engineers kept the same freedom, and the bill dropped.

The claim has a second half: you can only set those defaults well if the layer understands the work. Aaron Levie’s phrase for the goal is “more intelligence per dollar.” Cost work at this layer is applied AI with a bill attached.

Caps fight the wrong tail

Armstrong’s thread carries the number that should end most cap debates: 91% of their engineers never hit their usage caps. A cap is a wall in front of the other 9%, and those are usually the people who get the most out of the tool. Lower the cap and you create approval friction and alert noise for your best users, while 91% of the spend keeps flowing.

Spend at scale is calls, times tokens per call, times price per token. A cap trims the far end of one distribution. A default resets the price term for everyone at once, silently, with nobody to argue with. That asymmetry is the whole game: caps discipline the tail, and defaults move the body.

Guidance fails the same way a cap does. A doc that says “please pick a cheaper model” reaches the few engineers who read docs and change habits. The default reaches all of them. If the question is fleet behavior, the honest move is to make the right thing happen on its own.

A cap is a wall you build in front of your most productive engineers.

Your gateway falls back to some model. It loads some tools without being asked, and it carries some context by habit. Each of those is a decision multiplied across your entire org. Set them well and the bill bends without anyone noticing. Set them carelessly and no cap will save you.

Route by the work, not by the request

Once the default is cheap, the next dollar lives in routing: match the model and the effort to the task. Armstrong describes preprocessing the prompt, then sending planning to a frontier model and execution to a cheaper one. The expensive model is overkill for the second job. There is a case for a less capable model on purpose before cost enters the picture. At fleet scale, the cost argument arrives on top.

One pattern sits under all routing: default-deny, escalate on signal. Pick the cheap, bounded option by default, and let a request earn the expensive one. The cleanest version measures instead of guesses.

Some teams want to pull the million-token context variants of their models entirely. Their argument: a long session left alone fills that window and re-bills it every turn, so a smaller window caps the damage. The instinct is right, and the remedy is too blunt. In an agentic loop, the whole context is re-sent as input on every turn, so context size is a per-turn tax. A bigger window does not change the price per token. It raises the ceiling the tax can climb to.

A million-token window only costs you when the context grows into it, because you pay only for the tokens you send. Delete the big window and you also delete the single-pass jobs where it is the cheapest option, like one read of a 600k-token corpus. A better rule keeps the big window and gates it. Default every request to the bounded model, measure the assembled input, and escalate only on overflow or a real large-context signal, at any turn, decided by code. Tight default, automatic escalation: the same shape as the model tier, applied to the window.

Humans shouldn’t be choosing models, and they shouldn’t be choosing context windows either.

Caching as a prefix discipline

Caching is where the defaults pay off fastest, and where they quietly fail. A model cache matches on the prompt prefix. The stable front of your prompt (system instructions, tool definitions, retrieved context) is paid for once and reused at a fraction of the price. Anthropic’s prompt caching reads cached tokens at roughly a tenth of the base input rate. Armstrong’s team moved their cache hit rate from 5% to 60%, and that move is most of where a halved bill comes from.

One moving token at the top breaks it. A timestamp, a session ID, or a reordered tool list above the cache breakpoint invalidates everything after it, and the call drops back to full price. I walked through the per-call version of this in The token leaks your dashboard doesn’t show. The fleet version is the same rule enforced once, in the default prompt assembly, instead of hoped for across a hundred call sites.

Caching also collides with compaction. Compaction rewrites the conversation to shrink it, which changes the prefix and throws away the warm cache. The turn after a compaction pays a full cache write again. Compact every turn and you re-pay the cache every turn, and you burn tokens on the summary itself, which is the worst of both. Compact rarely and decisively, keep the post-compaction prefix stable, and let the cache amortize in between.

Lean means signal, not less

Lean is not fewer tokens used. Lean is fewer tokens wasted, and the difference is whether you cut noise or signal. Drop the eleven irrelevant repos and the nine unused tool integrations and you get pure upside: the calls cost less, and the model reasons better when the relevant detail is not buried. Drop context the task actually required and you buy a hallucination instead.

At fleet scale, the largest invisible tax is tool definitions. Every integration you connect injects its full schema into the prompt on every call, used or not. The schemas sit in exactly the part of the prompt you wanted to cache. Connect a dozen by default and every engineer pays for a dozen on every message. The fix is a default again: ship a lean tool set, and load the rest on demand, names first and full schemas only when a tool is called.

The guard against cutting too far is not carrying everything just in case. That habit is expensive and noisy. The guard is retrieval: keep the default tight, and make more reachable when the work calls for it. The right context is a fact about the work, not about the model.

Visibility is a requirement

Every lever above needs a number to tune, which makes visibility the precondition rather than a peer of the others. You cannot pick a default model without the model-mix breakdown. You cannot justify a routing rule without the escalation rate, fix a cache without the read-versus-write ratio, or trim context without per-call token attribution. Armstrong’s team got the cache win after they could see the hit rate, not before.

Visibility is also the humane alternative to the cap you were about to ship. Keep usage uncapped but visible, and tie spend to expected impact. The distribution then regulates itself: spend what the work needs, in the open, where high spend should track high output. The proof does not require measuring anyone’s output. It is two lines on one chart heading in opposite directions, blended cost per token falling while total usage climbs. That is what Armstrong reported, and it is the only scoreboard that survives a budget review.

Output per token is the real prize and the hard one, because attributing engineering impact to spend is a quarters-long problem. Do not gate the cost win on solving it. The cheap proxy, cost per token down while usage is up, is available now and carries the argument on its own.

You don’t prevent hallucination by hoarding context, you prevent it by making the right context reachable.

The layer that understands the work

Read the five levers back to back and one requirement runs under all of them. You cannot set a sane default, route by task, place a cache breakpoint, trim context, or read the numbers without knowing what the work is. Levie’s argument is that the value lives in a layer between the work and the raw intelligence, a layer that understands your workflows and context well enough to make those calls.

A bare proxy in front of the models saves nothing; it only forwards calls. The savings come from knowledge. The layer knows that planning needs the big model and execution does not. It knows that this team’s nine repos are signal and the other org’s are noise, and that this prefix is stable enough to cache. Strip out the domain knowledge and the same layer either over-stuffs every prompt or starves it. Expensive or wrong: take your pick.

The cheap framing of this work, “turn on caching, set a budget,” undersells it. A team that builds this layer is doing applied AI for its own org. It evaluates which model fits which job, scopes context to the actual task, and tunes the defaults to how people actually work. Levie calls the output more intelligence per dollar. The orgs that get there will not be the ones with the strictest caps. They will be the ones whose layer understands the work.

I have built both halves of this. A deploy-support system I built routes each investigation to the right model, verifies failures against live evidence before it spends a frontier call, and skips a redundant second call on most investigations. Cost per investigation dropped 44 to 57%. None of those wins came off a pricing page. They came from knowing what a deploy investigation actually needs.