Defaults are the policy

When an organization’s token bill grows faster than the cost of the engineers themselves, the reflex for many is a spending cap. This typically targets the people spending the most, and misses where most of the spend actually is: the average engineer’s defaults. The lever that bends an AI token cost curve at fleet scale isn’t willpower or a quota.

It’s whatever the layer between your engineers and the model does when nobody makes a choice. That layer carries a default model, a default context budget, a default set of tools, a default fallback. Most engineers never touch any of it. So the default is applied thousands of times a day by people who never opted into it.

One public data point makes the case. Brian Armstrong reported cutting an engineering org’s AI spend nearly in half while token usage kept climbing, using better defaults, routing, and caching instead of caps. Same freedom for the engineers, lower bill.

What completes the claim is that you can only set those defaults well if the layer understands the work being done. Aaron Levie’s framing for it is “more intelligence per dollar,” and it’s why cost work at this layer is really applied AI with the bill attached.

Caps fight the wrong tail

Armstrong’s thread carries the number that should end most cap debates: 91% of their engineers never hit their usage caps. A cap is a wall you build in front of the other 9%, who are usually the people getting the most out of the tool. Lower it and you manufacture approval friction and alert noise for your best users while 91% of the spend keeps flowing untouched.

Spend at scale is roughly calls times tokens-per-call times price-per-token. A cap chops the far end of one distribution. A default resets the price-per-token term for everyone at once, silently, with nobody to argue with. That asymmetry is the whole game: caps discipline the tail, defaults move the body.

Guidance fails the same way. A doc or blind mandate that says “please pick a cheaper model” reaches the handful of engineers who read docs and change habits. The default reaches all of them. When the question is fleet behavior, the honest move is to make the right thing the thing that happens by itself.

A cap is a wall you build in front of your most productive engineers.

Whatever model your gateway falls back to, the tools it loads without being asked, the context it carries by habit: each is a decision multiplied across your entire org. Set them well and the bill bends without anyone noticing. Set them carelessly and no cap will save you.

Route by the work, not by the request

Once the default is cheap, the next dollar lives in routing: matching the model and the effort to the task instead of binding every request to one tier. Armstrong describes preprocessing the prompt and sending planning to a frontier model and execution to a cheaper one, because the expensive model is overkill for the second job. There’s a case for reaching for a less capable model on purpose even before cost enters the picture, and at fleet scale the cost argument arrives on top of it.

Underneath routing is one pattern: default-deny, escalate on signal. Pick the cheap, bounded option by default, and let a request earn the expensive one. Its cleanest version measures instead of guesses.

Consider the argument for pulling the million-token context variants of your models entirely: a long session left alone fills that window and re-bills it every turn, so capping the window caps the damage. The instinct is right and the remedy is too blunt. In an agentic loop the whole context is re-sent as input on every turn, so context size is a per-turn tax, and an unmanaged session bills its own history again and again as it grows. A bigger window doesn’t change price per token, it raises the ceiling that tax can climb to.

A million-token window only costs you if the context actually grows into it, and you pay only for the tokens you send. Delete the big window and you also delete the single-pass jobs where it’s the cheapest option, like reading a 600k-token corpus once and answering. A better rule keeps the big window available and gates it: default every request to the bounded model, measure the assembled input, and escalate only on overflow or a real large-context signal, at any turn, decided by code. Tight default, automatic escalation, same shape as the model tier applied to the window.

Humans shouldn’t be choosing models, and they shouldn’t be choosing context windows either.

Caching as a prefix discipline

Caching is where the defaults pay off fastest, and where they quietly fail. A model cache matches on the prompt prefix, so the stable front of your prompt (system instructions, tool definitions, retrieved context) gets paid for once and reused at a fraction of the price. Anthropic’s prompt caching reads cached tokens at roughly a tenth of the base input rate. Armstrong’s team moved their cache hit rate from 5% to 60%, which is most of where a halved bill comes from.

One moving token at the top breaks it. A timestamp, a session ID, a reordered tool list: anything dynamic above the cache breakpoint invalidates everything after it and drops the call back to full price. I walked through the per-call version of this in The token leaks your dashboard doesn’t show; the fleet version is the same rule enforced once, in the default prompt assembly, instead of hoped for across a hundred call sites.

Caching also collides with the context strategy from the last section. Compaction rewrites the conversation to shrink it, which changes the prefix and throws away the warm cache, so the turn after a compaction pays a full write again. Compact every turn and you re-pay the cache every turn while also burning tokens on the summary itself, which is the worst of both. Compact rarely and decisively, keep the post-compaction prefix stable, and let the cache amortize in between.

Lean means signal, not less

Leanness isn’t fewer tokens used, it’s fewer tokens wasted, and the difference is whether what you cut was noise or signal. Dropping the eleven irrelevant repos and the nine unused tool integrations is pure upside, because it’s cheaper and the model reasons better when the relevant detail isn’t buried under context it didn’t need. Dropping context the task actually required buys you a hallucination instead.

At fleet scale the largest invisible tax is tool definitions. Every integration you connect injects its full schema into the prompt on every call, used or not, and it sits in exactly the part of the prompt you wanted to cache. Connect a dozen by default and every engineer pays for a dozen on every message. Here the fix is a default again: ship a lean tool set, and load the rest on demand, names first and full schemas only when a tool is actually called.

Guarding against cutting too far isn’t carrying everything just in case, which is both expensive and noisy. It’s retrieval: keep the default tight and make more reachable when the work calls for it. Knowing which context is the right context is a fact about the work, not about the model.

Visibility is a requirement

Every lever so far needs a number to tune, which makes visibility the precondition rather than a peer of the others. You can’t pick a default model without the model-mix breakdown, justify a routing rule without the escalation rate, fix a cache without the read-versus-write ratio, or trim context without per-call token attribution. Armstrong’s team got the cache win after they could see the hit rate, not before.

Visibility also turns out to be the humane alternative to the cap you were about to ship. Keep usage uncapped but visible, tie spend to expected impact, and the distribution self-regulates without friction: spend what the work needs, but everyone can see it, and high spend should track high output. Proving the levers work doesn’t require measuring anyone’s output. It’s two lines on one chart heading opposite directions, blended cost per token falling while total usage climbs, which is exactly what Armstrong reported and the only scoreboard that survives a budget review.

Output per token is the real prize and also the hard one, because attributing engineering impact to spend is a quarters-long problem. Don’t gate the cost win on solving it. The cheap proxy, cost per token down while usage is up, is available now and carries the argument on its own.

You don’t prevent hallucination by hoarding context, you prevent it by making the right context reachable.

The layer that understands the work

Read the five levers back to back and one requirement runs under all of them: you cannot set a sane default, route by task, place a cache breakpoint, trim context, or read the numbers without knowing, concretely, what the work is. Levie’s argument is that none of these practices happen without a non-abstract understanding of the underlying work, and that the value lives in a layer between the work and the raw intelligence, one that understands your workflows and context well enough to make those calls.

A bare proxy in front of the models saves nothing; it only forwards calls. Savings come from the layer knowing that planning needs the big model and execution doesn’t, that this team’s nine repos are signal and the other org’s are noise, that this prefix is stable enough to cache. Strip out the domain knowledge and the same layer either over-stuffs every prompt or starves it, expensive or wrong, take your pick.

Which is why the cheap framing of this work, “turn on caching, set a budget,” undersells it. A team that can build this layer is doing applied AI for its own org: evaluating which model fits which job, scoping context to the actual task, tuning the defaults to how people actually work. Levie calls the output more intelligence per dollar, and the orgs that get there won’t be the ones with the strictest caps. They’ll be the ones whose layer understands the work.

I’ve built both halves of this. A deploy-support system I built routes each investigation to the right model, verifies failures against live evidence before spending a frontier call on them, and skips a redundant second call on most investigations, which took cost per investigation down 44 to 57%. None of those wins came off a pricing page. They came from knowing what a deploy investigation actually needs.