Key facts.
- The Berkeley Function-Calling Leaderboard evaluates whether models pick the right function and fill valid parameters, and includes hallucination categories, because tool-call errors are common and each one is a paid retry the cost model must account for. source
- NoLiMa found long context actively degrades performance, with most of 12 tested models falling below half their short-context baseline at 32k tokens, so carrying more context raises cost and can lower quality. source
- RULER evaluated 17 long-context models and found almost all exhibit large performance drops as context length grows, so a model's advertised context window overstates its effective length, and paying to fill it is a cost-modeling mistake, not a quality gain. source
What do the successful teams actually model?
Cost per successful outcome, early, and they let it shape the design. They estimate the per-run cost including the tool-call retries that BFCL predicts, they account for context cost knowing that NoLiMa shows longer is not better, and they divide by a realistic success rate to get the real per-outcome figure. Then they choose a scope where that figure is comfortably under the value of the outcome, with margin. This is the move the surviving projects make and the failing ones skip: cost is an input that constrains the scope, not a number checked at the end to see if the build they already committed to is affordable.
The design consequences are concrete. Short runs over long ones, because context cost rises and quality falls with length. Tight tool scoping over broad, because every available tool is a chance for a wrong, retried call. And a narrow task over an ambitious one, because a high success rate keeps the per-outcome cost low. None of these is a model choice. They are design choices that follow directly from taking cost seriously up front, and they are what separate the agents that stay in production from the ones that get pulled when the bill arrives.
Why is cost a design input and not a check?
Because by the time you check it, the design is fixed, and the expensive choices, broad scope, long context, many tools, are already baked in. Modeling cost first lets it shape those choices while they are still cheap to change. A team that knows its per-outcome budget designs a scope that fits it. A team that builds first and checks later discovers the scope it chose does not fit any affordable cost, and then faces a rebuild or a cancellation. The successful projects spend the modeling effort before the build, which is the cheapest moment to act on what it tells them.
| Approach | When cost is considered | Outcome |
|---|---|---|
| Cost as design input | Before scope is set | Scope fits the budget; sustains |
| Cost as final check | After build | Expensive choices already baked in |
| Cost ignored until invoice | In production | Surprise overrun; cancellation risk |
The Pattern Intelligence Layer is where cost per outcome, tool-call reliability, and context discipline are modeled and held as design constraints from the start. Those properties are tracked at the pattern level, so the scope you choose is one the economics actually support. Reliability at the pattern level is also the reliability of staying in production, which is the outcome cost modeling is really protecting.
Frequently asked questions
Do the surviving teams just buy a better model?
Survivors model cost per outcome early and design around tool reliability, since BFCL shows hallucinated parameters, and context, since NoLiMa halves scores at 32k. A bigger model won't change that scoping math. (arXiv:2502.05167)
Did the surviving teams just find a cheaper model?
No. They modeled cost per outcome early and scoped the agent to fit it. The model matters less than designing short, reliable, well-scoped runs.
Why design for short context?
Because longer is not free or better. NoLiMa shows most models lose over half their performance by 32k tokens, so long context raises cost and lowers quality at once.
What is the single discipline that matters most?
Modeling cost per outcome before setting scope, so cost shapes the design while the expensive choices are still cheap to change.