18 Times the Cost, the Same Backlog: Why AI Model Rightsizing Matters

Recently, a friend of mine asked me to help him out with some architecture-testing libraries he is working on as part of ArchUnit. The idea is satisfyingly concrete: express architectural decisions as fluent assertions, then run them alongside ordinary tests. Boundaries, cycles, dependencies, all the things our diagrams promise the code will definitely respect forever. ;)

It also gave me a convenient excuse to explore tools I do not use every day, particularly OpenCode, and see what would happen if I pointed an unattended agent harness at a real engineering backlog.

I am lucky enough to work for a company that encourages us to stay at the cutting edge of frontier models. That is a genuine privilege. It can also become a vice. When the biggest and cleverest model is always within reach, using it for everything starts to feel less like a decision and more like the default setting.

At AWS, Leadership Principles guide how decisions are approached, and one of the most relevant here is Frugality:

Accomplish more with less. Constraints breed resourcefulness, self-sufficiency, and invention. There are no extra points for growing headcount, budget size, or fixed expense. (Amazon Leadership Principles)

It is easy to discuss frontier models as if everyone has the same freedom to choose them. Many companies do not. They work with real financial constraints every day, and those constraints do not disappear just because the model at the top of a leaderboard is especially shiny. Constant access to frontier models can give those of us at the cutting edge a blind spot: we start optimizing for maximum capability before asking how much capability the task actually needs.

We humans have a wonderfully simple instinct when choosing AI models: find the biggest, cleverest one available and point it at everything. Architecture review? Biggest model. Rename a field? Biggest model. Reformat a table? You guessed it, summon the expensive oracle.

I understand the appeal. If you can speak to God, why chat to Einstein? Nobody wants to explain that the production incident happened because they tried to save twelve cents on tokens.

But there is another question hiding underneath all of this, and it became the question I really wanted to answer:

What is the least expensive model that can reliably clear the quality bar for the work in front of us?

That is model rightsizing. It sounds like something invented by a committee with a spreadsheet, but it becomes rather interesting once you let two very different models loose on the same real backlog.

ArchUnitDev gave me a way to try it. It is an unattended software-engineering harness that works through GitHub issues one by one. The same C# architecture-testing library was built twice from the same starting commit and the same 46-issue backlog. One run used DeepSeek V4 Flash. The other used Claude Opus 5.5 on Amazon Bedrock. Each model had to do the whole job: implement the feature, review the code, criticize the tests, check the idioms, and fix whatever its various personalities complained about.

Both builds landed 45 of 46 issues. Issue 44, publishing to NuGet, was held back because it needed credentials. Even agents should not be given publishing credentials simply because they asked nicely.

The interesting part is what the two journeys cost.

The Numbers, Because Eventually Someone Opens the Spreadsheet

Metric	DeepSeek V4 Flash	Claude Opus 5.5
Issues landed	45 of 46	45 of 46
Model spend	$19.86	$346.70
Model and estimated infrastructure	$41.34	$359.98
Model spend per landed issue	$0.44	$7.70
Issues landed on the first attempt	40	43
Issues abandoned at least once	5	2
Model turns	7,183	4,033
Output tokens	9.74 million	4.40 million

The benchmark results at a glance. Both models landed 45 of 46 issues, with a 17.5 times difference in model spend.

Cumulative model spend by issue. Both runs landed 45 issues; issue 44 was deliberately held back.

Yes, you read that correctly. Both models reached the same backlog completion result, but Opus cost 17.5 times as much in model spend.

Before anybody runs off to replace every frontier model with DeepSeek and buys a small yacht with the savings, there is more going on here. Opus landed three more issues on the first attempt, abandoned fewer issues, used 44% fewer turns, and produced 55% fewer output tokens. It was more direct, needed less correction, and finished in a much shorter operating window.

So the frontier model was better behaved. It simply did not finish more of the backlog.

For this particular workload, the premium bought efficiency and reliability rather than a different final outcome. That distinction is the entire article.

The Opus premium bought a smoother journey, not a different destination.

Rightsizing Does Not Mean Being Cheap

Let us get one thing out of the way: model rightsizing is not selecting the cheapest model and hoping that positive thinking will carry it through production.

Rightsizing means matching the model to the task, surrounding it with suitable controls, watching what happens, and escalating when the evidence says the first choice is out of its depth. Think of it less like finding one champion model and more like assembling a mildly eccentric toolbox. Sometimes you need the huge power drill. Sometimes you need the little screwdriver that costs almost nothing and does the job perfectly well.

ArchUnitDev was unusually well suited to this approach. Every issue was bounded. The repository contained explicit architecture and style rules. A deterministic gate compiled the code and ran formatting, tests, vulnerability checks, and cross-platform checks before a model reviewer was allowed to have an opinion. Three read-only critics then looked at correctness, test strength, and idiomatic conformity. Invalid verdicts failed closed. Findings went back into repair rounds, and work that did not converge was parked instead of being quietly waved through.

In other words, the model was not wandering through the codebase alone at night with a flashlight and good intentions.

The harness turned mistakes into visible, recoverable events. DeepSeek could afford to take more turns because every turn was inexpensive, while the surrounding machinery stopped an inexpensive mistake from becoming an inexpensive production incident.

This leads to the important bit: the model capability you need depends partly on the strength of the system around the model. A frontier model can compensate for vague instructions and weak verification, but using intelligence as a substitute for engineering discipline is quite an expensive hobby.

Chinese Open-Weight Models Are No Longer Sitting at the Children's Table

DeepSeek is operated by Hangzhou DeepSeek Artificial Intelligence Co., Ltd., which identifies its registered address as being in China. Its official V4 release describes DeepSeek V4 Flash as a Mixture-of-Experts model with 284 billion total parameters and 13 billion active parameters. So, despite being the cheaper option here, it is not exactly a small woodland creature.

The model is positioned for fast and economical agentic workloads, and its weights are published under the MIT license. DeepSeek also publishes its own benchmark results, although vendor benchmarks should always be consumed with the same healthy skepticism as a restaurant describing its own lasagna as world famous.

The ArchUnitDev result is more useful because it asks a narrower and less glamorous question: can this model complete a long, dependency-ordered software build inside a fairly demanding agent harness?

In this experiment, yes, it can.

That does not make DeepSeek interchangeable with every proprietary frontier model. It does mean Chinese open-weight models deserve to be evaluated as serious production candidates rather than dismissed as interesting science projects. Where tasks are testable, failures are contained, data-handling requirements are understood, and inference happens often enough for cost to matter, they may be more than good enough. And whether we like it or not, they are going to get better.

"Good enough" sounds underwhelming until it delivers the same backlog outcome for a seventeenth of the model spend. There is an inflection point somewhere between 18 times the cost and double the cost where these models start to become very appealing.

So What Did the Extra Money Buy?

It would be easy, and wrong, to conclude that Opus brought nothing to the party except a more impressive bar tab. It is a monster of a model and frontier for a reason.

Opus needed less correction and fewer attempts. That matters when the task is ambiguous, the specification is thin, mistakes carry real consequences, or no deterministic test can tell you whether the answer is nonsense. In those situations, fewer retries and stronger first-pass performance can be worth considerably more than the token bill suggests.

The cost ratio was also not remotely consistent. Across the 45 completed issues, Opus cost between 3.2 and 70.1 times as much as DeepSeek, with a median ratio of 18.1. Extra review rounds explain much of that spread. Every additional round starts another context-heavy model session, and expensive context remains expensive even when the agent is using it to rediscover something it forgot five minutes ago.

Prompt caching adds another wrinkle. Amazon Bedrock documentation notes that cache reads receive a discounted rate, while cache writes may cost more than normal input. In the Opus run, cache writes were the largest measured cost component at $165.64, or 47.8% of model spend. Output cost $96.81 and cache reads cost $84.22.

Nearly half the Opus model bill came from writing context into caches. If your model-routing spreadsheet looks only at the headline input and output price you may have carefully optimized the smaller half of the problem.

A Routing Strategy for People Who Enjoy Sleeping at Night

You do not need a grand artificial-intelligence parliament to apply this. A practical escalation ladder works rather well:

Start economical, verify deterministically, and escalate when the evidence asks you to.

Classify the task. Consider ambiguity, business impact, context size, testability, and reversibility. Renaming an internal field is not the same thing as rewriting an authorization boundary, even if both arrive as innocent-looking tickets.
Start with the least expensive qualified model. Use economical models for bounded implementation, extraction, transformation, and other work where automated checks can catch mistakes.
Escalate when the evidence asks you to. Repeated gate failures, contradictory critic findings, stalled progress, or a high-risk decision are good reasons to call in the expensive oracle.
Keep deterministic checks independent. Compilation, tests, schemas, policy checks, and security controls should not depend on the same model declaring itself brilliant.
Measure the whole workflow. Track accepted outcomes, retries, elapsed time, human intervention, cache behavior, and infrastructure. Price per token is interesting, but cost per useful result pays the bill.
Recalibrate regularly. Models and prices change at a slightly absurd pace. Today's premium model may be tomorrow's sensible default, and today's benchmark champion may be forgotten by next quarter.

The goal is not to keep frontier models locked in a glass cabinet for special occasions. It is to spend their capability where it changes the result.

Before We Get Carried Away

Now for the caveats, because every benchmark eventually reaches the section where the author admits reality is complicated.

First, each model reviewed its own work. A landed issue means its own critics were satisfied and the deterministic gate passed. It does not mean an independent judge applied exactly the same semantic quality standard to both repositories. Asking a model whether its own work is excellent has certain annual performance review vibes.

Second, both final test suites were green, but they were structured differently. DeepSeek produced 1,852 tests and Opus produced 1,534. More tests do not automatically mean better tests. You can write ten excellent tests, or two thousand tests that enthusiastically confirm the code does whatever the code currently does. Mutation testing or an independent hidden suite would tell us much more.

Third, the runs were not operationally identical. They happened about a month apart, used different regions and instance types, and the first seven DeepSeek issues ran on an earlier harness revision. Some early DeepSeek abandonments may therefore belong partly to the harness rather than the model. Software blaming AI for its own childhood mistakes would be a little unfair.

Fourth, DeepSeek's $19.86 came from provider pricing reported through OpenCode and did not have the same provider-side cross-check available for Opus. The Opus token counts were compared with Amazon CloudWatch and came within roughly 3% to 5%.

These limitations stop us from claiming universal quality equivalence. They do not make the main observation disappear: inside this controlled, fail-closed harness, a much less expensive open-weight model completed the same issue backlog.

The Answer Is a Portfolio, Not a Winner

Model conversations have a habit of turning into horse races. Somebody produces a leaderboard, everybody picks a favorite, and a week later a new model arrives to knock over the refreshments.

Production systems need something less dramatic: a portfolio. Some tasks deserve the strongest model available. Others need a model that is strong enough because deterministic software, explicit constraints, and escalation policies provide the rest of the assurance.

The ArchUnitDev benchmark suggests a practical default. Start with the least expensive model that has demonstrated success on the actual workload. Watch the correction signals. Escalate deliberately. In this case, DeepSeek required more turns and a little more patience, but it still reached the target for 17.5 times less model spend.

This is not an argument against frontier models. It is an argument against paying frontier prices before proving that the workload needs one.

So the next time somebody reaches automatically for the largest model in the catalog to rename a field, perhaps gently move their hand away from the expensive oracle. There is probably a perfectly good screwdriver nearby.

Sources

Benchmark evidence

Model and platform sources

Last verified. Benchmark reports and linked official sources checked on 2026-09-26. The quantitative comparison is specific to this workload and harness configuration.

The views expressed here are my own and do not necessarily reflect the views of Amazon Web Services (AWS) or Amazon.com.