Open-weight models got good, and they got cheap. DeepSeek V4 Flash costs a fraction of a frontier model, takes a long context, and caches the prompt automatically. If you run agents on a budget, the question writes itself: if the model can read the whole operations manual on every turn and still follow it, why bother with anything else?
I wanted numbers on that, so I ran my instruction-following benchmark on DeepSeek V4 Flash, with the knowledge base growing from 30K to 188K tokens. I wrote down the design and the tests before spending a cent.
The model earned its reputation: a stuffed prompt followed the rules as well at 188K tokens as at 30K. jitprompt still made the same agent cheaper, flat as the manual grew, and more careful about when a rule applies, and with a better search plan it matched the stuffed prompt’s accuracy.
Results at 188K tokens, the whole knowledge base:
| Full prompt | RAG | jitprompt | |
|---|---|---|---|
| Rules followed | 92% | 77% | 83% |
| Rule applied where it shouldn’t be | 10% | 3% | 4% |
| Cost per conversation | 0.84¢ | 0.33¢ | 0.32¢ |
| Rules followed, with plan-aware search | — | — | 93% |
The setup
The benchmark is the same as at launch: τ²-bench’s banking knowledge base with 23 business rules planted in it, written in the bank’s own style. Each rule has two tasks: in one the rule changes the right answer, in the other one fact differs and applying the rule is a mistake. Grading comes from the agent’s actual tool calls. No AI judge.
For this study:
- The agent: DeepSeek V4 Flash, thinking off.
- The customer: also DeepSeek V4 Flash, thinking on. Before the main runs I checked that it reached each rule’s decision point at least as often as the launch customer did (100% vs 96%).
- Trials: 5 per task, so 115 paired runs per comparison.
- Sizes: 30K, 60K, 120K and 188K tokens. Every size contains the smaller one plus filler documents, and 188K is the whole knowledge base: the same 716 documents that Claude’s tokenizer counts as 303K.
- Arms: the full knowledge base in the system prompt, BM25 RAG, and jitprompt. Plus one extra cell with the rule documents shuffled to random positions in the prompt.
A capable model handles a long prompt
- The full prompt held up. 91% at 30K, 92% at 188K, and no size scored below the smallest one.
- jitprompt was flat too, at 83–85% across all sizes.
- On accuracy alone, the full prompt was ahead by 4 to 10 points; only the gap at 188K is significant (16 runs to 5, p = 0.03).
- RAG trailed both at 77%.
At launch, the same documents in Claude Haiku 5.5’s prompt took it down to 50% of rules followed. I wouldn’t read that as a head-to-head between the two models, since the customer and the number of trials changed too, but it’s a clear sign that how well a stuffed prompt holds up depends on the model. This one holds up well.
What jitprompt adds
Cost that doesn’t grow with the manual. With a stuffed prompt, every model call re-reads the whole knowledge base:
The full prompt went from 0.28¢ to 0.84¢ per conversation as the knowledge base grew. jitprompt stayed at 0.32¢ at every size, 2.6× less at 188K, and the gap widened at every step up in size. Stuffing stays this cheap at all only because DeepSeek’s cache served 99% of its input: without it, the full knowledge base would cost about 3¢ per model call, and a conversation makes several.
Fewer wrong calls. On the control tasks, where one fact makes the rule wrong, the full prompt applied it anyway 10% of the time, and 17% with the rules shuffled. jitprompt did it 4% of the time. An agent that sees every rule on every turn tends to apply every rule it sees, and part of the full prompt’s accuracy lead comes from exactly that.
No middle of the prompt. Shuffling the rule documents dropped the full prompt from 92% to 83%, and rules that landed in the middle third were followed 72% of the time, against 92–97% at either end. The rules that moved there are also some of the hardest, and the drop isn’t significant after correcting for multiple tests, so treat it as a hint of lost in the middle, not as proof. jitprompt doesn’t have a middle: the agent gets the section it asks for, wherever it sits in the knowledge base.
Closing the accuracy gap
To see where jitprompt’s remaining misses came from, I checked each one against the rule’s own text. Of 77 triggered misses across all sizes:
- 34: the rule text never reached the agent.
- 25: only part of it did, for example the base rule but not its exception or its update.
- 18: the agent had all of it and didn’t apply it.
So most misses were search misses, with the same shape: the agent searched for what it was doing, not for whom. A typical one was an exception for Plus members, where the agent searched for the base policy and left out the customer fact the exception depends on. That pointed to two fixes:
- A plan-aware agent prompt. Put the customer’s facts into every search, search again when a lookup reveals a new one, and search for the exact action before changing an account.
- Linked sections. When a section comes back, jitprompt also returns the exceptions and updates linked to it.
Run the same day at 188K tokens, the new prompt took jitprompt from 85% to 89%, and linked sections took it to 93%: level with the full prompt, at less than half its cost and with fewer wrong calls. The plan-aware prompt is the recommended agent prompt in the docs. Linked sections missed one of my pre-registered guards by a single trial, so they stay off by default until they pass it.
Of course, this study has limits: one model, 23 rules, and intervals about ±10 points wide, so differences under 10 points are not resolved. The fixes are also in-sample: I wrote the prompt after reading these misses and tested it on the same rules, so they need a fresh set of rules before I claim them. The pre-registration, data and code are public, so feel free to poke holes in it.
In a nutshell
- Cheap open-weight models can follow a long prompt. DeepSeek V4 Flash followed 92% of the rules with the whole 188K-token knowledge base in its prompt.
- jitprompt keeps the cost flat as the manual grows: 0.32¢ per conversation at every size, 2.6× less than stuffing at 188K.
- It applies rules more carefully: 4% wrong calls against 10%, and no dependence on where a rule sits.
- With plan-aware search it matched the stuffed prompt’s accuracy on this benchmark, at less than half the cost.
A good model makes stuffing workable; it doesn’t make it the cheapest or the most careful option. Next, I’ll test the fixes on a fresh set of rules, so stay tuned.
Try it free Benchmark harness on GitHub
Are you running agents on open-weight models? I’d love to hear how you’re feeding them their instructions: support@jitprompt.com.