Prompt stuffing vs RAG vs jitprompt: 23 business rules, 300K tokens

Most agents get their instructions one of two ways: the whole manual in the system prompt, or RAG over it. Prompt caching made the first one cheap enough that many teams stopped questioning it. I wanted numbers on both accuracy and cost, so I built a benchmark around the kind of instructions operations teams actually write, and ran the same agent through it three ways.

Results at 300K tokens:

	Full prompt	RAG	jitprompt
Rules followed	50%	67%	85%
Cost per conversation	$0.148	$0.0093	$0.0086
Tokens read per conversation	2.35M	165K	160K
Cost per rule followed	$0.297	$0.014	$0.010

Setup

Benchmark. τ²-bench banking_knowledge: an agent plays a bank’s support rep, with tools that change account state, a simulated customer, and a 698-document knowledge base. Tools, user simulator and grader are unchanged.

Planted rules. I added 23 conditional business rules, written in the knowledge base’s own style, across six types:

Type	Example
Threshold	Late-fee reversals on the Bronze card are capped at $25.
Prohibition	Never close a card with a pending replacement order; transfer instead.
Exception	A retention offer in one document, its eligibility override in another.
Superseded	An older policy, and a newer dated update that replaces it.
Look-alike products	The same rule for Gold and Platinum, with different values.
Ordering	Check the customer’s dispute history before applying a dispute credit.

Each rule has two tasks. In the triggered task the rule changes the correct action. In the control task one fact differs, so applying the rule is wrong. Adherence is scored deterministically from the agent’s tool calls: arguments, amounts, order. No LLM judge.

Corpus sizes.

Size	Documents	Prompt tokens
Small	49 (every document the tasks need)	29,848
Large	716 (the full knowledge base, plus the rule documents)	303,500

Arms. Same agent (Claude Haiku 5.5, thinking off), same base prompt, same tools, same simulated customer (Claude Sonnet 5.5). Prompt caching on everywhere. Only the delivery of knowledge differs:

• Full prompt: the whole knowledge base as a cached system-prompt prefix.
• RAG: τ²-bench’s BM25 retriever (top-10 documents) plus grep.
• jitprompt: the same documents, ingested as-is, served over MCP. The agent keeps its core procedures in view, reads a table of contents, and searches by its plan; results are whole sections.

46 tasks × 2 trials per cell: 92 conversations per arm and size.

Accuracy: the full prompt degrades, jitprompt doesn’t

On the small knowledge base the three arms are level. At 10× the size, the full prompt drops 30 points, even though every rule is in its context on every turn. On paired (task, trial) runs at 300K tokens:

Comparison	Wins – losses	Exact McNemar p
jitprompt vs full prompt	17 – 1	< 0.001
jitprompt vs RAG	10 – 2	0.039
Full prompt, small vs large	16 – 2	0.001

For reference, an agent handed exactly the right documents for each task follows 98% of the rules, and one with the rule text removed follows 4%. The rules are clear, and they change behaviour.

By rule type, at 300K tokens:

Rule type	Full prompt	RAG	jitprompt
Exception (override in another doc)	0%	62%	75%
Superseded (newer policy wins)	38%	38%	88%
Ordering (check X before Y)	25%	50%	75%
Threshold	75%	75%	100%
Look-alike products	75%	88%	75%
Prohibition	100%	100%	100%

The rules that break under stuffing are the ones that need the agent to connect two pieces of text, and those are most of what an operations manual is.

Cost: where the tokens go

An agent conversation is not one model call. Here a conversation averaged about eight, and with a full prompt every call re-reads the whole knowledge base:

tokens per conversation ≈ prompt size × model calls
full prompt, large KB:  303,500 × ~7.7 ≈ 2.35M tokens
jitprompt,   large KB:  small prompt + ~2 retrievals ≈ 160K tokens

Caching makes those re-reads cheaper per token (98% of the full prompt’s input was cache reads) but doesn’t make them go away. And past 100K tokens the prompt is billed at the long-context rate, which is why cost grew faster than the knowledge base: 10× the documents, 15× the cost.

Retrieval keeps cost flat: jitprompt cost $0.0090 per conversation on the small knowledge base and $0.0086 on the large one. Divide cost by the share of rules followed and the gap widens, because the full prompt pays more and gets less right:

Full numbers:

Size	Arm	Rules followed	$/conversation	Tokens read	Retrieval calls
Small	Full prompt	80%	$0.0098	291K	—
Small	RAG	80%	$0.0092	144K	1.6
Small	jitprompt	85%	$0.0090	155K	2.5
Large	Full prompt	50%	$0.1484	2,346K	—
Large	RAG	67%	$0.0093	165K	2.1
Large	jitprompt	85%	$0.0086	160K	2.2

How jitprompt serves instructions

You upload documents as they are. jitprompt splits them into sections with a title and a one-line summary, and the agent gets three MCP tools:

get_toc(document_id?)      → map of every document and section
search_instructions(plan)  → whole sections matching the plan
get_section(section_id)    → one section in full

The agent searches with its plan (“refunding a late fee for a Bronze card customer”), not the customer’s words, and a small core of procedures stays in view from the first turn, so it doesn’t depend on knowing what to ask for. A typical map:

## Card Fees and Credits  [doc 3f9a1c20]
How to handle fee reversals, credits and caps by card.
- [e812d01e] Late-fee reversals — caps by card, eligibility
- [9b04c7aa] Annual-fee waivers — retention offers, exceptions
- [1d7e5f32] Statement credits — reasons, approval order

Any MCP client connects with one URL:

claude mcp add --transport http jitprompt https://app.jitprompt.com/mcp \
  --header "Authorization: Bearer imcp_..."

Takeaways

• Caching fixes the price of a token, not the cost of re-reading. A conversation re-reads the system prompt on every call.
• A stuffed prompt degrades as it grows. Here, 80% → 50% of rules followed at 10× the size.
• Retrieval keeps cost flat; retrieval driven by the agent’s plan keeps accuracy too. 85% at 300K tokens, for under a cent per conversation.

The free tier is enough to try this on your own rules file. I’d like to hear how you’re handling instruction bloat: support@jitprompt.com.