Every token in a system prompt is paid for on every request, and an agent instruction file is read again on every turn. A prompt that is 20% padding costs 20% more forever, and takes up context the model could have used for the actual task.
So prompt compression is worth doing. The hard part is not making a prompt shorter. It is making it shorter without removing the one word that mattered.
Two ways to compress a prompt
Model-based compression uses a language model to decide what to drop. The best-known example is Microsoft's LLMLingua, which uses a small model to find and remove non-essential tokens and reports up to 20x compression with minimal performance loss. Its successors, LongLLMLingua and LLMLingua-2, target long RAG contexts and speed. For a long retrieved document, where most tokens really are low-value, this is a strong approach.
Rule-based compression applies fixed rewrites: drop filler, shorten verbose phrases, remove repeats. It compresses far less. What it gives you in return is that the same input always produces the same output, and you can read every rule that fired.
For instructions, rather than retrieved context, we think the second kind is the right default. A system prompt is not mostly filler. It is mostly rules, and the words that carry a rule are exactly the small, high-frequency words a token-level model is most tempted to drop: not, only, unless, never. A compressed document that loses one sentence is slightly worse. A compressed instruction that loses one not does the opposite of what you wrote.
Where the wasted tokens actually are
Look at a typical prompt and the padding is not evenly spread. It clusters in a few places:
- Politeness and hedges. "Please", "try to", "make sure that". The model does not need to be asked nicely.
- Second-person scaffolding. "You need to classify…" carries the same instruction as "Classify…".
- Intensifiers. "Very carefully", "really important". If everything is important, nothing is.
- Verbose phrases. "In order to", "the following", "at this point in time".
- Articles. Mostly removable in imperative instructions, at the cost of prose that reads like a telegram.
- Repeats. The same rule pasted twice, usually by two people editing the file months apart.
- Whitespace and table padding. Aligned Markdown tables are mostly spaces.
None of these carry an instruction. All of them cost tokens.
What must never be touched
The other half of the job is a list of things compression is not allowed to change, however long they are:
- Code blocks and inline code
- Front matter and headings
- Quoted text, which is often an exact string the model must reproduce
- URLs, file paths, and command-line flags
- Template placeholders like
{{customer_name}} - Numbers and identifiers
- Negations and limiters: not, never, only, unless, must
The last item is the important one. An edit next to a negation is the most likely place for a shorter sentence to mean something different, so a safe compressor skips any edit that would drop one, rather than trying to be clever about it.
A real before and after
Here is a short support-triage prompt, written the way prompts usually get written:
You are a helpful assistant. Please make sure that you always read the entire ticket very carefully before you respond.
You need to classify each ticket into exactly one of the following categories: `billing`, `bug`, `account`, `other`.
It is really important that you never include the customer's email address in your reply. If the ticket mentions a refund over $500, escalate it to a human and do not reply.
Please try to keep your answer short and to the point, ideally under 120 words.
Please make sure that you always read the entire ticket very carefully before you respond.
And here is what Rubrkit's compaction returns for it, unedited:
You are helpful assistant. Ensure you always read entire ticket before you respond.
Classify each ticket into exactly one of following categories: `billing`, `bug`, `account`, `other`.
It is important that you never include customer's email address in your reply. If ticket mentions refund over $500, escalate it to human and do not reply.
Try to keep your answer short and to point, ideally under 120 words.
Ensure you always read entire ticket before you respond.
That is 586 characters down to 470, about 20% fewer and roughly 29 tokens, with nothing changed that carries a rule. Never, do not, exactly one, $500, the four category names, and the 120-word limit are all intact. What went: every "please", "make sure that", "you need to", "very carefully", "really", and nine articles.
Two things are worth noticing. First, the output reads worse to a person. That is the trade: you are writing for a model that has to read this on every request, not for a colleague reading it once.
Second, the repeated rule survived. Rule-based deduplication removes exact duplicate lines, and duplicate sentences within the same paragraph. Here the first copy shares a paragraph with "You are helpful assistant", so the two paragraphs are not identical and both stay. Merging rules that say the same thing in different places takes judgement, which is where a model earns its place.
When to let a model help
Rubrkit's compaction has two levels. Standard is the rule-based pass shown above. It is deterministic, running it twice changes nothing more, and it is free in the workspace, the API, and the MCP server. Deep runs Standard first, then uses a model to merge repeated rules and tighten wordy prose, with code, quotes, headings, tables, and links locked.
The model pass is where meaning can slip, so Deep does not trust it. A second model checks every instruction in the result against the original. If anything would be lost, nothing changes and the credits are refunded. Either way, you see the diff before anything is saved, and in the workspace compaction saves a new version, so you can restore the previous one at any time.
How much either level saves depends entirely on how padded the prompt was. Prompts full of politeness, hedges, and repeats often shrink 20–40%. Instructions that were already tight save a few percent, mostly whitespace.
Compression is not the same as caching
Provider-side prompt caching makes a repeated prefix cheaper to send. It does not make it shorter: the model still reads every token, and the prompt still takes the same share of the context window. The two stack. Compress the prompt, then cache the compressed version.
Check that it still works
A shorter prompt that behaves differently is not a saving. Even with every guard in place, treat compression like any other change to an instruction and run it against your golden set before and after. If you do not have one yet, how to test a prompt covers building it from real failures. It is the same discipline that catches a teammate's well-meant edit, and it catches a compressor's too.
Cut the padding, keep every rule, and check the result. That is all prompt compression should be.