Case study · Creator affiliate agent
A trust filter for agent-written product copy
An agent matches a creator's post to products in a merchant's catalog and drafts the affiliate captions. A trust filter sits between what the model writes and what gets published, and only lets through statements it can trace to the merchant's own records. This page describes how the filter works and how it is graded.
Planted errors
24 kinds
of known error, planted in captions the agent really wrote. Some are built to get through
True captions
20
written to trip naive filters, so blocking good copy is measured too
Creator posts
40
in three markets, labeled before the agent existed. Six ask for something the catalog cannot serve
A wrong detail reads exactly like a right one
Affiliate captions are advertising. When an agent writes "100% cashmere" about a blend, quotes last season's price, or promises a jacket is waterproof, that is a false product claim published under a creator's name about a merchant's product.
Language models make this worse in a specific way. An invented detail is as fluent as a correct one, so nothing on the page looks wrong, and review by eye does not catch it reliably.
Asking the model to be careful does not close the gap, because the model cannot tell which of its fluent sentences are grounded. Requiring it to cite each claim does, because a citation can be checked by a program.
Traceability, not truth
The filter does not know whether a dress is really linen. It knows whether the caption matches the merchant's record. The guarantee is that every factual statement traces to the party accountable for it.
That still works where ground truth is impossible. The question stops being "is this true?" and becomes "who stands behind this claim, and can we show it?"
Four layers, each catching what the one before cannot
Code checks
The agent lists every factual statement as a claim pointing at a catalog field. Code verifies each one in the market's own terms: euro prices for Spain, EU sizes, "lino" rather than "linen". Uncited wording is verified too. If it contradicts the record, or cannot be checked, it fails. Performance, sustainability, health, and temperature claims fail unless the record makes them. Disclosure is enforced per market, including where in the caption it appears.
Model audit
A model reads each caption against the record and lists product facts the record does not support. It exists for what code cannot catch: "keeps you dry in any downpour" makes a water-resistance claim with none of the words a rule could match. Its output is checked too. Any quote it flags must appear verbatim in the caption, or it is discarded.
Every statement points at a field, or it does not pass
#publi Mi vestido para la boda de mi prima: cruzado, de 100% lino, 145 €, tallas 32 a 46. Totalmente impermeable.
Plant errors with a known answer, and count both kinds of mistake
A filter can fail in two directions: it lets a false statement through, or it blocks a true one. The grading measures both, for each layer separately, because the layers are trusted differently.
The trust filter
- Caught: a planted error produces at least one blocking finding. Starting captions must pass every check first, so the finding can only come from the planted error.
- False block: a true caption produces a blocking finding. Warnings are counted separately, since the reviewer only sees them.
- Model audit: graded on what the code missed. A flag only counts if its quote appears verbatim in the caption. Flags on clean or true captions are false alarms.
- Real defects: every caption the agent actually got wrong is kept, and must fail.
The agent
- Each post is labeled with acceptable products, specifically wrong ones, or "nothing fits".
- Precision: the share of picks that were acceptable. Hit: at least one acceptable pick.
- Forbidden picks are counted apart from misses, because recommending something sold out, over budget, or the wrong fiber is worse than a weak pick.
- No-match is only correct with zero picks. Offering a blazer for a tuxedo brief is wrong, however close.
- Check pass rates are recorded on first submission and after revisions, to separate what the agent gets right from what the filter makes it fix.
| Kind | Planted error | Designed to |
|---|---|---|
| Cited lie | Cited price changed to a different amount | be caught |
| Cited lie | Cited price shown in the other market's currency | be caught |
| Cited lie | Cited material replaced with a fiber the product does not contain | be caught |
| Cited lie | A blend cited as 100% of its main fiber | be caught |
| Cited lie | Cited size range extended past what is stocked | be caught |
| Cited lie | EU market caption cites US sizes | be caught |
| Cited lie | Cited color the product does not come in | be caught |
| Uncited | A wrong price added with no claim | be caught |
| Uncited | A fiber the product does not contain, mentioned with no claim | be caught |
| Uncited | Scarcity language on an item that is in stock | be caught |
| Uncited | A blend called pure, with no claim | be caught |
| Uncited | Claims it costs under an amount it does not | be caught |
| Uncited | A size that is not stocked, with no claim | be caught |
| Uncited | Wrong price written out in words | test the edge of pattern matching |
| Disclosure | Affiliate disclosure deleted | be caught |
| Disclosure | Disclosure moved to the end, after the caption truncates | be caught |
| Unsupported | Waterproof claim the catalog does not make | be caught |
| Unsupported | Sustainability claim the catalog does not make | be caught |
| Unsupported | Specific warmth rating the catalog does not make | be caught |
| Unsupported | Wrinkle-free claim the catalog does not make | be caught |
| Unsupported | Health claim the catalog does not make | be caught |
| Unsupported | Water resistance paraphrased, no keyword | get past the code |
| Unsupported | Sustainability paraphrased, no keyword | get past the code |
| Style | Em-dash inserted | be caught |
The paraphrase rows exist to fail. "Keeps you dry in any downpour" makes a water-resistance claim with none of the words a rule could match. A test set where every error is catchable would only show that the author's blind spots match the checks.
A number measured on the data you tuned against is a regression test, not a result
The checks, the agent's prompt, and one eval label were all adjusted while looking at these same test sets. Every miss that surfaced was fixed. The planted errors were written by the same person who wrote the checks. So the scores these sets produce mostly show coverage of what was anticipated. They are useful for catching regressions, and they say little about captions nobody has seen yet.
A fair measure would need:
- Errors planted by someone who has not read the checks.
- Fresh captions from a different run, model, or real creators, never used during development.
- Labels written by someone other than the builder.
- Harder and messier posts. The current agent eval is small, and the agent now gets nearly every case right, which means it can no longer tell a good agent from a better one.
- A held-out split that stays unseen until the end.
Five findings from building it
The first bottleneck was not the model
The agent's early misses traced to the search tool. It only understood English, could not search by material, and let one unmatched keyword empty a filtered search. Fixing search removed most of the agent's misses. Reading the agent's traces found that. Prompt tuning would not have.
Strictness has a cost
The first version failed any attribute word without a claim. It blocked true captions ("the Cashmere Crewneck"), and many of the agent's first drafts needed a revision round over nothing. Verifying uncited wording against the record, failing on contradiction and warning on agreement, removed both false blocks without letting anything false through. "Pure wool" on a 70% wool coat still fails, because "pure" is checked as a claim of its own.
A trust layer can make an agent unhelpfully cautious
Once the rules tightened, the agent started declining a sneaker brief ("I need to walk 20k steps") because the record makes no comfort claim. It happened repeatedly. An unverifiable hope in the post is a reason to leave the claim out of the caption, not to withhold the product. One prompt line separating recommending from claiming fixed it, and the regression it could have caused, forced matches on unservable briefs, did not appear.
Code verifies facts, not coherence
A handful of captions came back garbled, all German, all at a non-ASCII character: "heißt" became "hei\resultat", and "für" became "f\ti\ber". Two contained the model correcting itself ("Nein, korrigiert:"). Some passed every code check and entered the clean fixtures. The model audit surfaced them. A cheap code check for malformed output now catches them.
Word lists cannot catch paraphrases, and models should not have the last word
"Keeps you dry in any downpour" has none of the words a rule could match. The model audit catches most of these, and also flags clean captions, often on debatable points ("elegant drape", "linen wrinkles"). The layering follows from that: code decides what it can verify, the model points at what it cannot, and a person makes the call.
Limits
- The guarantee is consistency with the record. A wrong record produces wrong copy that passes.
- Unsupported claim detection is word lists per language. Coverage is what the lists contain.
- The model audit varies between runs, and its flags need a human to judge.
- One eval label was corrected after seeing agent output: es-03 asks for a red dress, the catalog has none, and the original label accepted near misses. It now expects no match, like the rest of the set.
- Disclosure rules are the configured requirement per market, written conservatively. They are not legal advice.
- The catalog is hand-written for a demo store. No real merchants, creators, or sales are behind any number here.
Reproduce
These reproduce the grading. Read their output as regression checks on the tested sets.
npm test # 73 tests
npm run trust-eval # code checks, free
npm run trust-eval -- --audit # adds model audit, ~$0.07
npm run eval -- --yes --model claude-haiku-5-5 # agent, ~$0.08