Articles

We measured how often LLMs quote part of a redacted value. With realistic stand-ins, it's most of the time.

1,800 Claude replies: how often restore leaves customers with wrong digits, and why the obvious prompt fix trades one problem for another.

Reversible redaction rests on a simple loop: replace sensitive values with stand-ins, let the model work on the stand-ins, then swap the real values back into its reply. The loop has a gap. If the model quotes only part of a stand-in — "your card ending in 1461" — there is nothing whole to swap back. The fragment stays in the reply, and the customer reads digits that were never theirs.

We had seen this in a small test and documented it. This time we measured it properly.

What we ran

1,800 replies: 5 synthetic support scenarios x 3 strategies x 3 Claude models (Haiku 5.5, Sonnet 5.5, Opus 5.5, effort medium) x 2 system prompts x 20 runs; deterministic grading.

The scenarios are ordinary support tasks: confirm a refund, acknowledge a card dispute, confirm a KYC check, arrange a callback, confirm a replacement card. Every ticket uses synthetic data — test card numbers, example.com addresses, invented names.

The three strategies:

  • fpe replaces each value with an encrypted stand-in of the same format: a card becomes a different, Luhn-valid card number.
  • synthetic replaces each value with a realistic fake, including a fake name.
  • tokenize replaces each value with a placeholder such as [CREDIT_CARD_1].

Each scenario ran with a plain system prompt, and again with one added line: "When you mention a card number, email or phone number, repeat it exactly and in full as given. Never abbreviate, mask or quote part of it."

Grading is deterministic, with no model as judge. After restore, a reply counts as wrong if it contains a run of four or more digits, or part of an email address, that came from a stand-in and from no original value. We read a sample of flagged and unflagged replies by hand to check the grader.

Result 1: with realistic stand-ins, wrong digits are the norm

StrategyWrong digits after restore, plain prompt
fpe204/300 (68%)
synthetic173/300 (58%)
tokenize0/300

The typical failure is exactly the one you'd expect from a careful support agent: "We've confirmed the duplicate charge on your card ending in 1461." The model abbreviates out of politeness or caution, and the abbreviation is of a number the customer has never seen.

All three models did it. With the plain prompt, across fpe and synthetic: plain prompt, fpe + synthetic combined: Sonnet 5.5 104/200 (52%), Haiku 5.5 127/200 (64%), Opus 5.5 146/200 (73%). We didn't investigate why the rate differs by model.

tokenize never produced wrong digits: a model can't take the "last four" of [CREDIT_CARD_1].

Result 2: the obvious fix trades one problem for another

The one-line instruction works on its own terms. Wrong digits fell to 3/300 (1%) for fpe and 4/300 (1%) for synthetic.

But look at what the replies say instead. With the instruction, the reply printed the full card or Aadhaar number in 199/300 (66%) fpe replies, 180/300 (60%) synthetic replies and 195/300 (65%) tokenize replies. The models did exactly what they were told: repeat the value in full. After restore, that means a customer email containing their complete card number — something most card-security policies are written to prevent.

tokenize has a version of this even without the instruction: in 123/300 (41%) replies, the model wrote the placeholder where a person would write a masked number ("card ending in [CREDIT_CARD_1]"), and restore filled in the whole card.

So the question isn't only "does restore return the right value?" It is also "should this value be in the output at all?"

Result 3: fake names leak through first names

With synthetic, names are replaced with realistic fake names, and models greet customers by first name. The reply addressed the customer by the stand-in's first name in 241/300 (80%) plain replies and 148/300 (49%) with the instruction. A first name on its own doesn't match the whole stand-in, so it isn't restored: Priya gets an email that opens "Hi Arjun".

fpe and tokenize had none of these (0/600). Their name tokens are opaque, so models either avoided the name or used the whole token, which restores correctly.

What we'd do with this

Tested in this run:

  • Use tokenize for anything a customer will read. It produced no wrong digits in either prompt condition.
  • Don't rely on a "repeat it in full" instruction as a fix for customer-facing text. It removes the wrong digits by putting complete identifiers into the output.

Not tested in this run — ideas we'd try next:

  • Ask the model not to restate identifiers at all, and have your application render any masked form ("ending in 4242") from the original value after restore.
  • Keep fpe for data that machines read — logs, analytics, payloads that must pass validators — where its format preservation is the point and nobody is reading "the last four".

What this changes for us

Our docs used to describe a one-line prompt fix for partial values, based on a 3-run test. With 20 runs per condition, the fix holds for wrong digits but has the side effect above, so we've changed that guidance: the docs now recommend tokenize for anything a person reads. fpe's strengths are unchanged: no mapping table, valid checksums, and an exact restore of every value the model repeats whole. But "the stand-in looks real" is exactly what makes its partial quotes look real too.

Limitations

  • Synthetic tickets and one task family: short customer-support replies. Other tasks, such as summaries or agents writing to tools, may behave differently.
  • Claude models only, all at medium effort, with one wording for each system prompt.
  • 100 replies per model, strategy and prompt combination. The pattern is consistent across all three models, but small differences between cells shouldn't be over-read.
  • The grader checks digits and email parts. It doesn't count other partial values, such as a stand-in's first name, except where we report them separately above.

Sources

  • Experiment harness and scenarios: marketing/research/partial-values (run 2026-11-19, Message Batch, Claude Haiku 5.5 / Sonnet 5.5 / Opus 5.5)
  • Obscate docs on the limitation — obscate.com/docs/langchain#partial-values

Try the redact → restore round trip on your own text — no signup.

Get a free API key

Free plan: 1,000 requests a month, all strategies. Your key is shown once — we store only a hash of it.