Trust

Benchmark & methodology

We publish our evaluation method and numbers because an unverifiable accuracy claim is worthless to a security team. Everything below is reproducible from the eval harness (eval/) with one command.

HeadlineResultRead with
Structured & checksum IDsF1 1.00Upper bound for clean, synthetic input
All entity types (micro)F1 0.92Names 0.80, places 0.77 pull it down
Exact restores200/200For fpe, tokenize and synthetic, each
Median fpe round trip2.02 msEngine time on a laptop, not API latency

What we test

  1. Detection — does Obscate find the sensitive data? (precision, recall, F1)
  2. Restoration — after redact → restore, is the output identical to the input? (exact-match rate, per reversible strategy)
  3. Latency — how long redaction and a full round trip take per document.

The dataset

  • 200 synthetic documents from eval/dataset.py (fixed seed, so every run is identical), 484 labelled sensitive spans.
  • Each document is filled from one of 10 templates (support tickets, KYC notes, invoices, login alerts, refund requests…), and every span’s exact position and type is recorded at generation time — no human-labelling ambiguity.
  • Entity types covered: credit cards, Aadhaar, PAN, GSTIN, IBAN, US SSN, email, phone, IPv4, MAC address, person names, locations. (Company names appear in the text but aren’t scored: they’re generally not personal data. AWS keys and IPv6 are detected by Obscate but not in this dataset.)
  • Synthetic by design, so the dataset can be published without exposing anyone’s real data.

Read this caveat before the numbers. The structured values are generated by Obscate’s own synthetic-value module, so they are well-formed by construction, and the templates are short and clean (80 characters on average). That makes the structured-ID results an upper bound for clean input, not a claim about messy real-world text. We’ll publish results on additional corpora as we build them.

Metrics, defined

  • Precision — of the spans Obscate flagged, the share that were sensitive.
  • Recall — of the sensitive spans present, the share Obscate found.
  • F1 — the harmonic mean of the two.
  • Micro-average pools every span before computing the scores, so each span counts once. On this dataset that’s the conservative headline: the largest class (PERSON, 137 of 484 spans) is also the hardest, so micro-F1 (0.92) is lower than the macro-average (0.96, every entity type weighted equally). We report both.
  • Exact-restore rate — the share of documents where redact → restore gives back the input as an identical string.
  • A prediction counts as correct when its type matches and its span overlaps the labelled span.

Results

Detection

EntityPrecisionRecallF1Spans
Aadhaar1.001.001.0029
Credit card1.001.001.0037
Email1.001.001.0057
GSTIN1.001.001.0016
IBAN1.001.001.0024
IPv41.001.001.0021
MAC address1.001.001.0021
PAN1.001.001.0029
Phone1.001.001.0056
US SSN1.001.001.0014
Person0.790.800.80137
Location0.770.770.7743
Micro average0.920.920.92484
Macro average F10.96

Micro average: TP 447 · FP 39 · FN 37.

Why the structured IDs score 1.00. Cards (Luhn), Aadhaar (Verhoeff), IBAN (mod-97) and GSTIN (check character) are checksum-validated: a match that fails its checksum is rejected, which is what keeps precision at 1.00. PAN, SSN, email, phone, IPv4 and MAC are pattern-matched without a checksum; they also score 1.00 here, helped by the clean synthetic input (see the caveat).

Restoration

StrategyExact restores
fpe (map-free, key only)200 / 200
tokenize (with token map)200 / 200
synthetic (with token map)200 / 200

Beyond the benchmark, randomized property tests (Hypothesis) check the round-trip on adversarial generated input:

  • 3,000 generated texts through fpe and synthetic: no failures;
  • 10,000 generated nested JSON payloads across tokenize, synthetic and fpe, with keys both reported and redacted: no failures.

These tests have caught and fixed real bugs before release, for example an IBAN followed by a number that went undetected.

Latency (per document)

StrategyRedact medianRedact p95Round-trip medianRound-trip p95
fpe1.86 ms2.23 ms2.02 ms2.47 ms
tokenize1.39 ms1.56 ms1.39 ms1.57 ms
synthetic1.42 ms1.64 ms1.45 ms1.68 ms

Measured in-process on a single thread with no network, on an Apple M5 laptop (Python 3.12, macOS), over the benchmark’s short documents (80 characters on average), with the spaCy NER model loaded. This is engine time, not API latency: a hosted API call adds network round-trip and server overhead, and longer documents take longer. Hosted-API latency is not published yet.

Limitations (stated plainly)

  • Names and places are best-effort (PERSON F1 0.80, LOCATION 0.77). They rely on a small NER model (spaCy en_core_web_sm); single first names with little surrounding context are the main miss. For JSON, field-name hints (name, first_name, …) close much of the gap. A larger NER model or the optional LLM detection tier improves recall. The LLM tier exists in the library today and is coming to hosted plans.
  • Malformed identifiers are missed by design. A card or Aadhaar number that fails its checksum isn’t flagged: we favour precision over guessing. If you need to catch mistyped IDs, add them through a deny-list or field map.
  • Synthetic data. See the caveat above. Real text varies.

Reproduce it yourself

pip install -e ".[ner]" && python -m spacy download en_core_web_sm
python -m eval.benchmark

The dataset generator (eval/dataset.py) and scorer (eval/benchmark.py) are both in the repo, so you can see exactly what was tested and run it again. Latency figures will differ on other hardware.

Get a free API key

Free plan: 1,000 requests a month, all strategies. Your key is shown once — we store only a hash of it.

Prefer the terminal?
curl -s API/v1/signup \
  -H 'content-type: application/json' \
  -d '{"email":"you@company.com"}'

Contact

Our contact address is being set up.