Trust
Benchmark & methodology
We publish our evaluation method and numbers because an unverifiable accuracy claim is worthless to a security team. Everything below is reproducible from the eval harness (eval/) with one command.
| Headline | Result | Read with |
|---|---|---|
| Structured & checksum IDs | F1 1.00 | Upper bound for clean, synthetic input |
| All entity types (micro) | F1 0.92 | Names 0.80, places 0.77 pull it down |
| Exact restores | 200/200 | For fpe, tokenize and synthetic, each |
| Median fpe round trip | 2.02 ms | Engine time on a laptop, not API latency |
What we test
- Detection — does Obscate find the sensitive data? (precision, recall, F1)
- Restoration — after redact → restore, is the output identical to the input? (exact-match rate, per reversible strategy)
- Latency — how long redaction and a full round trip take per document.
The dataset
- 200 synthetic documents from
eval/dataset.py(fixed seed, so every run is identical), 484 labelled sensitive spans. - Each document is filled from one of 10 templates (support tickets, KYC notes, invoices, login alerts, refund requests…), and every span’s exact position and type is recorded at generation time — no human-labelling ambiguity.
- Entity types covered: credit cards, Aadhaar, PAN, GSTIN, IBAN, US SSN, email, phone, IPv4, MAC address, person names, locations. (Company names appear in the text but aren’t scored: they’re generally not personal data. AWS keys and IPv6 are detected by Obscate but not in this dataset.)
- Synthetic by design, so the dataset can be published without exposing anyone’s real data.
Read this caveat before the numbers. The structured values are generated by Obscate’s own synthetic-value module, so they are well-formed by construction, and the templates are short and clean (80 characters on average). That makes the structured-ID results an upper bound for clean input, not a claim about messy real-world text. We’ll publish results on additional corpora as we build them.
Metrics, defined
- Precision — of the spans Obscate flagged, the share that were sensitive.
- Recall — of the sensitive spans present, the share Obscate found.
- F1 — the harmonic mean of the two.
- Micro-average pools every span before computing the scores, so each span counts once. On this dataset that’s the conservative headline: the largest class (PERSON, 137 of 484 spans) is also the hardest, so micro-F1 (0.92) is lower than the macro-average (0.96, every entity type weighted equally). We report both.
- Exact-restore rate — the share of documents where redact → restore gives back the input as an identical string.
- A prediction counts as correct when its type matches and its span overlaps the labelled span.
Results
Detection
| Entity | Precision | Recall | F1 | Spans |
|---|---|---|---|---|
| Aadhaar | 1.00 | 1.00 | 1.00 | 29 |
| Credit card | 1.00 | 1.00 | 1.00 | 37 |
| 1.00 | 1.00 | 1.00 | 57 | |
| GSTIN | 1.00 | 1.00 | 1.00 | 16 |
| IBAN | 1.00 | 1.00 | 1.00 | 24 |
| IPv4 | 1.00 | 1.00 | 1.00 | 21 |
| MAC address | 1.00 | 1.00 | 1.00 | 21 |
| PAN | 1.00 | 1.00 | 1.00 | 29 |
| Phone | 1.00 | 1.00 | 1.00 | 56 |
| US SSN | 1.00 | 1.00 | 1.00 | 14 |
| Person | 0.79 | 0.80 | 0.80 | 137 |
| Location | 0.77 | 0.77 | 0.77 | 43 |
| Micro average | 0.92 | 0.92 | 0.92 | 484 |
| Macro average F1 | 0.96 |
Micro average: TP 447 · FP 39 · FN 37.
Why the structured IDs score 1.00. Cards (Luhn), Aadhaar (Verhoeff), IBAN (mod-97) and GSTIN (check character) are checksum-validated: a match that fails its checksum is rejected, which is what keeps precision at 1.00. PAN, SSN, email, phone, IPv4 and MAC are pattern-matched without a checksum; they also score 1.00 here, helped by the clean synthetic input (see the caveat).
Restoration
| Strategy | Exact restores |
|---|---|
fpe (map-free, key only) | 200 / 200 |
tokenize (with token map) | 200 / 200 |
synthetic (with token map) | 200 / 200 |
Beyond the benchmark, randomized property tests (Hypothesis) check the round-trip on adversarial generated input:
- 3,000 generated texts through
fpeandsynthetic: no failures; - 10,000 generated nested JSON payloads across
tokenize,syntheticandfpe, with keys both reported and redacted: no failures.
These tests have caught and fixed real bugs before release, for example an IBAN followed by a number that went undetected.
Latency (per document)
| Strategy | Redact median | Redact p95 | Round-trip median | Round-trip p95 |
|---|---|---|---|---|
fpe | 1.86 ms | 2.23 ms | 2.02 ms | 2.47 ms |
tokenize | 1.39 ms | 1.56 ms | 1.39 ms | 1.57 ms |
synthetic | 1.42 ms | 1.64 ms | 1.45 ms | 1.68 ms |
Measured in-process on a single thread with no network, on an Apple M5 laptop (Python 3.12, macOS), over the benchmark’s short documents (80 characters on average), with the spaCy NER model loaded. This is engine time, not API latency: a hosted API call adds network round-trip and server overhead, and longer documents take longer. Hosted-API latency is not published yet.
Limitations (stated plainly)
- Names and places are best-effort (PERSON F1 0.80, LOCATION 0.77). They rely on a small NER model (spaCy
en_core_web_sm); single first names with little surrounding context are the main miss. For JSON, field-name hints (name,first_name, …) close much of the gap. A larger NER model or the optional LLM detection tier improves recall. The LLM tier exists in the library today and is coming to hosted plans. - Malformed identifiers are missed by design. A card or Aadhaar number that fails its checksum isn’t flagged: we favour precision over guessing. If you need to catch mistyped IDs, add them through a deny-list or field map.
- Synthetic data. See the caveat above. Real text varies.
Reproduce it yourself
pip install -e ".[ner]" && python -m spacy download en_core_web_sm
python -m eval.benchmark
The dataset generator (eval/dataset.py) and scorer (eval/benchmark.py) are both in the repo, so you can see exactly what was tested and run it again. Latency figures will differ on other hardware.