Teaching VETR Which Clauses Actually Bind
VETR Proposal (Internal R&D)
+0.08–0.14 F1
Binding-set accuracy gain over regexes
70–79%
Specificity vs 8–19% at equal recall
~13.8%
Phantom regex entries caught
The Challenge
Status-quo clause detection is regexes: every mention becomes a "match." Measured across 6,472 solicitations, 13.5% of the entries those regexes list are clause numbers whose every mention is a checklist item with an empty box —rising to 31% in checklist-heavy documents. Each phantom clause becomes a compliance task somebody works, or noise somebody learns to ignore. Both outcomes are bad: one wastes capture hours, the other trains teams to distrust the matrix.
The Solution
A clause ledger instead of a clause list. Deterministic checkbox rules (a checked box binds; an empty one doesn't — verified 100/100 against human readers) plus a small logistic-regression classifier for narrative mentions, each number scored with a probability and the exact evidence lines. Full text, incorporated-by-reference, and checklist selections are separated from narrative mentions. Anything uncertain is routed to a person, never silently resolved. The model (857 KB) and the eCFR registry snapshot run as CPU-only code inside our VPC — no GPU, no LLM call, no data leaving the boundary — with the old regexes kept as the fallback.
Problem. Compliance matrices inherit every clause number the parser's regexes touch. Our measurement: ~13.8% of those entries are clauses no one selected. In checklist-heavy solicitations it's roughly one in three.
Method. FedProc-Ledger, developed as open research: rule layer for checkboxes, paragraph-A references, lists and exclusions; a role classifier (logistic regression over mention features, temperature-calibrated) for the rest; max-over-mentions aggregation; tiering at q ≥ 0.7 BINDING / q ≤ 0.3 NOT_BINDING, UNDETERMINED between. Evidence capped at five lines per entry. Nothing human-labeled in training.
Evidence. Four consecutive fresh, pre-registered test rounds (26, 24, 24, 27 new documents; majority gold of one coding agent + two LLM judges): binding-set F1 +0.142, +0.097, +0.107, +0.084 over the regex baseline, every interval above zero; specificity 76%, 78%, 73%, 70% against 10%, 19%, 8%, 14% at equal-or-better recall. Gains concentrate where they should: +0.33 F1 in checklist-heavy documents, +0.03 in short ones without checklists.
In product. Every parsed solicitation in VETR now gets a ledger row: binding set drives the compliance matrix, ruled-out clauses are shown (not hidden), uncertain ones are flagged for review. Sidecar errors, scanned documents, and any disabled flag reproduce the old regex output byte-for-byte.
Limits, stated plainly. No human-expert labels yet — the gold above is agent + LLM majority, and a 200-document reviewer-gold acceptance gate (precision/recall/F1 non-inferiority) stands between the pilot and any customer-visible reliance. Clause ranges ("52.219-1 through 52.219-4") and alternates' dates are not resolved in this version; scanned PDFs return no ledger, never "no clauses."
Links. Model: huggingface.co/raihan-js/fedproc-ledger-v1 · Data: huggingface.co/datasets/raihan-js/fedproc-ledger-bench · License: Apache-2.0.
Snippet
Regexes list every clause they see — including ones nobody checked. FedProc-Ledger decides which FAR/DFARS clauses actually bind, with evidence. Now live in VETR.
Ready to put VETR on your next pursuit?
Walk through the platform with our team and see how it fits your capture cycle.
Schedule a free demo