VeRDiCT: Verifying Real Docket Citations in Tiers

1 Introduction

Citation to authority is the load-bearing structure of legal argument, and generative models break it in ways that are expensive to discover. When asked verifiable questions about federal cases, general-purpose models produced legal hallucinations between 58% of the time for ChatGPT 4 and 88% for Llama 2 (Dahl et al., 2024), and retrieval-augmented legal research products marketed as reliable still hallucinated between 17% and 33% of the time (Magesh et al., 2025). The consequences now reach the courts. The AI Hallucination Cases Database (Charlotin, 2026) recorded 2,185 decisions in its export of 10 October 2026, of which 1,492 list the United States alone; the yearly count rose from 16 in 2023 and 61 in 2024 to 859 in 2025 and 1,248 in 2026 up to 8 October. Pro se litigants are the sole party responsible in 885 of the US entries and lawyers in 579.

Figure 1: VeRDiCT on two fabricated authorities from a widely reported 2023 sanction (United States District Court for the Southern District of New York, 2023). Both use real reporter series and real volume numbers; (b) borrows the volume of the genuine Supreme Court decision and moves it to the Federal Reporter. An index lookup with a caption check flags both in milliseconds, while the genuine decision passes and only its proposition would be left for a model or a person to read.

In this work, citation verification is defined as answering four questions about each authority a filing relies on: whether a case exists at the cited location, whether it is the case the filing names, whether quoted language appears in that case at the cited page, and whether the case supports the proposition it is cited for. The first three questions have deterministic answers given the right data; the fourth requires reading. The study is restricted to United States case law: statutes, regulations, secondary sources and the record are out of scope.

Existing approaches treat all four questions as one reading task. Closed-book models are not a remedy: on LegalCiteBench, even the strongest of 21 models scored below 7 out of 100 on closed-book citation retrieval and completion, and for 20 of them more than 94% of the low-scoring answers on retrieval-heavy tasks named a concrete authority rather than abstaining (Chen et al., 2026). Agentic verifiers do better at a cost. On the Legal_Phantom_Citation benchmark, a GPT-5 agent with legal-database and web tools reached 84.4 recall but 40.8 precision (55.0 F1), averaged 15.3 steps per excerpt and exhausted its 30-step budget in 29.5% of episodes; Claude Code with Opus 4.8 reached 68.8 F1 with 15.4 tool calls per excerpt (Liu et al., 2026). Both rely on the CourtListener lookup service, and 19.9% of the opinions the agents retrieved had no usable text or pagination (Liu et al., 2026). Claim-level work isolates the fourth question and supplies the full opinion to the model (Demir and Canbaz, 2026), and pinpoint-level work finds that even strong models miss many wrong page references (Verma, 2026). These benchmarks share a further limit: their errors are constructed, by injection or by prompting a model, rather than found in filings. Legal_Phantom_Citation, for example, generates non-existent citations from a list of reporter names that do not exist (Liu et al., 2026), a pattern that does not occur among the fabrications courts have documented (Section 5).

The authors propose VeRDiCT, which answers the first three questions with deterministic checks over open legal data and passes only the residue to a reader. The cascade checks reporter series and their publication dates, resolves each citation against an index of 18,130,235 CourtListener citations with an explicit abstention when the index does not cover a volume, matches the caption party by party, bounds pin cites by the cited case's pages, and locates quoted language in page-labelled opinion text from the Caselaw Access Project. Each tier sees only what earlier tiers could not settle.

The contributions of this work are as follows:

• A deterministic, open-data verification cascade for court filings, with coverage-aware abstention, that runs on two CPU cores without model calls.
• A test set of 7,515 citation instances from US court decisions that addressed hallucinated authority: 869 citations from the curated hallucination items of a public tracker and 6,646 authorities that the deciding courts themselves relied on.
• An evaluation on Legal_Phantom_Citation against eleven published agent and model configurations, with ablations of each tier and of the main thresholds, and a held-out check on test entries never inspected during development.
• Evidence that benchmark and real-world difficulty diverge: every non-existent citation in Legal_Phantom_Citation that VeRDiCT finds is caught by the reporter-series rule alone, whereas none of the 588 fabricated citations extracted for Dataset A uses a non-existent series, 32.8% cite proprietary numbers, and the open index misses some genuine authority that courts rely on.

2 Related Works

2.1 Hallucinated Authority in Legal Practice

Dahl et al. (2024) profiled legal hallucinations across models and tasks and found them pervasive for questions about real federal cases, and Magesh et al. (2025) showed that commercial retrieval-augmented tools reduce but do not remove the problem. Charlotin (2026) catalogues the decisions in which courts addressed hallucinated material, with the party, the tool where known, the nature of each item and the sanction. These works measure the problem; VeRDiCT is a check that a filer, a clerk or an opposing party can run before the problem reaches a judge.

2.2 Benchmarks for Citation Verification

LegalCiteBench evaluates closed-book citation retrieval, completion and verification over opinions from the Caselaw Access Project (Chen et al., 2026). Legal_Phantom_Citation injects five error types into excerpts of 245 federal appellate briefs filed between 2012 and 2021, adds model-generated holdings from Dahl et al. (2024), and evaluates agentic and non-agentic detectors (Liu et al., 2026). Demir and Canbaz (2026) frame support checking as three-way inference over 3,396 parenthetical claims, and Verma (2026) corrupts citations by replacing the case or the pinpoint page. None of the four draws its errors from filings that courts have examined. VeRDiCT is evaluated on one of them and on citations recorded in such decisions.

2.3 Citation Infrastructure

eyecite extracts and normalises case citations (Cushman et al., 2021); reporters-db records reporter series and their date ranges (Free Law Project, 2026c); CourtListener publishes a citation lookup service and quarterly bulk data (Free Law Project, 2026b,a); and the Caselaw Access Project publishes page-labelled opinion text (Harvard Law School Library Innovation Lab, 2024). These are components rather than detectors and have not, to the authors' knowledge, been evaluated as a verification system against labelled hallucinations.

2.4 Cascades and Cost-Aware Routing

Attentional cascades reject easy negatives with cheap tests before expensive ones (Viola and Jones, 2001), and FrugalGPT applies the idea to language models by routing queries through models of increasing cost (Chen et al., 2024). VeRDiCT places deterministic, zero-cost tiers before any model and routes by what each tier can decide rather than by model confidence. To our knowledge, no publicly available method verifies citations in court filings with deterministic open-data checks evaluated on real sanctioned filings, or reports the cost of each verification stage.

3 Methodology

VeRDiCT processes a document in five tiers (Figure 2). Offline, it builds three resources from open data. Online, each citation passes through the tiers in order, and each tier either flags it, clears it, or leaves it undecided for the next tier.

Figure 2: The VeRDiCT pipeline. Offline (blue), three open resources are indexed once: reporter series with their date ranges, 18,130,235 CourtListener citations with 10,174,551 case names from the snapshot of 30 September 2026, and Caselaw Access Project page ranges and text, cached on first use. Online (pink), each citation passes the tiers in order; a flag ends its path, and anything a tier cannot decide moves on. What survives every deterministic tier, together with quotations absent from the cited opinion, forms the escalation set for a reading tier (grey), which this paper sizes but does not evaluate.

3.1 Problem Formulation

A document $d$ contains citation instances $c_1, \ldots, c_n$. Each instance carries a volume $v$, a reporter $r$, a first page $p$, an optional pin cite $\pi$, the caption $w$ written before it, and any quotations $Q(c)$ attached to it. A short form ("Smith, 437 U.S. at 587") or an Id. citation inherits $(v, r, p)$ from its antecedent. Gold labels mark spans as non-existent citations, case-name mismatches, incorrect pin cites, altered quotations or misrepresented holdings. A system outputs flagged spans; Section 4.4 gives the matching rule.

3.2 T0: Reporter Series and Dates

Let $\mathcal{E}$ be the set of reporter editions in reporters-db and $\mathcal{B}$ the set of their base names ("N.E." for "N.E.2d"). A string of the form "volume, abbreviation, page" whose abbreviation has a base in $\mathcal{B}$ but is not itself in $\mathcal{E}$, such as "N.E.4th", is flagged. A citation to a numbered later series whose parenthetical year precedes the series' start by more than a year is also flagged. Year checks are limited to numbered series because start dates for unnumbered reporters are unreliable in the source data.

3.3 T1: Existence with Coverage-Aware Abstention

Let $I$ be the set of $(v, r, p)$ triples in the citation index and $V_I$ the set of indexed $(v, r)$ volumes. The existence decision is

where $R_{\mathrm{prop}}$ is the set of proprietary reporters (Westlaw and Lexis numbers), $N(v, r, p)$ is the set of names of the up to four cases that begin within 150 pages before $p$ in the same volume, and $s$ is the caption similarity of Eq. 2. The third case admits a common drafting habit, citing a pin page in full form ("467 U.S. 842" for a case that begins at 837), provided the written caption matches a case that contains the page. Without a caption, such citations abstain. For closed series with at least 200 indexed volumes, a volume beyond the last indexed one is flagged. Abstention is reported, never counted as a flag.

3.4 T2: Party-wise Caption Identity

For a citation found in the index, let $C$ be the names CourtListener and CAP give the case. Captions are split into parties at "v.", parties that are generic ("United States", a state, "People", "Commonwealth") are dropped, abbreviations are expanded with a table of common reporter abbreviations, and dotted initialisms are collapsed ("E.P.A." to "epa"). The similarity is

where $P(\cdot)$ is the set of distinctive parties and $\sigma$ is the token-set ratio of RapidFuzz (Bachmann, 2026), set to 100 when $a$ is the initialism of $b$. A citation is flagged as a case-name mismatch when $s(w, C) < \theta_n$. Matching on either party tolerates the substitution of public officials across a case's history, while requiring a distinctive party prevents "United States v. Smith" from matching "United States v. Jones".

3.5 T2b: Pin-Cite Range

For a full or short citation with pin $\pi$ resolving to a case with first page $p$ and last page $\ell$ from CAP metadata, the citation is flagged when $\pi < p$ or $\pi > \ell$. Short forms whose antecedent lies outside the excerpt are resolved by their short name within the cited volume, and are skipped when the name does not identify a unique case. Id. citations are excluded from this tier because they frequently refer to record documents that the extractor cannot see.

3.6 T3: Quotation Location

Each quotation of at least $m$ words is attached to a citation by one of three rules, tried in order: the quotation sits inside a parenthetical that opens after a citation; the next citation follows within 220 characters with no record cite, statute or other quotation between; or the quotation follows a citation in the same sentence. A quotation $q$ is split into fragments $f_1, \ldots, f_k$ at ellipses and bracketed alterations, and each fragment is located in the cited opinion's normalised text $T$. If $f_j$ is a substring of $T$ it is found. Otherwise let $a_j$ be the best partial alignment score of $f_j$ against $T$ and $\delta_j$ the number of content words that differ between $f_j$ and the aligned window, ignoring word pairs whose similarity is at least $\omega$ (optical-character-recognition noise). The verdict for $q$ is

Altered quotations are flagged. Absent quotations are escalated rather than flagged, because a deterministic attachment rule cannot distinguish a fabricated quotation from one drawn from another source. For verbatim quotations with a pin cite, CAP page labels give the pages on which each fragment occurs; when no occurrence lies on the pin page or the page before it, the citation is flagged as an incorrect pin cite.

3.7 Escalation Set

Citations that survive all deterministic tiers contribute their proposition, the text from the start of the citing sentence to the citation, to an escalation set, together with every absent quotation. This set is what a reading tier, whether a model or a person, must examine. The paper reports its size; evaluating a reading tier is left to future work.

4 Experiments

4.1 Datasets

Legal_Phantom_Citation. The benchmark of Liu et al. (2026) provides a 390-entry test split (eval) and a 910-entry auxiliary split (aux_train), released under CC BY 4.0. Of the test entries, 90 are model-generated holdings from Dahl et al. (2024) rather than brief excerpts; they hold 77 of the misrepresented-holding spans. The test split contains 331 gold spans: 131 misrepresented holdings, 68 case-name mismatches, 55 incorrect pin cites, 46 altered quotations and 31 non-existent citations, plus 22 optional spans. All rules and thresholds were developed on aux_train. The first 60 test entries were inspected while debugging citation extraction early in development, so results on the remaining 330 entries are reported separately as a held-out check.

Court decisions on hallucinated authority (Dataset A). The authors built a citation-level set from the AI Hallucination Cases Database (Charlotin, 2026), downloaded on 10 October 2026. Its entries are decisions that addressed hallucinated material in a filing, from warnings to monetary and professional sanctions. Of its 2,185 rows, 1,492 are from the United States and 1,365 record at least one hallucinated item about case law. Positives are citations that appear inside the curated item descriptions, labelled with the item's category: fabricated, false quotation or misrepresented. The primary set keeps items that contain exactly one citation, so the label attaches without ambiguity, giving 869 positives (588 fabricated, 192 misrepresented, 89 false quotations); 331 citations from multi-citation items form an extended set. Negatives are authorities that the deciding court relied on in its own decision. The authors converted 1,267 of the source decisions to text, removed every sentence (and the sentence after it) that discusses fabricated, false, misattributed or AI-generated authority, removed citations matching any positive, sampled up to six distinct citations per decision, and excluded decisions in which a judge was among the parties who used AI, giving 6,646 negatives. Appendix B lists the exclusion vocabulary and counts.

4.2 Implementation Details

All experiments ran on a cloud container with two virtual CPU cores and 7 GB of memory, Python 3.13, eyecite 2.7.8, RapidFuzz 3.14.6, PySBD (Sadvilkar and Neumann, 2020) for sentence boundaries, and SQLite. The CourtListener bulk snapshot of 30 September 2026 was indexed into 18,130,235 citation rows (139 s) and 10,174,551 case-name rows (389 s). CAP volume metadata and opinion HTML were fetched on first use at no more than four requests per second and cached. Hyperparameters, selected on aux_train (Appendix A), are: caption threshold $\theta_n = 80$; quotation absence threshold $\theta_q = 70$; OCR word similarity $\omega = 80$; minimum changed words $k_\delta = 1$; minimum quotation length $m = 8$ words; proprietary reporters $R_{\mathrm{prop}}$ = WL and the four LEXIS forms. No component is trained, so results are deterministic and no seeds apply.

4.3 Baselines

Six cascade variants isolate each tier: T0 alone; T0 and T1; T0 and T1 without coverage-aware abstention (misses in proprietary reporters and unindexed volumes are flagged); T0 to T2; T0 to T2b; and the full cascade T0 to T3. External baselines are the eleven configurations reported by Liu et al. (2026) on the same split with the same matching rule: Gemini 2.5 Flash, GPT-5, GPT-OSS 120B, Qwen3-8B and Qwen3.6-27B, each with and without their agentic harness, and Claude Code with Opus 4.8. Their numbers are quoted from that paper, not re-run.

4.4 Metrics

On Legal_Phantom_Citation a gold span is found when it is a substring of a predicted span or the reverse, case-insensitively, with ellipsis fragments allowed (Liu et al., 2026). Precision is the share of predicted spans matching any gold span, recall the share of required gold spans found, micro-averaged over entries; optional spans count as true positives when predicted and are not required (Liu et al., 2026, App. A1.4). Under the released evaluation code, which counts optional spans as required, the full cascade scores 62.5 precision, 28.0 recall and 38.7 F1. Ninety-five per cent intervals come from 1,000 bootstrap resamples of entries. On Dataset A the unit is the citation: the flag rate on positives, the false-alarm rate on negatives, and the abstention rate.

5 Results

Against published systems. Table 1 compares VeRDiCT with the published configurations. The full cascade reaches 62.5 precision and 29.0 recall (39.6 F1, 95% interval 33.3 to 45.6). Its precision is second only to Claude Code with Opus 4.8 (76.1) and above every other configuration, agentic or not; its recall is the lowest of all, because it makes no attempt on the 131 misrepresented holdings. Its F1 is above seven of the eleven configurations and below GPT-5 with and without tools, Claude Code, and the Qwen3.6-27B agent (40.0). Liu et al. (2026) report ± values of 2.2 to 7.8 points for these rows without stating what they denote, so gaps of a few points should be read with care. Restricted to the four citation-level types, VeRDiCT finds 48.0% of gold spans. The 330 held-out entries give 39.0 F1, against 39.6 on all 390. One post-hoc change to the output format, reporting a flagged citation's caption and parenthetical with it, lets a flag match case-name spans that the benchmark marks on the caption alone (16 of the 68); it raises case-name matches from 40 to 44, F1 to 41.3 on the test split and F1 from 50.2 to 50.9 on aux_train, but was chosen after the first test run and is not part of the main result.

Table 1: Legal_Phantom_Citation test split (390 excerpts). Deterministic rows are the authors' runs; parentheses give 95% bootstrap intervals for the full cascade. Model and agent rows are quoted from Liu et al. (2026, Table 2), which reports ± values of 2.2 to 7.8 points without defining them. In bold: the best published value per column.

By error type. Table 2 sets VeRDiCT's recall beside the agents' per-type recall (Liu et al., 2026, Table A6). VeRDiCT finds 80.6% of non-existent citations, all through the reporter-series rule of T0, which reflects how the benchmark generates them (Liu et al., 2026) rather than a property of real fabrications; the six it misses all use "Mass. App. 4th", whose base is not a reporter series in reporters-db, so neither T0 nor the extractor recognises them. It finds 58.8% of case-name mismatches, below every agent except Qwen3-8B, partly because 16 of the 68 gold spans mark the caption alone. It finds 41.3% of altered quotations, close to GPT-OSS 120B (42.9) and well below GPT-5 (95.2). Incorrect pin cites remain hard for every system: VeRDiCT finds 21.8%, against 52.8 for GPT-5 and 3.6 for Claude Code, because a small change of page is detectable only when a quotation fixes the true page.

Table 2: Recall (%) by error type on the test split. VeRDiCT is the authors' run; Tier lists the tiers that produced its matches, most frequent first. Agent columns are the agentic configurations of Liu et al. (2026, Table A6), which reports recall only because its systems do not label error types.

Cost. Table 3 reports the cost per excerpt. The full cascade takes 90 ms per excerpt and makes no model calls: extraction and tiers T0 to T2 take 25 ms, the pin-range tier adds 31 ms and the quotation tier 34 ms, and a warm cache fetches nothing. The GPT-5 agent averages 15.3 steps per excerpt, each a model turn with a growing context, and Claude Code 15.4 tool calls, which its harness can issue several per turn; the source paper reports no dollar costs. On the test split, the deterministic tiers leave 956 propositions for a reading tier, an average of 2.5 per excerpt.

Table 3: Cost per excerpt on the test split, two virtual CPU cores, warm cache, median of three runs. Agent figures are averages reported by Liu et al. (2026): model steps, or tool calls for Claude Code, whose harness can issue several per turn; n/r: not reported. Index build covers both CourtListener tables.

Court decisions on hallucinated authority. Table 4 reports Dataset A. VeRDiCT flags 46.6% of the 588 fabricated citations in the primary set and abstains on 44.4%. Of the fabrications, 32.8% cite Westlaw or Lexis numbers, which open data cannot confirm or refute; on the remaining fabrications that cite published reporters, the flag rate is 69.1%. None of the fabrications uses a reporter series that does not exist, the pattern behind every non-existent citation that VeRDiCT finds in Legal_Phantom_Citation. Misrepresented and falsely quoted authorities are flagged at 13.0% and 3.4%: these are real cases, and the citation-level tiers are not meant to catch them. On the 6,646 authorities the courts relied on, the alarm rate is 5.9%, with 29.0% abstentions. A random sample of 59 of these alarms was categorised by reading each decision's text with AI assistance (the categories are released for confirmation by a person): 35 were genuine authorities that the 30 September snapshot does not hold at the cited location (Mata v. Avianca's own sanctions opinion, at 678 F. Supp. 3d 443, accounts for four); 14 came from text-extraction noise in the PDF orders; 3 were negatives mislabelled by the construction rules, sentences in which a court discussed a fabricated citation without the excluded vocabulary; 2 were Arizona paragraph cites read as pages; 2 were caption variants below the threshold; and 3 had other causes. The coverage-aware rule of Eq. 1 assumes that an indexed volume is complete, and in this sample it often was not.

Table 4: Dataset A, primary set: citation-level tiers T0 to T2b on citations recorded in US decisions that addressed hallucinated authority (%). The 95% bootstrap interval for the fabricated flag rate is 42.7 to 50.5.

5.1 Ablations

Table 5 varies one component at a time on the test split, with the other settings fixed at the values selected on aux_train; Appendix A gives the full sweeps on aux_train. These ablations isolate three findings. First, coverage-aware abstention is what makes an open index usable: flagging lookup misses in proprietary reporters and unindexed volumes lowers precision from 82.0 to 37.6 for T0 and T1 without finding any additional error, because Westlaw and Lexis citations and unindexed volumes are common in real briefs. Second, the quotation tier contributes the largest share of the recall beyond format and existence checks, and its precision depends on treating absent quotations as escalations: flagging them adds 6 true and 23 false positives. Third, the pin-range tier is nearly neutral on this benchmark, 39.4 F1 without it against 39.6 with it, because the benchmark draws each injected pin from between the case's first page and the original pin (Liu et al., 2026, App. A1.2), so it always lies inside the case.

Table 5: Ablations on the test split, one change at a time from the settings selected on aux_train. Best F1 in bold.

6 Conclusion and Future Works

VeRDiCT shows that a large share of citation verification needs no model. Over open data, five deterministic tiers find non-existent citations, mismatched captions and altered quotations at 90 ms per excerpt and no model cost, with precision above every published configuration but one. On citations from court decisions that addressed hallucinated authority, the same tiers catch 46.6% of documented fabrications and expose three limits that synthetic benchmarks hide: fabrications cite valid reporter series, a third cite proprietary numbers that no open resource can check, and the open index is incomplete even for well-known decisions. The practical implication is to place deterministic tiers in front of any agent, so that model calls are spent on the propositions and absent quotations that only reading can settle. Future work will evaluate reading tiers on the escalation set, replace the quarterly snapshot with the live lookup service and a density test for partly indexed volumes, extend coverage to Westlaw and Lexis citations through licensed lookups, label quotation and proposition errors in the filings these decisions address, and release the verifier as a filing-time check.

Limitations

The authors outline several limitations of VeRDiCT and its evaluation, and encourage future work to address them.

No reading tier. VeRDiCT does not judge whether a case supports a proposition. Misrepresented holdings are 40% of the gold spans in Legal_Phantom_Citation and 22% of the primary positives in Dataset A, and VeRDiCT finds almost none of them. Its recall is therefore capped at the share of gold spans that are citation-level errors, and the comparison in Table 1 should be read per error type.

Labels in Dataset A. Positives come from a curator's summary of each decision, not from the filing itself, and negatives assume that a deciding court's own authorities are real; the inspected sample shows that the exclusion vocabulary misses some sentences in which a court discusses a fabricated citation. Neither assumption has been audited by hand; an audit sample of 100 positives and 100 negatives is released with the data and is pending. Quotation and proposition checks cannot be evaluated on Dataset A because the database does not carry the filings' text.

Coverage of open data. The CourtListener snapshot ends on 30 September 2026, does not index every volume, and is incomplete within volumes it does index; CAP text ends with its digitisation. VeRDiCT abstains on proprietary citations and missing volumes, which leaves 44.4% of real fabrications undecided: 191 of the 193 Westlaw and Lexis citations, and published-reporter citations amounting to 11.9% of the fabrications. It cannot detect a missing case inside an indexed volume, which caused most of the inspected alarms on genuine authority.

Development on the test split. The first 60 test entries were inspected during early debugging of extraction. The held-out 330 entries give 39.0 F1, close to the full-split figure, but the risk of tuning to the test split is not zero.

Comparison with published numbers. Agent results are quoted from Liu et al. (2026) rather than re-run, and their models, harnesses and search services may have changed since. The benchmark's labels also assume that the original briefs are free of errors: T1 raises 9 flags that no gold span covers, and for 5 of them a case with the written caption begins one digit away (Appendix A), so measured precision understates the share of flags that point at real mistakes.

Scope. Only United States case law in English is covered. Statutes, regulations, secondary sources, record citations and foreign authority are out of scope, although the same tracker lists decisions from more than 40 other jurisdictions (Charlotin, 2026).

Use of AI Assistants

The authors used Claude (Anthropic) as a research and engineering assistant: to search the literature, write and debug the code, run the experiments and draft the text. Every experimental result in the paper is written into the text by a script from the released results files; figures quoted from other work and dataset construction counts were typed by hand and checked against their sources. The sample of alarms in Section 5 was categorised with AI assistance and is released for confirmation by a person.

References

• Max Bachmann. 2026. RapidFuzz: rapid fuzzy string matching. https://github.com/rapidfuzz/RapidFuzz. Version 3.14.6.
• Damien Charlotin. 2026. AI Hallucination Cases Database. https://www.damiencharlotin.com/hallucinations/. CSV export of 10 October 2026 (page last updated 9 October 2026), CC BY 4.0. Accessed: 2026-10-10.
• Lingjiao Chen, Matei Zaharia, and James Zou. 2024. FrugalGPT: How to use large language models while reducing cost and improving performance. Transactions on Machine Learning Research.
• Sijia Chen, Hang Yin, and Shunfan Zhou. 2026. LegalCiteBench: Evaluating citation reliability in legal language models. Preprint, arXiv:2605.10186.
• Jack Cushman, Matthew Dahl, and Michael Lissner. 2021. eyecite: A tool for parsing legal citations. Journal of Open Source Software, 6(66):3617.
• Matthew Dahl, Varun Magesh, Mirac Suzgun, and Daniel E. Ho. 2024. Large legal fictions: Profiling legal hallucinations in large language models. Journal of Legal Analysis, 16(1):64–93.
• M. Mikail Demir and M. Abdullah Canbaz. 2026. When Citations Mislead? A Claim-Level Benchmark for Legal Hallucination Detection. In Proceedings of the 21st International Conference on Artificial Intelligence and Law (ICAIL 2026), Singapore. ACM.
• Free Law Project. 2026a. CourtListener bulk data: citations and opinion clusters, snapshot of 2026-09-30. https://com-courtlistener-storage.s3-us-west-2.amazonaws.com/list.html?prefix=bulk-data/. Files citations-2026-09-30.csv.bz2 and opinion-clusters-2026-09-30.csv.bz2. Accessed: 2026-10-10.
• Free Law Project. 2026b. CourtListener citation lookup API (v4). https://wiki.free.law/c/courtlistener/help/api/rest/v4/citation-lookup. Accessed: 2026-10-10.
• Free Law Project. 2026c. reporters-db: a database of court reporters and their editions. https://github.com/freelawproject/reporters-db. Python package version 3.2.66.
• Harvard Law School Library Innovation Lab. 2024. Caselaw Access Project: static case data. https://static.case.law/. Accessed: 2026-10-10.
• Patty Liu, Dominik Stammbach, and Peter Henderson. 2026. Who Checks the Citations? Benchmarking Legal Hallucination Detection. Preprint, arXiv:2606.21155. Version 3, 4 October 2026. Dataset: https://huggingface.co/datasets/ai-law-society-lab/Legal_Phantom_Citation.
• Varun Magesh, Faiz Surani, Matthew Dahl, Mirac Suzgun, Christopher D. Manning, and Daniel E. Ho. 2025. Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. Journal of Empirical Legal Studies, 22(2):216–242.
• Nipun Sadvilkar and Mark Neumann. 2020. PySBD: Pragmatic sentence boundary disambiguation. In Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS), pages 110–114, Online. Association for Computational Linguistics.
• United States District Court for the Southern District of New York. 2023. Mata v. Avianca, Inc., No. 22-cv-1461 (PKC), Opinion and Order on Sanctions, ECF No. 54. https://storage.courtlistener.com/recap/gov.uscourts.nysd.575368/gov.uscourts.nysd.575368.54.0_8.pdf. Filed 22 June 2023. Accessed: 2026-10-10.
• Apurv Verma. 2026. Is this citation on point? Preprint, arXiv:2608.12571.
• Paul Viola and Michael Jones. 2001. Rapid object detection using a boosted cascade of simple features. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2001), volume 1, pages I-511–I-518, Kauai, HI, USA. IEEE.

A Detailed Ablation Analysis

A.1 Selection on aux_train

All thresholds were chosen on the 910-entry aux_train split before the first test run. Table 6 sweeps the caption threshold with tiers T0 to T2 only; $\theta_n = 80$ gave the best F1 and was kept. Table 7 compares quotation settings with the full cascade (these runs used $\theta_n = 60$, the setting at the time). Treating absent quotations as escalations raised F1 by 1.5 points, and requiring two changed words instead of one traded recall for precision with no gain in F1, so $m = 8$ and $k_\delta = 1$ were kept. With every selected setting, the cascade reaches 70.1 precision, 39.1 recall and 50.2 F1 on aux_train.

Table 6: Caption threshold on aux_train, tiers T0 to T2.

Table 7: Quotation settings on aux_train, full cascade with $\theta_n = 60$.

A.2 Coverage-aware Abstention

A lookup miss is weak evidence on its own. On the test split, flagging every miss lowers the precision of T0 and T1 from 82.0 to 37.6 without finding any additional error, because briefs cite Westlaw and Lexis numbers ("2015 WL 1240182, *8") and volumes the open index does not hold. Liu et al. (2026) report the same pressure on agents: 132 correct citations in the test split return no CourtListener result, and weaker models flagged up to 65.9% of them.

A.3 Unlabelled Errors Found by T1

T1 raises 9 flags on the test split that match no gold span. For 5 of them, a case with the written caption begins at a citation one digit away in the volume or the page (a digit changed, added, removed or transposed), which marks the flag as a likely typing error in the original brief rather than a false alarm. Examples are "216 F.3d 659" where the captioned case is at 276 F.3d 659, "616 F.2d 153" for 616 F.2d 1153, and "553 S.W.2d 903" for 554 S.W.2d 903. Liu et al. (2026, App. A1.4, Table A1) list "216 F.3d 659" among the typos they added to the gold labels after inspecting GPT-5's false positives, but the released labels do not contain it; the other four are not listed.

A.4 Caption Matching

Party-wise matching mainly guards against generic parties. "United States v. Smith" cited where "United States v. Jones" begins scores 50 on whole-caption similarity but 20 party-wise, and "People v. Garcia" against "People v. Martinez" scores 71 and 43. Abbreviation expansion and the initialism rule let "Nat'l Lime Ass'n v. EPA" match "National Lime Association v. Environmental Protection Agency" (93 whole, 100 party-wise), and matching on either party accepts "Heartland Regional Med. Ctr. v. Sebelius" against the indexed "Heartland Regional Medical Center v. Leavitt", where the Secretary changed between the decision and the brief. The rule has a blind spot: when one party is a frequent litigant that is not on the generic list, a fabricated opponent passes, so "Smith v. City of Chicago" cited where "Jones v. City of Chicago" begins scores 100 party-wise (80 whole). On aux_train, raising $\theta_n$ from 80 to 90 more than doubles T2's false alarms (20 against 9) for one more true positive.

A.5 Quotation Location

On the test split, for example, the altered quotation "constitutional torts, like their common law kin, require a showing of both actual cause and proximate cause" aligns with the cited opinion at a score of 79, with five content words counted as changed. Inspection of its false positives on aux_train found three recurring sources: quotations the brief itself altered without brackets, quotations attached to the wrong authority when a sentence cites several, and page labels that place a quotation spanning a page break on the earlier page. On the test split, the tier's misquotation flags are 22 correct and 25 incorrect, and its pin-cite flags 10 correct and 17 incorrect. Disabling the pin check gives 37.5 F1 against 39.6.

A.6 Pin-Cite Range

CAP metadata occasionally records a case's last page as equal to its first; such ranges are ignored. Bounding a case by the next indexed case in the volume instead of CAP's last page made no difference on the test split (39.6 F1): the benchmark draws each injected pin from between the case's first page and the original pin (Liu et al., 2026, App. A1.2), so a range check cannot catch it by construction.

B Dataset A Construction

Source. The AI Hallucination Cases Database (Charlotin, 2026) was exported as CSV on 10 October 2026 (2,185 rows; the web page reported 2,180 cases and a last update of 9 October 2026). Each row lists one or more hallucination items in the form "Category: Subcategory | description", separated by double bars. Only US rows and items with the subcategory Case Law and the categories Fabricated, False Quotes or Misrepresented were used.

Positives. Citations were extracted from each item's description with eyecite and with the reporter-series rule of T0, and deduplicated by volume, reporter and page within the item. Of 4,116 case-law items in US rows, 1,029 contain at least one citation, giving 1,200 citations. Items with exactly one citation form the primary set (588 fabricated, 192 misrepresented, 89 false quotations); the remaining 331 citations form the extended set, which is released but not used in Table 4.

Negatives. For each US row whose source is a PDF decision hosted by the database and whose items include case law, the decision was converted to text with pdftotext and unwrapped. Of the 1,272 such decisions, 1,271 downloaded; two came from rows in which a judge was the party who used AI and were not used, two yielded no readable text, and 1,267 were converted. A sentence was excluded, together with the sentence after it, when it matched the pattern below; 3,834 candidate citations fell in such sentences, and a further 1,729 matched a positive by volume, reporter and page. Up to six distinct remaining citations per decision were sampled with a fixed seed. Finally, the 18 sampled citations from three decisions in which a judge was among the parties who used AI were removed, leaving 6,646 negatives.

fictitious, fabricat*, hallucinat*, non-existent,
nonexistent, does / do / did not exist, no such
case, no such opinion, could not / unable to /
cannot locate or find, not exist, misrepresent*,
mischaracteri*, does / did not stand for, support,
say or contain, not appear in, not found in,
inaccurate, incorrect citation, wrong citation,
miscit*, phantom, fake, bogus, generative,
artificial intelligence, AI, ChatGPT

Presentation to the verifier. Each instance is presented as "Caption, volume reporter page, pin." using the caption written before the citation in its source text, or "See volume reporter page." when no caption was found. Only tiers T0 to T2b apply; quotation and proposition checks need the filing's own text, which the database does not carry.

Audit sample. A fixed-seed sample of 100 primary positives and 100 negatives, with the source text of each, is released for independent annotation. At the time of writing it has not been annotated.

C Reproduction

The code, the processed Dataset A, the audit sample and every results file behind this paper's numbers are released with it. Rebuilding the indexes from the 30 September 2026 CourtListener snapshot takes 139 s and 389 s on the hardware above; the CAP cache is filled on the first run. src/run_phantom.py reproduces Tables 1 to 5, src/eval_dataset_a.py reproduces Table 4, and paper/make_numbers.py writes every number in the text from those results.

@misc{moinuddin2026verdict,
  title  = {{VeRDiCT}: Verifying Real Docket Citations in Tiers},
  author = {Moinuddin, Syed},
  year   = {2026},
  month  = oct,
  note   = {Preprint},
  url    = {https://syedmoinuddin.com/research/verdict}
}

Last updated on October 11, 2026