10:15 - 11:45
Location: Multi-Function Room 1 (19/F LAU)
Chair/s:
Emilie Tran
Discussant: King-wa FU

Lisa Lechner
- Detecting Absence in Large Language Models: Missing Precedents and Omitted Provisions in Legal Text
Runyi Ma - Normative Rigidity vs. Probabilistic Fragility: Identifying the Phase Transition of Systematic Failure in Large Language Models via Structured Legal Logic
Emilie Tran, Eric Sautede - Governing the Lingua Franca of Power: Middle Eastern Agency in the Regulation of Large Language Models
James Wong - Generative AI in Policymaking: Bias, Limitations, and Implications for Policy Change
Submission 109
Detecting Absence in Large Language Models: Missing Precedents and Omitted Provisions in Legal Text
Panel 2-Multi-Function Room 1 (19/F LAU)-01
Presented by: Lisa Lechner
Lisa Lechner
University of Innsbruck
Large language models are increasingly used to summarize, annotate, and reason over legal texts — treaties, statutes, and court decisions — and courts and counsel are already being sanctioned for relying on LLM-fabricated case citations that do not exist. This paper traces that failure to a general and under-examined limitation: models are far better at detecting what is present than at registering what is absent. We evaluate existing absence-detection and uncertainty-estimation tools on legal corpora and argue that absence-sensitivity is a validity precondition for using LLMs as instruments in legal and political research.

We distinguish two registers of absence. Sampling absence is a gap in the training distribution; recent work (AbsenceBench; Fu et al. 2025) shows that models fail to detect even conspicuous surface omissions, because transformer attention has no gap to anchor on, and instead fill them with confident fabrication (Kalai et al. 2025). Produced absence is a gap that was made — a provision negotiated away, a precedent left uncited, a dissent unaddressed — which legal and political theory treats as an exercise of power rather than a neutral void (Bachrach and Baratz 1962; Lukes 2005; Trouillot 1995; Fricker 2007).

Legal texts are an ideal and feasible testbed: they are structured and versioned, so clean original-versus-omitted pairs can be constructed, and their meaning often lies precisely where they deviate from templated defaults. Using existing tools — the AbsenceBench protocol and semantic-entropy uncertainty estimation (Farquhar et al. 2024) across several open-weight models — we test, on constitutional court decisions, whether models detect a removed precedent or instead reconstruct a citation that was never there; and, on tax treaties built from the OECD and UN Model Conventions, whether they notice a modified article or default to the standard clause the treaty in fact negotiated away.

We test the hypothesis that fabrication concentrates precisely on the deviations that carry legal meaning — so an LLM used to read case law or treaties would systematically misreport the precedents and provisions that matter most. Absence-sensitivity should therefore be a standard axis of LLM evaluation for legal and social research.