Benchmark 02 · legal questions (report, PL)#
Piotrowski A., Koman A. (2026). LexAlpha Lab · Benchmark 02 · pytania prawne [legal questions]. Version 1.1, 29.10.2026, in Polish (1.0: 27.10.2026; 1.1 adds a reference to the Jagiellonian University study in the introduction and references). PDF
Scope. Eight models of three providers (OpenAI, Anthropic, Google), ten questions on Polish law, two conditions (without tools and with web search), 160 answers; 1,436 of 1,454 citations checked against the official source as of 1.10.2026.
Main findings. Hallucinated answers: 51 of 80 without tools, 40 of 80 with web search; 19 and 27 correctly grounded. The most frequent error class is content falsely attributed to an existing provision or ruling (66 answers), followed by a former wording presented as in force (35); invented sources are rare (8), as are distorted identifiers (9); one answer may carry several classes. In this run, W1 had fewer hallucinated answers for five of eight models and the same number for three; the cost of the W1 run was 1.40 to 7.70 times that of W0 (part of the W1 costs known only as a lower bound). One run on different days cannot separate the effect of tools from answer variability. The four model judges that scored the full set reach κ of 0.58 to 0.90 with the cards (the fifth, GPT-6 Astra, 0.95 on 43 pilot cards); a verification report given to the judge as an aid raises agreement with the cards for GPT-6.1 Sol and lowers it for Gemini 4 Argon.
Contents. 1 Introduction · 2 The question set: ten questions from practice, keys approved before the run (keys P09 and P10 changed after the answers were collected, see methodology) · 3 Systems and run conditions · 4 Scoring method · 5 Model results · 6 Model judges · 7 Verification layer · 8 Limitations · 9 Conclusions and next steps · appendices A–E (questions and keys in short, results question × system, judges: agreement and settings, glossary, confusion matrices and uncertainty intervals for every κ).
@techreport{piotrowski2026benchmark02, author = {Piotrowski, Adam and Koman, Andrzej}, title = {LexAlpha Lab · Benchmark 02 · pytania prawne}, institution = {LexAlpha Lab, KTZR}, year = {2026}, month = {10}, note = {wersja 1.1, 29 października 2026}, url = {https://lawbenchmark.pl/} }
"Hallucination" Is Not Enough (methodology paper, EN)#
Piotrowski A. (2026). "Hallucination" Is Not Enough: Why Legal LLM Benchmarks Should Measure "Correctly Grounded". LexAlpha Lab, working paper v1.2, 26.10.2026, 37 pages (not peer-reviewed). doi:10.5281/zenodo.23674794 · SSRN 7672779 · full text on ktzr.pl · PDF
What it is about. The paper justifies the metric used on this site. The label "hallucinated" measures only whether the model said something untrue about the law; it says nothing about whether the answer is complete or what it rests on. The paper proposes the measure "correctly grounded" (full core, no false citations, at least one citation checked at the source) and the F/T/W/S error typology with the origin of the error, and shows on the pilot cards how model judges assess the plausibility of an answer rather than its truth, and how a verification report changes their verdicts. It includes the judges' κ with matrices and confidence intervals and the full scoring specification. The author also argues why such a measure is needed especially in legal systems where legislation is the principal source of law.
@techreport{piotrowski2026correctly, author = {Piotrowski, Adam}, title = {"Hallucination" Is Not Enough: Why Legal LLM Benchmarks Should Measure "Correctly Grounded"}, institution = {LexAlpha Lab, KTZR}, year = {2026}, month = {10}, note = {v1.2, 26 October 2026}, doi = {10.5281/zenodo.23674794} }
Other studies of language models on Polish law#
Porębski A., Figura J. (Jagiellonian University). Free LLMs Hallucinate and Rarely Signal Their Limitations in Solving Legal Problems. In: Progress in Polish Artificial Intelligence Research 6, University of Silesia Press, 2025, pp. 294–300. SSRN · full text (RUJ) · dataset (RODBUK, 2026).
The closest earlier empirical study: 12 prompts in Polish from constitutional, criminal and inheritance law, with a true or false premise, put five times in January 2025 to two free models (GPT-4o mini, Bielik-11B-v2); 120 answers scored by two assessors, and 369 cited rulings checked separately. The main part of the answer was correct in 91 of 120 cases; of the 369 rulings, 16 matched court and date, 8 were on the topic of the question, 280 had a case number belonging to a different case (numbers recomputed from the published spreadsheets). The authors did not count citations towards correctness; our measure "correctly grounded" joins both axes in one label. The error pattern matches ours: the identifier exists, the case or the content belongs elsewhere, pure invention is rarer.
Do you know another published study of language models on Polish law with an explicit scoring method? Write to a.piotrowski@ktzr.pl.