What is in the dataset#

Directory	Content	Records
`SPEC/`	scoring specification v1.0.4 (scoring as in 1.0.1) with the register of dated decisions (internal document as used, in Polish, reviewed for personal data and internal identifiers)	1
`keys/`	keys of the ten questions: core, rubric, penalties, required bases, sources checked, version and status	10
`answers/`	the systems' answers as returned, with the tool trace where the tool records one	160
`cards/`	reference cards: A/B scores, error classes, origin and scope, the result of checking every citation, version date	160
`judges/`	outputs of five model judges and of the "judge with verification report" variants	1 319
`verification/`	verification-layer reports: citation check (code) and support check (model), per card and aggregate	2 184
`costs/`	price list of 18.10.2026 with sources and cost per answer (API equivalent)	—
`cases/skd-01/`	Benchmark 01 (pilot): the case, key, pilot metric, 17 answers from general-purpose models with their cards, `wyniki.json`; 12 cards marked preliminary	17
`data/`	machine-readable layer: `questions`, `keys`, `answers`, `cards`, `judgments`, `verification` as JSONL (English field names, Polish content) with a dataset card	6 files
`reports/`	the Benchmark 02 report (PDF, Markdown sources, `dane.py`) and the paper v1.2	2
`tools/`	scripts for running the systems and the judges, comparing judges with the cards, the verification layer, the JSONL export	—

The dataset on Hugging Face#

The same machine-readable layer is available as the dataset LexAlphaLab/lawbenchmark-b02 on Hugging Face: six tables (cards, answers, questions, keys, judgments, verification layer) as a test split. Downloading requires accepting a short notice asking not to train models on these data; acceptance is automatic. The files in the GitHub repository are available without a gate.

from datasets import load_dataset cards = load_dataset("LexAlphaLab/lawbenchmark-b02", "cards", split="test")

How to reproduce the results#

Tables of the report and of this site: python3 -I reports/benchmark-02/src/dane.py — reads cards/, keys/, judges/, costs/ and recomputes every table from them.
JSONL layer: python3 -I tools/export_jsonl.py — rebuilds data/ from the sources and checks the totals against dane.py.
Judges vs cards: python3 -I tools/porownaj_sedziow.py argon4v101 (identifiers: argon4v101, sol61v101, gemini31v101, gemini38flashv101, astra6v101).
Your own run on the questions: tools/eval_*.py — needs the provider's CLI or Vertex AI access. Repeating the run will not reproduce the answers of the study: models, their configurations and search results change over time.
The verification layer (tools/verification/) needs the legal-cite-pl library to read ISAP/ELI.

The scripts are in the repository as used, with paths adapted to its layout. The working names of the systems in the files (e.g. chatgpt_codex = GPT-6.1 Sol, argon4_high_vertex = Gemini 4 Argon) are spelled out in the repository README and in appendix D of the report.

Licences and the canary#

Material created by the project (specification, questions as formulated for the benchmark, keys, cards, judge instructions, verification-layer reports, reports and texts): CC BY 4.0; attribution "LexAlpha Lab, LawBenchmark.pl".
Code: MIT.
Model answers are outputs of third-party systems reproduced verbatim for evaluation, criticism and research; we claim no rights in them and do not relicense them. Fragments of other authors' works that a model may have repeated in an answer remain their authors' property.
Statutes and court rulings quoted in answers and keys are official documents and materials not protected by copyright (art. 4(2) of the Polish Copyright Act); this does not extend to commentaries and studies.
Canary: every file and record carries a canary string with a request not to use the dataset for model training. The licence does not forbid such use; it is a methodological request so that future measurements stay meaningful.

How to cite the dataset#

Piotrowski A., Koman A. (2026). LawBenchmark.pl · Benchmark 02 · pytania prawne (data). Version 1.1, 29.10.2026. LexAlpha Lab, KTZR. https://lawbenchmark.pl/dane

Report and methodology paper: reports. Contact: a.piotrowski@ktzr.pl.