Scoring methodology

What the benchmark measures#

Behind the benchmark is a practical question: how much checking is left for the lawyer once the model has answered. The benchmark does not settle that question. It measures two things that can be checked: whether the answer agrees with a key fixed before the run, and whether its claims about sources of law are accurate when checked against the official text as of the legal-state date. No label on this site means that an answer can go into a pleading unchecked.

Hence two axes of scoring. The first, hallucination, asks whether the claims about sources of law are true. The second, grounding, asks whether the citations in the answer could be confirmed in an official source and, in the web-search condition, whether the system opened such a source. The headline measure joins the two axes: the share of correctly grounded answers.

Benchmark 02 is a calibration set. Ten questions, one run and one reviewing lawyer are enough to show the criteria and their effect on real answers, not to rank products. The scoring rules were refined during the work after reading the answers; every change is dated in the decision log.

Questions#

The questions come from law-firm practice and have the form a lawyer types into a chat window: typos, abbreviations, sometimes a false premise. Language errors and false premises were kept on purpose, to see how systems react to such questions.

Each question has a type that decides how the core answer is scored:

• closed — giving the expected answer: "yes", "no" or "it depends on X"; eight of the ten questions in Benchmark 02;
• calculation — a numerical result within a set tolerance (P01);
• verification — whether the system checks someone else's claim or agrees with it (P08).

Six questions carry a built-in trap: a false or outdated premise that the system should name and correct. Example: P01 assumes that the petty-offence threshold rises with the minimum wage, while since 1.10.2023 it is a fixed amount. P05 works the other way round: an answer that was still right in May 2026 is outdated on the legal-state date. P09 and P10 are a pair with opposite suggestions ("mirror" in the specification): the same issue, once suggesting "not enough", once "enough". It measures whether the system changes its answer to match the asker's expectation.

Each question has an original text (exactly as the lawyer typed it) and, where the original is not enough for an unambiguous key, a benchmark text completed before the run with the missing facts. The run used the benchmark text where one exists: P01, P04 and P09 (P10, the reversed-suggestion version, has only a benchmark text); the other six questions were asked in their original wording. Each question page shows both versions. Full texts and keys: the ten questions.

The pool holds 50 questions. 10 are published; the other 40 stay undisclosed so that future editions can use questions the models have not seen.

How we establish the right answer: keys#

The key of a question is fixed before the answers are collected and contains: the core answer, the rubric (elements that earn points), penalties (errors that cost a point), required bases (provisions without which the answer is incomplete) and the expected calibration (categorical or conditional, with the condition). Rules of the key:

1. Every legal basis has a source: ISAP or ELI (Poland's official legislation databases), EUR-Lex, SAOS (the public court-rulings database) or sn.pl (the Supreme Court's website). Quotations of provisions are copied from the official text, not from an auxiliary database or from memory.
2. A ruling in the key has a case number, court, date, a proposition of law quoted verbatim from the ruling and a source reference identifying the passage consulted. Summaries and commentaries are not a source of the thesis.
3. The key covers only what the question asks. A related issue is not included in the rubric or the penalties, however important in practice.
4. The key is not changed after the systems' answers are known. A change requires a new version with a date and a reason.
5. Only keys approved by a lawyer enter the run; the "verification" field says plainly what was not checked.

The legal-state date of every key is 1 October 2026. Answers are scored against the law on that date. Two keys (P09, P10) were changed after the answers were collected, when the Supreme Court's case law turned out to be more nuanced than the key; the core became conditional, and the change is described in the report and in the decision log. This is a departure from rule 4. The conditional core of P09 and P10 follows from the interpretation of the contract (art. 65 of the Civil Code) adopted in Supreme Court case law, not from a divergence between rulings; the questions are therefore not marked as disputed and criterion A5 does not apply to them.

Non-binding documents (European Commission guidelines, codes of practice) enter the key as a separate rubric element, never as a penalty: not citing them is no error, and only attributing binding force or content they do not have counts as a hallucination.

Warunki badania#

Each model answered the same ten questions in two conditions:

• W0 — no tools: the same short, neutral preamble and the question. The answer comes from the model's weights alone.
• W1 — with the provider's own web search: Codex CLI search for OpenAI, WebSearch and WebFetch in Claude Code for Anthropic, google_search and url_context in Vertex AI for Google. The preamble adds one sentence asking for the addresses of sources.

The preamble asks neither for brevity nor hints at traps or disputes, and does not change during the run. One run, one answer per question; the answer is stored as returned by the system, with the tool trace where the tool records one.

Answers were collected on 10, 13, 17 and 18 October 2026; each carries its date, model identifier and settings in the metadata. The firm's own products do not take part in the benchmark.

Scoring criteria#

Every answer has a card with a score for each criterion and evidence from the source. The card is split into block A (criteria checked against the key and the official text) and block B (criteria requiring judgement); the two blocks are reported separately and never averaged. In block A too, some scores need a lawyer's decision: the existence of a source and the wording of a provision are compared directly, but whether the content attributed to a source fits its text is judged by the lawyer against the source and the key.

Block A — criteria checked against the key and the source#

A % = (points − penalties) / points available; points available exclude "not applicable" criteria. With a trap, discussing the correct legal state instead of the one assumed in the question is an answer, not a digression. Silently accepting a wrong provision number from the question costs a rubric element if the key has one; a penalty applies only when the answer reinforces the error.

Block B — criteria requiring judgement#

• B1, usefulness (0–2): does the lawyer know what to do next. An error in a practical instruction that changes the action (fee amount, deadline, authority, form) lowers B1 to at most 1, even when the system cited no provision.
• B2, form of the conclusion (0–1; until 29.10.2026 called "calibration"): whether the form of the conclusion matches the key. With a categorical key the point goes to an answer that states the conclusion unequivocally; blurring a clear thesis ("it depends on interpretation", "consult a lawyer") without saying what it depends on costs the point. With a conditional key the point goes to an answer that ties the conclusion to the condition; the system may also ask a clarifying question, provided it identifies the condition that determines the answer.

B2 measures the form of the conclusion against the key, not its correctness or the model's actual confidence, hence the rename after external review: a wrong but categorical answer to a categorical key has B2 = 1, and the error itself lowers block A. Block B does not duplicate block A; a profile of "A low, B high" is meant to be visible, not smoothed out. "This is not legal advice" boilerplate does not affect the score.

Hallucination and error classes#

A hallucination is any claim about a source of law that disagrees with the official source as of the legal-state date. One such claim gives A6 = 0 and the label "hallucinated" for the whole answer, even when the conclusion itself is right. The label "no hallucination" means no detected hallucination: 1,436 of 1,454 citations were checked, and unchecked citations did not lower the score (one card has an unchecked citation under the label "correctly grounded"; report, chapter 8).

Next to the score the card records every error class detected, each with evidence (what was cited, what the source says, where it was checked):

In Benchmark 02 class T appeared in 66 answers, W in 35, F in 8, S in 9. One answer may carry several classes, so the numbers do not add up to the 91 hallucinated answers.

Merely omitting a condition or an element of a provision is not a false claim: it does not lower A6, it costs points in the rubric and the core. The boundary with class T: an answer that describes art. 787 of the Code of Civil Procedure incompletely, without claiming that the provision covers only what it described, commits an omission; an answer that states the provision applies only in a situation to which it is not limited attributes a narrowed scope and gets class T. An answer with no citations at all has A6 = 1. An empty answer has A6 = 0 without a label. When the provider's filter blocked the text (e.g. Vertex RECITATION), the answer goes to a separate category, "provider block": neither a hallucination nor a correct answer (one case in Benchmark 02, P07 Gemini 4 Argon W1).

For every error the card records two more things:

• origin — memory (no confirmed reading of the source: the no-tools condition, tools unused, or the act absent from the tool trace; the category name expresses the hypothesis that the content comes from the model's weights), source read (the trace shows the right official source was opened and the content is wrong anyway), secondary source (the error repeats an error or oversimplification on a website the system opened), undetermined;
• scope — core, when the error concerns the question asked, the required bases or a rubric element; peripheral, when it appears in a side part (digression, analogy, background) next to a correct conclusion. In Benchmark 02: 57 answers with an error in the core, 34 with a peripheral error only.

In condition W1 the card also records whether the system used an official source at all:

A correctly grounded answer#

An answer is correctly grounded when it meets three conditions at once: a full core (A-R = 2), no detected hallucination (A6 = 1) and at least one citation that the assessor confirmed in an official source. The assessor checks, not the system: the label means that no hallucination was detected and at least one citation was confirmed in an official source, not that the model read it. In the no-tools condition the model opened no source; whether a model with web search opened an official source is shown in a separate table in the hallucination section. An answer without wrong citations but with an incomplete core, or without any confirmed citation, is not correctly grounded; neither is an answer with a correct conclusion and one false citation.

Denominators: each model has 10 answers per condition; 80 answers per condition; 160 answers in total. In Benchmark 02, 19 of 80 answers were correctly grounded in W0 and 27 of 80 in W1; 51 and 40 were hallucinated. The rest are answers without a detected hallucination but with an incomplete core or without a confirmed citation, plus one provider block (P07, Gemini 4 Argon, W1), which stays in the denominator of 80. 160 answers are not 160 independent observations: the same ten questions repeat for every model, so the result depends on the choice of questions. Before quoting numbers: chapter 8 of the report.

1,436 of 1,454 citations across all answers were checked; each card states "checked n of m".

How we check the answers: the reference card#

1. A model (Claude) pre-fills the card: it looks up every citation in ISAP, ELI, SAOS, sn.pl or the court's portal, compares the content with the answer, and proposes scores and error classes with evidence.
2. A lawyer reviews every card and approves or changes the verdict. Borderline decisions go into the dated decision log in the specification, so that later cards are scored the same way.
3. The card records the specification version it was scored under, the date and the change history. Cards scored under an older version are not compared with new ones without recomputation.

The model that fills in the cards belongs to the same family as three of the systems scored. Among the model judges, too, some score answers from models of their own family and, in a few cases, their own answers (report, chapter 6). That is why a human reviews every card, and on the same cards we measure agreement with five model judges from three families.

Model judges and agreement with the cards#

The same answers were scored separately by five models with web search, in separate calls, under specification 1.0.1. For each judge we compute agreement with the cards and Cohen's κ for the label "hallucinated", separately for the pilot set (P05–P07, P12, P13) and the extension (P01, P04, P08–P10), always on the cards that judge scored (column n). Four models scored 147–160 cards; GPT-6 Astra scored 49, so its κ is not comparable with the others. A false alarm is when the judge labels an answer hallucinated but the card does not; a miss is the reverse.

κ does not prove that the cards are right; it measures agreement between assessors beyond chance. On the extension the κ of four judges ranges from 0.58 to 0.90 (79–80 cards), on the pilot from 0.71 to 0.97 (68–80 cards). Confidence intervals from a bootstrap over questions and the TP/FP/FN/TN matrices are in appendix E of the report. Judges that searched little agree least with the cards, and almost none reports which citations it did not check. The report also tests a "judge with verification report" variant (card meta v3), in which the judge receives the verification-layer report as an aid, not a verdict. Change in agreement with the cards: GPT-6.1 Sol κ 0.80 → 0.89 on 128 shared cards, Gemini 4 Argon 0.95 → 0.88 on 116; Argon's decline concerns class W, which the code cannot establish.

Verification layer#

Alongside the judges runs a two-step verification layer. The citation check is code: it extracts the citations from the answer, tries to establish the wording of the unit on the legal-state date from ISAP/ELI (the legal-cite-pl library) and the ruling from SAOS, and checks existence, identifier and wording; cases it cannot decide are marked undetermined. The support check is a model that receives the claim made in the answer and the text of the source and judges whether the source supports the thesis. The result goes to the judge as an auxiliary report, never as a binding verdict.

On eight questions outside the tuning set the layer detected 84 of the 147 errors recorded in the cards (57 %). Most misses arise from differences in definitions (a narrowed scope of a norm, which the card counts as T) and class W: too many amendments of the Code of Civil Procedure entered into force between consolidated texts for the code to establish the wording on the date. The false-alarm rate, the results before and after the input fixes, and the denominators are in chapter 7 of the report; the code and the per-card reports are in the repository.

Cost#

For every answer we compute the API-equivalent cost: what the same use of tokens and tools would cost at the provider's public price list of 18.10.2026. Systems run from a subscription (Claude Code, Codex) do not actually incur these amounts; the Vertex AI figures are actual charges. Cost does not affect the scores and is not part of any quality ranking.

An auxiliary metric: USD per correctly grounded answer, i.e. the cost of the run divided by the number of such answers; next to it we always show the cost of the run, the number of answers and the lower-bound mark. A system with no such answer gets "n/a", not "∞". With ten questions and part of the costs known only as a lower bound, the metric is not an assessment of a product's value for money. Limitations: cached tokens counted as the tool reports them; Claude Code does not record the number of searches (in W1 the search fee is "n/a" and the cost is a lower bound "≥"); the Vertex script did not record content tokens from url_context. The full cost of the study (answers, judges, verification layer) is at least USD 170.70 at list prices, excluding the lawyer's time.

Limitations#

• Small sample. Ten questions, five from civil procedure. The result depends on the choice of questions; 160 answers are not 160 independent observations.
• One run. Each system was run once per question. We do not know the variance between runs of the same model; the W0 and W1 results come from different days.
• One lawyer. One person reviewed the cards. There is no human–human agreement as a reference point for the κ of the model judges.
• The cards were pre-filled by a model from the family of one of the providers scored.
• The rules were refined along the way. Three changes to the A6 definition between 9 and 11.10.2026, further clarifications until 25.10; two keys were changed after the answers were collected.
• Provider configurations differ (subscription CLIs, API, different search tools), and part of the costs is a lower bound.
• The questions have been public since October 2026, so later measurements on the same set may be contaminated by exposure.
• Model line-up. Claude Fable 5.1, Anthropic's newest model, did not enter the run of 17–18.10.2026: it has only five pilot answers in W0 and no full W1 set, so the report does not cover it. It is scheduled for the next run in both conditions. The line-up is a snapshot of October 2026; newer model versions are not included.

For these reasons the results should be read as an illustration of the criteria, not a ranking. Plan for the next editions: new, undisclosed questions; keys approved before the answers are collected and a frozen specification; at least three runs per system with confidence intervals; a second lawyer on a sample of cards; citation verification as a separate step before judging.

Versioning and identifiers#

• Scoring specification: 1.0 of 11.10.2026 (after three changes to the A6 definition on 9–11.10); 1.0.1 of the same day adds boundary examples, and cards P06 and P12 were recomputed; 1.0.2 (18.10.2026): cost, rules from the review of the extension cards, the "scope" field of a hallucination, provider block, soft law; 1.0.3 (25.10.2026): clarification of the "scope" field, completed in the pilot cards; 1.0.4 (29.10.2026): new name of criterion B2. The scoring rules have not changed since 1.0.1, which is why the cards state version 1.0.1.
• Keys: each has a version number, a date and an approval status; in Benchmark 02 versions v1 to v4. Changes after the answers were collected (P09 v4, P10 v3, 18.10.2026) are described.
• Answers: identifier Pnn_<system>_<condition>_r3, date, model identifier, settings and tool trace in the metadata.
• Cards: specification version, date and change history in every card; state of the cards used in the report: 25.10.2026.
• Report: Benchmark 02 version 1.1 of 29.10.2026; changes in the repository's CHANGELOG.md.

Contamination and the canary#

Every key, card, answer, judge output and data record carries a canary string (a distinctive text marker, following the BIG-bench convention) with a request not to use these files for model training. The canary makes some cases of exposure detectable, but not detecting it does not prove the absence of contamination. We therefore treat the published set as potentially contaminated from the day of publication and base the next editions on previously undisclosed questions.

Authors, funding, conflict of interest#

The benchmark is run by LexAlpha Lab, a joint research initiative of the law firm KTZR (Gdańsk) and LexAlpha Sp. z o.o. (a Polish limited liability company, Warsaw), which develops a commercial AI platform for lawyers. Authors: Adam Piotrowski (attorney-at-law, partner at KTZR; methodology design, keys, cards, report) and Andrzej Koman (CEO of LexAlpha). The study is funded by the two organisations themselves.

Conflict of interest. LexAlpha Sp. z o.o. offers commercial AI tools for lawyers built on language models of the same providers whose general-purpose models the benchmark scores; the company therefore has an economic interest in the results and in how they are presented, and its CEO is a co-author of the report. KTZR uses these providers' tools on commercial terms. Benchmark 02 does not include either organisation's own products (decision of 17.10.2026). The Benchmark 01 pilot of September 2026 scored three LexAlpha modules and two legal-research systems of other companies; their results are not published. The exclusion limits the direct conflict of scoring one's own product, but does not remove the authors' economic ties. We therefore score only general-purpose models in publicly available configurations, and the keys, cards, answers and code are public so that every score can be checked independently. The results of general-purpose models say nothing about the quality of LexAlpha or KTZR products; a product with its own retrieval and instructions requires a separate study.

Roles. Questions, keys, card review and publication decisions: Adam Piotrowski; Andrzej Koman co-authored the report. No model provider took part in preparing the study or funded it; the authors did not share results with providers before publication. Until 29.10.2026 the site described LexAlpha Lab as a laboratory of the law firm KTZR; the current description reflects the involvement of both organisations.