This is a pilot, not part of Benchmark 02. The case was scored with a different metric and through different interfaces, so its numbers should not be compared with the results of the ten questions. Five cards were reviewed by a lawyer in September 2026; the 12 cards of 8–9.10.2026 are preliminary scores filled in by a model, marked as such in the table and in each card. The pilot also scored five legal-research systems: three LexAlpha modules (LexAlpha co-organises the project) and two systems of other companies. Their results are not part of this publication.
The case#
The facts and the instruction exactly as given to the systems (in Polish, typos kept). The case is fictional.
SKD ##1 1.1 Stan faktyczny 1) Kredytobiorca zawarl umowę kredytu konsumenckiego w czerwcu 2022 r. na kwotę 50 000 zł, przy czym całkowita kwota kredytu wynosiła 40 000 zł 2) 10 000 zł to kredytowane koszty, od których bank również naliczał odsetki. 3) W czerwu 2024 r. kredytobiorca złozyl reklamacje do banku, wskazując, że postanowienia które pozwalają naliczać bankowi odsetki od kredytowanych kosztów są abuzywne, w z.w czym bank powinien rozliczyć nadpłacone środki z przeszłości i zmienić harmonogram na przyszłość. 4) Bank odmówił uwzględnienia reklamacji oraz zmiany rozliczeń umowy. 5) Kredytobiorca sam dokonał nowego przeliczenia haramogramu i od lipca 2024 r. spłacał do banku niższe raty tj. z odsetkami liczonymi od kwoty pomnniejszone o kredytowane koszty. 6) W grudniu 2024 r. ze względu na (zdaniem banku) niedopłatę bank wypowiedział umowę. 7) W maju 2026 r. kredytobiorca złożył do banku oświadczenie o skorzystaniu z sankcji kredytu darmowego, powołując się na najnowszy wyrok TSUE, zgodnie z którym bank może naliczać odsetki tylko od calkowitej kwoty kredytu. 8) Bank odmówił uwzględnienia stanowiska konsumenta wskazując, że TSUE nie przesądził od braku możliwości pobierania odsetek od kredytowancyh kosztów a ponadto nawet gdyby uznać stanowisko konsumenta za uzasadnione to i tak od momentu wypowiedzenia umowy - czyli jej wykonania zgodnie z ustawą minął rok i termin na złożenie oświadczenia wygasł 9)kredytobiorca nie zgodził się z tak przedstawionym stanowiskiem banku. 1.2 W tak przedstawionym stanie faktczynym. 1) Ustal na jaką datę trzeba ustalic treść przepisów dla poszczególnych zdarzeń prawnych. 2) Ustal przepisy prawnej mające zastosowanie w tej sprawie. 3) Podaj dokładną treść przepisu (zacytuj), który będzie mieć zastosowanie w konkretnej dacie przy analizie 4) Przenalizuj staowisko każdej ze stron, 5) Wskaż w świetle orzecznictwa oraz poglądów doktryny, które stanowisko jest bardziej uzasadnione i ma większe szanse na obronę w sądzie. 6) Dla każdego ze stanowisko przedstaw tok rozumowania oraz odpowiednią argumetnację prawną. 1.3 Oczekiwany format odpowiedzi: 1) Przepisy mające zastosowanie w sprawie 2) Daty na jaką należy ustalić treść przepisów 3) dosłowna treść przepisów (cytat) 4) analiza stanowiska każdej ze stron (max 10 zdań dla każdego stanowiska) 5)wskazanie, które stanowisko ma większą szansę na obronę w sądzie wraz z uzasadnieniem i przywołaniem odpowiedniego stanowiskka doktryny i orzecznictwa 1.4 Założenia 1) Skup się wyłącznie na zagadnieniu terminu do złożenia oświadczenia SKD. Kwestię kredytowania kosztów uwzględniaj tylko jeżeli ma to wpływ na zagadnienie terminu do złożenia oświadczenia SKD. 2) Przyjmij, że wypowiedzenie nie zawierało braków formalnych i dla jego skuteczności może mieć ew. wpływ zagadnienie kredytowanych kosztów. 3) Jeżeli któreś z zagadnień prawnych nie jest jednolicie przesądzone i wskaż na możliwe rozwiązania
What the case measures#
The instruction tells the system to focus solely on the time limit under art. 45(5) of the Consumer Credit Act and to consider the credit charges financed through the loan only where they affect that limit. Whether interest on financed credit charges was lawful may decide whether there were arrears, and so whether the termination was effective. Whether the termination starts the time limit depends in turn on how "performance of the agreement" in art. 45(5) is construed. The task requires taking this dependency into account. Add to this the dates relevant to each event, verbatim quotations of the provisions and the identification of disputed issues.
The pilot metric#
| Criterion | Question | Points |
|---|---|---|
| A1 | Was the date for determining the applicable wording of the provisions set correctly, separately for each event? | 0/1 |
| A2 | Were the applicable provisions identified, including art. 75(1) of the Banking Law and art. 385¹ of the Civil Code? | 0/1 |
| A3 | Are the quotations of provisions verbatim or marked as excerpts? | 0/1 |
| A4 | Did the model notice that the financed credit charges matter for the time limit, and name the mechanism? | 0/1 |
| A5 | Were the disputed issues identified candidly? | 0/1 |
| A6 | Does the answer avoid citing non-existent sources or sources with content they do not contain? Since 11.10.2026 binary, with classes F, T, W, S as in Benchmark 02. | 0/1 |
| A7 | Did the model address every point of the instruction, in the required format? | 0/1 |
| A8 | Did the model stay within the scope of the task? | 0/1 |
| B1 | Analysis of the consumer's and the bank's positions, separately: none, a summary, or an analysis of the arguments. | 0–2 per party |
| B2 | Indication of which position is more likely to prevail: without reasons, with superficial reasons, or with reasoning grounded in case law and legal scholarship. | 0–3 |
Block A has 8 points, so one point is 12.5 percentage points; block B has 7 points. The blocks are not summed. The difference Δ = A − B, computed from the rounded percentages, shows the gap between the quality of the argument and the block A criteria; the reason is visible in the card. The label "hallucinated" follows from A6 = 0. Error classes: F, the source does not exist; T, a real source with content it does not contain; W, wording from another date; S, a distorted identifier. "S (unconfirmed)" means the case number was not found, without deciding whether it is distorted or invented. Full metric: file METRYKI.md in the repository.
Results#
Cards reviewed by the lawyer
Answers from the providers' chat apps, cards of 13.09.2026, reviewed by the lawyer in September. 3 of 5 answers were labelled hallucinated. The numbers differ from the September article for two reasons: on 11.10.2026 the cards were recomputed under the binary A6 rule approved by the lawyer, and on 29.10.2026 the lawyer classified the case number "judgment of the Court of Appeal in Warsaw of 10 May 2022, V ACa 810/21" as non-existent (class F): it is absent from SAOS, from the internal case-law database and from the court's own portal. The change history is in each card.
| Model | Condition | Block A | Block B | Δ A−B pp | Hallucinated (per card) | Files |
|---|---|---|---|---|---|---|
| Claude Opus 5.0 (High) Anthropic · chat app | with web search | 7/8 · 88 % | 7/7 · 100 % | -12 | no | card · answer |
| Claude Fable 5.1 (High) Anthropic · chat app | with web search | 6/8 · 75 % | 7/7 · 100 % | -25 | yes T, S | card · answer |
| Gemini 3.1 Pro Google · chat app | without web search | 6/8 · 75 % | 5/7 · 71 % | +4 | no | card · answer |
| Gemini 3.6 Flash Google · chat app | with web search | 3/8 · 38 % | 5/7 · 71 % | -33 | yes T | card · answer |
| Gemini 3.6 Flash Thinking Google · chat app | without web search | 2/8 · 25 % | 4/7 · 57 % | -32 | yes F | card · answer |
Preliminary scores, not reviewed by the lawyer
Answers collected on 8–9.10.2026 by scripts: GPT-6.1 Sol through Codex CLI and open models through Ollama. The cards were filled in by a model (Claude); 7 of 12 were labelled hallucinated. The results may change after the lawyer's review.
| Model | Condition | Block A | Block B | Δ A−B pp | Hallucinated (per card) | Files |
|---|---|---|---|---|---|---|
| Bielik 11B v3 (Q4_K_M) SpeakLeash · Ollama, local | no tools | 1/8 · 13 % | 4/7 · 57 % | -44 | yes T | card · answer |
| Bielik 11B v3 (Q8_0) SpeakLeash · Ollama, GCP cloud | no tools | 1/8 · 13 % | 3/7 · 43 % | -30 | yes T | card · answer |
| Bielik 11B v3 with web tools SpeakLeash · Ollama, local | search available, not used | 1/8 · 13 % | 3/7 · 43 % | -30 | yes T, S (unconfirmed) | card · answer |
| Bielik 11B v3, forced search SpeakLeash · Ollama, local | instructed to search; did not use the tool | 0/8 · 0 % | 3/7 · 43 % | -43 | yes T, S (unconfirmed) | card · answer |
| GPT-6.1 Sol, reasoning medium OpenAI · Codex CLI | with web search | 8/8 · 100 % | 7/7 · 100 % | +0 | no | card · answer |
| GPT-6.1 Sol, reasoning high OpenAI · Codex CLI | with web search | 8/8 · 100 % | 7/7 · 100 % | +0 | no | card · answer |
| GPT-6.1 Sol, reasoning medium (repeat) OpenAI · Codex CLI | with web search | 7/8 · 88 % | 7/7 · 100 % | -12 | no | card · answer |
| GPT-6.1 Sol, reasoning medium OpenAI · Codex CLI | no tools | 6/8 · 75 % | 6/7 · 86 % | -11 | no | card · answer |
| GPT-6.1 Sol, reasoning medium (repeat) OpenAI · Codex CLI | no tools | 6/8 · 75 % | 7/7 · 100 % | -25 | no | card · answer |
| Qwen 3.5 9B Alibaba · Ollama, local | no tools | 1/8 · 13 % | 4/7 · 57 % | -44 | yes F, T, S | card · answer |
| Qwen 3.5 9B with web tools Alibaba · Ollama, local | with web search | 1/8 · 13 % | 4/7 · 57 % | -44 | yes T, S | card · answer |
| Qwen 3.8 27B Alibaba · Ollama, GCP cloud | no tools | 1/8 · 13 % | 6/7 · 86 % | -73 | yes T, S | card · answer |
One answer per system and condition. "Repeat" variants are a second attempt of the same model under the same conditions, shown to make the run-to-run variability visible. The no-tools answer of GPT-6.1 Sol with ChatGPT account apps enabled was left out, because the apps gave access to a search engine.
The key in short#
- Relevant dates: unfairness of the terms as at the conclusion of the agreement (June 2022, art. 385² of the Civil Code); effectiveness of the termination as at December 2024; art. 45(5) of the Consumer Credit Act with reasons for the date chosen (the provision has not changed since 2011).
- Required bases: art. 45(1) and (5) of the Consumer Credit Act, art. 75(1) of the Banking Law, art. 385¹ § 1 of the Civil Code.
- Disputed as at the date of the declaration (May 2026): what starts the time limit under art. 45(5) (performance by the lender, by both parties, or termination; proceedings on the question of law III CZP 15/25 before the Supreme Court stayed); whether the consumer could lower the instalments on their own without a ruling on unfairness; whether termination is "performance" of the agreement.
- Case law: the CJEU judgment C-744/24 of 23.04.2026 concerns interest on financed credit charges and does not decide the time limit; the preliminary references C-763/25 and C-828/25 on the time limit were pending as of 13.09.2026.
The key describes the state as of 13.09.2026 and has not been updated since. Matters pending then may have been decided; the case is not a description of the current line of case law and is not legal advice.
Data#
The case, the key, the pilot metric, the 17 answers and cards and the summary wyniki.json are in the repository under cases/skd-01/, licensed like the rest of the dataset (licences). Every file carries a canary, a unique string that makes some uses of these data for model training detectable. The case is now public, so it is no longer suitable for testing new models.