Providers and builders of legal AI systems#
This concerns law-firm assistants, systems using retrieval-augmented generation (RAG) and other tools built on language models. Results of such systems do not enter the table of general-purpose models: access to sources, the document corpus and system instructions change the conditions, so we publish them in a separate category with a description of the configuration.
Testing on undisclosed questions
We offer to test a specified version of a system on the 40 undisclosed questions of the pool of 50, under the published scoring specification. Before we start we agree in writing: the system's configuration and version, access to tools and sources, the legal-state date, how the answers are produced, and scoring with the same card we apply to general-purpose models.
Terms agreed individually
Funding of the study, the time for the provider's comments on the result and the manner of publication are agreed individually, in writing, before the first call to the system. Two rules are fixed: the result and its description do not depend on who funds the study, and every published report states who funded it, whether the provider made the system available free of charge and what publication terms were adopted. The provider receives the result before publication and may report factual and technical errors.
System version and repeats
The report identifies the version tested, the configuration and the date of the run. A repeat test is identified as a separate run and does not replace the earlier result; we keep a register of all tests, including those aborted or voided, with the reason. An infrastructure failure justifies a repeat; a poor answer by the system does not.
Protecting the questions
We do not give the provider the question set or the keys to prepare the system; the questions are submitted by the researcher running the test. If the questions reach the provider's system through its interface, they may be visible in its logs: we therefore require an undertaking not to use them for training or tuning and a stated retention period, and where this cannot be assured we note it in the report. The undisclosed set is not secret indefinitely; after a few tests some questions are replaced. We use undisclosed control markers to detect leakage of the material; suspicions are verified before any conclusion is drawn.
Independence of the scoring
The funder of a study does not choose the answers scored, does not set the scoring and does not approve the conclusions. The report discloses the ties of the authors and assessors to the provider tested; where a tie is material, the assessor is excluded or the scoring is done by an external person. KTZR's and LexAlpha's own products stay outside the published results (conflict of interest).
The test is not a certification, an audit or a purchase recommendation. The result concerns the specified version and configuration of the system under the described conditions and does not confirm the correctness of answers in other matters.
Universities and researchers#
An independent second assessment
We invite lawyers to score answers independently under the published specification. The sample and the way inter-assessor agreement is computed are fixed before the scoring; κ is computed from the scores before disagreements are reconciled, and the disagreements are kept for analysis. A second assessment does not establish objectivity; it measures how repeatable the scoring is.
Replications and joint publications
Possible areas: replication on other models or dates, error analysis by the F/T/W/S classes, development of scoring and citation-verification methods, comparison with benchmarks from other legal systems. Authorship of publications follows actual contribution.
Data for reuse
The published questions, keys, cards, answers and judge outputs are publicly available and require no contact with the authors; the licences and reuse terms, including the separate status of model answers, are on data and code. The forty undisclosed questions and their keys stay outside the open set; we share them only within a joint study, under a non-disclosure undertaking.
Law firms and legal departments#
The results show what kinds of errors general-purpose models make and how web search changes them. They help plan your own check of a tool on the tasks done in your organisation. They do not replace an assessment of a specific deployment, its sources, configuration and oversight, and the result of a general-purpose model does not confirm the quality of a product built on it. No result relieves a lawyer of checking an answer before use.
We accept proposals of question categories and comments on the scoring criteria. In your first message do not send client data or material covered by professional secrecy.
Media and citation#
When citing, give the name and version of the study, the date, the configuration tested and a link to the results and methodology; citation templates are on the reports page. Charts and tables may be used under CC BY 4.0 with the attribution "LexAlpha Lab, LawBenchmark.pl". We explain the scope of the study and accept error reports. The result of a single study should not be presented as a general assessment of a system across all legal uses; a safe formulation is: "in the study [version, date], under [conditions], system X scored [result] on [tasks]".
Contact#
E-mail: a.piotrowski@ktzr.pl. In the subject line put one of: "System test", "Research cooperation", "Methodology comments", "Media".
Correspondence is handled by the law firm KTZR (Gdańsk), which is the controller of the senders' personal data; information on processing: KTZR privacy policy (Polish). The contracting party (KTZR or LexAlpha Sp. z o.o.) is agreed individually; "LexAlpha Lab" is the name of the initiative, not a legal entity.