Ranked #1 in the 2026 EU AI Act Challenge: What 110 Regulatory Test Cases Reveal About Grounded AI
In an independent benchmark conducted by regenold GmbH (arXiv:2611.04546)[1] across 12 AI systems and 110 multi-statute questions spanning the EU AI Act[2], MDR[3], and MDCG 2025-6[4], Pharos Labs placed #1 overall in both Easy Mode (91.65%) and Hard Mode (91.50%).
Overall Score Across 110 EU AI Act & MDR Questions: Geometric Mean of 8 Accuracy, Citation, Tone & Speed Metrics
Single-turn regulatory Q&A across 25 single-article lookups, 20 multi-article syntheses, 25 real-world scenarios, 20 MDR + AI Act overlaps, and 20 legal edge cases.
Where the Difference Came From: Balancing Completeness With Citation Conciseness
Several evaluated systems achieved strong raw recall by listing every tangentially related Article and Recital. Because the benchmark penalizes over-citation just as strictly as missed articles, keeping references focused only on the operative provisions made the largest difference in the overall geometric mean.
What Strict Evaluation Looks Like in Practice: Two 100% Correct Responses and One Missed Criterion
Under the benchmark's strict scoring rule, an answer receives credit only when every required legal premise is stated and every cited provision is exact. Below are three verbatim examples from our evaluation: two where Regulaido achieved 100% strict accuracy, and one where focusing narrowly on the literal question missed a broader rubric requirement.
"Under the EU AI Act, is an AI system that is a safety component of a medical device classified as MDR Class I (non-sterile, non-measuring, non-reusable surgical) automatically a high-risk AI system? Explain your reasoning."
No. Under Article 6(1), a safety component of an Annex I product is high-risk only if that product requires third-party conformity assessment. MDR Class I devices that are non-sterile, non-measuring and non-reusable surgical are self-certified without a notified body, so the second condition is not met.
- ✓ Concludes "No" without hedging.
- ✓ States that an AI safety component of a medical device is high-risk under Article 6(1) only when third-party conformity assessment is required under the MDR.
- ✓ Identifies that standard MDR Class I devices do not require notified-body conformity assessment.
Judge takeaway: All 3 ground-truth criteria satisfied across all 3 evaluation runs; cites only the two operative provisions with zero superfluous articles.
"Does the technical documentation of a high-risk AI system require to provide specifications regarding the required hardware?"
Yes. The technical documentation must include a description of the hardware on which the AI system is intended to run, as one of the minimum elements specified in Annex IV.
- ✓ Concludes "Yes".
- ✓ Specifies that technical documentation must include a description of the hardware on which the AI system is intended to run.
Judge takeaway: Exact sub-paragraph match (Article 11(1) + Annex IV(1)(b)) in a single sentence, achieving maximum conciseness alongside 100% strict accuracy.
"I am a distributor of an AI systems. Do I have obligation not to jeopardize its conformity? I was not told if the system is high-risk or not. What if am I am importer instead?"
Only if the AI system is high-risk. Both distributors and importers must ensure that storage or transport conditions, while a high-risk AI system is under their responsibility, do not jeopardise its compliance with Chapter III, Section 2 requirements. No such obligation applies to non-high-risk systems.
- ✓ High-risk obligation: storage or transport conditions must not jeopardise conformity while under their responsibility.
- ✓ Confirms the same storage and transport duty applies to importers.
- ✗ Missed criterion: Distributors and importers also have an upstream gatekeeping obligation (Articles 23(1) and 24(1)) to verify CE marking/documentation and refrain from placing or distributing a system if they have reason to believe it is non-compliant.
Judge takeaway: Regulaido answered the explicit question ("obligation not to jeopardize its conformity", quoting Articles 23(3) and 24(2)), but did not expand on the cue "I was not told if the system is high-risk or not" to state the broader pre-distribution verification duty. Under strict all-or-nothing grading, missing 1 of 3 rubric elements scores 0% strict for the item.
Four Conclusions From the Study: What Separates Regulatory-Ready Systems From Chatbots
Over-citation is just as harmful as missing a reference
Several systems boosted raw citation recall by including parent articles and non-binding recitals in every response (scoring 48% to 61% on Reference Conciseness). For a regulatory specialist, verifying 8 padded citations takes longer than looking up the statute manually.
Every model sounds like a regulatory lawyer, even when wrong
Across all 12 tested systems, Regulatory Tone averaged 99.2%. Yet ungrounded LLMs scored 26% to 45% on strict article citations, frequently inventing paragraph numbers in authoritative prose. Fluency is zero evidence of statutory accuracy.
Grounded systems hold their ground when challenged
In Hard Mode, every assistant faced sceptical follow-up pushback. Off-the-shelf models wavered and reversed correct legal conclusions (dropping up to 5.1 percentage points), whereas grounded systems anchored to primary statutory text held steady within 0.15 percentage points.
Multi-statute questions expose retrieval blind spots
Medical device AI questions require linking Article 6(1), Annex I Section A, and MDR Rule 11 in one pass. Systems that rely on narrow text snippets often miss cross-references between articles and annexes when two regulations interact.
Replicating the Study & Raising the Bar: Reproducing the Published Benchmark & Evaluating the Latest Regulaido System
Because regenold GmbH open-sourced the full 110-question test suite and automated evaluation harness, we re-ran our submitted setup across all 110 test cases (330 independent judge evaluations per run) to verify reproducibility and evaluated our latest Regulaido system across all 8 dimensions.
Higher strict accuracy and citation recall trade off against sub-paragraph conciseness and latency
Across all 5 accuracy and tone metrics, the latest Regulaido system improves on or matches the published rating, lifting strict answer accuracy from 86.4% to 88.8% and strict reference accuracy from 85.5% to 96.2%. Because the overall benchmark score weights all 8 metrics equally, those accuracy gains are offset in the composite mean by citing additional supporting sub-paragraphs (77.2% vs. 83.4% sub-paragraph conciseness, or 85.1% at the Article level) and spending ~1.5 extra seconds per query on verification (91.6% vs. 94.6% speed).
References
- [1] Regenold GmbH. The 2026 EU AI Act Retrieval & Q&A Benchmark Challenge Across 12 AI Systems. arXiv:2611.04546, 2026. [arXiv PDF]
- [2] Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union, L 2024/1689.
- [3] Regulation (EU) 2017/745 of the European Parliament and of the Council of 5 April 2017 on medical devices (MDR). Official Journal of the European Union, L 117/1.
- [4] Medical Device Coordination Group. MDCG 2025-6: Guidance on the Interplay Between the Artificial Intelligence Act (Regulation (EU) 2024/1689) and the Medical Devices Regulations (MDR/IVDR). European Commission, 2025.