LLMs are increasingly used in legal work, yet their reliability outside the United States is poorly documented. This expert-validated benchmark has 1,042 items across ten areas of law and three formats, and was used to test 15 models. Closed-question accuracy reaches 0.905, but free-text correctness never exceeds 0.45, and only about half of cited norms are correct. Models often sound responsive while being wrong, which is risky for non-experts.

Authors: Rubén Manrique, Michelle Castellanos, Jorge Morales, Juan David Gutiérrez, Antonio Barreto Rozo, Joaquín Vélez Navarro

Paper: https://arxiv.org/abs/2610.03639v1

Rss
Podden och tillhörande omslagsbild på den här sidan tillhör Craig Spencer Smith. Innehållet i podden är skapat av Craig Spencer Smith och inte av, eller tillsammans med, Poddtoppen.