EuroExec: Frontier Language Models Fall Short of Expert Judgment on European Executive Decision Tasks
Identification
- Clé publique
- radar:r-875fb71644bc99e84d6391c35c32d274
- Auteurs
- Arnal, Pau · Denfir, Khaled · Smahliuk, Danylo · Avhad, Amrut · Castro, Marcus A.
- Année déclarée
- 2026
- Langue déclarée
- und
- Type source
- Preprint
- Première détection
- 2026-11-13T08:56:58.903006+00:00
- Dernière observation des métadonnées
- 2026-11-13T08:56:58.903006+00:00
- Dernière vérification HTTP
- Non effectuée
Résumé documentaire — langue originale
Frontier LLMs are increasingly put to use on open-ended complex questions, different in nature from the ones they are typically evaluated on. We dedicate more than 4,000 human expert hours to evaluate a selection of six frontier LLMs on a member of this class of problems: EuroExec, our introduced human expert-based benchmark composed of 413 open-ended long-form European executive tasks authored by 47 vetted domain experts, each question drawn from experience in a real case. Every response is manually evaluated through a multi-attribute rubric, an item-specific checklist of requirements, and a preference rank ordering, extracting an aggregate metric "Solve Rate". The strongest model solves only 56.9% of tasks, while expert-written reference answers judged blindly are solved at near-ceiling levels and are preferred over every model response in 74% of direct rankings, placing frontier generative systems well below the professional standard of work they are already used for. We see that the best way to extract this kind of conclusion is by employing human evaluators, carefully checking their consistency through rigorous statistical analysis, and observe that automatic measurements also fall short when evaluating on this case of real-world open-ended problems with a subjective ground truth.
Identifiants
- DOI
- 10.48550/arxiv.2608.04549
- ARXIV
- 2608.04549
Repères documentaires
- Famille documentaire
- unknown
- Types déclarés par les sources
- Preprint
- Éditeurs
- arXiv
Les mentions d’entités ne constituent pas une preuve de reconnaissance juridique. La juridiction étudiée est distincte du pays d’affiliation des auteurs.
Sources
DataCite · Document 1
Citer cette référence
Arnal, Pau; Denfir, Khaled; Smahliuk, Danylo; Avhad, Amrut; Castro, Marcus A.. EuroExec: Frontier Language Models Fall Short of Expert Judgment on European Executive Decision Tasks. 2026. https://doi.org/10.48550/arxiv.2608.04549
JSON · CSL-JSON · BibTeX · RIS · CSV · JSON-LD ·
APA, Chicago, ISO 690 : formats bibliographiques fondés sur les champs disponibles. OSCOLA / Bluebook : données structurées uniquement ; aucune conformité juridique complète n’est revendiquée sans vérification des éléments nécessaires.
- APA
- Arnal, Pau; Denfir, Khaled; Smahliuk, Danylo; Avhad, Amrut; Castro, Marcus A.. (2026). EuroExec: Frontier Language Models Fall Short of Expert Judgment on European Executive Decision Tasks. https://doi.org/10.48550/arxiv.2608.04549
- Chicago
- Arnal, Pau; Denfir, Khaled; Smahliuk, Danylo; Avhad, Amrut; Castro, Marcus A.. EuroExec: Frontier Language Models Fall Short of Expert Judgment on European Executive Decision Tasks. 2026. https://doi.org/10.48550/arxiv.2608.04549
- ISO 690
- Arnal, Pau; Denfir, Khaled; Smahliuk, Danylo; Avhad, Amrut; Castro, Marcus A.. EuroExec: Frontier Language Models Fall Short of Expert Judgment on European Executive Decision Tasks. 2026. https://doi.org/10.48550/arxiv.2608.04549
Warning: Undefined array key "last_link_verified" in /home/clients/e27d81e12f04d2bfd747aa264540ac56/sites/_UDTR_APP/legal_library_reference.php on line 31
UDTR — Bibliothèque juridique mondiale des droits de la Nature. EuroExec: Frontier Language Models Fall Short of Expert Judgment on European Executive Decision Tasks. Notice documentaire: https://www.declarationuniverselledesdroitsdelarbre.org/global-rights-of-nature-radar/reference/r-875fb71644bc99e84d6391c35c32d274/ · Dernière vérification HTTP : non effectuée
Données de cette version figée
Dossiers et relations documentaires
Rôles des sources
Source bibliographique / publication d’origine · https://arxiv.org/abs/2608.04549
Un lien vers une autorité ne prouve pas à lui seul une affirmation juridique. Les résumés et les textes intégraux restent soumis aux conditions de leur source.
Statut documentaire
Pertinence à examiner
Bibliothèque
Cette identité n’est pas encore intégrée à la Bibliothèque juridique.