Detection & evals
LLMEval3
LLMEval is a research initiative from Fudan NLP Lab dedicated to building comprehensive, fair, and robust evaluation frameworks for large language models.
What is LLMEval3?
LLMEval is a research initiative from Fudan NLP Lab dedicated to building comprehensive, fair, and robust evaluation frameworks for large language models.It benchmarks LLMs across 13+ academic disciplines, medical AI, and over 220,000 generative questions, addressing critical concerns regarding robustness and fairness in current evaluation practices.
The series features key contributions such as LLMEval-Fair, which conducts a 30-month longitudinal study using dynamic sampling of unseen test sets, and LLMEval-Med, a physician-validated benchmark for real-world clinical tasks.By employing contamination-resistant data curation and a calibrated LLM-as-a-judge process, LLMEval exposes performance ceilings and data contamination vulnerabilities that static benchmarks often miss.
With publications in top conferences like AAAI, EMNLP, and ACL, LLMEval provides open access to its code and datasets.Its evaluation pipeline combines human expert validation with automated scoring, achieving high agreement with human annotators and offering a reliable reference for model capabilities.
Key features
- Multi-DisciplineCovers professional questions across 13+ academic disciplines, providing a comprehensive assessment of general capabilities.
- Large-Scale QuesBuilt on a proprietary bank of over 220,000 graduate-level questions, ensuring depth and breadth in evaluation.
- Contamination ReUses dynamic sampling of unseen test sets and contamination-resistant data curation to prevent cheating via memorization.
- Real-World CliniLLMEval-Med includes 2,996 questions from real electronic health records and expert-designed scenarios, validated by physicians.
- Expert-Level AgrCalibrated LLM-as-a-judge process achieves 90% agreement with human experts, ensuring credible scoring.
- Open Source & OpCode, datasets, and papers are publicly available on GitHub and HuggingFace, supporting community collaboration and research.
How to use it
- Visit the official LLMEval website to explore the latest research papers, leaderboards, and blog posts.
- Browse the GitHub organization page to access code implementations for LLMEval-Fair and LLMEval-Med.
- Download LLMEval-1, LLMEval-2, and related datasets from HuggingFace for local testing.
- Refer to the published papers to understand the evaluation methodology, anti-cheating architecture, and detailed experimental results.
- Contact the Fudan NLP Lab team via email or WeChat to discuss collaboration or participate in future research projects.
Where it fits
- Academic ResearcResearchers use the LLMEval framework to conduct fair, robust performance comparisons and longitudinal analysis of new LLMs.
- Medical AI EvaluHealthcare institutions or developers use LLMEval-Med to validate model performance in knowledge, reasoning, and safety ethics within real clinical tasks.
- Model Security ADetect whether models suffer from data leakage or overfitting to specific benchmarks using contamination-resistant test sets.
Pricing and free quota
Pricing and any free quota follow the official site.
Who it is for
- AI Researchers
- LLM Developers
- Medical AI Specialists
- Data Scientists
FAQ
What is the main contribution of LLMEval?
It builds comprehensive, fair, and robust evaluation frameworks, specifically addressing robustness and fairness issues through LLMEval-Fair and providing a physician-validated clinical benchmark via LLMEval-Med.
How can I access LLMEval datasets?
You can find LLMEval-1, LLMEval-2, and other related datasets on HuggingFace, and view the associated code on GitHub.
What does LLMEval-Med include?
LLMEval-Med includes 2,996 questions from real electronic health records and expert-designed scenarios, covering five core areas: Medical Knowledge, Language Understanding, Reasoning, Ethics & Safety, and Text Generation.