METR Blog
Example autonomy evaluation protocol
An example protocol for the whole evaluation process, based on our task suite, elicitation protocol, and scoring methods.
Community feed
evald.ai
A focused stream of recent stories from the sources curated for this community. Latest: Example autonomy evaluation protocol, GitHub - METR/public-tasks, and Guidelines for capability elicitation. Page 28.
Search
METR Blog
GitHub - METR/public-tasks
Contribute to METR/public-tasks development by creating an account on GitHub.
METR Blog
Guidelines for capability elicitation
Priorities for approximating the full potential capability of an AI agent, and recommended checks for evaluation validity.
METR Blog
Measuring the impact of post-training enhancements
A brief research report outlining quantitative research which could inform the "safety margin" to add to take into account further post-training enhancements to agent capability.
METR Blog
Portable Evaluation Tasks via the METR Task Standard
METR has published a standard way to define tasks for evaluating the capabilities of AI agents.
Hugging Face Evaluation Filter
Introducing the Red-Teaming Resistance Leaderboard
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
Hugging Face Evaluation Filter
Introducing the Open Ko-LLM Leaderboard: Leading the Korean LLM Evaluation Ecosystem
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
METR Blog
2023 Year In Review
A summary of what METR accomplished in 2023 – our first full year of operation.
Hugging Face Evaluation Filter
NPHardEval Leaderboard: Unveiling the Reasoning Abilities of Large Language Models through Complexity Classes and Dynamic Updates
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
OpenAI Evaluation Filter
Building an early warning system for LLM-aided biological threat creation
We’re developing a blueprint for evaluating the risk that a large language model (LLM) could aid someone in creating a biological threat. In an evaluation involving both biology experts and students, we found that GPT-4 provides at most a mild uplift in...
Hugging Face Evaluation Filter
Introducing the Enterprise Scenarios Leaderboard: a Leaderboard for Real World Use Cases
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
Hugging Face Evaluation Filter
The Hallucinations Leaderboard, an Open Effort to Measure Hallucinations in Large Language Models
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
More stories
More stories load automatically as you scroll.