An update on our general capability evaluations
More tasks, human baselines, and preliminary results for GPT-4 and Claude.
ERROR: The request could not be satisfied
Comments on NIST’s draft document “AI Risk Management Framework: Generative AI Profile.”
ML Engineers Needed for New AI R&D Evals Project
METR is hiring ML engineers and researchers.
Emma Abele is METR’s new Executive Director
Emma moves from President to Executive Director, Beth moves to Head of Research.
Autonomy Evaluation Resources
A collection of resources for evaluating potentially dangerous autonomous capabilities of frontier models.
Example autonomy evaluation protocol
An example protocol for the whole evaluation process, based on our task suite, elicitation protocol, and scoring methods.
GitHub - METR/public-tasks
Contribute to METR/public-tasks development by creating an account on GitHub.
Guidelines for capability elicitation
Priorities for approximating the full potential capability of an AI agent, and recommended checks for evaluation validity.
Measuring the impact of post-training enhancements
A brief research report outlining quantitative research which could inform the "safety margin" to add to take into account further post-training enhancements to agent capability.
Portable Evaluation Tasks via the METR Task Standard
METR has published a standard way to define tasks for evaluating the capabilities of AI agents.
2023 Year In Review
A summary of what METR accomplished in 2023 – our first full year of operation.
Bounty: Diverse hard tasks for LLM agents
METR (formerly ARC Evals) is looking for (1) ideas, (2) detailed specifications, and (3) well-tested implementations for tasks to measure performance of autonomous LLM agents.