METR Blog
Vivaria
Vivaria is METR's tool for running evaluations and conducting agent elicitation research. Vivaria is a web application with which users can interact using a web UI and a command-line interface.
METR Blog
Details about METR's preliminary evaluation of GPT-4o
We measured the performance of GPT-4o given a simple agent scaffolding on 77 tasks across 30 task families testing autonomous capabilities.
Topics: Testing Tools
Entities: Testing Tools
METR Blog
An update on our general capability evaluations
More tasks, human baselines, and preliminary results for GPT-4 and Claude.
Entities: Claude
METR Blog
ERROR: The request could not be satisfied
Comments on NIST’s draft document “AI Risk Management Framework: Generative AI Profile.”
Topics: Safety Evals
Entities: Safety Evals
METR Blog
ML Engineers Needed for New AI R&D Evals Project
METR is hiring ML engineers and researchers.
METR Blog
Emma Abele is METR’s new Executive Director
Emma moves from President to Executive Director, Beth moves to Head of Research.
METR Blog
Autonomy Evaluation Resources
A collection of resources for evaluating potentially dangerous autonomous capabilities of frontier models.
METR Blog
Example autonomy evaluation protocol
An example protocol for the whole evaluation process, based on our task suite, elicitation protocol, and scoring methods.
METR Blog
GitHub - METR/public-tasks
Contribute to METR/public-tasks development by creating an account on GitHub.
METR Blog
Guidelines for capability elicitation
Priorities for approximating the full potential capability of an AI agent, and recommended checks for evaluation validity.
METR Blog
Measuring the impact of post-training enhancements
A brief research report outlining quantitative research which could inform the "safety margin" to add to take into account further post-training enhancements to agent capability.
METR Blog
Portable Evaluation Tasks via the METR Task Standard
METR has published a standard way to define tasks for evaluating the capabilities of AI agents.