METR Blog
We find that roughly half of test-passing SWE-bench Verified PRs written by recent AI agents would not be merged into main by repo maintainers. A naive interpretation of benchmark scores may lead one to overestimate how useful agents are without more...
Topics: Benchmarks
Entities: Benchmarks
METR Blog
Nikola Jurkovic describes observations from tasking Opus 4.6 with reimplementing Slay the Spire and Balatro in the CLI.
METR Blog
Our second developer productivity study faces selection effects from wider AI adoption, prompting us to redesign our approach.
METR Blog
Luca Righetti shares takeaways on the role of randomized controlled trials in AI safety testing.
Topics: Safety EvalsTesting Tools
Entities: Safety EvalsTesting Tools
METR Blog
Our high-level approach to protecting confidential access and information
METR Blog
Amy Deng investigates whether coding agent transcripts could serve as an alternative for estimating AI productivity uplift, using 5305 Claude Code transcripts from METR technical staff.
Entities: ClaudeClaude Code
METR Blog
Nikola Jurkovic describes our measurements of time horizon using Claude Code and Codex scaffolds.
Entities: ClaudeClaude Code
METR Blog
Thomas Kwa describes a simple model for forecasting when AI will automate AI development, based on the AI Futures model but with only 8 parameters.
METR Blog
Miles Kodama and Michael Chen summarize key provisions from California's SB 53, the EU Code of Practice, and New York's RAISE Act covering frontier AI developers.
Topics: Safety Evals
Entities: Safety Evals
METR Blog
We’re releasing a new version of our time horizon estimates (TH1.1), using more tasks and a new eval infrastructure.
Topics: LLM Evaluation
Entities: LLM Evaluation
METR Blog
We show preliminary results on a prototype evaluation that tests monitors' ability to catch AI agents doing side tasks, and AI agents' ability to bypass this monitoring.
METR Blog
Thomas Kwa responds to some misinterpretations of our time horizon work, and explains limitations and the core finding.