Community feed
A focused stream of recent stories from the sources curated for this community. Latest: Evaluating frontier AI R&D capabilities of language model agents against human experts, Introducing the Open Leaderboard for Japanese LLMs!, and The Rogue Replication Threat Model. Page 23.
METR Blog
We’re releasing RE-Bench, a new benchmark for measuring the performance of humans and frontier model agents on ML research engineering tasks. We also share data from 71 human expert attempts and results for Anthropic’s Claude 3.5 Sonnet and OpenAI’s...
Hugging Face Evaluation Filter
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
METR Blog
Thoughts on how AI agents might develop large and resilient rogue populations.
Hugging Face Evaluation Filter
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
OpenAI Evaluation Filter
A factuality benchmark called SimpleQA that measures the ability for language models to answer short, fact-seeking questions.
OpenAI Evaluation Filter
We’ve simplified, stabilized, and scaled continuous-time consistency models, achieving comparable sample quality to leading diffusion models, while using only two sampling steps.
METR Blog
Red-teaming and security suggestions regarding proposed rule by the Bureau of Industry and Security, “Establishment of Reporting Requirements for the Development of Advanced Artificial Intelligence Models and Computing Clusters.”
OpenAI Evaluation Filter
We introduce MLE-bench, a benchmark for measuring how well AI agents perform at machine learning engineering.
METR Blog
Funding for Canary will enable research and implementation at scale
Hugging Face Evaluation Filter
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
METR Blog
We measured the performance of OpenAI's o1-mini and o1-preview models on our autonomy and AI R&D task suites, and found they did not exceed the capabilities of the best existing public model we've evaluated, though we could not confidently upper-bound...
METR Blog
Suggestions for expanded guidance on capability elicitation and robust model safeguards in the U.S. AI Safety Institute’s draft document “Managing Misuse Risk for Dual-Use Foundation Models” (NIST AI 800-1).
More stories load automatically as you scroll.