Community feed
A focused stream of recent stories from the sources curated for this community. Latest: Argilla 2.4: Easily Build Fine-Tuning and Evaluation Datasets on the Hub — No Code Required, Introducing SimpleQA, and Simplifying, stabilizing, and scaling continuous-time consistency models. Page 67.
Hugging Face Evaluation Filter
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
OpenAI Evaluation Filter
A factuality benchmark called SimpleQA that measures the ability for language models to answer short, fact-seeking questions.
Topics: Benchmarks
Entities: Benchmarks
OpenAI Evaluation Filter
We’ve simplified, stabilized, and scaled continuous-time consistency models, achieving comparable sample quality to leading diffusion models, while using only two sampling steps.
Topics: LLM Evaluation
Entities: LLM Evaluation
METR Blog
Red-teaming and security suggestions regarding proposed rule by the Bureau of Industry and Security, “Establishment of Reporting Requirements for the Development of Advanced Artificial Intelligence Models and Computing Clusters.”
OpenAI Evaluation Filter
We introduce MLE-bench, a benchmark for measuring how well AI agents perform at machine learning engineering.
Topics: Benchmarks
Entities: Benchmarks
METR Blog
Funding for Canary will enable research and implementation at scale
Hugging Face Evaluation Filter
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
Topics: Benchmarks
Entities: Benchmarks
METR Blog
We measured the performance of OpenAI's o1-mini and o1-preview models on our autonomy and AI R&D task suites, and found they did not exceed the capabilities of the best existing public model we've evaluated, though we could not confidently upper-bound their...
Entities: OpenAI
METR Blog
Suggestions for expanded guidance on capability elicitation and robust model safeguards in the U.S. AI Safety Institute’s draft document “Managing Misuse Risk for Dual-Use Foundation Models” (NIST AI 800-1).
Topics: Safety Evals
Entities: Safety Evals
Google News LLM Evaluation
A review of model evaluation metrics for machine learning in genetics and genomics Frontiers
Topics: LLM Evaluation
METR Blog
Vivaria is METR's tool for running evaluations and conducting agent elicitation research. Vivaria is a web application with which users can interact using a web UI and a command-line interface.
METR Blog
We measured the performance of GPT-4o given a simple agent scaffolding on 77 tasks across 30 task families testing autonomous capabilities.
Topics: Testing Tools
Entities: Testing Tools