Community feed
A focused stream of recent stories from the sources curated for this community. Latest: Stop “vibe testing” your LLMs. It's time for real evals.- Google Developers Blog, Findings from a pilot Anthropic–OpenAI alignment evaluation exercise: OpenAI Safety Tests, and Claude, GPT, and Gemini All Struggle to Evade Monitors. Page 60.
Hacker News LLM Evaluation
Explore Stax, an experimental developer tool that streamlines LLM evaluation with human labelling and scalable LLM-as-a-judge auto-raters for data driven decisions.
Topics: LLM EvaluationTesting Tools
Entities: Testing ToolsLLM EvaluationGoogle
OpenAI Evaluation Filter
OpenAI and Anthropic share findings from a first-of-its-kind joint safety evaluation, testing each other’s models for misalignment, instruction following, hallucinations, jailbreaking, and more—highlighting progress, challenges, and the value of cross-lab...
Topics: Testing Tools
Entities: AnthropicTesting ToolsOpenAI
METR Blog
Vincent Cheng and Thomas Kwa replicate a Google DeepMind paper on chain-of-thought monitoring, showing evidence that monitoring works on other companies' models.
Entities: ClaudeGeminiGoogleGoogle DeepMind
Google News LLM Evaluation
Signal and Noise: Unlocking Reliable LLM Evaluation for Better AI Decisions MarkTechPost
Topics: LLM Evaluation
Entities: LLM Evaluation
METR Blog
AI agents are improving rapidly at autonomous software development and machine learning tasks, and, if recent trends hold, may match human researchers at challenging months-long research projects in under a decade. Some economic models predict that...
Google News LLM Evaluation
Signal and Noise: Reducing uncertainty in language model evaluation | Ai2 Allen AI
Topics: LLM Evaluation
Entities: LLM Evaluation
Hacker News LLM Evaluation
Next generation LLM evaluation framework powered by Vitest.
Topics: LLM EvaluationTesting Tools
Entities: Testing ToolsLLM Evaluation
METR Blog
Many AI benchmarks use algorithmic scoring to evaluate how well AI systems perform on some set of tasks. However, AI systems often produce code that scores well but isn't production-ready due to issues with test coverage, formatting, and code quality. This...
Topics: BenchmarksLLM Evaluation
Entities: BenchmarksLLM Evaluation
METR Blog
How we think about tradeoffs when communicating surprising or nuanced findings.
METR Blog
Recent work from Anthropic and others claims that LLMs' chains of thoughts can be “unfaithful”. These papers make an important point: you can't take everything in the CoT at face value. As a result, people often use these results to conclude the CoT is...
Entities: Anthropic
METR Blog
We evaluate whether GPT-5 poses significant catastrophic risks via AI self-improvement, rogue replication, or sabotage of AI labs. We conclude that this seems unlikely. However, capability trends continue rapidly, and models display increasing eval awareness.
Topics: LLM Evaluation
Entities: LLM EvaluationOpenAI
Google News LLM Evaluation
Is your AI benchmark lying to you? Nature
Topics: Benchmarks
Entities: Benchmarks