evald.ai

Hacker News LLM Evaluation · August 27, 2025 18:52

Stop “vibe testing” your LLMs. It's time for real evals.- Google Developers Blog

Explore Stax, an experimental developer tool that streamlines LLM evaluation with human labelling and scalable LLM-as-a-judge auto-raters for data driven decisions.

LLM Evaluation Testing Tools

Testing Tools LLM Evaluation Google

OpenAI Evaluation Filter · August 27, 2025 10:00

Findings from a pilot Anthropic–OpenAI alignment evaluation exercise: OpenAI Safety Tests

OpenAI and Anthropic share findings from a first-of-its-kind joint safety evaluation, testing each other’s models for misalignment, instruction following, hallucinations, jailbreaking, and more—highlighting progress, challenges, and the value of cross-lab...

Testing Tools

Anthropic Testing Tools OpenAI

METR Blog · August 22, 2025 07:00

Claude, GPT, and Gemini All Struggle to Evade Monitors

Vincent Cheng and Thomas Kwa replicate a Google DeepMind paper on chain-of-thought monitoring, showing evidence that monitoring works on other companies' models.

Claude Gemini Google Google DeepMind

METR Blog · August 20, 2025 07:00

Forecasting the Impacts of AI R&D Acceleration: Results of a Pilot Study

AI agents are improving rapidly at autonomous software development and machine learning tasks, and, if recent trends hold, may match human researchers at challenging months-long research projects in under a decade. Some economic models predict that...

Hacker News LLM Evaluation · August 19, 2025 06:45

Viteval | Next generation LLM evaluation framework powered by Vitest.

Next generation LLM evaluation framework powered by Vitest.

LLM Evaluation Testing Tools

Testing Tools LLM Evaluation

METR Blog · August 13, 2025 07:00

Research Update: Algorithmic vs. Holistic Evaluation

Many AI benchmarks use algorithmic scoring to evaluate how well AI systems perform on some set of tasks. However, AI systems often produce code that scores well but isn't production-ready due to issues with test coverage, formatting, and code quality. This...

Benchmarks LLM Evaluation

Benchmarks LLM Evaluation

METR Blog · August 12, 2025 12:00

Notes on Scientific Communication at METR

How we think about tradeoffs when communicating surprising or nuanced findings.

METR Blog · August 08, 2025 07:00

CoT May Be Highly Informative Despite “Unfaithfulness”

Recent work from Anthropic and others claims that LLMs' chains of thoughts can be “unfaithful”. These papers make an important point: you can't take everything in the CoT at face value. As a result, people often use these results to conclude the CoT is...

Anthropic

METR Blog · August 07, 2025 07:00

Details about METR's evaluation of OpenAI GPT-5

We evaluate whether GPT-5 poses significant catastrophic risks via AI self-improvement, rogue replication, or sabotage of AI labs. We conclude that this seems unlikely. However, capability trends continue rapidly, and models display increasing eval awareness.

LLM Evaluation

LLM Evaluation OpenAI