evald.ai

METR Blog · April 01, 2026 07:00

Fine-tuning experiments on CoT controllability

We find that a small amount of fine-tuning on instruction following in the CoT generalizes to meaningful increases in CoT controllability on an out-of-distribution set of tasks. We fine-tune four reasoning models on small datasets of instruction-following...

Mitchell Bryson AI Reliability Articles · March 30, 2026 00:00

OpenAI just showed everyone what an AI company looks like when the math stops working - Mitchell Bryson

In a single week, OpenAI killed Sora ($15M/day burn, $2.1M lifetime revenue), blindsided Disney on a $1B deal, shelved its adult chatbot, renamed its product org to 'AGI Deployment,' moved safety oversight away from the CEO, and bet everything on a model...

Entities: OpenAI

Mitchell Bryson AI Reliability Articles · March 29, 2026 00:00

The real AI war is over your memory - Mitchell Bryson

Google launched tools to import your ChatGPT memories and chat histories. Apple is turning Siri into a marketplace where every AI assistant plugs in — for a 30% cut. OpenAI is wiring Codex into every work tool you touch. Shopify made every AI conversation a...

Topics: LLM Evaluation

Entities: LLM EvaluationOpenAIGoogleChatGPT

Mitchell Bryson AI Reliability Articles · March 28, 2026 00:00

The leak that repriced cybersecurity - Mitchell Bryson

Anthropic's accidental Mythos reveal crashed cybersecurity stocks — but the market was catching up to a reality that was already here. On the same day, CISA warned of active exploitation of AI agent frameworks, researchers disclosed basic vulnerabilities in...

Entities: AnthropicMythos

Mitchell Bryson AI Reliability Articles · March 27, 2026 00:00

The text box was just the prototype - Mitchell Bryson

In a single 48-hour stretch, Apple revealed plans to open Siri to rival chatbots, Mistral shipped an open-source voice model rivaling ElevenLabs, Google launched studio-quality AI music generation, and IBM embedded voice AI into enterprise agents. The chat...

Topics: LLM Evaluation

Entities: LLM EvaluationGoogle

METR Blog · March 26, 2026 07:00

Red-Teaming Anthropic's Internal Agent Monitoring Systems

A METR staff member spent three weeks red-teaming a subset of Anthropic's internal agent monitoring and security systems, discovering several novel vulnerabilities.

Entities: Anthropic

Google DeepMind Evaluation Filter · March 25, 2026 16:46

Protecting People from Harmful Manipulation

Google DeepMind releases new findings and an evaluation framework to measure AI's potential for harmful manipulation in areas like finance and health, with the goal of enhancing AI safety.

Topics: Safety EvalsTesting Tools

Entities: Safety EvalsTesting ToolsGoogleGoogle DeepMind