evald.ai

OpenAI Evaluation Filter · February 18, 2025 10:00

Introducing the SWE-Lancer benchmark

Can frontier LLMs earn $1 million from real-world freelance software engineering?

Topics: Benchmarks

Entities: Benchmarks

Mitchell Bryson AI Reliability Articles · February 17, 2025 00:00

EU/UK AI compliance in 2025: mapping the ICO risk toolkit to EU AI Act deadlines for product teams - Mitchell Bryson

What the EU AI Act requires in 2025–2027, how it lines up with the UK ICO's AI & Data Protection Risk Toolkit, and the exact outputs your team should ship.

Topics: Safety Evals

Entities: Safety Evals

METR Blog · February 14, 2025 08:00

Measuring Automated Kernel Engineering

We measured the performance of frontier models at writing GPU kernels. With a small amount of scaffolding, we found that the best model can provide an average speedup on KernelBench of 1.8x.

Hugging Face Evaluation Filter · February 14, 2025 00:00

Fixing Open LLM Leaderboard with Math-Verify

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

Topics: Benchmarks

Entities: Benchmarks

METR Blog · February 12, 2025 08:00

Details about METR's preliminary evaluation of DeepSeek-V3

We evaluated DeepSeek-V3 for dangerous autonomous capabilities and found no evidence of dangerous capabilities beyond those of existing models such as Claude 3.5 Sonnet and GPT-4o. We also confirmed that its performance on GPQA is not due to training data...

Entities: Claude

Hugging Face Evaluation Filter · February 10, 2025 00:00

The Open Arabic LLM Leaderboard 2

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

Topics: Benchmarks

Entities: Benchmarks

METR Blog · February 08, 2025 16:00

Frontier AI Safety Policies

Model Evaluation & Threat Research

Topics: Safety EvalsLLM Evaluation

Entities: Safety EvalsLLM Evaluation

Hugging Face Evaluation Filter · February 04, 2025 00:00

DABStep: Data Agent Benchmark for Multi-step Reasoning

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

Topics: Benchmarks

Entities: Benchmarks

OpenAI Evaluation Filter · January 31, 2025 11:00

OpenAI o3-mini System Card

This report outlines the safety work carried out for the OpenAI o3-mini model, including safety evaluations, external red teaming, and Preparedness Framework evaluations.

Topics: Safety Evals

Entities: Safety EvalsOpenAI

METR Blog · January 31, 2025 08:00

An update on our preliminary evaluations of Claude 3.5 Sonnet and o1

Preliminary evaluations of Claude 3.5 Sonnet (New) and o1, as well as some discussion of challenges in making capability-based safety arguments for AI models.

Entities: Claude