arXiv||1 min read
What Matters in Quality-Aware VLA Pretraining: A Benchmark and Empirical Study
Focuses on What Matters in Quality-Aware VLA Pretraining: A Benchmark and Empirical Study.
Topic archive
AI Evaluation AI news
AI evaluations, benchmarks, model testing, and agent reliability. This page collects the latest briefings that match the topic so readers can follow one area without scanning the full feed.
Indexed briefings
373
Latest source-linked updates, ordered newest first.
Latest
Recent briefings
arXiv||1 min read
RepProbe: Can Video Language Models Count Repeated Actions? A Diagnostic Benchmark
Focuses on RepProbe: Can Video Language Models Count Repeated Actions? A Diagnostic Benchmark.
arXiv||1 min read
LIBERO-RIG: A Hardware-Setup Robustness Benchmark for Vision-Language-Action Models
Focuses on LIBERO-RIG: A Hardware-Setup Robustness Benchmark for Vision-Language-Action Models.
arXiv||1 min read
OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding
Focuses on OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding.
arXiv||1 min read
WorldOracle: Benchmarking Action-Conditioned World Models against Real-World Executions
Focuses on WorldOracle: Benchmarking Action-Conditioned World Models against Real-World Executions.
arXiv||1 min read
Towards Physically Grounded Articulated 3D Assets: A Multi-Agent Pipeline and Benchmark
Focuses on Towards Physically Grounded Articulated 3D Assets: A Multi-Agent Pipeline and Benchmark.
arXiv||1 min read
Neurons++: A Unified Visual-Stream Framework for Image and Video Reconstruction from fMRI with Neural-to-Visual Interpretability
Focuses on Neurons++: A Unified Visual-Stream Framework for Image and Video Reconstruction from fMRI with Neural-to-Visual Interpretability.
arXiv||1 min read
SeoulMMOD: A Large-Scale Multimodal Origin-Destination Flow Benchmark
Focuses on SeoulMMOD: A Large-Scale Multimodal Origin-Destination Flow Benchmark.
arXiv||1 min read
OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning
Focuses on OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning.
arXiv||1 min read
Pumpire: Unified Benchmark for Metric Distance Estimation
Focuses on Pumpire: Unified Benchmark for Metric Distance Estimation.
arXiv||1 min read
Searching for "Harmful Refusal": A Psychometric Audit of an AI Safety Benchmark
Focuses on Searching for "Harmful Refusal": A Psychometric Audit of an AI Safety Benchmark.
arXiv||1 min read
AirGroundVLN: A Large-Scale Benchmark for Goal-Oriented Air-Ground Collaborative Vision-and-Language Navigation
Focuses on AirGroundVLN: A Large-Scale Benchmark for Goal-Oriented Air-Ground Collaborative Vision-and-Language Navigation.
arXiv||1 min read
AirGroundVLN: A Large-Scale Benchmark for Goal-Oriented Air-Ground Collaborative Vision-and-Language Navigation
Focuses on AirGroundVLN: A Large-Scale Benchmark for Goal-Oriented Air-Ground Collaborative Vision-and-Language Navigation.
arXiv||1 min read
RobotWorld: Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodiments
Focuses on RobotWorld: Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodiments.
arXiv||1 min read
Benchmarking Adult Addressee Classification Across Child- and Adult-Directed Speech Datasets
Focuses on Benchmarking Adult Addressee Classification Across Child- and Adult-Directed Speech Datasets.
arXiv||1 min read
OpenViTac: Learning and Benchmarking Visuo-Tactile Policies in a Unified Sim-and-Real Framework
Focuses on OpenViTac: Learning and Benchmarking Visuo-Tactile Policies in a Unified Sim-and-Real Framework.
arXiv||1 min read
RobotWorld: Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodiments
Focuses on RobotWorld: Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodiments.
arXiv||1 min read
TaoD2C-Bench: Benchmarking MLLMs for Industrial UI Code Generation Beyond Visual Fidelity
Focuses on TaoD2C-Bench: Benchmarking MLLMs for Industrial UI Code Generation Beyond Visual Fidelity.
arXiv||1 min read
HeiCo-FOCUS: A Clinically Grounded Dataset for Long-Context Video Understanding
Focuses on HeiCo-FOCUS: A Clinically Grounded Dataset for Long-Context Video Understanding.
arXiv||1 min read
DuRe-ST: Dual-Relation Spectro-Temporal Modeling for Speech Deepfake Detection
Focuses on DuRe-ST: Dual-Relation Spectro-Temporal Modeling for Speech Deepfake Detection.
arXiv||1 min read
AdSpark: A Large-Scale Dataset and Benchmark for Product-Centric Advertisement Video Generation
Focuses on AdSpark: A Large-Scale Dataset and Benchmark for Product-Centric Advertisement Video Generation.
arXiv||1 min read
CARES: A Controlled Synthetic Benchmark of Speaker Reactions to Sound
Focuses on CARES: A Controlled Synthetic Benchmark of Speaker Reactions to Sound.
arXiv||1 min read
Benchmarking Behavioral Steerability in Behavior Foundation Models
Focuses on Benchmarking Behavioral Steerability in Behavior Foundation Models.
arXiv||1 min read
On the Reliability of LLM-Based Vulnerability Patching Benchmarks
Focuses on On the Reliability of LLM-Based Vulnerability Patching Benchmarks.
arXiv||1 min read
BanglaRhet: Benchmarking Classical and Transformer Models for Rhetorical and Persuasion Detection in Bangla Political Speech
Focuses on BanglaRhet: Benchmarking Classical and Transformer Models for Rhetorical and Persuasion Detection in Bangla Political Speech.
arXiv||1 min read
cua-speedrun: Standardized Benchmarking of the Speed of Computer-Use Agents
Focuses on cua-speedrun: Standardized Benchmarking of the Speed of Computer-Use Agents.
arXiv||1 min read
Index-Translate: A Multilingual Translation Model Family -- Text, Speech, Controlled Dubbing, and Long-Document Translation
Focuses on Index-Translate: A Multilingual Translation Model Family -- Text, Speech, Controlled Dubbing, and Long-Document Translation.
arXiv||1 min read
JuryFlow: Disagreement-Guided Human-in-the-Loop Multi-Agent Evaluation
Focuses on JuryFlow: Disagreement-Guided Human-in-the-Loop Multi-Agent Evaluation.
arXiv||1 min read
HEXIS: Compiling Skills into Extended Finite State Machines
Focuses on HEXIS: Compiling Skills into Extended Finite State Machines.
arXiv||1 min read
COSED: Setting the Bar for Open-Vocabulary Sound Event Detection
Focuses on COSED: Setting the Bar for Open-Vocabulary Sound Event Detection.
arXiv||1 min read
SWE-Serve: Benchmarking Agentic Engineering For Production Inference Serving
Focuses on SWE-Serve: Benchmarking Agentic Engineering For Production Inference Serving.
arXiv||1 min read
Era by Eon: Benchmarking Enterprise Agents on Hidden Knowledge
Focuses on Era by Eon: Benchmarking Enterprise Agents on Hidden Knowledge.
arXiv||1 min read
Advancing Model Research in AgentX: Long-Horizon Autonomy for Industrial Recommender Systems
Focuses on Advancing Model Research in AgentX: Long-Horizon Autonomy for Industrial Recommender Systems.
arXiv||1 min read
Trajectory-Aware Benchmark Subset Selection for Cost-Efficient Software Engineering Agent Regression Testing
Focuses on Trajectory-Aware Benchmark Subset Selection for Cost-Efficient Software Engineering Agent Regression Testing.
Hugging Face||1 min read
How UK AISI and EvalEval Are Making Benchmark Results Reproducible
Hugging Face published How UK AISI and EvalEval Are Making Benchmark Results Reproducible.
arXiv||1 min read
Benchmarking World Models for Continual Learning on Compositional Tasks
Focuses on Benchmarking World Models for Continual Learning on Compositional Tasks.