Claims Browser

Google DeepMind is conducting the world's first double-blind AI evaluations

"Piloting the world's first double-blind AI evaluations"

Adaptive domain normalization fine-tuning of pretrained Contrastive Predictive Coding models yields a 23.6% relative improvement on across-speaker ABX scores for accented speech

"the proposed method yields a relative improvement of 23.6% on across-speaker ABX scores on average compared to non adapted models"

ABX-Accent benchmark provides a standardized way to evaluate representation learning methods on accented speech with 10 different English accents

"We introduce ABX- Accent, a benchmark based on the AESRC dataset that features 10 different accents of English. It includes a small (< 10 hours) unlabelled training set in each of the accents and adaptations of the Zero Resources Challenge ABX evaluation metrics to each of the accents"

Compositional generalization is typically assessed using model accuracy as the primary metric

"Compositional generalization is usually evaluated through model accuracy."

An alternative approach to evaluating compositional generalization examines which structural or lexical identifications enable held-out examples to be admissible based on training structures

"We instead ask which structural or lexical identifications make held-out COGS examples admissible from the structures observed in training."

Across 21 COGS generalization types, admissibility patterns follow distinct identification profiles and residual failures reveal unsupported structural templates

"Across 21 COGS generalization types, admissibility follows distinct identification profiles, while residual failures separate unsupported structural templates."

Data-side diagnostic approaches can characterize what a training corpus licenses under specified identifications without requiring training of a predictive model

"These data-side diagnoses characterize what the training corpus licenses under specified identifications, without training a predictive model."

Human evaluation for non-verifiable tasks is reliable but expensive, while automatic metrics are scalable but often biased

"Across various non-verifiable tasks, human evaluation is reliable but expensive, while automatic metrics are more scalable but often biased."

Prediction-powered evaluation provides data-efficient system comparisons that are provably unbiased by combining limited human judgments with large-scale automatic scores

"we propose prediction-powered evaluation, a framework that combines limited human judgments with large-scale automatic scores to obtain data-efficient system comparisons that are provably unbiased."

PPSR yields more discriminative and stable metric rankings than existing system-level meta-metrics

"PPSR directly targets metric utility for prediction-powered evaluation and yields more discriminative and stable metric rankings than existing system-level meta-metrics."

Automatic metrics should be reframed as tools for reducing human annotation cost rather than replacing human judgment

"our new paradigm reframes automatic metrics as tools for reducing human annotation cost rather than replacing human judgment"

The prediction-powered evaluation framework applies broadly to non-verifiable tasks

"applies broadly to non-verifiable tasks"

AraMS-28k is the largest publicly released line-level dataset of genuine historical Arabic manuscripts with 28,600 annotated text lines

"We introduce AraMS-28k, the largest publicly released line-level dataset of genuine historical Arabic manuscripts, comprising 14 books, 3,043 pages, and 28,600 annotated text lines"

This is the first annotation released for a historical Arabic manuscript corpus that recovers non-linear reading order at line-level granularity

"margin lines that have an unambiguous attachment point in the main text are further annotated with an insertion anchor, recovering the manuscript's true non-linear reading order at line-level granularity -- to our knowledge the first such annotation released for a historical Arabic manuscript corpus"

RefLAM pipeline combines multimodal-LLM OCR with human review to construct high-quality manuscript datasets

"The dataset was constructed with RefLAM, a reference-grounded annotation pipeline that aligns multimodal-LLM OCR against independently sourced clean transcriptions and routes every line through human review, combining automatic verification with expert oversight"

AraMS-28k supports reproducible research on Arabic manuscript recognition, layout analysis, and reading-order recovery

"AraMS-28k is released with page images, line-level annotations, and fixed train/val/test splits under CC BY-NC-SA 4.0 to support reproducible research on Arabic manuscript recognition, layout analysis, and reading-order recovery"

FID's first-two-moment summary can miss distributional differences between generative models and real data

"FID's first-two-moment summary can miss distributional differences"

On ImageNet, visually unrecognizable images optimized to match reference Inception statistics achieve FID 24.7 versus 58.6 for real images, demonstrating FID can be gamed

"on ImageNet, visually unrecognizable images optimized only to match the reference Inception mean and covariance obtain FID $24.7$ versus $58.6$ for held-out real images (lower is better)"