CLIP Models Generalize Less Than Compositional Benchmarks Suggest

Shuman Peng1, Arnas Uselis2, Darina Koishigarina2, Martin Ester1, Seong Joon Oh3
1Simon Fraser University 2Tübingen AI Center, University of Tübingen 3KAIST AI
ICML 2026 2nd Workshop on Compositional Learning: Safety, Interpretability, and Agents
ICML 2026 Workshop on Combining Theory and Benchmarks
↗ Read the paper ↓ Download BindSplit eval data (coming soon)
TL;DR

A high compositional benchmark score doesn't tell you what you think it does.

CLIP's high scores on compositional benchmarks largely reflect a familiarity asymmetry and memorized seen (familiar) bindings, not generalization to unseen (novel) bindings. We introduce BindSplit, a protocol that splits any benchmark into seen and unseen (attribute, object) bindings, and find the benchmarks barely test unseen generalization. We should report the splits separately — the aggregate benchmark score is hiding potential shortcuts and a generalization gap.

Problem

A higher benchmark score doesn't tell you memorization or generalization.

  • When CLIP picks "a red apple and a ceramic mug" over the attribute-swapped negative "a ceramic apple and a red mug," is it generalizing to novel bindings or recognizing ones it saw in training? The aggregate score can't tell you which.
  • Every new method reports a higher benchmark score than the last, but does a higher score reflect better generalization to novel bindings or more memorization of familiar bindings? The benchmarks don't separate them.
The swap-attribute task. Left: seen bindings — a red apple and a ceramic mug, with the attribute-swapped negative 'a ceramic apple and a red mug.' Right: unseen bindings — a wooden apple and a fluffy mug, with the swapped negative 'a fluffy apple and a wooden mug.'
Protocol

BindSplit: Split the benchmark by what the model has seen.

BindSplit parses the (attribute, object) bindings in each caption, labels them seen or unseen against the training distribution (or a proxy like COCO for real benchmarks), and partitions the benchmark accordingly into seen and unseen (bindings) evaluation splits. It requires no new data collection, and it works on benchmarks you already use.

Three evaluation splits produced by BindSplit. Full evaluation split shows familiarity asymmetry between positive and negative captions in 82.5% of ARO samples. Seen eval split keeps samples where positive and negative captions are equally familiar (red apple / green mug example). Unseen eval split keeps samples whose bindings are entirely novel (wooden apple / fluffy mug example).
Findings

Two things become visible.

1CLIP degrades on unseen bindings

  • On synthetic data (PUG:SPARE), image-to-text retrieval drops monotonically as more of a sample's bindings are unseen.
PUG:SPARE R@1 (FT). Order-invariant retrieval accuracy drops from fully seen to partially unseen to fully unseen across five CLIP backbones (CLIP OpenAI, OpenCLIP LAION-2B, NegCLIP COCO-ft, CLIP CC12M, SPARO CC12M).

2The benchmarks barely test for generalization to unseen bindings

  • Across ARO VG-A, BiVLC, and VisMin, fewer than 2.5% of samples involve unseen bindings.
  • Bindings in correct captions are about twice as COCO-familiar as bindings in their corresponding negatives captions (79.8% vs. 41.8% on ARO VG-A), so a model can score high by exploiting the familiarity asymmetry rather than binding correctly.
  • Removing the shortcut reorders the leaderboard.
ARO I2T accuracy (FT). Accuracy on the full ARO benchmark, the seen split, and the unseen split (n=314), across the same five CLIP backbones. Removing the familiarity-asymmetry shortcut reorders the leaderboard.
Table 1: split composition for ARO VG-A, BiVLC, and VisMin based on binding overlap with COCO (train). Unseen samples are 1.2% of ARO VG-A, 1.8% of BiVLC, and 2.3% of VisMin — all under 2.5% of each benchmark.
Recommendation

Report the splits, not the aggregate.

  • Use BindSplit and report model performance on the shortcut-free seen and unseen splits, not just the aggregated benchmark score.
  • Otherwise reported improvements likely overstate how much CLIP has learned to bind.
Abstract

From the paper.

Compositional benchmarks track progress on CLIP-based compositional reasoning. Each new method reports higher scores than the last, but it is unclear whether the improvements reflect generalization to novel bindings or memorization of bindings already seen during training. To find out, we run two analyses: a synthetic ground-truth study with curated fully-seen, partially-unseen, and fully-unseen binding splits; and extension to three real compositional benchmarks (ARO VG-A, BiVLC, VisMin) using binding-overlap with COCO as a proxy for the alignment-training distribution. On the synthetic dataset, accuracy drops monotonically from fully-seen to fully-unseen across nine CLIP backbones. On ARO VG-A, positive captions overlap COCO bindings nearly twice as often as their attribute-swapped negatives (79.8% vs. 41.8%); only 1.2% of samples have no COCO-overlapping bindings. Restricting evaluation to the shortcut-free seen split, where positive and negative captions are equally COCO-familiar, reorders the top of the leaderboard relative to the full benchmark. The accuracy drop from seen to unseen bindings broadly replicates on BiVLC and VisMin, though with greater noise. Compositional benchmarks should report performance on these shortcut-free splits; otherwise reported improvements likely overstate how much CLIP has learned to bind.

Cite

BibTeX

@inproceedings{
peng2026clip,
title={{CLIP} Models Generalize Less Than Compositional Benchmarks Suggest},
author={Shuman Peng and Arnas Uselis and Darina Koishigarina and Martin Ester and Seong Joon Oh},
booktitle={2nd Workshop on Compositional Learning: Safety, Interpretability, and Agents},
year={2026},
url={https://openreview.net/forum?id=C8MlQkr4bw}
}