Abstract
As multimodal models like CLIP become integral to downstream systems, the need to remove sensitive information is critical. However, machine unlearning for contrastively trained encoders remains underexplored, and existing evaluations fail to diagnose fine-grained, association-level forgetting. We introduce SALMUBench (Sensitive Association-Level Multimodal Unlearning), a benchmark built upon a synthetic dataset of 60K persona-attribute associations and two foundational models: a Compromised model polluted with this data, and a Clean model without it. Both are trained from scratch on a 400M-pair retain set to isolate unlearning effects. We propose a novel evaluation protocol with structured holdout sets (holdout_identity, holdout_association) to precisely measure unlearning efficacy and collateral damage. Our benchmark reveals that while utility-efficient deletion is feasible, current methods exhibit distinct failure modes: they either fail to forget effectively or over-generalize by erasing more than intended. SALMUBench sets a new standard for comprehensive unlearning evaluation, and we publicly release our dataset, models, evaluation scripts, and leaderboards to foster future research.
Overview
SALMUBench is a benchmark for evaluating association-level multimodal unlearning in CLIP-style models.
Unlike prior unlearning evaluations that focus on coarse metrics, SALMUBench targets fine-grained association-level forgetting.
It provides:
- A synthetic dataset of about 60K persona-attribute associations
- A Compromised CLIP model trained on these sensitive associations
- A Clean CLIP model trained without them
- A structured evaluation protocol with holdout splits that diagnose forgetting and collateral damage
SALMUBench evaluates unlearning along two main pillars:
- Forgetting Efficacy: has the model removed the targeted sensitive associations?
- Utility Impact: has the model preserved general capability and avoided collateral forgetting?
Dataset
SALMUBench is built on a synthetic dataset of fictitious personas associated with sensitive private information such as names, locations, email addresses, phone numbers, and financial identifiers. Images are identity-preserving synthetic portraits, and captions are paraphrased to create semantically varied associations, making unlearning substantially harder than simple string or identity removal, as associations must be removed without harming related concepts.
Examples of Sensitive Associations
The examples below are fully synthetic and illustrate the type of sensitive persona-attribute associations used in SALMUBench.
Evaluation Protocol
The protocol is based on four key evaluation splits: forget, retain_synth, holdout_identity, and holdout_association.
SALMUBench evaluates unlearning across two pillars:
Forgetting Efficacy
These metrics measure whether the model has actually removed the targeted sensitive associations from the forget set. They capture failures such as incomplete forgetting or persistent memorization.
Utility Impact
These metrics measure whether unlearning causes collateral damage. They evaluate:
- Preservation of general knowledge
- Damage to unrelated identities (
holdout_identity) - Damage to other associations of the same identity (
holdout_association) - Preservation of generic visual identity information and weaker retained knowledge
This protocol goes beyond a simple forget-vs-retain setup and enables diagnosis of distinct failure modes, including ineffective forgetting, catastrophic damage, and over-generalized forgetting.
Key Findings
SALMUBench reveals that current multimodal unlearning methods exhibit distinct failure modes:
- Ineffective forgetting: some methods fail to sufficiently erase the targeted sensitive associations
- Catastrophic damage: some methods forget effectively but severely degrade general utility
- Over-generalized forgetting: some methods erase more than intended, harming unrelated identities or non-target associations
Overall, our benchmark shows that utility-efficient deletion is achievable, but no current method fully resolves the trade-off between precise forgetting and collateral damage.
Leaderboard
A public leaderboard will be released soon to track the performance of unlearning methods on SALMUBench.
Resources
All SALMUBench resources are publicly available.
Datasets
Models
Code
Collection
Getting Started
- Download the SALMUBench benchmark dataset and the Compromised model
- Apply an unlearning method
- Evaluate using the provided scripts and benchmark splits
- Compare against the Clean reference model
Citation
@misc{selvassala2026salmubenchbenchmarksensitiveassociationlevel,
title={SALMUBench: A Benchmark for Sensitive Association-Level Multimodal Unlearning},
author={Cai Selvas-Sala and Lei Kang and Lluis Gomez},
year={2026},
eprint={2603.26316},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2603.26316},
}