<aside>
Authors: Hadi Reisizadeh¹, Jiajun Ruan¹, Yiwei Chen², Soumyadeep Pal², Sijia Liu²³, Mingyi Hong¹
(1 University of Minnesota, 2 Michigan State University, 3 IBM Research)
</aside>
📄 Paper 🤗 Hugging Face 💻 Code
Figure: Pipeline of Leak@k evaluation.
Large Language Models (LLMs) have rapidly transformed natural language processing, enabling powerful capabilities across many applications. However, their training on massive, indiscriminately web datasets introduces significant privacy, legal, and ethical risks. In practice, LLMs may memorize and reproduce sensitive information, including private data, toxic content, illegal material, or copyrighted text when prompted.
LLM unlearning has emerged as a critical approach to mitigate these risks. The goal is to remove the influences of undesired data (the forget set) from a model while preserving its overall capabilities on the retain set. Various works have been proposed to achieve the unlearning goal including Gradient Ascent (GA), Gradient Difference (GradDiff), Negative Preference Optimization (NPO), SimNPO, BLUR-NPO, and Representation Misdirection (RMU).
Different benchmarks have been introduced for LLM unlearning such as TOFU (Task of Fictitious Unlearning), MUSE ****(Machine Unlearning Six-Way Evaluation), and ****WMDP (Weapons of Mass Destruction Proxy). The evaluation in these benchmarks is conducted almost exclusively under deterministic decoding, most often greedy decoding. The above unlearning methods achieve partial success in mitigating unwanted information while preserving model utility under greedy decoding.
Such a evaluation creates a significant mismatch between evaluation protocols and real-world deployment. While greedy decoding facilitates convenient evaluation and fair comparison for different methods, it poorly reflects deployed systems, where probabilistic decoding strategies such as temperature sampling or nucleus sampling are widely adopted, especially in domains such as conversational agent and code generation.
As the following figure shows, ALL models appear to have forgotten sensitive passages under greedy decoding readily regenerate them verbatim once sampled multiple times under the probabilistic decoding.
Figure: Examples of generated text on the MUSE-News dataset for GradDiff, NPO, and BLUR-NPO. The left column shows greedy decoding outputs, while the right column shows the worst-case result among 64 probabilistic samples (temperature 0.2, top-p 1.0). Failed unlearning is highlighted in red, and successful unlearning in green.
The gap between algorithmic advances in unlearning and their evaluations under greedy decoding identified above raises a critical question:
<aside> 💡
How do LLMs unlearned with SOTA algorithms behave under probabilistic decoding?
</aside>
The greedy decoding always selects the highest-probability token implying the model produces safe responses and appears to have successfully forgotten the sensitive information. However, real-world LLM systems rarely use greedy decoding. Instead, they rely on probabilistic decoding to generate diverse outputs. Deterministic outputs can mask low-probability memorized strings. The figure below illustrates the differences between greedy decoding and probabilistic decoding, highlighting how each approach generates distinct responses.
