Threat Models for Memorization: Privacy, Copyright, and Everything In-Between

Typical evaluations of heuristic privacy defenses are misleading because they implicitly report an average-case notion of privacy, failing to reflect leakage from the most vulnerable data. This often includes outliers, mislabeled samples, or out-of-distribution data, which are precisely what one would want to protect most. Additionally, these evaluations often use weak attacks and compare against unfairly private (and useless) differentially private (DP) baselines.

The speaker demonstrates that when evaluated correctly, the actual privacy leakage for heuristic defenses was 7 to over 50 times larger than reported, and none provided meaningful privacy in practice. This is shown by shifting from averaging over the entire dataset to auditing 'canaries' that mimic the most vulnerable data.

Even when relaxing the threat model from pathological worst-case datasets to more natural data, privacy should still be treated as a worst-case metric. It is crucial to report the privacy leakage of the most vulnerable data points within a natural dataset, as leakage can be highly non-uniform and sensitive data (e.g., rare medical cases) is often the most vulnerable.

The speaker illustrates this by comparing average-case privacy (low leakage) with worst-case privacy for individual samples (high leakage, especially for outliers), showing a 'very non-uniform' leakage pattern. The example of medical settings with rare diseases highlights the importance of protecting these vulnerable individuals.

Differentially Private Stochastic Gradient Descent (DP-SGD), even when configured with very high epsilon parameters that provide 'meaningless' theoretical privacy guarantees (e.g., epsilon in the order of 10^8), can still act as a very strong heuristic defense. It empirically outperforms other heuristic approaches in protecting privacy, particularly when considering the worst-case leakage of vulnerable data.

When DP-SGD baselines are trained for high utility (comparable to heuristics) and evaluated on worst-case privacy leakage, they perform 'not too bad' and can even be 'the best thing you can do' depending on constraints, surprising the speaker.

LLMs can inadvertently reproduce verbatim training data, including copyrighted material, even when used by benign users for everyday tasks (e.g., creative writing, factual explanations). This 'non-adversarial reproduction' challenges the argument that copyright violations only occur under adversarial usage.

Examples show LLMs generating text with many long snippets (50+ characters) found verbatim online, while human-written text for the same prompts contains far fewer. For factual tasks like explanations, up to 25% of LLM output can consist of such snippets. This phenomenon is long-tailed, with rare instances of very long reproductions (e.g., entire Wikipedia articles or code snippets).

Compared to human writers, LLMs systematically reproduce more existing online text. While human-generated text for creative, persuasive, and factual tasks often has zero median overlap with existing online snippets, LLMs consistently show non-trivial median and mean reproduction rates.

A comparison using prompts from Reddit and IMDb shows that the median fraction of reproduced text for all human baselines is zero, whereas for LLMs, the median and mean are 'much much larger or at least very non-trivially non zero.'