Privacy Auditing of Large Language Models

Current industry practices for privacy auditing in LLMs, such as those used for Palm, Gemma, and Llama 3, tend to underestimate worst-case data memorization. These methods typically involve inserting random canaries (random strings or samples from the training corpus) and measuring average-case memorization. This approach fails to capture the full extent of potential privacy leakage, especially in scenarios with legal consequences or impact on data providers.

Model releases like Palm, Gemma, and Llama 3 conclude their models don't memorize much data, but this tends to underestimate worst-case effects. Looking at average-case memorization isn't enough to tell us how much data is memorized in the worst case, which has legal and data availability consequences.

The 'New Token Canary' strategy involves using one of the tokenizer's unused special tokens (or adding a new one) and appending it to normal training sequences. Since this token has never been seen in the model's entire pre-training data set, the model assigns near-zero probability to generating it. This makes it extremely easy to detect if the canary was part of the fine-tuning data, providing a robust upper bound estimate for the worst possible privacy leakage. This method is primarily useful for model developers who can modify model embeddings.

We're going to actually use one of the unused special tokens in the tokenizer... it also hasn't been seen anywhere else in the model's entire pre-training data set. So, our model should assign basically zero probability to ever generating one of these special tokens. As a result, it should be very, very easy to tell if our canary contains one of these tokens whether or not that canary was actually in the fine-tuning data set.

The 'Search Token Canary' strategy involves finding tokens that the pre-trained model is least likely to predict given a normal sequence of training data. This method is more sophisticated than random token canaries as it leverages the model's existing biases. It requires blackbox access to the model (to perform inference passes) and is more computationally expensive, but it provides a stronger signal for privacy leakage compared to random tokens, bridging the gap between random and new token canaries.

We're going to again sample normal sequences of training data and for each sequence we're going to find tokens such that the model's likelihood of ever predicting that token given the sequence is minimized... this tells us what sequences this model is least likely to predict correctly.

The 'Biogram Canary' method is a 'model-free' approach that only requires access to the fine-tuning data set. It involves fitting a biogram model on the data and then mining sequences where the probability of a final canary token is very low, conditioned on the preceding token. This method consistently outperforms random token baselines and is a practical solution for auditing when model access is limited or unavailable, such as during pre-training planning.

We can sort of level up a bit and try and create a biogram model... we'd like to find a canary sequence such that the probability of that final last canary token is very low given the data that comes before it... this still doesn't have any knowledge of the pre-training data. So in that sense it is you know really quite model free.

Training LLMs using a Supervised Fine-Tuning (SFT) objective, where the prompt is masked and the loss is only calculated over the answer, significantly increases empirical privacy leakage. This is a critical finding because SFT is a common practice in fine-tuning LLMs for specific tasks (e.g., Q&A, instruction following).

Doing this sort of SFT style training where we mask out the prompt is going to significantly increase privacy leakage.