POPri: Private Federated Learning using Preference-Optimized Synthetic Data

Traditional federated learning, which involves sending model weights to clients for local training and then aggregating noisy updates, is becoming impractical for increasingly large foundation models (LLMs). These models are too big to train efficiently on client devices, yet privacy constraints prevent collecting raw data centrally.

Foundation models are getting too big to train on device these days. We can't send the models to the silos because the models are too big and we also can't collect data from the silos because that would violate basic privacy constraints.

POPri proposes a solution where an LLM generates synthetic data. This synthetic data is sent to client devices, which then provide privacy-preserving feedback (scores) on its quality by comparing it to their local private data using an embedding model. Instead of using these scores for in-context learning, POPri employs Direct Preference Optimization (DPO) to fine-tune the LLM directly, teaching it to generate higher-quality synthetic data aligned with client preferences.

The proposal here is we can make synthetic client data. The approach here is first you send candidate outputs to the clients. Then you receive scores on the output quality and we can iteratively refine. The clients will calculate which synthetic data is closest to their own data. And then you can use DPO on each of the P rankings that you have to align the foundation model towards the private data.

POPri significantly outperforms previous synthetic data generation methods like Private Evolution and traditional on-device federated learning (DP Fed Average, DP FTRL, DPSG). On a next-token prediction task, POPri closed 58% of the performance gap between zero privacy and full data access. For a classification task, it closed 43% of the gap, demonstrating its effectiveness in generating high-quality, privacy-preserving synthetic data.

POPri... we're able to close 58% of the gap between epsilon equals zero and epsilon equals infinity. We're able to close 43% of the gap. It's a pretty significant gain.

POPri is more computationally expensive on the server side than Private Evolution due to RL fine-tuning, and has slightly higher communication costs due to sending more samples per round. However, it offers substantial advantages over traditional on-device federated learning by drastically reducing client-side computation (no on-device training, just similarity calculations) and communication (sending synthetic data/histograms instead of model weights).

Across the board POPri is more expensive than private evolution. We've replaced the in context learning stuff which is pretty cheap to a RL fine-tuning stuff which is going to on the server side is going to increase the burden a lot. The communication cost again is much better for POPri because we're not sending model weights and the client computation cost is also much cheaper because we don't have to do ondevice training.