Dataset Distillation via Drifting

George Cazenavette, Angelina Quan, Giannis Daras, Antonio Torralba, Vincent Sitzmann

Dataset Distillation via Drifting: 3 upvotes on Hugging Face Daily Papers, #32 of 42 papers on 2026-11-17. Day-by-day upvote history.

Upvotes over time

It had 3 upvotes by its first day, 3 upvotes by the next day, 3 upvotes by day 3, 3 upvotes by day 7, and has 3 now. It ranked #98 of 146 papers in its week and #405 of 520 in its month.

The task of dataset distillation concerns the learning of a small set of synthetic samples that can serve as an effective replacement for a large real dataset when used for downstream tasks. While many solutions have been proposed since the problem's inception, most of them primarily seek to optimize the distilled data with respect to a classifier. In this work, we propose a new distillation method that learns a set of samples such that they directly approximate the per-class data distribution. Specifically, we "drift" our distilled data in the latent space of a feature extractor such that they move both toward the real data and away from each other without using a classification objective. We find that our drifted data cover the real distribution significantly better than existing methods while still serving as powerful training samples for classification. Furthermore, we showcase our method's use as a visualization tool for better interpreting models and extend it beyond images to other modalities. Our work advances the frontier of dataset distillation, offering a novel synthesis method and proposing distribution-based metrics as a complementary evaluation. More broadly, our findings help address the growing need to understand both data and models by uncovering more about how they interact.

Paper page on Hugging Face · arXiv

Data: hysts-bot-data/daily-papers-stats and the Daily Papers API. Open data: tardellirs/paper-pulse-data. Sister project: Model Pulse, the download history of every model on the Hub.