RLCD: Reinforcement Learning from Contrast Distillation for Language Model Alignment

Kevin Yang, Dan Klein, Asli Celikyilmaz, NANYUN (Violet) PENG, Yuandong Tian

RLCD: Reinforcement Learning from Contrast Distillation for Language Model Alignment: 11 upvotes on Hugging Face Daily Papers, #8 of 12 papers on 2023-07-25. Day-by-day upvote history.

We propose Reinforcement Learning from Contrast Distillation (RLCD), a method for aligning language models to follow natural language principles without using human feedback. RLCD trains a preference model using simulated preference pairs that contain both a high-quality and low-quality example, generated using contrasting positive and negative prompts. The preference model is then used to improve a base unaligned language model via reinforcement learning. Empirically, RLCD outperforms RLAIF (Bai et al., 2022b) and context distillation (Huang et al., 2022) baselines across three diverse alignment tasks--harmlessness, helpfulness, and story outline generation--and on both 7B and 30B model scales for preference data simulation.

Paper page on Hugging Face · arXiv

Data: hysts-bot-data/daily-papers-stats and the Daily Papers API. Open data: tardellirs/paper-pulse-data. Sister project: Model Pulse, the download history of every model on the Hub.