Steerable Multi-Objective Group-Relative Preference Optimization with Jointly Trained Low-Rank Adapters

Constantinos Karouzos, Nikolaos Aletras

Steerable Multi-Objective Group-Relative Preference Optimization with Jointly Trained Low-Rank Adapters: 11 upvotes on Hugging Face Daily Papers, #18 of 34 papers on 2026-11-09. Day-by-day upvote history.

Upvotes over time

It had 7 upvotes by its first day, 11 upvotes by the next day, 11 upvotes by day 3, 11 upvotes by day 7, and has 11 now. It ranked #54 of 133 papers in its week and #160 of 340 in its month.

Language model deployment often requires different trade-offs between competing objectives as user preferences change, yet standard post-training methods typically produce a single policy for a fixed objective weighting. We introduce LambdaMix, a new parameter efficient method for learning a preference conditioned family of policies with group-relative online reinforcement learning. LambdaMix jointly trains one low-rank adapter per objective over a frozen base model. One preference vector selects the adapter mixture that generates responses and weights the separately normalized rewards used to update the adapters. A single training run supports changing preferences at inference without further optimization. We experiment with four models on helpfulness and harmlessness and on solving math problems while following instructions. We also evaluate the two largest models on rewriting comments to preserve wording while reducing clues about personal attributes. On the first two tasks, LambdaMix achieves higher hypervolume than all evaluated conditioned and policy merging baselines across all four models. LambdaMix uses fewer training completions and stores fewer adapter parameters than a set of GRPO specialist policies. With architecture and initialization fixed, varying adapter mixtures and objective weights together gives wider score ranges and higher coverage than varying either alone.

Paper page on Hugging Face · arXiv

Data: hysts-bot-data/daily-papers-stats and the Daily Papers API. Open data: tardellirs/paper-pulse-data. Sister project: Model Pulse, the download history of every model on the Hub.