Reducing Representational Interference in Retrieval Embeddings with Trainable Prefix Tokens

Seongtae Hong, Youngjoon Jang, Jungseob Lee, Chanjun Park, Heuiseok Lim

Reducing Representational Interference in Retrieval Embeddings with Trainable Prefix Tokens: 7 upvotes on Hugging Face Daily Papers, #22 of 32 papers on 2026-11-06. Day-by-day upvote history.

Upvotes over time

It had 2 upvotes by its first day, 6 upvotes by the next day, 7 upvotes by day 3, 7 upvotes by day 7, and has 7 now. It ranked #134 of 207 papers in its week and #201 of 340 in its month.

Embedding models compress variable-length text into fixed-size vector representations, but this compression can entangle semantic content with residual token-level information, such as artifacts arising from attention sink behavior. We propose a simple method that prepends trainable [DUMP] tokens to the input during embedding model training. The method keeps the model architecture, pooling strategy, and training loss unchanged, while optimizing the added tokens through the standard contrastive retrieval objective. By introducing trainable prefix positions that interact with regular tokens during encoding, [DUMP] tokens help reduce the influence of residual or uninformative token-level signals on the final retrieval representation. Experiments on standard retrieval and long-context benchmarks show that our method consistently improves over finetuned baselines across encoder and decoder embedding models. Further analysis shows that [DUMP] tokens alter token-level activation and attention patterns associated with low-information tokens. These results suggest that trainable prefix tokens provide a lightweight way to improve retrieval representations.

Paper page on Hugging Face · arXiv

Data: hysts-bot-data/daily-papers-stats and the Daily Papers API. Open data: tardellirs/paper-pulse-data. Sister project: Model Pulse, the download history of every model on the Hub.