Dual-Alignment Pre-training for Cross-lingual Sentence Embedding

Ziheng Li, HUANG SHAOHAN, Zihan Zhang, Zhi-Hong Deng, Qiang Lou, Huang, Jian Jiao, Furu Wei, deng, Qi Zhang

Dual-Alignment Pre-training for Cross-lingual Sentence Embedding: 1 upvotes on Hugging Face Daily Papers, #10 of 10 papers on 2023-05-17. Day-by-day upvote history.

Recent studies have shown that dual encoder models trained with the sentence-level translation ranking task are effective methods for cross-lingual sentence embedding. However, our research indicates that token-level alignment is also crucial in multilingual scenarios, which has not been fully explored previously. Based on our findings, we propose a dual-alignment pre-training (DAP) framework for cross-lingual sentence embedding that incorporates both sentence-level and token-level alignment. To achieve this, we introduce a novel representation translation learning (RTL) task, where the model learns to use one-side contextualized token representation to reconstruct its translation counterpart. This reconstruction objective encourages the model to embed translation information into the token representation. Compared to other token-level alignment methods such as translation language modeling, RTL is more suitable for dual encoder architectures and is computationally efficient. Extensive experiments on three sentence-level cross-lingual benchmarks demonstrate that our approach can significantly improve sentence embedding. Our code is available at https://github.com/ChillingDream/DAP.

Paper page on Hugging Face · arXiv

Data: hysts-bot-data/daily-papers-stats and the Daily Papers API. Open data: tardellirs/paper-pulse-data. Sister project: Model Pulse, the download history of every model on the Hub.