Ruby Teaming: Improving Quality Diversity Search with Memory for Automated Red Teaming

Vernon Toh Yan Han, Rishabh Bhardwaj, Soujanya Poria

Ruby Teaming: Improving Quality Diversity Search with Memory for Automated Red Teaming: 6 upvotes on Hugging Face Daily Papers, #16 of 27 papers on 2024-06-24. Day-by-day upvote history.

We propose Ruby Teaming, a method that improves on Rainbow Teaming by including a memory cache as its third dimension. The memory dimension provides cues to the mutator to yield better-quality prompts, both in terms of attack success rate (ASR) and quality diversity. The prompt archive generated by Ruby Teaming has an ASR of 74%, which is 20% higher than the baseline. In terms of quality diversity, Ruby Teaming outperforms Rainbow Teaming by 6% and 3% on Shannon's Evenness Index (SEI) and Simpson's Diversity Index (SDI), respectively.

Paper page on Hugging Face · arXiv

Data: hysts-bot-data/daily-papers-stats and the Daily Papers API. Open data: tardellirs/paper-pulse-data. Sister project: Model Pulse, the download history of every model on the Hub.