Training Vision Transformers to Focus: Emergence of Spatial Attention Head Specialization

Yuji Nozawa, Yu-Chieh Lin, Youyang Ng

Training Vision Transformers to Focus: Emergence of Spatial Attention Head Specialization: 5 upvotes on Hugging Face Daily Papers, #21 of 39 papers on 2026-11-10. Day-by-day upvote history.

Upvotes over time

It had 4 upvotes by its first day, 4 upvotes by the next day, 5 upvotes by day 3, 5 upvotes by day 7, and has 5 now. It ranked #76 of 133 papers in its week and #223 of 340 in its month. Its code repository has 2 GitHub stars.

Self-supervised learning in Vision Transformer models, such as DINO, enables robust visual perception without requiring labeled data by leveraging large-scale image datasets. However, these models lack a focusing mechanism during training, resulting in perceptual representations that are often ambiguous or overly similar. In this work, we introduce two visual focusing loss functions designed to establish correspondence between the model's attention and specific regions within images. Specifically, we leverage the multi-head attention mechanism within the model to selectively steer the model's focus, leading to the emergence of spatial attention head specialization, characterized by diverse and spatially focused perceptual capabilities without requiring annotations. Through both qualitative and quantitative evaluations, we demonstrate that our method substantially increases the spatial selectivity and diversity of attention heads, improving both interpretability and performance on dense prediction tasks such as zero-shot image segmentation.

Paper page on Hugging Face · arXiv

Data: hysts-bot-data/daily-papers-stats and the Daily Papers API. Open data: tardellirs/paper-pulse-data. Sister project: Model Pulse, the download history of every model on the Hub.