This paper proposes PoolFormer, a Vision Transformer architecture that replaces the attention mechanism with simple spatial pooling. Surprisingly, PoolFormer achieves competitive performance on ImageNet, object detection, and semantic segmentation tasks, suggesting the general architecture of Vision Transformers matters more than the specific token mixer used.

The authors report working at Sea AI Lab and the National University of Singapore, however they state their work was completed during an internship. The authors may still have close ties to these organizations.

This paper proposes an extremely simple architecture, PoolFormer, which achieves competitive performance on image classification tasks. The paper’s strength lies in its simplicity and the strong message: the core architecture of Vision Transformers may be more important than the token mixers commonly used. While it doesn't surpass state of the art in many cases, its efficiency makes it quite promising for real-world applications.