Generalized Sum Pooling for Metric Learning

• Main Authors: Gurbuz, Yeti Z., Sener, Ozan, Alatan, A. Aydın
• Format: Text
• Language: unknown
• Published: 2023
• Subjects: Computer Science - Computer Vision and Pattern Recognition; Computer Science - Machine Learning; Statistics - Machine Learning; DML
• Online Access: http://arxiv.org/abs/2308.09228

A common architectural choice for deep metric learning is a convolutional neural network followed by global average pooling (GAP). Albeit simple, GAP is a highly effective way to aggregate information. One possible explanation for the effectiveness of GAP is considering each feature vector as representing a different semantic entity and GAP as a convex combination of them. Following this perspective, we generalize GAP and propose a learnable generalized sum pooling method (GSP). GSP improves GAP with two distinct abilities: i) the ability to choose a subset of semantic entities, effectively learning to ignore nuisance information, and ii) learning the weights corresponding to the importance of each entity. Formally, we propose an entropy-smoothed optimal transport problem and show that it is a strict generalization of GAP, i.e., a specific realization of the problem gives back GAP. We show that this optimization problem enjoys analytical gradients enabling us to use it as a direct learnable replacement for GAP. We further propose a zero-shot loss to ease the learning of GSP. We show the effectiveness of our method with extensive evaluations on 4 popular metric learning benchmarks. Code is available at: GSP-DML Framework Comment: Accepted as a conference paper at International Conference on Computer Vision (ICCV) 2023