Characterizing Generalization under Out-Of-Distribution Shifts in Deep Metric Learning

• Main Authors: Milbich, Timo, Roth, Karsten, Sinha, Samarth, Schmidt, Ludwig, Ghassemi, Marzyeh, Ommer, Björn
• Format: Article in Journal/Newspaper
• Language: unknown
• Published: arXiv 2021
• Subjects: Machine Learning cs.LG; Computer Vision and Pattern Recognition cs.CV; FOS Computer and information sciences; DML
• Online Access: https://dx.doi.org/10.48550/arxiv.2107.09562; https://arxiv.org/abs/2107.09562

Deep Metric Learning (DML) aims to find representations suitable for zero-shot transfer to a priori unknown test distributions. However, common evaluation protocols only test a single, fixed data split in which train and test classes are assigned randomly. More realistic evaluations should consider a broad spectrum of distribution shifts with potentially varying degree and difficulty. In this work, we systematically construct train-test splits of increasing difficulty and present the ooDML benchmark to characterize generalization under out-of-distribution shifts in DML. ooDML is designed to probe the generalization performance on much more challenging, diverse train-to-test distribution shifts. Based on our new benchmark, we conduct a thorough empirical analysis of state-of-the-art DML methods. We find that while generalization tends to consistently degrade with difficulty, some methods are better at retaining performance as the distribution shift increases. Finally, we propose few-shot DML as an efficient way to consistently improve generalization in response to unknown test shifts presented in ooDML. Code available here: https://github.com/CompVis/Characterizing_Generalization_in_DML. : 35th Conference on Neural Information Processing Systems (NeurIPS 2021)