Self-distillation with Online Diffusion on Batch Manifolds Improves Deep Metric Learning
Recent deep metric learning (DML) methods typically leverage solely class labels to keep positive samples far away from negative ones. However, this type of method normally ignores the crucial knowledge hidden in the data (e.g., intra-class information variation), which is harmful to the generalizat...
Main Authors: | , , , |
---|---|
Format: | Text |
Language: | unknown |
Published: |
2022
|
Subjects: | |
Online Access: | http://arxiv.org/abs/2211.07566 |
id |
ftarxivpreprints:oai:arXiv.org:2211.07566 |
---|---|
record_format |
openpolar |
spelling |
ftarxivpreprints:oai:arXiv.org:2211.07566 2023-09-05T13:19:05+02:00 Self-distillation with Online Diffusion on Batch Manifolds Improves Deep Metric Learning Zeng, Zelong Yang, Fan Liu, Hong Satoh, Shin'ichi 2022-11-14 http://arxiv.org/abs/2211.07566 unknown http://arxiv.org/abs/2211.07566 Computer Science - Computer Vision and Pattern Recognition text 2022 ftarxivpreprints 2023-08-16T17:23:27Z Recent deep metric learning (DML) methods typically leverage solely class labels to keep positive samples far away from negative ones. However, this type of method normally ignores the crucial knowledge hidden in the data (e.g., intra-class information variation), which is harmful to the generalization of the trained model. To alleviate this problem, in this paper we propose Online Batch Diffusion-based Self-Distillation (OBD-SD) for DML. Specifically, we first propose a simple but effective Progressive Self-Distillation (PSD), which distills the knowledge progressively from the model itself during training. The soft distance targets achieved by PSD can present richer relational information among samples, which is beneficial for the diversity of embedding representations. Then, we extend PSD with an Online Batch Diffusion Process (OBDP), which is to capture the local geometric structure of manifolds in each batch, so that it can reveal the intrinsic relationships among samples in the batch and produce better soft distance targets. Note that our OBDP is able to restore the insufficient manifold relationships obtained by the original PSD and achieve significant performance improvement. Our OBD-SD is a flexible framework that can be integrated into state-of-the-art (SOTA) DML methods. Extensive experiments on various benchmarks, namely CUB200, CARS196, and Stanford Online Products, demonstrate that our OBD-SD consistently improves the performance of the existing DML methods on multiple datasets with negligible additional training time, achieving very competitive results. Code: \url{https://github.com/ZelongZeng/OBD-SD_Pytorch} Comment: 14 pages Text DML ArXiv.org (Cornell University Library) |
institution |
Open Polar |
collection |
ArXiv.org (Cornell University Library) |
op_collection_id |
ftarxivpreprints |
language |
unknown |
topic |
Computer Science - Computer Vision and Pattern Recognition |
spellingShingle |
Computer Science - Computer Vision and Pattern Recognition Zeng, Zelong Yang, Fan Liu, Hong Satoh, Shin'ichi Self-distillation with Online Diffusion on Batch Manifolds Improves Deep Metric Learning |
topic_facet |
Computer Science - Computer Vision and Pattern Recognition |
description |
Recent deep metric learning (DML) methods typically leverage solely class labels to keep positive samples far away from negative ones. However, this type of method normally ignores the crucial knowledge hidden in the data (e.g., intra-class information variation), which is harmful to the generalization of the trained model. To alleviate this problem, in this paper we propose Online Batch Diffusion-based Self-Distillation (OBD-SD) for DML. Specifically, we first propose a simple but effective Progressive Self-Distillation (PSD), which distills the knowledge progressively from the model itself during training. The soft distance targets achieved by PSD can present richer relational information among samples, which is beneficial for the diversity of embedding representations. Then, we extend PSD with an Online Batch Diffusion Process (OBDP), which is to capture the local geometric structure of manifolds in each batch, so that it can reveal the intrinsic relationships among samples in the batch and produce better soft distance targets. Note that our OBDP is able to restore the insufficient manifold relationships obtained by the original PSD and achieve significant performance improvement. Our OBD-SD is a flexible framework that can be integrated into state-of-the-art (SOTA) DML methods. Extensive experiments on various benchmarks, namely CUB200, CARS196, and Stanford Online Products, demonstrate that our OBD-SD consistently improves the performance of the existing DML methods on multiple datasets with negligible additional training time, achieving very competitive results. Code: \url{https://github.com/ZelongZeng/OBD-SD_Pytorch} Comment: 14 pages |
format |
Text |
author |
Zeng, Zelong Yang, Fan Liu, Hong Satoh, Shin'ichi |
author_facet |
Zeng, Zelong Yang, Fan Liu, Hong Satoh, Shin'ichi |
author_sort |
Zeng, Zelong |
title |
Self-distillation with Online Diffusion on Batch Manifolds Improves Deep Metric Learning |
title_short |
Self-distillation with Online Diffusion on Batch Manifolds Improves Deep Metric Learning |
title_full |
Self-distillation with Online Diffusion on Batch Manifolds Improves Deep Metric Learning |
title_fullStr |
Self-distillation with Online Diffusion on Batch Manifolds Improves Deep Metric Learning |
title_full_unstemmed |
Self-distillation with Online Diffusion on Batch Manifolds Improves Deep Metric Learning |
title_sort |
self-distillation with online diffusion on batch manifolds improves deep metric learning |
publishDate |
2022 |
url |
http://arxiv.org/abs/2211.07566 |
genre |
DML |
genre_facet |
DML |
op_relation |
http://arxiv.org/abs/2211.07566 |
_version_ |
1776199897115000832 |