Similarity‐based adversarial knowledge distillation using graph convolutional neural network

• Published in: Electronics Letters
• Main Authors: Sungjun Lee, Sejun Kim, Seong Soo Kim, Kisung Seo
• Format: Article in Journal/Newspaper
• Language: English
• Published: Wiley 2022
• Subjects: Electrical engineering. Electronics. Nuclear engineering; TK1-9971; DML
• Online Access: https://doi.org/10.1049/ell2.12543; https://doaj.org/article/9c39d2c5854c435ca81726a22f5637c9

Abstract This letter presents an adversarial knowledge distillation based on graph convolutional neural network. For knowledge distillation, many methods have been proposed in which the student model individually and independently imitates the output of the teacher model on the input data. Our method suggests the application of a similarity matrix to consider the relationship among output vectors, compared to the other existing approaches. The similarity matrix of the output vectors is calculated and converted into a graph structure, and a generative adversarial network using graph convolutional neural network is applied. We suggest similarity‐based knowledge distillation in which a student model simultaneously imitates both of output vector and similarity matrix of the teacher model. We evaluate our method on ResNet, MobileNet and Wide ResNet using CIFAR‐10 and CIFAR‐100 datasets, and our results outperform results of the baseline model and other existing knowledge distillations like KLD and DML.