Learning Distinct and Representative Styles for Image Captioning ...

• Main Authors: Chen, Qi, Deng, Chaorui, Wu, Qi
• Format: Report
• Language: unknown
• Published: arXiv 2022
• Subjects: Computer Vision and Pattern Recognition cs.CV; FOS Computer and information sciences; DML
• Online Access: https://dx.doi.org/10.48550/arxiv.2209.08231; https://arxiv.org/abs/2209.08231

Over the years, state-of-the-art (SoTA) image captioning methods have achieved promising results on some evaluation metrics (e.g., CIDEr). However, recent findings show that the captions generated by these methods tend to be biased toward the "average" caption that only captures the most general mode (a.k.a, language pattern) in the training corpus, i.e., the so-called mode collapse problem. Affected by it, the generated captions are limited in diversity and usually less informative than natural image descriptions made by humans. In this paper, we seek to avoid this problem by proposing a Discrete Mode Learning (DML) paradigm for image captioning. Our innovative idea is to explore the rich modes in the training caption corpus to learn a set of "mode embeddings", and further use them to control the mode of the generated captions for existing image captioning models. Specifically, the proposed DML optimizes a dual architecture that consists of an image-conditioned discrete variational autoencoder (CdVAE) ... : NeurIPS 2022 ...