Abstract

Analogical reasoning is fundamental to human cognition and holds an important place in various fields. However, previous studies mainly focus on single-modal analogical reasoning and ignore taking advantage of structure knowledge. Notably, the research in cognitive psychology has demonstrated that information from multimodal sources always brings more powerful cognitive transfer than single modality sources. To this end, we introduce the new task of multimodal analogical reasoning over knowledge graphs, which requires multimodal reasoning ability with the help of background knowledge. Specifically, we construct a Multimodal Analogical Reasoning dataSet (MARS) and a multimodal knowledge graph MarKG. We evaluate with multimodal knowledge graph embedding and pre-trained Transformer baselines, illustrating the potential challenges of the proposed task. We further propose a novel model-agnostic Multimodal analogical reasoning framework with Transformer (MarT) motivated by the structure mapping theory, which can obtain better performance. We hope our work can deliver benefits and inspire future research.

Task Definition

We introduce the task of Multimodal Analogical Reasoning that can be formulated as link prediction without explicitly providing relations. We further divide the task into Single Analogical Reasoning and Blended Analogical Reasoning according different modalities the entities:

1. Single Analogical Reasoning: In this setting, the analogy example and the question-answer entity pair involve only one modality. This setting can be further divided into (I_h, I_t) : (T_q, ?) and (T_h, T_t) : (I_q, ?), where T_* and I_* represent the modality is textual and visual respectively.
2. Blended Analogical Reasoning: This task can be formalized as (I_h, T_t) : (I_q, ?).

Demo

This is a demo for the multimodal analogical reasoning task, you can also try it on the hugginface space MKG_Analogy.

If this paper or datasets was helpful for your research, please use the following citation:

@inproceedings{zhang2023multimodal,
    title={Multimodal Analogical Reasoning over Knowledge Graphs},
    author={Ningyu Zhang and Lei Li and Xiang Chen and Xiaozhuan Liang and Shumin Deng and Huajun Chen},
    booktitle={The Eleventh International Conference on Learning Representations },
    year={2023},
    url={https://openreview.net/forum?id=NRHajbzg8y0P}
}