Multimodal Analogical Reasoning over Knowledge Graphs

About MarKG

Analogical reasoning is fundamental to human cognition and holds an important place in various fields. Thus, we introduce the multimodal analogical reasoning task, which can be formulate as link prediction without explicitly providing relations. To support it, we collect an Multimodal Analogical Reasoning dataSet (MARS) and a multimodal knowledge graph (MarKG). Among them, MARS has 10,685 training, 1,228 validation and 1,415 test instances. MarKG contains 11,292 entities, 192 relations and 76,424 images, include 2,063 analogy entities and 27 analogy relations.

For more tails about the task and the datasets, please refer to our ICLR 2023 paper:

ZHANG, LI, CHEN ET AL.

Getting Started

MarKG and MARS

MarKG and MARS are distributed under CC BY-SA 4.0 license , download the textual data of them by following links:

Image data

The image data (18.72G) of MARS and MarKG can be be downloaded through the Google Drive or the Baidu Yun(TeraBox)(code:7hoc) .

Checkpoints

We provide the best checkpoints of our transformer-based models during the fine-tuning and pre-training phrases at this link.

For more introduction and guidence, please check out our Github repository.

Connection

If you have any question about the multimodal analogical reasoning task and the datasets MARS and MarKG, or you want to submit your results to update the leaderboard, feel free to emtail to us:

If you use or extend our work, please cite the paper as follows:

@inproceedings{zhang2023multimodal,
    title={Multimodal Analogical Reasoning over Knowledge Graphs},
    author={Ningyu Zhang and Lei Li and Xiang Chen and Xiaozhuan Liang and Shumin Deng and Huajun Chen},
    booktitle={The Eleventh International Conference on Learning Representations },
    year={2023},
    url={https://openreview.net/forum?id=NRHajbzg8y0P}
}

Leaderboard

Rank	Model	Hit@1	Hit@3	Hit@10	MRR
1 Oct 01, 2022	MKGformer (MART) (SIGIR '22)	0.301	0.367	0.408	0.341
2 Oct 01, 2022	ViLBERT (MART) (NeurIPS '19)	0.256	0.312	0.347	0.292
3 Oct 01, 2022	FLAVA (MART) (CVPR '22)	0.264	0.303	0.319	0.288
4 Oct 01, 2022	RSME (ANALOGY) (ACM MM '21)	0.266	0.298	0.311	0.285
5 Oct 01, 2022	VisualBERT (MART) (arxiv '19)	0.261	0.292	0.321	0.284
6 Oct 01, 2022	IKRL (ANALOGY) (IJCAI '17)	0.266	0.294	0.310	0.283
7 Oct 01, 2022	TransAE (ANALOGY) (IJCNN '19')	0.261	0.285	0.293	0.276
8 Oct 01, 2022	ViLT (MART) (ICML '21)	0.245	0.275	0.303	0.266

MKG Analogy