Recent endeavors towards directly using large language models (LLMs) as agent models to execute interactive planning tasks have shown commendable results. Despite their achievements, however, they still struggle with brainless trial-and-error in global planning and generating hallucinatory actions in local planning due to their poor understanding of the "real" physical world. Imitating humans' mental world knowledge model which provides global prior knowledge before the task and maintains local dynamic knowledge during the task, in this paper, we introduce parametric World Knowledge Model (WKM) to facilitate agent planning. Concretely, we steer the agent model to self-synthesize knowledge from both expert and sampled trajectories. Then we develop WKM, providing prior task knowledge to guide the global planning and dynamic state knowledge to assist the local planning. Experimental results on three complex real-world simulated datasets with three state-of-the-art open-source LLMs, Mistral-7B, Gemma-7B, and Llama-3-8B, demonstrate that our method can achieve superior performance compared to various strong baselines. Besides, we analyze to illustrate that our WKM can effectively alleviate the blind trial-and-error and hallucinatory action issues, providing strong support for the agent's understanding of the world. Other interesting findings include: 1) our instance-level task knowledge can generalize better to unseen tasks, 2) weak WKM can guide strong agent model planning, and 3) unified WKM training has promising potential for further development
Figure 1: Overview of our WKM. We train a world knowledge model on the knowledge synthesized by the agent model itself from both expert and explored trajectories, providing prior task knowledge to guide global planning and dynamic state knowledge to assist local planning.
Table 1: Main Results. The best results are marked in bold and the second-best results are marked with underline. All the prompt-based baselines () are evaluated under one-shot prompting and all the fine-tuning- based baselines () are trained through LoRA. Red represents the changes of WKM relative to the optimal results in the baselines. WKM and agent model are different LoRAs sharing the same backbone.
Figure 2: Ablation Study on Mistral-7B. w/o all means the vanilla experienced agent model training with pure expert trajectories. w/ state is testing agent model with only state knowledge base constraints. w/ task stands for guiding agent model with only task knowledge. w/ task&state is our WKM with both task knowledge guidance and state knowledge constraints.
Table 2&3: Average Steps (left) and Hallucinatory Action Rates (right) on ALFWorld. The maximum number of steps in ALFWorld and WebShop is 40 and 10. In ScienceWorld, the number of steps ranges from 10 to 120 depending on the task type, with an average of around 40. We calculate the proportion of trajectories containing invalid actions regardless of their correctness.
Figure 3: Our instance-level knowledge can generalize better to unseen tasks. We compare the performance of dataset-level knowledge with our instance-level task knowledge (WKM w/o state) on ALFWorld and ScienceWorld. It can be observed that our model-generated instance- level knowledge not only surpasses human-designed knowledge on seen tasks but also exhibits even more remarkable performance on unseen tasks, with the improvement in performance on unseen tasks significantly greater than that on seen tasks. This phenomenon straightly reflects the strong generalization ability of our knowledge model compared to rigidly designed knowledge by humans.
Table 4: Weak knowledge model guides strong agent model planning. The results of both ChatGPT and GPT-4 show distinct advances after being guided by the Mistral-7B world knowledge model, indicating the weak world knowledge model also contains knowledge that the strong model may lack. In the era of LLMs, this inspires us with a new agent learning paradigm: weak-guide-strong. Due to its lightweight nature, the weak knowledge model can flexibly adjust its parameters based on the needs of the agent model, which can address the difficulty of large agent models in adapting to new environments through fine-tuning.
Figure 4: Unified World Knowledge Model Training. We can observe that multi-task WKM not only does not lead to performance degradation but also exhibits visible improvements compared to single-task WKM, especially on WebShop and ScienceWorld. This observation inspires us with the potential of training a unified world knowledge model that can be applied to help various held-in agent models and also generalize to guide held-out agent models. A more daring idea is whether a unified agent model combined with a unified world knowledge model is the key to Artificial General Intelligence (AGI).
Figure 5: Explicit state knowledge will hurt the planning performance. The performance of explicit state knowledge is far inferior to our approach of retrieving from a state knowledge base and utilizing probabilistic constraints. It even performs worse than when we remove state knowledge and only include task knowledge. This clearly indicates that blindly extending prompts with a large amount of explicit natural language feedback is lose-more-than-gain for agent planning, and implicit knowledge constraints may be sometimes more prudent.
@article{qiao2024agent,
title={Agent Planning with World Knowledge Model},
author={Qiao, Shuofei and Fang, Runnan and Zhang, Ningyu and Zhu, Yuqi and Chen, Xiang and Deng, Shumin and Jiang, Yong and Xie, Pengjun and Huang, Fei and Chen, Huajun},
journal={arXiv preprint arXiv:2405.14205},
year={2024}
}
This website is adapted from Nerfies, licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.