On the Utility of Accounting for Human Beliefs about AI Behavior in Human-AI Collaboration

The paper presents a study on enhancing human-AI collaboration by designing AI agents that consider human beliefs about AI behavior, leading to improved performance, as evidenced by real-world experiments.

Abstract

To enable effective human-AI collaboration, merely optimizing AI performance while ignoring humans is not sufficient. Recent research has demonstrated that designing AI agents to account for human behavior leads to improved performance in human-AI collaboration. However, a limitation of most existing approaches is their assumption that human behavior is static, irrespective of AI behavior. In reality, humans may adjust their action plans based on their observations of AI behavior. In this paper, we address this limitation by enabling a collaborative AI agent to consider the beliefs of its human partner, i.e., what the human partner thinks the AI agent is doing, and design its action plan to facilitate easier collaboration with its human partner. Specifically, we developed a model of human beliefs that accounts for how humans reason about the behavior of their AI partners. Based on this belief model, we then developed an AI agent that considers both human behavior and human beliefs in devising its strategy for working with humans. Through extensive real-world human-subject experiments, we demonstrated that our belief model more accurately predicts humans' beliefs about AI behavior. Moreover, we showed that our design of AI agents that accounts for human beliefs enhances performance in human-AI collaboration.

Method

The authors used a multi-faceted methodology that included developing models of human behavior and beliefs, designing AI agents that incorporate these models, and conducting extensive human-subject experiments to evaluate the effectiveness of their approach. They utilized behavioral cloning to model human behavior, Bayesian inference to model human belief updating, and reinforcement learning techniques such as Proximal Policy Optimization (PPO) to train collaborative AI agents. The experiments involved both simulations and interactions with real human participants to assess the performance of AI agents trained with different assumptions about human behavior and beliefs.

Main Finding

The authors discovered that incorporating models of human behavior and beliefs into the design of AI agents can significantly improve the performance of human-AI collaboration. Their experiments showed that AI agents that account for human beliefs about AI behavior are more effective in working with humans than those that do not. This was evidenced by the higher collaborative performance achieved by AI agents trained with models that consider both human behavior and beliefs, compared to those trained without such considerations. The study also found that human behavior deviates significantly from optimality, highlighting the importance of using realistic models of human behavior in the design of collaborative AI systems.

Conclusion

The conclusion of the paper is that designing AI agents to account for human beliefs about AI behavior can significantly improve the performance of human-AI collaboration. The authors' experiments demonstrate the effectiveness of their approach in enhancing collaboration through the use of AI agents that are better aligned with human partners' expectations and actions. This suggests a path forward for creating AI systems that are more intuitive and efficient in collaborative tasks with humans.

Keywords

Human-AI collaboration, AI performance, human behavior, human beliefs, AI agent design, collaborative performance, multi-player goal-oriented Markov decision process (MDP), level-k reasoning framework, behavioral level-0 model, behavioral level-1 model, Bayesian inference, Proximal Policy Optimization (PPO), human-subject experiments, grid world environments, goal inference, explicable AI policy, collaborative AI agents, human models, belief models, societal impacts, transparency, reliance, team performance, behavioral cloning, inverse reinforcement learning, imitation learning, decision-making environment, goal-oriented MDP