A Survey on Compositional Learning of AI Models: Theoretical and Experimetnal Practices

This survey paper investigates the state of compositional learning in AI, exploring how computational models can learn to combine simple concepts into more complex ones, and evaluates the theoretical and practical challenges in achieving this, drawing from cognitive science and linguistics to inform future research directions.

Abstract

Compositional learning, mastering the ability to combine basic concepts and construct more intricate ones, is crucial for human cognition, especially in human language comprehension and visual perception. This notion is tightly connected to generalization over unobserved situations. Despite its integral role in intelligence, there is a lack of systematic theoretical and experimental research methodologies, making it difficult to analyze the compositional learning abilities of computational models. In this paper, we survey the literature on compositional learning of AI models and the connections made to cognitive studies. We identify abstract concepts of compositionality in cognitive and linguistic studies and connect these to the computational challenges faced by language and vision models in compositional reasoning. We overview the formal definitions, tasks, evaluation benchmarks, variety of computational models, and theoretical findings. We cover modern studies on large language models to provide a deeper understanding of the cutting-edge compositional capabilities exhibited by state-of-the-art AI models and pinpoint important directions for future research.

Method

The authors of the paper utilized a survey methodology to examine the literature on compositional learning in AI models. They conducted a comprehensive review of existing research, identifying key concepts, methodologies, and findings related to the ability of AI models to learn and reason compositionally. This included an analysis of cognitive and linguistic studies, computational models, evaluation benchmarks, and theoretical frameworks. The survey also covered modern studies on large language models to understand their cutting-edge compositional capabilities and to pinpoint important directions for future research.

Main Finding

The authors of the paper employed a survey-based methodology to systematically review the literature on compositional learning within AI models. This approach involved identifying, analyzing, and synthesizing relevant studies, theories, and empirical findings from the fields of cognitive science, linguistics, and computational modeling. They examined the cognitive aspects of compositionality, the formal definitions and tasks associated with it, the evaluation benchmarks used, and the variety of computational models developed to address compositional reasoning challenges. The survey also covered the theoretical underpinnings of compositional learning, the practical applications of these models, and the experimental results obtained from benchmark datasets. Through this method, the authors aimed to provide a comprehensive overview of the current state of research in compositional learning and to highlight areas for future investigation.

Conclusion

The paper concludes that compositional learning, a critical aspect of human cognition essential for generalizing to unobserved situations, is under-researched in AI models. Traditional symbolic AI models support compositional reasoning but falter with noisy data, while modern neural models, like large language models, handle noisy data better but struggle with compositional generalization due to data contamination and memorization. Neuro-symbolic modeling, which integrates neural networks with symbolic reasoning, is a promising avenue for future research. Theoretical studies indicate that the compositional generalization capabilities of current AI models are not fully understood, highlighting the need for more systematic research to develop models that can effectively generalize to new situations.

Keywords

Compositional learning, AI models, cognitive studies, linguistic studies, computational challenges, systematic research, generalization, unobserved situations, formal grammars, neural networks, large language models, neuro-symbolic models, systematicity, productivity, substitutivity, localism, overgeneralization, theoretical analysis, experimental analysis, benchmarks, datasets, cognitive aspects, compositional reasoning, generalization capabilities, systematic generalization, neuro-symbolic computing, theoretical findings, emerging abilities, data contamination, memorization, cognitive science, linguistics, psychology, neural module networks, neurocompositional computing, transformers, language models, compositional generalization.