MLLM-SR: Conversational Symbolic Regression base Multi-Modal Large Language Models

The paper presents MLLM-SR, a novel approach to symbolic regression that uses multimodal large language models to generate mathematical expressions tailored to user-defined criteria expressed in natural language, offering a more intuitive and user-friendly alternative to traditional methods by allowing for the inclusion or exclusion of specific functions and properties in the generated expressions.

Abstract

Formulas are the language of communication between humans and nature. It is an important research topic of artificial intelligence to find expressions from observed data to reflect the relationship between each variable in the data, which is called a symbolic regression problem. The existing symbolic regression methods directly generate expressions according to the given observation data, and we cannot require the algorithm to generate expressions that meet specific requirements according to the known prior knowledge. For example, the expression needs to contain sin or be symmetric, and so on. Even if it can, it often requires very complex operations, which is very inconvenient. In this paper, based on multi-modal large language models, we propose MLLM-SR, a conversational symbolic regression method that can generate expressions that meet the requirements simply by describing the requirements with natural language instructions. By experimenting on the Nguyen dataset, we can demonstrate that MLLM-SR leads the state-of-the-art baselines in fitting performance. More notably, we experimentally demonstrate that MLLM-SR can well understand the prior knowledge we add to the natural language instructions. Moreover, the addition of prior knowledge can effectively guide MLLM-SR to generate correct expressions.

Method

The paper's main methodological approach for creating MLLM-SR, a conversational symbolic regression method, involved generating a substantial dataset of question-answer pairs that included observations and expressions represented as binary tree preorder traversals. A SetTransformer was trained with contrastive learning to extract features from the data, with its parameters frozen after training. A projection layer was then trained to align these features with the word embeddings of a large language model (LLM), which was used to generate expressions based on natural language instructions. The LLM's parameters were trained using the LoRA technique. A two-stage training process was employed, starting with pre-training the projection layer to align features with the LLM's embeddings while keeping the SetTransformer and LLM frozen, followed by end-to-end fine-tuning. For expressions containing constants, numerical optimization algorithms like BFGS were used to refine the constants after the LLM generated the expression's preorder traversal. The final model architecture integrated the SetTransformer for data encoding, the projection layer for feature alignment, and the LLM for expression generation, with the SetTransformer's parameters remaining frozen. The effectiveness of MLLM-SR was validated through experiments on the Nguyen dataset, demonstrating its ability to generate expressions that meet user-specified criteria and its superior performance compared to existing methods.

Main Finding

The authors discovered that MLLM-SR, their proposed conversational symbolic regression method, was capable of generating mathematical expressions that conform to user-defined requirements described in natural language. This was a significant advancement over traditional symbolic regression methods, which typically do not account for specific user preferences or prior knowledge. They also observed that MLLM-SR could effectively understand and apply prior knowledge provided in natural language instructions to guide the generation of correct expressions. MLLM-SR could be trained to handle a variety of conditions, such as generating expressions that include or exclude certain functions, exhibit specific properties like symmetry or periodicity, or adhere to other user-specified criteria. This flexibility and the ability to incorporate user-provided constraints into the expression generation process make MLLM-SR a powerful and user-friendly tool for non-experts in the field of symbolic regression.

Conclusion

The conclusion of the paper is that the authors have successfully developed MLLM-SR, a conversational symbolic regression method that utilizes multimodal large language models to generate mathematical expressions based on natural language instructions. This method not only outperforms existing state-of-the-art baselines in terms of fitting performance but also demonstrates the ability to understand and apply user-provided prior knowledge to guide the generation of expressions. MLLM-SR's capability to handle a variety of conditions and constraints in expression generation makes it a versatile and accessible tool for users who may not be experts in symbolic regression, thereby lowering the barrier to entry for utilizing such advanced AI algorithms in scientific and mathematical research.

Keywords

Symbolic Regression, Multimodal Large Language Models, Conversational AI, Expression Generation, Natural Language Instructions, Prior Knowledge, Fitting Performance, Nguyen Dataset, SetTransformer, Feature Alignment, Projection Layer, Two-Stage Training, Constant Optimization, BFGS Algorithm, LoRA Technique, Model Architecture, User-Friendly Interface, Scientific Research, Mathematical Formulas, Artificial Intelligence, Machine Learning, Neural Networks, Genetic Programming, Reinforcement Learning, Pre-Training, End-to-End Training, Noise Robustness, Versatility, Efficiency, Benchmarking, Coefficient of Determination (R2), Full Recovery Rate, Question-Answer Pairs, Data Feature Extraction, Binary Tree, Preorder Traversal, Arity Function, Generation Constraints, Training Data Collection, Instruction Effect Test, Discussion and Conclusion, References.