EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Models

The paper introduces EmbSpatial-Bench, a benchmark that assesses the spatial understanding of LVLMs in embodied tasks, identifies their current limitations, and proposes EmbSpatial-SFT, an instruction-tuning dataset, to enhance their spatial reasoning capabilities.

Abstract

The recent rapid development of Large Vision-Language Models (LVLMs) has indicated their potential for embodied tasks.However, the critical skill of spatial understanding in embodied environments has not been thoroughly evaluated, leaving the gap between current LVLMs and qualified embodied intelligence unknown. Therefore, we construct EmbSpatial-Bench, a benchmark for evaluating embodied spatial understanding of LVLMs.The benchmark is automatically derived from embodied scenes and covers 6 spatial relationships from an egocentric perspective.Experiments expose the insufficient capacity of current LVLMs (even GPT-4V). We further present EmbSpatial-SFT, an instruction-tuning dataset designed to improve LVLMs' embodied spatial understanding.

Method

The authors used a methodology that involved constructing a benchmark called EmbSpatial-Bench, which is derived from embodied 3D scenes and evaluates six spatial relationships from an egocentric perspective. This benchmark is in the form of multiple-choice questions and is used to assess the spatial understanding capabilities of LVLMs. To address the identified deficiencies in these models, the authors created an instruction-tuning dataset named EmbSpatial-SFT, which includes tasks for spatial relationship identification and object localization. This dataset is used to fine-tune LVLMs, specifically MiniGPT-v2, to improve their performance on the benchmark. The fine-tuning process involved adjusting the visual connection module and LoRA modules in the LLM backbone, and the effectiveness of the instruction-tuning was validated through experiments that showed improved performance across different embodied environments.

Main Finding

The authors discovered that current Large Vision-Language Models (LVLMs), including advanced models like GPT-4V, exhibit insufficient spatial understanding capabilities when evaluated in embodied tasks using their newly constructed benchmark, EmbSpatial-Bench. This benchmark assesses spatial relationships from an egocentric perspective, which is crucial for embodied AI scenarios. The authors also found that by fine-tuning these models on their instruction-tuning dataset, EmbSpatial-SFT, which includes tasks for spatial relationship identification and object localization, the models' performance in spatial understanding tasks can be significantly improved. The fine-tuned models showed better spatial perception abilities across various scenarios, indicating that instruction tuning is an effective approach to enhance the spatial reasoning capabilities of LVLMs.

Conclusion

The conclusion of the paper is that while current Large Vision-Language Models (LVLMs) show promise in embodied AI tasks, they have significant deficiencies in understanding spatial relationships from an egocentric perspective, as revealed by the EmbSpatial-Bench benchmark. However, the authors demonstrate that these models can be effectively improved through instruction tuning using the EmbSpatial-SFT dataset. The fine-tuned models exhibit enhanced spatial understanding abilities, suggesting that instruction tuning is a viable method for advancing the spatial reasoning capabilities of LVLMs in embodied tasks.

Keywords

EmbSpatial-Bench, Large Vision-Language Models (LVLMs), spatial understanding, embodied tasks, egocentric perspective, instruction tuning, EmbSpatial-SFT, object localization, spatial relationship identification, benchmark, embodied AI, MiniGPT-v2, LoRA modules, LLM backbone, fine-tuning, performance improvement, embodied environments, 3D scenes, multiple-choice questions, dataset, evaluation, AI systems, intelligent agents, visual contexts, planning, embodied scenarios, embodied AI systems, spatial relationships