Evaluating Multimodal Large Language Models on Identifying Image Rotation
A comprehensive benchmark for assessing the spatial reasoning capabilities of multimodal AI systems
RotBench is a novel evaluation framework designed to test the ability of Multimodal Large Language Models (MLLMs) to identify and reason about image rotations. This benchmark addresses a critical gap in evaluating spatial understanding capabilities of modern AI systems.
The benchmark consists of carefully curated images across multiple categories, each rotated at different angles, challenging models to demonstrate true visual understanding beyond simple pattern recognition.
Comprehensive collection of images across diverse categories including objects, scenes, and abstract patterns with controlled rotation angles.
Multiple evaluation criteria including accuracy, confidence calibration, and robustness across different rotation angles and image types.
Extensive testing across state-of-the-art MLLMs to provide comparative analysis and identify strengths and weaknesses.
Our evaluation reveals significant variations in performance across different MLLMs, with many models struggling with rotated images despite strong performance on standard benchmarks. The results highlight the importance of spatial reasoning capabilities in multimodal understanding.
The detailed methodology, comprehensive results, and analysis are available in our arXiv preprint:
The RotBench dataset, evaluation code, and results are openly available on GitHub. Contribute to the project or use it for your own research:
View on GitHub