Enhancing Vision-Language Models with Synthetic Motion Data for Motion Risk Prediction

PDF view of the paper entitled Drivemrp: Enhancing Vision Language Models with Artificial Movement Data to Father Risk, by Zhiyi Hou and 13 other authors

PDF HTML (experimental) view

a summary:Independent leadership has seen remarkable progress, driven by real, large -scale data. However, in long length scenarios, the prediction of the future movement of the ego is still a major challenge due to uncertainty in dynamic environments and restrictions in data coverage. In this work, we aim to explore whether it is possible to enhance the ability to predict the risk of movement of VLM models by manufacturing high -risk movement data. Specifically, we offer the method of simulating movement to the bird’s eye (Bev) to the risk modeling of three aspects: the ego, other vehicles, and the environment. This allows us to collect high-risk motion data suitable for VLM training, which we call Drivemrp-10K. Moreover, we design a frame of the risk of VLM movement, called Drivemrp-Agent. This framework includes a new strategy to inject information for the global context, the perspective of the ego, and the projection of the path, enabling VLMS to think effectively about spatial relationships between road points and the environment. Intensive experiments show that through settling using Drivemrp-10K, our DrivemrP framework can significantly improve the performance of multiple risk predictions of multiple VLM agents, with an accuracy of accident recognition from 27.13 % to 88.03 %. Moreover, when testing it by evaluating the world’s high movement data set, Drivemrp-Agent achieves a big leap in performance, which enhances the accuracy of our way in the real world from 29.42 % to 68.50 %, which exposes the powerful generalization capabilities in the real world scenario.