Project title: Bi-VLA: Vision-Language-Action Model-Based System for Bimanual Robotic Dexterous Manipulations

Description:
The research introduces the Bi-VLA (Vision-Language-Action) model, a novel system designed for bimanual robotic dexterous manipulations that seamlessly integrates vision, language understanding, and physical action. Starling-LM-7B-alpha, an open-source large language model trained by Reinforcement Learning from AI Feedback (RLAIF) and Qwen-VL VLM by Alibaba are embedded in the proposed architecture. Bi-VLA demonstrates the ability to interpret complex human instructions, perceive and understand the visual context of ingredients, and execute precise bimanual actions to assemble the requested salad. You will collect the datasets and study the VLM/LLM architecture for bimanual collaborative robots. Through a series of experiments, Intern will evaluate the system’s performance in terms of accuracy, efficiency, and adaptability to various salad recipes and human preferences.

Candidates requirements:
➔ basic programming skills in Python or ROS / Gazebo

Supervisor: Associate Professor Dzmitry Tsetserukou

Internship duration: 1-2 months

The start date of the internship: July 12

End day: August 31

Contact person: Dzmitry Tsetserukou d.tsetserukou@skoltech.ru