Project title: CognitiveDog: Large Multimodal Model Based System to Translate Vision and Language into Action of Qadruped Robot

Description:
The project will focus on CognitiveDog, a pioneering development of quadruped robot with Large Multi-modal Model (LMM) that is capable of not only communicating with humans verbally but also physically interacting with the environment through object manipulation. The system is realized on Unitree Go1 robot-dog equipped with a custom gripper and demonstrated autonomous decision-making capabilities, independently determining the most appropriate actions and interactions with various objects to fulfill user-defined tasks. These tasks do not necessarily include direct instructions, challenging the robot to comprehend and execute them based on natural language input and environmental cues. Key to this development is the robot’s proficiency in navigating space using Visual-SLAM, effectively manipulating and transporting objects, and providing insightful natural language commentary during task execution. The Intern will develop architecture of Large Language Models and Visual Language model to achieve intelligent interaction of robotic dog with human and environment. The dataset used to fine-tune the robot-dog behavior generation model is provided at the following link.

Candidates requirements:
➔ basic programming skills in Python or ROS / Gazebo

Supervisor: Associate Professor Dzmitry Tsetserukou

Internship duration: 1-2 months

The start date of the internship: July 12

End day: August 31

Contact person: Dzmitry Tsetserukou d.tsetserukou@skoltech.ru