| KEYWORD |
DAUIN - GR-03 - COMPUTER NETWORKS GROUP - NETGROUP
Programming Robotic Arms via NLP
Tesi all'estero
Parole chiave DEEP LEARNING, DEEP LEARNING, VIDEO ANALYSIS, MACHINE LEARNING, NATURAL LANGUAGE PROCESSING
Riferimenti GUIDO MARCHETTO, ALESSIO SACCO
Riferimenti esterni Prof. Flavio Esposito, Saint Louis University, USA
Gruppi di ricerca DAUIN - GR-03 - COMPUTER NETWORKS GROUP - NETGROUP
Descrizione This activity focuses on enhancing the control capabilities of a robotic arm through a language-based vision control system. The implementation involves setting up the Robot Operating System 2 (ROS2) to control the robot, and the integration of speech-to-text functionality is achieved by selecting a compatible speech recognition service/library for ROS, capturing audio input, converting it to text, and utilizing recent LLMs, e.g., ChatGPT, to generate corresponding commands. The integration process involves obtaining API keys, programmatically sending user inputs to the LLM, and processing the model's outputs to determine subsequent actions. A software bridge is coded to parse this responses and convert them into ROS commands. This bridge interprets natural language input, extracts actionable commands, and translates them into ROS-compatible messages to control the robot.
Finally, the last step involves creating a custom General Behavior Tree (GBT) with specific data and incorporating deep learning models for real-time object detection and recognition (such as YOLO).
Scadenza validita proposta 05/02/2025
PROPONI LA TUA CANDIDATURA