Technische Universität München
TUM professor uses ChatGPT for choreographies with flying robots
TECHNICAL UNIVERSITY OF MUNICH
PRESS RELEASE
Combining robotics and ChatGPT
TUM professor uses ChatGPT for choreographies with flying robots
- Prof. Angela Schoellig has proved that large language models can be used safely in robotics.
- ChatGPT develops choreographies for up to nine flying robots performing to selected music.
- An additional algorithm guarantees that the drones will fly safely.
Prof. Angela Schoellig from the Technical University of Munich (TUM) uses ChatGPT to develop choreographies for swarms of drones to perform along to music. An additional safety filter prevents mid-air collisions. The researchers' results demonstrate the first time that large language models (LLMs) such as ChatGPT can be used in robotics.
The web interface is easy to use: Doctoral student Martin Schuck selects a music track and enters a text requesting a suggested choreography. Additional instructions can be given to the drone swarm via another prompt in the ChatGPT tool before an algorithm checks whether the suggested flight paths are feasible. The screen in the Learning Systems and Robotics Lab now shows a simulated airfield with six drones flying in circles to the music. If the scientist from the Chair of Safety, Performance and Reliability for Learning Systems at TUM likes this choreography, he logs it in. Soon afterwards, six palm-sized drones take off from the floor of the robotics lab.
Choreography with three flying robots is completed within five minutes
In Prof. Angela Schoellig's laboratory, the research team has installed six ceiling-mounted cameras in a room measuring around 40 square meters and three meters high. Crosses are marked on the floor with insulating tape. These indicate the starting positions of the drones. Once the computer has verified a possible choreography, they can take off. The cameras detect the position of the quadrocopters, which are equipped with four propellers and motors, 200 times a second. The system compares them with the desired position. The "airshows" in the Learning Systems and Robotics Lab, which the research team realizes with up to nine drones, are 100 percent safe today. Without the special safety filter, only one in four demonstrations is accident-free.
SwarmGPT: additional algorithm ensures safety of flying robots
Prof. Angela Schoellig combined ChatGPT with the safety filter for the "Dance of the Flying Robots". "The ChatGPT AI tool was primarily created to generate texts, but it can also suggest choreographies," says the professor. "However, it initially knows nothing about the properties of drones and physical limits for the flight paths. So it is clear that ChatGPT makes mistakes."
The additional safety algorithm closes this gap by mapping out flight paths for the proposed choreography so precisely that mid-air collisions are completely avoided. Drones can even approach each other diagonally. Prof. Schoellig calls the overall concept of ChatGPT and security filter designed to use several flying robots "SwarmGPT". The tool generates the processes in the air and at the same time serves as an interface between the robot and the human, who does not require any expert knowledge.
A quantum leap through ChatGPT
When Angela Schoellig began her drone research almost 15 years ago, choreographies were hand-crafted. It took more than three years to develop the first six choreographies for six drones and get them up and running. "ChatGPT has brought about a quantum leap," says Prof. Schoellig. Over the past three months, the researchers have experimented with over 30 choreographies for up to nine drones. Today, it takes the researchers only around five minutes to develop a safe choreography for 30-second music clips with three drones. The more drones are added, the more time ChatGPT needs for its calculations and the longer it takes to propose a choreography. But Schoellig is certain: "The concept is scalable."
Robotics: ChatGPT becomes an interface for non-experts
Can other robots be used with a similar interface via ChatGPT? For robots that use voice control to pick up objects, lay cables or open doors, the success rate of those actions is currently only 63, 56, and 80 percent. So far, applications in other robotics scenarios have been somewhat unreliable. For Prof Angela Schoellig, this is an incentive: "I assume that our approach will keep getting better in other scenarios, too." It may soon be possible to reprogram suction-based and industrial robots simply by voice command without the need for expert knowledge or programming skills.
More information
- Prof. Angela Schoellig is Head of the Chair of Safety, Performance and Reliability for Learning Systems at TUM and Director of the Munich Institute of Robotics and Machine Intelligence (MIRMI). At TUM's Robotics and AI Institute, more than 70 professors are focusing on developing robotic and intelligent solutions in the fields of health, the environment, mobility, and work.
- Swarm-GPT: Combining Large Language Models with Safe Motion Planning for Robot Choreography Design; NeurIPS 2023 workshop paper
Publications
Vivek K. Adajania, Siqi Zhou, Arun Kumar Singh, Angela P. Schoellig; AMSwarm: An Alternating Minimization Approach for Safe Motion Planning of Quadrotor Swarms in Cluttered Environments; IEEE; https://ieeexplore.ieee.org/document/10161063
Additional editorial information
Photo material: https://mediatum.ub.tum.de/image/1740784
Video: https://youtu.be/aKnhSnJHNpw
Scientific contact
Prof. Angela Schoellig
Chair of Safety, Performance and Reliability for Learning Systems
Technical University of Munich (TUM)
Angela.schoellig@tum.de
TUM Corporate Communications Center contact
Andreas Schmitz
Press Officer, Robotics and Machine Intelligence
0162-27 46 193
The Technical University of Munich (TUM) is one of the world’s leading universities in terms of research, teaching and innovation, with around 650 professorships, 52,000 students and 12,000 staff. TUM’s range of subjects includes engineering, natural and life sciences, medicine, computer sciences, mathematics, economics and social sciences. As an entrepreneurial university, TUM envisages itself as a global hub of knowledge exchange, open to society. Every year, more than 70 start-ups are founded at TUM, which acts as a key player in Munich’s high-tech ecosystem. The university is represented around the world by its TUM Asia campus in Singapore along with offices in Beijing, Brussels, Mumbai, San Francisco and São Paulo. Nobel Prize laureates and inventors such as Rudolf Diesel, Carl von Linde and Rudolf Mößbauer have conducted research at TUM, which was awarded the title of University of Excellence in 2006, 2012 and 2019. International rankings regularly cite TUM as the best university in the European Union.