Technische Universität München
AI on board: earch robot thinks for itself
TECHNICAL UNIVERSITY OF MUNICH
NEWS RELEASE
Tracking objects with the help of language models
Search robot thinks for itself
- Researchers combine 3D image recognition with language models.
- Artificial intelligence can be integrated into any robot.
- Robots use continuous probability calculations to guide searches.
A robot that can locate lost items on command – this is the latest development at the Technical University of Munich (TUM). It combines knowledge from the internet with a spatial map of its surroundings to efficiently find the objects being sought.
The new robot from Prof. Angela Schoellig’s TUM Learning Systems and Robotics Lab looks like a broomstick on wheels with a camera mounted at the top. It is one of the first robots that not only integrates image understanding but also applies it to a clearly defined task.
To find a pair of glasses misplaced in the kitchen, for example, the robot has to look around and build a three-dimensional image of the room. The camera initially provides two-dimensional images, but these pixels also contain depth information. This creates a spatial map of the environment that is accurate to the centimeter and is constantly updated. A laptop also provides the robot with information about which objects are visible in the image and what significance they have for humans.
“We have taught the robot to understand its surroundings,” says Prof. Angela Schoellig. The head of the Robotics Lab at the TUM Chair of Safety, Performance and Reliability for Learning Systems aims to develop robots that can navigate any environment independently. Humanoid robots working in factories or robots in care settings in private homes require this newly developed basic understanding, which, as Schoellig explains, “is important for all robots that move in spaces that are constantly changing”.
Internet knowledge translated into the robot’s language
The robot therefore understands that a table or window sill can be used to briefly set down a pair of glasses, whereas a stovetop or a sink are not suitable for this purpose. “The language model captures the relationships between the objects and we convert this information into the robot’s language,” explains Prof. Schoellig. Two-digit numbers appear on the three-dimensional map of the environment, constantly recalculating the likelihood that the object being searched for is located there. According to the research results, the robot then searches the probable locations almost 30 per cent more efficiently than if it searched randomly throughout the room. Artificial intelligence is used in two ways: on the one hand in image recognition and on the other hand through the use of a language model.
Another special capability of the robot: it remembers previous images and is able to compare them with new images of its surroundings. If a new object suddenly appears in the kitchen, it recognizes the change with a high degree of certainty (95 per cent) and marks these areas as “highly probable” search locations.
Next step: searching behind cupboard doors
In the next step, the TUM scientist and board member at the Munich Institute of Robotics and Machine Intelligence (TUM MIRMI) also wants to search for objects that are in a drawer or behind a door. To do this, however, the robot will not merely have to draw on knowledge from the internet but will also have to interact with its surroundings. Robotic arms and hands must open a cupboard and determine whether it opens upwards or sideways and how best to grasp the handle. This will enable the robot to search even in closed spaces such as cupboards or drawers.
Publications:
Where did I leave my glasses? Open-Vocabulary Semantic Exploration in Real-World Semi-Static Environments; Benjamin Bogenberger, Oliver Harrison, Orrin Dahanaggamaarachchi, Lukas Brunke, Jingxing Qian, Siqi Zhou, Angela P. Schoellig; IEEE Robotics and Automation Letters, 3 March 2026; https://ieeexplore.ieee.org/document/11359697
Further information:
- Prof. Angela Schoellig is a member of the board of the Munich Institute of Robotics and Machine Intelligence (TUM MIRMI). The institute is an integrative research institute at the Technical University of Munich (TUM) that focuses on robotics and AI. The institute brings together expertise in key areas of robotics, including perception and data science. Nearly 80 TUM chairs are networked within MIRMI to develop innovative robotic and AI-supported solutions for the environment, health, mobility, work, as well as security and defense. TUM MIRMI is headed by Prof. Lorenzo Masia. Further information can be found at https://www.mirmi.tum.de/.
- A demonstration of the search robot can be seen until Friday, 13 March at the German Robotics Congress in Cologne: https://robotics-institute-germany.de/event/grc/
Additional material for media outlets:
- Photos for download: https://mediatum.ub.tum.de/1846659
- Scientific video: https://utiasdsl.github.io/semi-static-semantic-exploration/
Subject experts:
Prof. Angela Schoellig
Chair of Safety, Performance and Reliability for Learning Systems
Technical University of Munich (TUM)
TUM Corporate Communications Center — Media contact:
Andreas Schmitz
0162-27 46 193
andreas.schmitz@tum.de
The Technical University of Munich (TUM) is one of the world’s leading universities in terms of research, teaching and innovation, with around 700 professorships, 52,000 students and 12,000 staff. TUM’s range of subjects includes computer sciences, engineering, natural and life sciences, medicine, economics and social sciences. As an entrepreneurial university, TUM envisages itself as a global hub of knowledge exchange, open to society. Every year, around 100 start-ups are founded at TUM, which acts as a key player in Munich’s high-tech ecosystem. The university is represented around the world by its TUM Asia campus in Singapore along with offices in Beijing, Brussels, Mumbai, San Francisco, São Paulo and Shenzhen. Nobel Prize laureates and inventors such as Rudolf Diesel, Carl von Linde and Rudolf Mößbauer have conducted research at TUM, which was awarded the title of University of Excellence in 2006, 2012, 2019 and 2026. International rankings regularly cite TUM as the best university in the European Union.