Robot TIAGo learned to bring 20 types of beer from the refrigerator

The robot recognizes the desired beer from 20 samples.

One of the participants of the competition of AI-systems Nvidia Jetson Challenge has found an extraordinary use of the system of artificial intelligence. While other participants tried to use machine vision in health systems, security, industrial IoT systems and innovative robotics solutions, German developers taught their robot PAL Robotics TIAGo ... to bring beer from the fridge!

The free robot TIAGo (“free” not in the sense of free will, but in the sense of open software and published design of some components for independent production and assembly) from the Spanish company PAL Robotics is a service robot designed to serve indoor people. The software is built on the standard Robot Operating System (ROS) open framework, so this model is ideal for various studies and experiments in the field of robotics.

Detailed documentation is available for TIAGo, including how to use OpenCV, PCL, offline navigation, MoveIt! and systems of interaction with people. There is a ready-made Gazebo simulator for quick testing of algorithms, robot design, regression testing, and learning of machine vision systems in realistic scenarios.

A group of German programmers - Rafael Memesheimer (team leader), Lukas Buchhold (semantic segmentation) and Ivanna Mihalchishin (manipulation) - connected a modular computer Nvidia Jetson TX2 , specifically designed for calculating AI tasks, including machine vision, to TIAGo.


Nvidia Jetson TX2. Photo: Nvidia

Jetson TX2 - energy-efficient built-in module for fast AI-computing, built on the architecture of Pascal . Thanks to a miniature form factor, low power consumption of 7.5 W, it is great for robots, drones, multikopter, smart cameras and portable medical devices, the manufacturer believes. The module is equipped with 8 GB of RAM, with a memory bus bandwidth of 59.7 GB / s. For reference, here are the technical specifications of the Jetson TX2, as well as the cheaper Jetson TX1 module of the previous generation on the Maxwell architecture.

	Jetson tx2	Jetson tx1
GPU	NVIDIA Pascal, 256 CUDA Core	NVIDIA Maxwell, 256 CUDA Core
CPU	HMP Dual Denver 2/2 MB L2 + Quad ARM A57 / 2 MB L2	Quad ARM A57 / 2 MB L2
Video	4K x 2K 60 Hz coding (HEVC) 4K x 2K 60 Hz decoding (12 bits)	4K x 2K 30 Hz (HEVC) 4K x 2K 60 Hz decoding (10 bits)
Memory	8 GB 128 bit LPDDR4 59.7 GB / s	4 GB 64 bit LPDDR4 25.6 GB / s
Display	2x DSI, 2x DP 1.2 / HDMI 2.0 / eDP 1.4	2x DSI, 1x eDP 1.4 / DP 1.2 / HDMI
CSI	Up to 6 cameras (two lines) CSI2 D-PHY 1.2 (2.5 Gbit / s per line)	Up to 6 cameras (two lines) CSI2 D-PHY 1.1 (1.5 Gbit / s per line)
PCIE	Gen 2 | 1x4 + 1x1 OR 2x1 + 1x2	Gen 2 | 1x4 + 1x1
Storage device	32 GB eMMC, SDIO, SATA	16 GB eMMC, SDIO, SATA
Other	CAN, UART, SPI, I2C, I2S, GPIOs	UART, SPI, I2C, I2S, GPIOs
USB	USB 3.0 + USB 2.0
Connection	1 Gigabit Ethernet, 802.11ac WLAN, Bluetooth
Dimensions and mechanics	50 × 87 mm (400-pin connector for connecting boards)

These toys cost $ 499 and $ 344, and bundled with a dev-kit is even more expensive (for educational institutions in the United States and Canada, a discount of $ 300 ).

So, on the basis of the TIAGo robot and the Jetson TX2 module, German students developed an autonomous control program that finds the way to the refrigerator, opens the door,



recognizes beer using the HomeNet semantic segmentation neural network ( source code ),



captures it with a manipulator,



closes the door



and lucky beer mistress.


Now, students seriously expect to receive one of the prizes at the competition - $ 10,000, NVIDIA TITAN Xp graphics card, Jetson TX2 developer kit or a free course at the Deep Learning Institute. The contest finalists will also be paid for tickets to the GTC 2018 (GPU Technology Conference) - a conference on GPU-innovation and Artificial Intelligence, which will be held March 25-29 in California.