In his search for a suitable camera, he came across IDS NXT - a complete system for the use of intelligent image processing. It fulfilled all requirements and, thanks to artificial intelligence, much more besides pure gesture recognition.

Robot plays "Rock, Paper, Scissors" - Part 1/3

Case Study from | IDS Imaging

Gesture recognition with intelligent camera

Sebastian Trella is passionate about technology and robotics. In his own blog, he is always taking on new tasks. But he has hardly ever worked with image processing. However, a colleague's LEGO® MINDSTORMS® robot, which uses several different sensors to detect the gestures of a hand, gave him an idea: "The robot should be able to "see”." Until now, the gesture had to be made at a specific point in front of the robot in order to be reliably recognized. Several sensors were needed for this, which made the system inflexible and dampened the joy of playing. Can image processing solve this task more "elegantly"?

 

From the idea to implementation

In his search for a suitable camera, he came across IDS NXT - a complete system for the use of  intelligent image processing. It fulfilled all requirements and, thanks to artificial intelligence, much more besides pure gesture recognition. Trella's interest was piqued. This is because the evaluation of the images and the communication of the results took place directly on or through the camera - without an additional PC. The IDS NXT Experience Kit also came with all the components to get started with the application right away - no prior AI knowledge required.

Trella took the idea further and began to develop a robot that would play the game "Rock, Paper, Scissors" in the future - with a process similar to the classical sense: The (human) player is asked to perform one of the familiar gestures (scissors, rock, paper) in front of the camera. The virtual opponent has already randomly determined his gesture at this point. The move is evaluated in real time and the winner is displayed.

 

The first step: Gesture recognition by means of image processing

But until then, some intermediate steps were necessary. Trella began implementing gesture recognition using image processing - new territory for the robotics fan. However, with the help of IDS lighthouse - a cloud-based AI vision studio - this was easier to realize than expected. Here, ideas evolve into complete applications. For this purpose, neural networks are trained by application images with the necessary product knowledge, such as in this case the individual gestures from different perspectives, and packaged into a suitable application workflow.

"The training process was super easy and I just used IDS Lighthouse's step-by-step wizard after taking several hundred pictures of my hands using rock, scissor, or paper gestures from different angles against different backgrounds. The first trained AI was able to reliably recognize the gestures directly," explains Sebastian Trella. This works for both left- and right-handed users with a recognition rate of about 95%. Probabilities are returned for the labels "Rock", "Paper", "Scissor" or "Nothing". A satisfactory result. But what happens now with the data obtained?

 

Further processing

The further processing of the recognized gestures could be carried out by means of a specially created vision app. For this, the captured image of the respective gesture - after evaluation by the AI - must be forwarded to the app. The latter "knows" the rules of the game and can thus decide which gesture beats another. It then determines the winner. In the first stage of development, the app will also simulate the opponent. All this is currently in the making and will be implemented in the next step to the "Rock, Paper, Scissors"-playing robot.

 

From play to everyday use

Initially, the project is more of a gimmick. But what could come out of it? A slot machine? Or maybe even an AI based sign language translator?

 

To be continued...  Read Part 2/3 here.

 

The content & opinions in this article are the author’s and do not necessarily represent the views of RoboticsTomorrow
IDS Imaging Development Systems Inc.

IDS Imaging Development Systems Inc.

IDS is a leading manufacturer of industrial cameras "Made in Germany" with USB or GigE interfaces. Equipped with state of the art CMOS sensors, the extensive camera portfolio ranges from low-cost project cameras, to small, powerful models with PoE functionality or robust cameras with housings that fulfill the prerequisites for protection code IP65/67. For quick, easy and precise 3D machine vision tasks IDS offers the Ensenso series. With the novel vision app-based sensors and cameras of IDS NXT the company opens up a new dimension in image processing. Whether in an industrial or non-industrial setting: IDS cameras and sensors assist companies worldwide in optimizing processes, ensuring quality, driving research, conserving raw materials, and serving people. They provide reliability, efficiency and flexibility for your application.

Other Articles

2D cameras for positioning and inspecting ultra-fine wires in semiconductor production
Wire bonding is a key process in semiconductor production. Extremely fine wires with diameters of 15 to 75 micrometers are used to create tiny electrical connections between a semiconductor chip and other components.
Picking the right chart - (Semi-)Autonomous surface and underwater mapping for rivers and lakes
On the camera side, the Fraunhofer Institute relies on two uEye FA industrial cameras from IDS. The robust and resilient models with PoE are ideal for demanding environments.
Pallet by pallet - Intelligent robotic vision system destacks up to 800 objects per hour
One of the locations where RODE adds value is for DHL eCommerce in Rotterdam. In this machine, two Ensenso 3D cameras from IDS Imaging Development Systems GmbH are implemented to provide the required image data.
More about IDS Imaging Development Systems Inc.

Comments (0)

This post does not have any comments. Be the first to leave a comment below.


Post A Comment

You must be logged in before you can post a comment. Login now.

Featured Product

Discover how human-robot collaboration can take flexibility to new heights!

Discover how human-robot collaboration can take flexibility to new heights!

Humans and robots can now share tasks - and this new partnership is on the verge of revolutionizing the production line. Today's drivers like data-driven services, decreasing product lifetimes and the need for product differentiation are putting flexibility paramount, and no technology is better suited to meet these needs than the Omron TM Series Collaborative Robot. With force feedback, collision detection technology and an intuitive, hand-guided teaching mechanism, the TM Series cobot is designed to work in immediate proximity to a human worker and is easier than ever to train on new tasks.