Robots Learn to Perform Tasks by “Watching” YouTube Videos

DARPA program advances robots’ ability to sense visual information and turn it into action

Robots can learn to recognize objects and patterns fairly well, but to interpret and be able to act on visual input is much more difficult. Researchers at the University of Maryland, funded by DARPA's Mathematics of Sensing, Exploitation and Execution (MSEE) program, recently developed a system that enabled robots to process visual data from a series of "how to" cooking videos on YouTube. Based on what was shown on a video, robots were able to recognize, grab and manipulate the correct kitchen utensil or object and perform the demonstrated task with high accuracy—without additional human input or programming.


"The MSEE program initially focused on sensing, which involves perception and understanding of what's happening in a visual scene, not simply recognizing and identifying objects," said Reza Ghanadan, program manager in DARPA's Defense Sciences Offices. "We've now taken the next step to execution, where a robot processes visual cues through a manipulation action-grammar module and translates them into actions."

Another significant advance to come out of the research is the robots' ability to accumulate and share knowledge with others. Current sensor systems typically view the world anew in each moment, without the ability to apply prior knowledge.

"This system allows robots to continuously build on previous learning—such as types of objects and grasps associated with them—which could have a huge impact on teaching and training," Ghanadan said. "Instead of the long and expensive process of programming code to teach robots to do tasks, this research opens the potential for robots to learn much faster, at much lower cost and, to the extent they are authorized to do so, share that knowledge with other robots. This learning-based approach is a significant step towards developing technologies that could have benefits in areas such as military repair and logistics."

The DARPA-funded researchers presented their work today at the 29th meeting of the Association for the Advancement of Artificial Intelligence. The University of Maryland paper is available here: http://ow.ly/I30im

Featured Product

Motion Controls Robotics, Inc. - The Random Robotic Case Erector System

Motion Controls Robotics, Inc. - The Random Robotic Case Erector System

Random Case Erectors are ideal for fulfillment centers and other end-of-line packaging needs. Save money by eliminating changeover downtime. Reduce ergonomic injuries by eliminating repetitive manual tasks. The Random Robotic Case Erector System includes: • FANUC M20iA/20 Robot • Combi gravity case magazine - holds 2 box sizes • Combi bottom flap folder with integrated 3M case sealing head. • Fixed robot base, anchored to the floor • Flexible end of arm tool to pick and erect multiple case sizes to eliminate changeover downtime • System wiring and controls • Color 7.5" touchscreen HMI display for selecting case size • Perimeter guarding with safety interlocks on doors • Programming for Robot, PLC/PMC, HMI and Vision • Complete documentation in electronic format • On-site training and support • Compliance with all applicable robotic safety standards. Alternate safety configuration may be available to allow an operator to changeover a case size in one magazine while the robot continues to work in other magazines. Contact Motion Controls Robotics to learn more about how to implement this system into your current end-of-line operations.