Show, don't tell your robot what you want

It would be valuable to get a robot to do what you want without you having to program it.  If you could just show it what you want, and have it understand that, much programming effort could be saved.

I've recently taken a first step towards creating such a system.  It is very much a prototype at the moment.  You can read my recent paper for the details.  Here is an outline of the basic idea:

• Drive a camera-equipped robot through a target area.  
• Have the robot build a Growing Neural Gas network of the images it acquires as it drives around.  
• Growing Neural Gas is an unsupervised learning algorithm.
• It generates clusters of its inputs using a distance metric.  
• It is closely related to k-means and the Kohonen Self-Organizing Map.
• The number of clusters is determined by the algorithm.
• For each cluster, select a desired action.
• Set the robot free.  
• As it acquires each image, it determines the matching cluster.
• It then executes the action for that cluster.

Here is a screenshot of the "programming" interface.  This example was from a 26-node GNG.  Each cluster has a representative image.  The action is selected from a drop-down menu.  For these first experiments, we are training the robot on an obstacle-avoidance task.  The goal is to maximize forward motion while avoiding obstacles.

Needless to say, the system is not perfect at this stage.  The biggest problem is that some of the clusters are host to ambiguous actions.  So sometimes the robot reacts to "ghosts", and other times it hits obstacles.  Going forward, a major research focus will be to find a way to automate additional growing of the network to compensate for this problem.  

I'd like to share two videos of the robot in action.  In the first video, the robot does a good job of avoiding obstacles until it gets stuck under a table:

In the second video, the robot sees a number of ghosts, but for the most part it maintains good forward motion for over five minutes: