Coffee Chat: Robot Learning

School life is more interesting than ever. With no courses or TA duties, I have full luxury to learn new trends in robot learning. Amir massoud gave a talk on his ICRA 2009 paper in our group meeting today. Afterwards, we headed to SUB (the Student Union's Building @ Univ. of Alberta) for a coffee and a nice chat about robot learning, visual servoing, parametric vs. non-parametric, and model-based vs. model-free reinforcement learning.

In addition to Amir massoud's reading list on estimation and control of robotic systems, I added the following papers to my to-read list:

• Sethu Vijayakumar, Aaron D'Souza and Stefan Schaal, Incremental Online Learning in High Dimensions, Neural Computation, vol. 17, no. 12, pp. 2602-2634 (2005). [pdf] [More detailed version of the paper Tech Report EDI-INF-RR-0284 of UEDIN]

• Manuel Lopes and Bruno Damas, A Learning Framework for Generic Sensory-Motor Maps, Proc. IEEE/RSJ Intl. Conf. on Intelligent Robotics and Systems, pp. 1533-1538, San Diego, CA, USA, Oct 29 - Nov 2, 2007. [pdf]

Also of general interest to robot learning researchers are the publications of Stefan Schaal, Michael Mistry, and Jan Peters. Also check the rest of Vijayakumar's publications and his research group page at University of Edinburgh. The future of robot learning is very exciting.

Current trends in Robotic Visual Servoing

While I was searching for a sufficiently narrow topic for my thesis research, I thought it would be a good idea to find the current trends in our field by doing a statistical literature exploration. Our field, robotic visual servoing, has an extensive literature starting some 30 years ago with the pioneering work of Shirai and Inoue. Of course there weren't too many researchers working on vision-based control of manipulators those days, but this isn't true anymore. The past 2 decades has observed a significant advancement in vision-based control of robots.

A scholar.google search for the "visual servoing" keyword, shows that there are more than 9,000 relevant papers with more than 6,000 of them written after 1988. The interest in this field is growing rapidly (see figure). This diagram shows that in 2007 alone, there has been more than 600 papers that studied visual servoing in some form. However, the diagram doesn't show what the current trends in visual servoing are. It is very difficult and rather impossible to find the current trends using a search engine alone (I also tried google sets, but it didn't quite work out for our case). Therefore I tried to combine the "visual servoing" keyword with what I thought would be the future in visual servonig, to find the current trends.


When combined with "learning", the number of papers seems to be increasing year after year. The results of 2008 and 2009 would increase our confidence on the growth rate. BTW, I can't really explain what has happened in 2006, it might be due to the implementation overhead of robotics research. Or maybe it's been just a dull year for robotics.

I used to work on a completely calibrated positioin-based visual servo system, where even the CAD model of the objects to be manipulated were known. This was pretty much the typical robotics-in-automation or robotics-for-assembly set ups of the 1990's and earlier. Currently, visual servoing is applied to humanoids research and other settings where the robot needs to work in unstructured environments. The modeled world assumption is no longer valid and position-based approach is not quite applicable. The traditional image-based approaches also use a lot of a priori knowledge of the scene, camera, and the image Jacobian. If these are not known, can we still perform high-precision visual servoing?

To answer the above question, I combined "reinforcement learning" with "visual servoing". The results of 2007 and earlier aren't statistically significant. I predict that in the next 2-5 years we will notice the sudden impact of reinforcement learning in vision-based robotics.

My last statistical literature exploration was to combine "neural network" (NN) with "visual servoing". Neural networks were the flavour of the decade in the 1990's and many used techniques based on NNs to control robots. In the past few years, NNs had become out-of-fashion. It seems though, that NNs are getting some attention these days though.

P.S. This study is by no means complete, nor is meant to be. A thorough study would include more keywords such as ("adaptive control" OR "reinforcement learning") and/or ("visual servoing" OR (("robot" OR "robotics") AND ("motor control" OR "motor learning"))) and/or "hand-eye coordination" and/or "motion planning", etc.

On noise, parameter identification, and Jacobian estimation

Apparently, noise benefits system identification. Measurement noise, however, if more than a couple of pixels, ruins the Jacobian estimation. It's not rocket science, it's all obvious. But after artificial generation of all of the visual-motor experiments and verifying the correctness of the Jacobian initialization method, I can sleep better at night. The wrong estimation of the visual-motor Jacobian was only due to the visual tracking noise and one faulty recorded data.

For the MATLAB simulations that verified the above, I calibrated the camera (COSMICAR 6mm Lens, PointGrey Dragonfly Express Firewire Camera) with the Camera Calibration Toolbox. Then I created a camera and placed it on the WAM arm model that I'd created earlier using the Robotics Toolbox and guessed the 3D coordinates of the feature points that I placed on a virtual wall in the simulations. Since the joint angle readings from the WAM arm are very accurate, I used the exact same readings and projected the feature points on the camera image at the new camera poses. I noticed that when using a few data points, adding a zero-mean Gaussian noise with a very small variance reduces the error in the estimation of the next joint value. When I increased the variance, the estimation became erroneous, as expected. For example, the results were reliable when the standard deviation of the noise was in the order of 0.05, but when the standard deviation was 1.5, then the estimation error was very large.

To avoid the unavoidable measurement noise, I'd filter the visual tracking results every 4-5 readings. Before filtering, the system tracks at 100 Hz, and I guess I can sacrifice some bandwidth for now (the system still works in 20-25 Hz).

{modelChecking.robotics} Robotic Controllers

Any developer involved in developing complex interactive systems such as robotic controllers agrees that validation and verification of such systems is a huge contribution. Many developers prefer not to get involved in model checking when developing simple applications, but at some point they will realize that as the size of the project grows it will get more and more difficult to test the system. Prototyping a general method that works for all different languages and different robotics setups might not be feasible. We have a specific hardware setup, so I might be able to develop some architecture for our setup in which developing robotic applications is reliable and error-free. I just might.

Master of Science in Vision and Robotics

Here is a list of courses that I think would give the right background (mathematics, computer science, mechanics, signal processing) to do good work in computer vision and robotics. Probably good for a 5-year course based master's degree in an EECS department. The more I work in this area, the more I feel that Robotics and Computer Vision is a field by itself and can't fit the traditional EE or CS program, because it needs a strong math background, a solid and understanding of mechanics in addition to the traditional education given in EE and CS. For example, many courses in traditional EE are not useful (for example 3 courses in electronics) to robotics and computer vision in particular. The CS graduates don't traditionally learn about signal processing and control theory on the other hand. We should come up with a good mixture.

--------------------------
Math background
--------------------------
Logic
Calculus I, II
Real Analysis
Linear Algebra
Algebraic Geometry
Differential Algebra
Complex Analysis
Ordinary Differential Equations
Partial Differential Equations
Applied Probabilities and Statistics
Numerical Methods for Computation
Graph Theory
Complexity Theory
Mathematical Optimization
Differential Geometry
Variational Calculus
-- 18

---------------------------------------
Computer Science background
---------------------------------------
Algorithms I,II
Complexity Theory
Languages and Automata
Programming
Object-oriented programming
UNIX programming
Computer Organization
Operating Systems
Formal Methods
System Architecture
System Validation and Verification
-- 12

--------------------------------------
Signal Processing Background
--------------------------------------
Fourier Analysis
Signals and Systems
Information Theory
Digital Signal Processing
Digital Image Processing
-- 5
---------------------------------------
Computer Vision and Graphics
---------------------------------------
Foundations of Computer Vision
3D Computer Vision
Feature Extraction and Motion Tracking
Computer Graphics
--4

--------------------------------------------------
Control Theory and Machine Learning
--------------------------------------------------
Linear Control Theory
Digital Control Theory
Nonlinear Control
System Identificaton
Statistical Methods in Learning
Machine Learning
Optimal Control
-- 7

--------------------------------
Mechanics and Robotics
--------------------------------
Dynamics
Statics
Digital Design
Mechatronics (Microcontrollers, Servo motors, DC motors, necessary electronics)
Robot Mechanics
Mobile Robots and Navigation
Calibration Methods
-- 7

50 courses

{favorite.quote} do not re-invent the square wheel

I have heard that getting the Segway robot to work properly (navigation, localization, etc.) may take quite a while. There are others in the department who have spent years on Segway development. My initial reaction when I heard this was, "nice! There is no need to re-invent the wheel." I am not sure how much this could be true and how much different professors are willing to share their students time. I am just hoping that I do not re-invent a square wheel.

Ironically, today, I read an article by Henry Spencer, "How to Steal Code or Inventing the Wheel Only Once". Somewhere in the article, I found a nice quote,

If you re-invent the square wheel, you will not benefit when somebody else rounds off the corners.
--Collyer & Spencer

PS: Henry Spencer is fun to read. Have you read his "Ten Commandments for C Programmers"?

{robotics.back} After two weeks or so ...

I guess I have rambled on sports too long and right now my play-off fever seems to be gone. Well, Oilers are winnig the Western Conference Finals and I have been cheering too much with them. It is time to focus back on my stuff.

The above pix is the newest member of our lab. Although it is still in the box (as it arrived just last week), it will look more or less the same when put together. It is a Segway RMP 200. For the record, RMP stands for Robotic Mobility Platform. We intend to put a WAM arm on it and move it around to do some cool stuff. This platform would be twins with NASA's Robotnaut and that is exciting. I am sure that there would be tons of implications to make that thing work. But, nothing's impossible.

{dream.job} astronaut

Moving to University of Alberta got me one step closer to my dream job. Don't take me wrong. Earning a PhD (stands for Piled Higher & Deeper) is not a dream. It's a step. Don't take me wrong again. Being hired as a university professor is also not a dream. It's another step (maybe not!). I always wanted to become an astronaut and explore the unknown. Why, you asking? Donno!

Star Trek influence might it be;
Star Wars passion might it be (I adore you master Yoda);
Whatever it is, it's real folks.

I never wanted to become a surgeon (though I may end up doing Robotics Tele-Surgery at some point!) to serve humanity by saving (or not saving) lives. I am not a huge fan of bloodshed, nah! I always wanted to become an astronaut and destiny is helping me contribute (though very slightly) to building a system such as NASA's Robonaut. If I am good, they may want me to maintain it on site. Wait! Did the project proposal say autonomous?

{hobbies.robotics} NXT to be released in F'06

Robotic kits may be a toy for many big kids, but for me they are more. Sort of similar to the map of the treasure island, which is more than just a map or a puzzle. Robotic kits are the guide to a valuable treasure.