Teaching: New course in Robot learning

I will be teaching a course on robot learning, starting in January 2022! This course will focus on deep reinforcement learning methods and their application to robotics. Here is a link to where you can find the course offered on the DIRO web page as IFT 6095. If you are interested in taking the course please fill out this form as well.

Learning methods such as deep reinforcement learning have shown success in solving simulated planning and control problems but struggle to produce diverse, intelligent behaviour, on robots. This class aims to discuss these limitations and study methods to overcome them and enable agents capable of training autonomously, becoming learning and adapting systems that require little supervision. By the end of the course, each student should have a solid grasp of different techniques to train robots to accomplish tasks in the real world. These techniques covered in the course include but are not limited to reinforcement learning, batch RL, multi-task RL, model-based RL, Sim2Real, hierarchical RL, goal conditioned RL, multi-Agent RL, the fragility of RL, meta-level decision making and learning reward functions.

Target and objectives of the course

Learn the fundamental concepts of machine learning for robotics applications. Such concepts are considered advanced and require a good foundation in machine learning, deep learning and reinforcement learning. This involves:

  • Becoming familiar with the main types of machine learning models for a control policy (from model-based to model-free)
  • Developing the ability to read research articles, contextualize them and develop a critical mind;
  • Develop presentation skills;
  • Develop their research autonomy in machine learning.
  • Develop skills related to the strengths and weaknesses of current machine learning methods when being applied to real-world problems.

Prerequisites of the course

You must have completed the following courses before taking my course. - IFT 6390, Fundamentals of machine learning

The course will also use Python heavily. I will assume familiarity with linear algebra, probability, statistics, planning, optimization, and operating systems (e.g. multi-threading and memory management). You also need to be able to read and understand research papers from NeurIPS, RSS, ICRA, CoRL, and ICLR.

Lectures covering topics related to learning robots as well as programming assignments and a final project.

The objectives of the assignments and final projects will be:

  • Software knowledge: Learning about the available software that is used for deep reinforcement learning.
  • Analysis skills: Collecting proper statistics of results and using server computers and docker to reproduce experiments and validate results.
  • Proposal of a new idea to explore for the final project based on the lectures;
  • Learn how to understand the potential of a method in the real world (positive or negative, social and environmental);
  • Exploration with final project: More free-form investigation of advanced topics from class that produces reusable code, reproducible results and a written report.
  • Estimation of the impact that the new idea could have in an industrial context;

The final project will be achieved as a team of two. Research in academia and industry involves working with others to achieve research goals. This project will evaluate the students’ ideas, research process understanding, teamwork, and presentation.

Regarding the lectures, the instructor will provide live lectures on the topics each week, focusing on their understanding, application, and limitations.

EVALUATIVE APPROACH AND WEIGHTING (indicative only)

  • 50%: Programming assignments. (This is needed to provide skills necessary to perform a good final project in the class)
  • 10%: Class participation and discussing readings.
  • 40%: Final project

Programming Assignments

The programming assignments cover topics that are important to perform research on combining machine learning and robots. They will cover:

  • Behaviour cloning and imitation learning
  • Model-free (PPO) and model-based RL methods (DDPG or PETS).
  • Exploration and pretraining methods, such as HRL and Goal conditioned RL.
  • Learning reward functions (VICE, un/smi-supervised RL)

The assignments are also designed to familiarize students with the software needed to perform research:

  • Deep learning libraries, such as pytorch or tensorflow
  • Hardware constraints when working with real robots (power, compute, mechanical limits)
  • Distributed computing for running proper experiments
  • Visualization and analysis (the most important part)

TextBooks

Final Project

The final project is designed to let students spread there wings and apply the learn skills in an area of interest of the students while digging deeper into the concepts. Your project does not need to use real robot hardware but there are options to get hardware for this class. Each project will start with a proposal that will ensure the right scope for each project. However, to provide some ideas on the scope a few examples are given below.

  • Reimpliment a method in a new deeplearning framework
  • Reimpliment a method in a paper that did not release code (shame on them)
  • Choose a robot to get and train a model to make it solve a task, like walk.
  • Investigate a new method that improves exploration
  • Investigate a new method for learning an improved representation for learning on robots.
  • Study Sim2Real by learning and transferring policies to another simulation or real robot hardware.
  • Perform a literature review of Sim2Real papers.

Some robot ideas for projects