Visual Navigation for Autonomous Vehicles: An Open-source Hands-on Robotics Course at MIT

TL;DR

This paper describes the development and open-sourcing of a comprehensive robotics course at MIT focused on visual navigation for autonomous vehicles, integrating theory, practical skills, and simulation tools for effective education.

Contribution

It introduces a novel, open-source course that bridges computer vision and robotics, emphasizing embodied intelligence and hands-on learning with drones and simulators.

Findings

Course successfully taught at MIT from 2018-2021

Open-source materials enable scalable online education

Students gain practical skills in visual navigation and perception

Abstract

This paper reports on the development, execution, and open-sourcing of a new robotics course at MIT. The course is a modern take on "Visual Navigation for Autonomous Vehicles" (VNAV) and targets first-year graduate students and senior undergraduates with prior exposure to robotics. VNAV has the goal of preparing the students to perform research in robotics and vision-based navigation, with emphasis on drones and self-driving cars. The course spans the entire autonomous navigation pipeline; as such, it covers a broad set of topics, including geometric control and trajectory optimization, 2D and 3D computer vision, visual and visual-inertial odometry, place recognition, simultaneous localization and mapping, and geometric deep learning for perception. VNAV has three key features. First, it bridges traditional computer vision and robotics courses by exposing the challenges that are specific to embodied intelligence, e.g., limited computation and need for just-in-time and robust perception to close the loop over control and decision making. Second, it strikes a balance between depth and breadth by combining rigorous technical notes (including topics that are less explored in typical robotics courses, e.g., on-manifold optimization) with slides and videos showcasing the latest research results. Third, it provides a compelling approach to hands-on robotics education by leveraging a physical drone platform (mostly suitable for small residential courses) and a photo-realistic Unity-based simulator (open-source and scalable to large online courses). VNAV has been offered at MIT in the Falls of 2018-2021 and is now publicly available on MIT OpenCourseWare (OCW).