Perceiving the world has been one of the long-standing problems in computer vision and robotic intelligence. Whereas human perception works so effortlessly, even state-of-the-art algorithms experience difficulty in performing the same tasks. In this talk, I will talk about an online self-supervised learning framework for robotic visual perception, and present two case studies. In order to enable high-level or interactive tasks in the real 3D world, both semantic (‘what’) and spatial (‘where’) scene understanding are critical. Humans are known to have two distinct visual processing systems called  and  dorsal and stream, which are often called ‘what’ and ‘where’ pathways respectively. As opposed to conventional approaches in computer vision that have parallelized the two issues, my study is motivated by the interaction between the two systems.

I will present a study on monocular vision-based ground surface estimation and classification. The ground is the most important background object, which appears everywhere if on land. Being ubiquitous, the ground exhibits diverse visual features depending where you are and when it is. In this study, an online simultaneous geometric estimation and appearance-based classification of the ground is demonstrated using a large-scale dataset developed for autonomous driving car research.

I will also talk about an online self-supervised approach for 3D object tracking. Knowing the precise 3D pose of an object is crucial for interactive robotic tasks such as grasping and manipulation. The three complementary modules of shape, appearance, and motion of our framework enable the self-supervision mechanism to work without any pretraining. Our approach