Self-Supervised Optical Flow with Spiking Neural Networks and Event Based Cameras

Published in IROS 2021, 2021

Recommended citation: Kenneth Chaney, Artemis Panagopoulou, Chankyu Lee, Kaushik Roy, and Kostas Daniilidis (2021). "Self-Supervised Optical Flow with Spiking Neural Networks and Event Based Cameras." IROS 2021.

Abstract. Optical flow can be leveraged in robotic systems for obstacle detection where low latency solutions are critical in highly dynamic settings. While event-based cameras have changed the dominant paradigm of sending by encoding stimuli into spike trails, offering low bandwidth and latency, events are still processed with traditional convolutional networks in GPUs defeating, thus, the promise of efficient low capacity low power processing that inspired the design of event sensors. In this work, we introduce a shallow spiking neural network for the computation of optical flow consisting of Leaky Integrate and Fire neurons. Optical flow is predicted as the synthesis of motion orientation selective channels. Learning is accomplished by Backpropapagation Through Time. We present promising results on events recorded in real “in the wild” scenes that has the capability to use only a small fraction of the energy consumed in CNNs deployed on GPUs.

Recommended citation: Kenneth Chaney et. al. (2021). “Self-Supervised Optical Flow with Spiking Neural Networks and Event Based Cameras.” IROS 2021.