(Dec 9th) A Cross-Layer Approach to Faster CNN Training

High-performance DNN inference and training is essential for the ongoing ML revolution. Training of DNNs requires massive memory capacity and bandwidth, and is generally a huge pain, especially for researchers. While significant research effort has been dedicated to inference accelerator, less work has been done on training, especially work that crosses the algorithmic and implementation layers. The result is high-cost accelerators, in particular, with very expensive high-bandwidth memories. I will motivate and discuss some of our recent work on accelerating training (of CNNs) that combines an understanding of and changes to the algorithm with matching hardware architecture modifications. Specifically, I will present a technique that reduces the bandwidth requirements of training modern CNNs like ResNet and Inception by 4x and a second technique that can speed up the training of pruned networks by 2x or more even on current hardware. The algorithmic changes further motivate a new architecture that achieves the efficiency of a large systolic array without suffering low utilization when using our pruning while training approach.