Kangjoon Choi, Sungho Kim, Jeongmin Kim, and In-Cheol Park, “Hardware-friendly Approximation for Swish Activation and its Implementation,” IEEE Transactions on Circuits and Systems II: Express Briefs, Apr. 2024
Abstract: This paper addresses the challenges associated with implementing the Swish activation function in hardware. While Swish exhibits improved training performance over the ReLU activation function, its computational complexity poses difficulties in hardware implementation. To mitigate the drawback, an approximation called h-swish was introduced to reduce the computational complexity; however, there were still rooms for improvement. This paper proposes a novel hardware-friendly approximation of Swish and its implementation, which demonstrates significant improvements over h-swish in terms of delay, area, and power consumption.
