Neural Network Compression via Learnable Wavelet Transforms
Abstract
Wavelets are well known for data compression, yet have rarely been applied to the compression of neural networks. This paper shows how the fast wavelet transform can be used to compress linear layers in neural networks. Linear layers still occupy a significant portion of the parameters in recurrent neural networks (RNNs). Through our method, we can learn both the wavelet bases and cor-responding coefficients to efficiently represent the linear layers of RNNs. Our wavelet compressed RNNs have significantly fewer parameters yet still perform competitively with the state-of-the-art on synthetic and real-world RNN benchmarks. Wavelet optimization adds basis flexibility, without large numbers of extra weights
Keywords: network compression, wavelets
Source code available at https://github.com/v0lta/Wavelet-network-compression .
Images
![]() |
Bibtex
@INPROCEEDINGS{wolter2020neural, author = {Wolter, Moritz and Lin, Shaohui and Yao, Angela}, title = {Neural Network Compression via Learnable Wavelet Transforms}, booktitle = {International Conference on Artificial Neural Networks}, year = {2020}, publisher = {Springer}, keywords = {network compression, wavelets}, abstract = {Wavelets are well known for data compression, yet have rarely been applied to the compression of neural networks. This paper shows how the fast wavelet transform can be used to compress linear layers in neural networks. Linear layers still occupy a significant portion of the parameters in recurrent neural networks (RNNs). Through our method, we can learn both the wavelet bases and cor-responding coefficients to efficiently represent the linear layers of RNNs. Our wavelet compressed RNNs have significantly fewer parameters yet still perform competitively with the state-of-the-art on synthetic and real-world RNN benchmarks. Wavelet optimization adds basis flexibility, without large numbers of extra weights}, url = {https://arxiv.org/pdf/2004.09569.pdf} }