Approximate Outer Product Gradient Descent with Memory

Code for the numerical experiment of the paper Speeding-Up Back-Propagation in DNN: Approximate Outer Product with Memory.

Environment

To use the code, install Anaconda with the following libraries:

• conda install scikit-image
• conda install pytorch==1.7.0 torchvision==0.8.0 torchaudio==0.7.0 cudatoolkit=10.2 -c pytorch

Reproducing Results

To reproduce any of the results, run one of the following codes for their corresponding plot in the paper:

• plot_figure_2a.py
• plot_figure_2b.py
• plot_figure_2c.py
• plot_figure_3a.py
• plot_figure_3b.py
• plot_figure_3c.py

Training Models

To train a single layered network on any of the schemes, use any of the following with the desired argument parameters:

• mnist_simulations.py
• energy_efficiency_simulatons.py

To train a single layered network on all the schemes in one go, use any of the following with the desired argument paramters:

• mnist_run.py
• energy_efficiency_run.py

Plotting Results

To plot the results from simulations, enter the argument parameters in the following codes:

• mnist_plots.py
• energy_efficiency_plots.py

Reference

If you use this code, please cite the following paper:

@misc{hernandez2021speedingup,
      title={Speeding-Up Back-Propagation in DNN: Approximate Outer Product with Memory}, 
      author={Eduin E. Hernandez and Stefano Rini and Tolga M. Duman},
      year={2021},
      eprint={2110.09164},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}