Hold me tight! Influence of discriminative features on deep network boundaries

This is the source code to reproduce the experiments of the NeurIPS 2020 paper "Hold me tight! Influence of discriminative features on deep network boundaries" by Guillermo Ortiz-Jimenez*, Apostolos Modas*, Seyed-Mohsen Moosavi-Dezfooli and Pascal Frossard.

Abstract

Important insights towards the explainability of neural networks reside in the characteristics of their decision boundaries. In this work, we borrow tools from the field of adversarial robustness, and propose a new perspective that relates dataset features to the distance of samples to the decision boundary. This enables us to carefully tweak the position of the training samples and measure the induced changes on the boundaries of CNNs trained on large-scale vision datasets. We use this framework to reveal some intriguing properties of CNNs. Specifically, we rigorously confirm that neural networks exhibit a high invariance to non-discriminative features, and show that very small perturbations of the training samples in certain directions can lead to sudden invariances in the orthogonal ones. This is precisely the mechanism that adversarial training uses to achieve robustness.

Dependencies

To run our code on a Linux machine with a GPU, install the Python packages in a fresh Anaconda environment:

$ conda env create -f environment.yml
$ conda activate hold_me_tight

Experiments

This repository contains code to reproduce the following experiments:

• Train and compute the margin distribution on MNIST
• Train and compute the margin distribution on Frequency-flipped MNIST
• Train and compute the margin distribution on CIFAR10
• Train and compute the margin distribution on Frequency-flipped CIFAR10
• Train and compute the margin distribution on Low-Pass CIFAR10
• Compute the margin distribution on Robust MNIST
• Compute the margin distribution on Robust CIFAR10
• Compute the margin distribution on Robust ImageNet
• Compute the margin distribution on Frequency-flipped ImageNet

You can reproduce this experiments separately using their individual scripts, or have a look at the comprehensive Jupyter notebook.

Pretrained architectures

We also provide a set of pretrained models that we used in our experiments. The exact hyperparameters and settings can be found in the Supplementary material of the paper. All the models are publicly available and can be downloaded from here. In order to execute the scripts using the pretrained models, it is recommended to download them and save them under the Models/Pretrained/ directory.

Reference

If you use this code, or some of the attached models, please cite the following paper:

@InCollection{OrtizModasHMT2020,
  TITLE = {{Hold me tight! Influence of discriminative features on deep network boundaries}},
  AUTHOR = {{Ortiz-Jimenez}, Guillermo and {Modas}, Apostolos and {Moosavi-Dezfooli}, Seyed-Mohsen and Frossard, Pascal},
  BOOKTITLE = {Advances in Neural Information Processing Systems 34},
  MONTH = dec,
  YEAR = {2020}
}