Collection of NLP model explanations and accompanying analysis tools

The following error was raised for "mnli-xlnet-lgxa":

Traceback (most recent call last):
  File "run_explainer.py", line 57, in <module>
    explainer = getattr(thermex, f'Explainer{config["explainer"]["name"]}').from_config(config=config)
  File "/home/feldhus/thermostat/src/thermostat/explainers/grad.py", line 71, in from_config
    layer=res.model.base_model.embeddings)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1094, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'XLNetModel' object has no attribute 'embeddings'

Some CUDA-related type error occurred at attribution time.

2021-04-09 17:15:04,043 -explain - INFO - (Progress) Loaded explainer
2021-04-09 17:15:04,043 -explain - INFO - (Progress) Initialized data loader
  0%|                                                 | 0/12500 [00:00<?, ?it/s]2021-04-09 17:15:04,052 -explain - INFO - (Progress) Processing batch 0 / instance 0
50
  0%|                                                 | 0/12500 [00:12<?, ?it/s]
Traceback (most recent call last):
  File "/home/nfel/PycharmProjects/thermostat/venv/lib/python3.8/site-packages/captum/attr/_core/lime.py", line 415, in attribute
    curr_sample = self.perturb_func(inputs, **kwargs)
  File "/home/nfel/PycharmProjects/thermostat/src/thermostat/explainers/lime.py", line 43, in perturb_func
    return original_input + torch.randn_like(original_input)
RuntimeError: "normal_kernel_cuda" not implemented for 'Long'

Hello, I am experimenting with the attribution values of Layer Integrated Gradients on the AlbertV2 model on the imdb dataset. Along the way, I noticed a mismatch between the predictions saved in each instance in a dataset and the output of the model if loaded and called separately. I hope the following code sample makes clear what I mean:

import thermostat
import torch
from transformers import AutoModelForSequenceClassification

data = thermostat.load("imdb-albert-lig")

albert = AutoModelForSequenceClassification.from_pretrained(data.model_name, return_dict=False)
albert.eval()

sliced_data = data[:10]
thermostat_preds = [instance.predictions for instance in sliced_data]

batch_input = torch.tensor([instance.input_ids for instance in sliced_data])
preds = albert(batch_input)[0]

print("thermostat  ---  new inference".center(89, " "))
for m, n in zip(thermostat_preds, preds.tolist()):
    print(m, " --- ", n)

outputs:

Loading Thermostat configuration: imdb-albert-lig
Reusing dataset thermostat (/home/tim/.cache/huggingface/datasets/thermostat/imdb-albert-lig/1.0.1/0cbe93e1fbe5b8ed0217559442d8b49a80fd4c2787185f2d7940817c67d8707b)
                              thermostat  ---  new inference                             
[-2.9755005836486816, 3.422632932662964]  ---  [0.5287008285522461, 0.11485755443572998]
[-2.1383304595947266, 2.6063592433929443]  ---  [0.5254676938056946, 0.11936521530151367]
[-2.891936779022217, 3.3106441497802734]  ---  [0.5285712480545044, 0.11505177617073059]
[-3.0642969608306885, 3.240943670272827]  ---  [0.5282390117645264, 0.11541029810905457]
[-3.1076266765594482, 3.050632953643799]  ---  [0.5288912057876587, 0.11466512084007263]
[-2.8576371669769287, 3.023214101791382]  ---  [0.5283359885215759, 0.12018033862113953]
[-1.8885599374771118, 2.4350857734680176]  ---  [0.5438214540481567, 0.12777245044708252]
[-1.5720579624176025, 2.051628589630127]  ---  [0.5284878611564636, 0.11513140797615051]
[-3.2173707485198975, 3.510160207748413]  ---  [0.5282713174819946, 0.11538228392601013]
[-2.5653769969940186, 3.0336244106292725]  ---  [-2.5653772354125977, 3.033625841140747]

I only checked for that particular dataset and model, but I am wondering what causes the mismatch here? My suspicion is that either the model changed or the input_ids from the dataset do not correspond to the exact input used to create the dataset. Maybe you can shed some light on this? Thank you already for this useful library and if there are further questions please contact me, Tim

The assertion in the token_similarity_kernel function of ExplainerLimeBase

assert original_input.shape[0] == perturbed_input.shape[0]  == 1

https://github.com/nfelnlp/thermostat/blob/24177342945e834552a6df956ae59fdf1e69335b/src/thermostat/explainers/lime.py#L47

only works for IMDB so far. An error is thrown for MNLI, so I debugged it and found out that the two input shapes can still be equal, although they're not exactly 1. I assume this is because it has two text fields ("premise", "hypothesis") instead of one. The calculation below can still be performed with .shape[0]==2.

Do you think removing the == 1 at the end of the assertion would be fine?

Stuck at this error while implementing InputXGradient. Tested on DistilBERT and RoBERTa.

Traceback (most recent call last):
  File "/home/nfel/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_exec2.py", line 3, in Exec
    exec(exp, global_vars, local_vars)
  File "<input>", line 1, in <module>
  File "/home/nfel/PycharmProjects/thermostat/venv/lib/python3.8/site-packages/captum/log/__init__.py", line 35, in wrapper
    return func(*args, **kwargs)
  File "/home/nfel/PycharmProjects/thermostat/venv/lib/python3.8/site-packages/captum/attr/_core/input_x_gradient.py", line 117, in attribute
    gradients = self.gradient_func(
  File "/home/nfel/PycharmProjects/thermostat/venv/lib/python3.8/site-packages/captum/_utils/gradient.py", line 125, in compute_gradients
    grads = torch.autograd.grad(torch.unbind(outputs), inputs)
  File "/home/nfel/PycharmProjects/thermostat/venv/lib/python3.8/site-packages/torch/autograd/__init__.py", line 223, in grad
    return Variable._execution_engine.run_backward(
RuntimeError: One of the differentiated Tensors does not require grad
  0%|                                                  | 0/1821 [07:25<?, ?it/s]
Traceback (most recent call last):
  File "/home/nfel/PycharmProjects/thermostat/venv/lib/python3.8/site-packages/captum/_utils/gradient.py", line 125, in compute_gradients
    grads = torch.autograd.grad(torch.unbind(outputs), inputs)
  File "/home/nfel/PycharmProjects/thermostat/venv/lib/python3.8/site-packages/torch/autograd/__init__.py", line 223, in grad
    return Variable._execution_engine.run_backward(
RuntimeError: One of the differentiated Tensors does not require grad

IMDb and AG News accuracies when comparing true labels to predicted labels are reasonably high. However, almost all MNLI and XNLI models investigated so far have an extremely low accuracy.

Here are some random configs that I investigated: multi_nli-albert-occ : 6.48% multi_nli-roberta-occ : 28.72% multi_nli-xlnet-occ : 5.53% multi_nli-bert-lime : 6.46 % xnli-bert-lig : 7.05%

Curiously, the ELECTRA model (submitted by a different person than all the other models) are not affected: multi_nli-electra-lgxa : 88.7% xnli-electra-lig : 88.3%

It's not clear to me yet if this is simply an issue stemming from an outdated label order as documented here: https://github.com/huggingface/transformers/pull/10203/

However, the label orders that are mentioned there, both old and new, are also different from the label assignment given by datasets: https://huggingface.co/datasets/viewer/?dataset=xnli (multi_nli has the same labels)

Note that the datasets version we use in thermostat is 1.5.0, so this might be a problem with the downstream models predicting different labels.

Then I investigated in what way the predictions and labels of such a faulty model (xnli-bert-lgxa) align:

import thermostat
from collections import Counter

bert_lgxa = thermostat.load("xnli-bert-lgxa")
true_pred_comp_bert_lgxa = [(b_i.true_label['index'], b_i.predicted_label['index']) for b_i in bert_lgxa]
Counter(true_pred_comp_bert_lgxa)

>>> Counter({(2, 0): 1488,
         (0, 1): 1345,
         (1, 0): 153,
         (1, 2): 1411,
         (0, 2): 231,
         (0, 0): 93,
         (1, 1): 107,
         (2, 2): 153,
         (2, 1): 29})

This leads me to assume that (2, 0), (0, 1) and (1, 2) are actually the correct predictions and (0, 0), (1, 1) and (2, 2) are wrong ones. If we sum up (2, 0), (0, 1) and (1, 2) and divide it by the sum of all (or the length of the XNLI dataset), we end up at 84.71% which is a much more reasonable number in my opinion.

At the very least, this means that all thermostat subsets concerning MNLI and XNLI need to be redone (editing JSONL files and reuploading). Hopefully, this only means going through each JSONL and changing the values either by:

1. Changing the true labels to the old standard. However, this means that we do not use the vanilla data from datasets anymore.
2. Changing the predicted labels as well as the logits.

I'm pretty positive that we don't need to run the explanations again.

On a sidenote, I also considered the encode_pair function (which is only used by MNLI and XNLI in thermostat) not working correctly, but couldn't find any reference implementation stating that the way the two text fields are ingested might be wrong.

if name_model in ['bert-base-cased', 'xlnet-base-cased']:
    assert 'input_ids' in batch, f'Input ids expected for {name_model} but not found.'
    assert 'attention_mask' in batch, f'Attention mask expected for {name_model} but not found.'
    assert 'token_type_ids' in batch, f'Token type ids expected for model {name_model} but not found.'
    input_ids = batch['input_ids']
    additional_forward_args = (batch['attention_mask'], batch['token_type_ids'])
    return input_ids, additional_forward_args
elif name_model == 'textattack/roberta-base-imdb':  # TODO: Separate classes?
    assert 'input_ids' in batch, f'Input ids expected for {name_model} but not found.'
    assert 'special_tokens_mask' in batch, f'Special tokens mask expected for {name_model} but not found.'
    assert 'attention_mask' in batch, f'Attention mask expected for {name_model} but not found.'
    input_ids = batch['input_ids']
    additional_forward_args = (batch['special_tokens_mask'], batch['attention_mask'])
    return input_ids, additional_forward_args
else:
    raise NotImplementedError

This way of putting every case where the batch encoding is different together into one class is not manageable in the future.

see commit

experiment_in = [f for f in os.listdir(experiment_path) if "preprocess" in f and f.endswith('.jsonl')][0]

can accidentally select another empty explainer file (from previous, sometimes unsuccessful debug runs). This needs to be fixed! Probably by introducing config files (like in gxai) where the experiment inputs (processed data) is hard-coded, i.e. includes the specific files with timestamps.

see commit

For once, there could be an option to choose different colors instead of red and blue as inspired by this site.

https://github.com/nfelnlp/thermostat/blob/b7f8e829ca2927121c7fa24eb977ecb91a92d017/src/thermostat/visualize.py#L46

On top of that, the current color scheme is not perfect for readability, because some of the most salient words (very red or very blue) still have black text and cannot be read properly. Maybe it would be better to have a colored border around the word instead?

https://github.com/nfelnlp/thermostat/blob/b7f8e829ca2927121c7fa24eb977ecb91a92d017/src/thermostat/visualize.py#L147-L152

The alignment of the classification head with the number of labels in datasets has not been a problem so far, but I left a TODO here to insert an assert that should check if res.num_labels corresponds with the shape of the loaded model's classification head.

https://github.com/nfelnlp/thermostat/blob/b7f8e829ca2927121c7fa24eb977ecb91a92d017/src/thermostat/explain.py#L137-L138

https://github.com/nfelnlp/thermostat/blob/a8180a2d83e1c3ec5f873dbf0ce0ab14026cf6bf/src/thermostat/explain.py#L54

        def bert_forward(input_ids, attention_mask, token_type_ids):
            input_model = {
                'input_ids': input_ids.long(),
                'attention_mask': attention_mask.long(),
                'token_type_ids': token_type_ids.long(),
            }
            output_model = model(**input_model)[0]
            return output_model

        def roberta_forward(input_ids, attention_mask):
            input_model = {
                'input_ids': input_ids.long(),
                'attention_mask': attention_mask.long(),
            }
            output_model = model(**input_model)[0]
            return output_model

KernelShap is not working yet and gave the following error:

Traceback (most recent call last):
  File "/home/nfel/PycharmProjects/thermostat/venv/lib/python3.8/site-packages/captum/attr/_core/lime.py", line 484, in attribute
    self.interpretable_model.fit(DataLoader(dataset, batch_size=n_samples))
  File "/home/nfel/PycharmProjects/thermostat/venv/lib/python3.8/site-packages/captum/_utils/models/linear_model/model.py", line 303, in fit
    return super().fit(train_data=train_data, **kwargs)
  File "/home/nfel/PycharmProjects/thermostat/venv/lib/python3.8/site-packages/captum/_utils/models/linear_model/model.py", line 260, in fit
    return super().fit(
  File "/home/nfel/PycharmProjects/thermostat/venv/lib/python3.8/site-packages/captum/_utils/models/linear_model/model.py", line 117, in fit
    return self.train_fn(
  File "/home/nfel/PycharmProjects/thermostat/venv/lib/python3.8/site-packages/captum/_utils/models/linear_model/train.py", line 329, in sklearn_train_linear_model
    sklearn_model.fit(x, y, sample_weight=w, **fit_kwargs)
  File "/home/nfel/PycharmProjects/thermostat/venv/lib/python3.8/site-packages/sklearn/linear_model/_base.py", line 525, in fit
    X, y, X_offset, y_offset, X_scale = self._preprocess_data(
  File "/home/nfel/PycharmProjects/thermostat/venv/lib/python3.8/site-packages/sklearn/linear_model/_base.py", line 162, in _preprocess_data
    X_offset = np.average(X, axis=0, weights=sample_weight)
  File "<__array_function__ internals>", line 5, in average
  File "/home/nfel/PycharmProjects/thermostat/venv/lib/python3.8/site-packages/numpy/lib/function_base.py", line 409, in average
    raise ZeroDivisionError(
ZeroDivisionError: Weights sum to zero, can't be normalized