First, I would like to know if this project works with segmentation problems, namely Unet - this one in particular.

Also, I tried to run a simple pipeline but I am getting this error when creating the dataset. However, if I run that piece of code it works:

from torch.utils.data.dataloader import default_collate
for data, target in DataLoader(train_dataset, batch_size, True, num_workers=4,
                                           collate_fn=None):
    print(data)

My Dataset constructor:

class Dataset(BaseDataset): """CamVid Dataset. Read images, apply augmentation and preprocessing transformations.

Args:
    images_dir (str): path to images folder
    masks_dir (str): path to segmentation masks folder
    class_values (list): values of classes to extract from segmentation mask
    augmentation (albumentations.Compose): data transfromation pipeline 
        (e.g. flip, scale, etc.)
    preprocessing (albumentations.Compose): data preprocessing 
        (e.g. noralization, shape manipulation, etc.)

"""

    def __init__(
            self, 
            ids,
            images_dir, 
            masks_dir, 
            classes=None, 
            augmentation=None, 
            preprocessing=None,
            show_original=False
    ):
        self.ids = ids#os.listdir(images_dir)
        self.images_fps = [os.path.join(images_dir, image_id) for image_id in self.ids]
        self.masks_fps = [os.path.join(masks_dir, image_id) for image_id in self.ids]
        
        # convert str names to class values on masks
        self.class_values = [i+1 for i,c in enumerate(classes)]
        
        self.augmentation = augmentation
        self.preprocessing = preprocessing
        self.show_original=show_original
    
    def __getitem__(self, i):
        try:
            # read data
            #print(self.images_fps[i])
            filename=self.images_fps[i]
            if not filename.endswith(".png"):
                # Virtual openslide patch in form filename_col_x_row_y
                slide_name=Path(filename).stem.split("_col_")[0]
                roi_x,roi_y=filename.split("col_")[1].split("_row_")
                roi_xy_large=int(int(roi_x)*patch_size),int(int(roi_y)*patch_size)
            
                slide= openslide.open_slide(slide_paths[slide_name])
                image=slide.read_region(roi_xy_large,0,(patch_size,patch_size))
                #print((roi_xy_large,0,(patch_size,patch_size)))
                #plt.imshow(img)
                #plt.show()

                image=image.convert('RGB')
                if self.show_original:
                    plt.imshow(image)
                    plt.show()
                image = np.asarray(image)
                mask=None
                
            else:
                image = cv2.imread(filename)
                

                image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
                if self.show_original:
                    plt.imshow(image)
                    plt.show()

                #print(self.masks_fps[i])
                mask = cv2.imread(self.masks_fps[i], 0)

            

            # No mask, ex: false positives
            if mask is None:
                mask= PIL.Image.new('RGB', (256, 256), (0, 0, 0))
                mask=np.array(mask.convert("L"))
                
#             if image is None:
#                 image= PIL.Image.new('RGB', (256, 256), (0, 0, 0))
#                 image=np.array(image.convert("L"))


            #print('self.images_fps[i]',self.images_fps[i])
            #print('self.masks_fps[i]',self.masks_fps[i])

            #print('self.class_values:', self.class_values)
            #print('mask.max:', np.max(mask))
            #print('mask.min:', np.min(mask))
            #print('mask.unique:', np.unique(mask))

            # extract certain classes from mask (e.g. cars)
            masks = [(mask == v) for v in self.class_values]
            mask = np.stack(masks, axis=-1).astype('float')

            # apply augmentations
            if self.augmentation:
                sample = self.augmentation(image=image, mask=mask)
                image, mask = sample['image'], sample['mask']
#                 set_trace() if DEBUG else None
                
#                 mask[np.all(image == [0, 0, 0], axis=-1)]=0

            # apply preprocessing
            if self.preprocessing:
                sample = self.preprocessing(image=image, mask=mask)
                image, mask = sample['image'], sample['mask']
                
                

            return image, mask
            
        except Exception as error:
            print("error",error,self.images_fps[i],self.masks_fps[i])
            raise

    def __len__(self):
        return len(self.ids)

Regards

We have several possibilities:

• Move to Mkdocs Material
• Update our CSS

In all cases, we should redo the tree structure to be more "Profesh"

Hello I would like to get some information related to patch module, I am using baal library with hugging face for multilabel classification. I am using BALD as a heuristic and wrapping model in patch_module. Before, asking the main question I would like to specify that I am not using the library specified inside the blog post regarding NLP classification with hugging face. I have created my own custom function and the only thing I am using from the library is patch_module and heuristic for selecting samples. The problem is, I am running the active learning loop 26 times, and for iteration 7 and 12: The results seems bit confusing as shown:

---------------------------- iteration -7 --------------------------
{
    "'fixed_rate'-F1": 0,
    "'floating_rate'-F1": 1.37,
    "'other'-F1": 62.27,
    "'rates'-F1": 63.56
}

----------------------------- Iteration -6 ---------------------------

{
    "'fixed_rate'-F1": 78.26,
    "'floating_rate'-F1": 78.55,
    "'other'-F1": 79.27,
    "'rates'-F1": 74.03
}

-------------------------   Iteration -5 -------------------------------
{
    "'fixed_rate'-F1": 63.41,
    "'floating_rate'-F1": 77.32,
    "'other'-F1": 78.65,
    "'rates'-F1": 73.76
}

as shown above in the 6th iteration we getting fixed_rate f1: 78.26 and in 7th iteration we went suddenly to 0:

We also tried for this specific iteration where f1 is 0 to train the model without Active learning, and it works totally fine so it suggest that there is no problem related to dataset. But when we adding the Active Learning procedure this becoming 0. I want to mention that this problem is with iteration 7 and 12. So I am really confused why this works fine for other iteration and only for these two iteration the f1 going down to 0. Is it because of patch_module, or the way I am using the patch_module.

   initial train -> 200
   every time we add more 200 samples into the train.

   trainer = Trainer(
          model=patch_module(model),
          args=training_args,
          train_dataset=train_dataset if training_args.do_train else None,
          eval_dataset=eval_dataset if training_args.do_eval else None,
          compute_metrics=custom_compute_metrics, #[ADD] passo al Trainer la nuova funzione che calcola le metriche
          tokenizer=tokenizer,
          data_collator=data_collator,
      )

One more question so when add the special dropout in the model, during the testing time does it uses the special dropout for doing the predictions, testing I mean when the model is fully trained using active learning procedure, and if it does then we cannot trust the prediction because it will change everytime, and if it doesn't please can u let me know if it automatically disabled by doing model.eval() or we need to add some other things. Sorry for too much of question. Please let me know if something is not clear.

Would this project be interested in ensemble based techniques for active learning? As seen in [1] ensembles can be an effective technique for deep active learning achieving competitive/superior performance to competing techniques (ie MC-Dropout). More generally they have been widely used in non-deep based approaches to active learning as well (see [2] chapter 3 for a review).

All of the heuristics this library has already implemented could immediately be used for ensemble predictions.

I see that SWAG is on the road map and this could be a more general feature of different ensemble style approaches (bootstrap, FGE, SWAG, diversity encouraging, etc...)

If this is something you all would like to move forward with. I would be happy to implement it. I've been working on active learning and already have my own implementations. Since finding this library I am in the midst of porting my code over to it, since I like it so much.

My only concern is that when resource constrained it is difficult to hold all of the ensemble members in memory and can be expensive to move them in and out. In my own experiments I've been generating the members of the ensemble one at a time and predicting on all data points from the pool set and validation set deleting the member and start training the new member. This might be a bit awkward given the current implementation of the ModelWrapper and ActiveLearningLoop.

Model.train()
active_loop.set()

Essentially for the above workflow to still work it would require the below line to be already executed and cached by the model and the active_loop would use the cache.

 probs = self.get_probabilities(pool, **self.kwargs)

Thanks for your time.

[1] http://openaccess.thecvf.com/content_cvpr_2018/papers/Beluch_The_Power_of_CVPR_2018_paper.pdf

[2] https://www.morganclaypool.com/doi/abs/10.2200/S00429ED1V01Y201207AIM018

Summary:

Add some utils to keep track of metrics over time.

This is WIP, but what do you think of this API? It would be quite simple for the users to get a metrics for each dataset size.

We can also add utilities for that such as MetricMixin.get_active_metric(metric_name="precision") : Dict[int, float]

Features:

Fixes #220

Checklist:

• [ ] Your code is documented (To validate this, add your module to tests/documentation_test.py).
• [ ] Your code is tested with unit tests.
• [ ] You moved your Issue to the PR state.

This is problematic when building the conda package since it does not include the optional dep transformers.

I see two solutions:

• make transformers a required dep
• add a conditional check to https://github.com/ElementAI/baal/blob/26bde3a9ef68454cb2795de3784cde4032fc0391/src/baal/transformers_trainer_wrapper.py so transformers is not imported if missing that way baal can still be used without transformers.

Hi,I want to construct my own dataset like the following:

import torch
from torch.utils.data import Dataset
from baal.active.dataset import ActiveLearningDataset

a = []
for i in range(20):
    a.append(torch.rand(10))

data = Data(a)
pool = ActiveLearningDataset(data)

and I want to label several data in the dataset:

pool.label(index=4, value=torch.tensor([0]))

But I get the error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
 in 
----> 1 pool.label(index=4, value=torch.tensor([0]))

 in label(self, index, value)
    159             if self.can_label and val is not None:
    160                 print(self._dataset.label)
--> 161                 self._dataset.label(index, val)
    162                 self._labelled[index] = 1
    163             elif self.can_label and val is None:

TypeError: 'NoneType' object is not callable

I read the original code, and find the self._dataset.label is None.Is the above code right or is this a bug?

Assuming baal returns a list of datums to be labeled by humans after each epoch, how do I connect it to Amazon Mechanical Turk to get them actually labeled for the next epoch's training? Can the training process be stalled while humans label these examples?

Summary:

Add a way to set the seed for Dropout so that the execution is always the same.

Features:

• Add SeededDropout

Checklist:

• [x] Your code is documented (To validate this, add your module to tests/documentation_test.py).
• [x] Your code is tested with unit tests.
• [x] You moved your Issue to the PR state.

Is your feature request related to a problem? Please describe. In Segmentation, we need to augment both the target and the input. At test time, only the transform is updated.

We should swap most of our metrics for torchmetrics this would make our code more robust.

Only ECE is special I think and we should keep our own implementation.

Summary:

This is a demo/tutorial to use active learning with hugging face models in a production setting. Kindly find more about this at in the discussion at https://github.com/baal-org/baal/discussions/242

Features:

NA

Checklist:

• [ ] Your code is documented (To validate this, add your module to tests/documentation_test.py).
• [ ] Your code is tested with unit tests.
• [ ] You moved your Issue to the PR state.

Given that this is a notebook and I am not setting up any new modules there are no test cases. There is some pending type hinting pending which I will complete.

Opening a PR for your feedback, just to check if you want me to add/remove somethings

Additional Info

Challenges with current GPU

Seems like the pytorch version which baal uses does not support my current GPU. Although I have tested this on Colab and it works fine.

NVIDIA GeForce RTX 3050 Laptop GPU with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
If you want to use the NVIDIA GeForce RTX 3050 Laptop GPU GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))

More info about this on the pytorch forum in case someone runs into a similar issue

import torch

torch.__version__

1.12.1+cu102

torch.cuda.get_arch_list()

['sm_37', 'sm_50', 'sm_60', 'sm_70']

Although I have tested this on Colab and it works fine.

Challenges with Black Formatting

You might want to update your black version to black==22.3.0.

The make format command produces an error which is identical to the one mentioned at stack overflow here.

I have encountered this before and an upgrade does fix it

Discussion Related Paper

I see this as a new object we include with the following strategies:

• If the average uncertainty is below a threshold
• If the labeling budget has been exhausted
• If the model has converged.

We have results on CLINC-150 where we show that Entropy is better than Random. We learned a lot since our first tutorial on HuggingFace. We should redo the expeirment.

Hi, thanks for providing such great work! I am working a object detection task and I have already trained a object detection model with detectron2 framework. is it possible to integrate detectron2 model into this active learning method ? or more generally, can i integrate object detection model like FasterRCNN yolo?