I would like to run on my own dataset, and also, print the accuracy on the source and target domains. In the paper implementations, I don't see any accuracy metric. Please guide the amateur learner looking at this to change the dataset to their own, and to implement accuracy terms. Thanks

Suggested Feature

A) The addition of a new TargetDataset class for supervised domain adaptation.

or

B) The extension of the TargetDataset class to return labels when passed a supervised flag.

I think option B could be cleaner?

Implementation

A) Create a new class capable of returning target_labels named something like SupervisedTargetDataset.

or

B) Update the init function of the TargetDataset to include a supervised flag.

    def __init__(self, dataset: Dataset, domain: int = 1, supervised=False):
        """
        Arguments:
            dataset: The dataset to wrap
            domain: An integer representing the domain.
        """
        super().__init__(dataset, domain, supervised)

Update the getitem method to behave differently under supervised domain adaptation.

    def __getitem__(self, idx: int) -> Dict[str, Any]:
        
        if supervised:
            img, target_labels = self.dataset[idx]
            return {
                "target_imgs": img,
                "target_domain": self.domain,
                "target_labels": target_labels,
                "target_sample_idx": idx,
            }
        else:
            img, _ = self.dataset[idx]
            return {
                "target_imgs": img,
                "target_domain": self.domain,
                "target_sample_idx": idx,
            }

Reasoning

To run supervised domain adaptation we need to have labels in the target domain but I think it would still be useful to distinguish between the two domains using different classes. Rather than using SourceDataset on a TargetDataset to achieve the same functionality.

With this change validators such as AccuracyValidator could be used on target_val in a supervised domain adaptation setting.

BTW: With these feature suggestions I am happy to do code PRs along with the docs as I previously mentioned!

Hello,

Well done on putting together this library I think it will be extremely useful for many people undertaking domain adaptation projects.

I am wondering how to create a test dataset using the DataloaderCreator class?

Some background on my issue.

I am using the MNISTM example within a PyTorch lightning data-module.

Adapting the code from the examples/DANNLightning.ipynb I have the following code.

class MnistAdaptDataModule(LightningDataModule):
    def __init__(
        self,
        data_dir: str = "data/mnistm/",
        batch_size: int = 4,
        num_workers: int = 0,
        pin_memory: bool = False,
    ):
        super().__init__()

        # this line allows to access init params with 'self.hparams' attribute
        # it also ensures init params will be stored in ckpt
        self.save_hyperparameters(logger=False)

        self.data_train: Optional[Dataset] = None
        self.data_val: Optional[Dataset] = None
        self.data_test: Optional[Dataset] = None
        self.dataloaders = None

    def prepare_data(self):
        if not os.path.exists(self.hparams.data_dir):
            print("downloading dataset")
            get_mnist_mnistm(["mnist"], ["mnistm"], folder=self.hparams.data_dir, download=True)
        return


    def setup(self, stage: Optional[str] = None):
        if not self.data_train and not self.data_val and not self.data_test:
            datasets = get_mnist_mnistm(["mnist"], ["mnistm"], folder=self.hparams.data_dir, download=False)
            dc = DataloaderCreator(batch_size=self.hparams.batch_size, num_workers=self.hparams.num_workers)
            validator = IMValidator()
            self.dataloaders = dc(**filter_datasets(datasets, validator))
            self.data_train = self.dataloaders.pop("train")
            self.data_val = list(self.dataloaders.values())
            return            

    def train_dataloader(self):
        return self.data_train

    def val_dataloader(self):
        return self.data_val

   def test_dataloader(self):
        # how to make a test dataset?
        return

self.dataloaders produces the following object

{'src_train': SourceDataset(
  domain=0
  (dataset): ConcatDataset(
    len=60000
    (datasets): [Dataset MNIST
        Number of datapoints: 60000
        Root location: /home/eoghan/Code/mnist-domain-adaptation/data/mnist_adapt/
        Split: Train
        StandardTransform
    Transform: Compose(
                   Resize(size=32, interpolation=bilinear, max_size=None, antialias=None)
                   ToTensor()
                   <pytorch_adapt.utils.transforms.GrayscaleToRGB object at 0x7fd1badcbdc0>
                   Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
               )]
  )
), 'src_val': SourceDataset(
  domain=0
  (dataset): ConcatDataset(
    len=10000
    (datasets): [Dataset MNIST
        Number of datapoints: 10000
        Root location: /home/eoghan/Code/mnist-domain-adaptation/data/mnist_adapt/
        Split: Test
        StandardTransform
    Transform: Compose(
                   Resize(size=32, interpolation=bilinear, max_size=None, antialias=None)
                   ToTensor()
                   <pytorch_adapt.utils.transforms.GrayscaleToRGB object at 0x7fd1badcb6a0>
                   Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
               )]
  )
), 'target_train': TargetDataset(
  domain=1
  (dataset): ConcatDataset(
    len=59001
    (datasets): [MNISTM(
      domain=MNISTM
      len=59001
      (transform): Compose(
          ToTensor()
          Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
      )
    )]
  )
), 'target_val': TargetDataset(
  domain=1
  (dataset): ConcatDataset(
    len=9001
    (datasets): [MNISTM(
      domain=MNISTM
      len=9001
      (transform): Compose(
          ToTensor()
          Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
      )
    )]
  )
), 'train': CombinedSourceAndTargetDataset(
  (source_dataset): SourceDataset(
    domain=0
    (dataset): ConcatDataset(
      len=60000
      (datasets): [Dataset MNIST
          Number of datapoints: 60000
          Root location: /home/eoghan/Code/mnist-domain-adaptation/data/mnist_adapt/
          Split: Train
          StandardTransform
      Transform: Compose(
                     Resize(size=32, interpolation=bilinear, max_size=None, antialias=None)
                     ToTensor()
                     <pytorch_adapt.utils.transforms.GrayscaleToRGB object at 0x7fd125f69d60>
                     Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
                 )]
    )
  )
  (target_dataset): TargetDataset(
    domain=1
    (dataset): ConcatDataset(
      len=59001
      (datasets): [MNISTM(
        domain=MNISTM
        len=59001
        (transform): Compose(
            ToTensor()
            Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
        )
      )]
    )
  )
)}

This handles train and val for source and target as well as creating a conjoined train dataset.

Going by the example ipynb, the concat dataset for train (of source and target) is used as the training dataset for the model.

The validation set is a list of the remaining keys in the data-loader and has the following form.

[
<torch.utils.data.dataloader.DataLoader object at 0x7fd1063e6b80> {
    dataset: TargetDataset(
  domain=1
  (dataset): ConcatDataset(
    len=59001
    (datasets): [MNISTM(
      domain=MNISTM
      len=59001
      (transform): Compose(
          ToTensor()
          Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
      )
    )]
  )
)
}
]

I am not sure why this is the validation dataset, Do we validate on only the target domain? How would we handle this validation set if the target domain is unlabelled? If you could explain why this is the case I would appreciate some insight.

In summation I am looking for guidance on is how to use something like a torch.utils.data.random_split to take some of the source and target data and use the DataloaderCreator to pass back test sets along with train and val, is this possible within the framework?

Many thanks, Eoghan

Hi @KevinMusgrave,

How could I use DANN with target labels, I tried to do that:

from pytorch_adapt.hooks import DANNHook, CLossHook, FeaturesAndLogitsHook
G.count, C.count, D.count = 0, 0, 0
f_hook =  FeaturesAndLogitsHook(domains = ["src", "target"])
c_hook = CLossHook(f_hook=f_hook)
hook = DANNHook(opts,c_hook=c_hook)
model_counts = validate_hook(hook, list(data.keys()))
outputs, losses = hook({**models, **data})
print_info(model_counts, outputs, losses, G, C, D)

But I'm having this issue:

ValueError: in DANNHook: __call__
in ChainHook: __call__
in OptimizerHook: __call__
in ChainHook: __call__
in ChainHook: __call__
in CLossHook: __call__
too many values to unpack (expected 1)

Thanks in advance!

Hi @KevinMusgrave,

I would like to ask the following question, since I'm trying to play with the library and I think that what I want to do it's easily doable exploiting all the library features.

I would like to try to implement the following architecture, given two separate src (src_1, src_2) and given two independent generators (g_1, g_2) and two independent classifiers (C_1, C_2). Where features_1 = G_1 (src_1) is input of C_1, and likewise features_2 = G_2(src_2) is input of C_2. And both features_1 and features_2 are passed to D (DANN's discriminator) which is shared.

Many thanks in advance!

Hi, this is roughly the code that I am using for training my models:

models = Models({"G": G, "C": C, "D": D})
adapter = DANN(models=models)
validator = IMValidator()
dataloaders = dc(**filter_datasets(datasets, validator))
train_loader = dataloaders.pop("train")

L_adapter = Lightning(adapter, validator=validator)
trainer = pl.Trainer(gpus=1, 
                     max_epochs=1,
                     default_root_dir="saved_models",
                     enable_checkpointing=True)
trainer.fit(L_adapter, train_loader, list(dataloaders.values()))

which causes the latest model to be saved under saved_models/lightning_logs/version_0/checkpoints/epoch=1-step=2832.ckpt.

Question 1): Is it possible to restore all three models, G, C and D from this checkpoint, and if yes how? I know that Lightning provides the function load_from_checkpoint() but I can't get it to work. Question 2) If it is not possible to restore these models from the Lightning checkpoint, should I instead just manually save the state_dicts of G, C and D and then manually restore these, or is there a more elegant way?

I ran pip install pytorch-adapt but only to be stuck later as whenever I say "import pytorch_adapt" on the python shell (Linux) I am faced with this annoying error. Where am I going wrong?

Right now it's hardcoded to use src_logits. Adding a domain parameter (set to either src or target) would allow CLossHook to be used for supervised domain adaptation as well.

https://github.com/KevinMusgrave/pytorch-adapt/blob/3b2713c4860b325c79481f11307a193bb381d53f/src/pytorch_adapt/hooks/classification.py#L75-L88

Hi, I am the first author of ATDOC, thanks for including our method in such an impressive library.

There exists a typo in the paper of Eq.(6) (already updated the arxiv version today), where the index k should be replaced with i. That is to say, in Line 87 of this python file pytorch_adapt/layers/neighborhood_aggregation.py, the correct code would be "logits = (logits ** p) / torch.sum(logits ** p, dim=0)".

Best,

Jian

See: https://github.com/KevinMusgrave/pytorch-adapt/blob/97afa6d801e48b7e30854dbb11fc7ebae5abb3c3/src/pytorch_adapt/frameworks/ignite/checkpoint_utils.py#L86-L101

Hi I installed pytorch-adapt with pip. But when I tried from pytorch_adapt.datasets import ( CombinedSourceAndTargetDataset, SourceDataset, TargetDataset, ) "No module named 'pytorch_adapt'" occured.

My python version is 3.9.5.

Thank you.

The original OfficeHome dataset has very large images, so I downscaled them so that the shortest side is 256 pixels, but this isn't mentioned anywhere in the docs or the code.

Suggested Feature

The Lightning class could be extended to include two more functions

1. test_step
2. test_epoch_end

Implementation

The test_step and test_epoch end could operate in the same way as the validation_step and epoch_end and return a test_score.

Reasoning

This would allow lightning users to specify a test_dataloader containing a hold-out set in their datamodule. The best saved model on validation data can then be run against the test data using pytorch lightning trainer.test function call.