Audio Retrieval with Natural Language Queries: A Benchmark Study

Paper | Project page | Text-to-audio search demo

This repository is the implementation of Audio Retrieval with Natural Language Queries: A Benchmark Study which builds on the Audio Retrieval with Natural Language Queries repository and provides code for downloading the SoundDescs dataset and for reproducing all result from Audio Retrieval with Natural Language Queries: A Benchmark Study. The code is based on the Use What You Have: Video retrieval using representations from collaborative experts and MMT: Multi-modal Transformer for Video Retrieval repositories.

The datasets used in this paper are SoundDescs, AudioCaps, CLOTHO, Activity-Net and QuerYD.

Requirements and datasets

The required libraries for running this code can be found in requirements.txt. Cuda 10.1 and Python 3.7 were used.

conda create --name audio-retrieval python=3.7
conda activate audio-retrieval
pip install -r requirements.txt
conda install -c conda-forge ffmpeg

To be able to run the code below, features extracted from various datasets need to be downloaded. If there is not enough space in your working location to store some of these features (for SoundDescs and AudioCaps the files are larger than 6GB while the others are under 1GB) then you will need to create a folder called data inside this repository which should be a symlink to a folder where enogh memory exists. As an example, run the following from the audio-retrieval-benchmark base directory:

ln -s 
   
     data

   

To download features for the AudioCaps, Clotho, Activity-Net, and QuerYD datasets, follow the steps here. The SoundDescs features can be downloaded analogously:

python3 misc/sync_experts.py --dataset SoundDescs

In case you want to use the raw audio data for the SoundDescs, we explain how to download the SoundDescs dataset below.

SoundDescs dataset download and pre-processing

This is a tool to allow for easy download of audio files and text information from the https://sound-effects.bbcrewind.co.uk/search page.

Downloading audios
First download the download_links_renamed.txt or, if needed, the download_links.txt file. Save it in the folder that will be used for downloading audios. To be able to download the files the --download_folder flag must be set when running the commands below.

To only download a few audio files, use the --limit flag with non-zero values.

To download audio files in zip form for the SoundDescs dataset simply run the line below. To download multiple files at the same time use the processes flag. We recommend not using more than two processes to avoid being blocked by the website.

python sounddescs_download_audios.py --download_folder {location where to save files} --processes 2

To unzip the audio files to a new folder, run the line below. Here a larger number of processes can be used:

python sounddescs_download_audios.py --action unzipping --processes 20 --download_folder {location where to save files}

To re-sample the audio files at 16kHz and be put in the format needed to run CE, MoEE, and MMT, run the following command:

python sounddescs_wavs_transforms.py --exp resample --initial_folder {location where files were saved before} --dest_folder {location where resampled files are stored} --processes 20

Other files available that might prove useful are found in the sounddescs_data folder. The files are:

• categories.pkl - this file contains tags for most audio files. These tags can be Nature, Clocks, Sport etc. Some files have more than one tag and some have no tags.
• descriptions.pkl - this file contains the descriptions associated with the audio files. These are used as captions in our CE, MoEE and MMT experiments.
• extra_info.pkl - this file contains information about the audio content such as file type (e.g. MP3) or sample rate (e.g. 44.1KHz)

Terms and conditions for SoundDescs dataset

To download and use the SoundDescs dataset, you need to comply with the terms and conditions of the RemArc Licence.

This is from the official website that hosts the data:

By continuing, you agree to comply with the terms of the RemArc Licence for this and any future downloads.

Commercial use of this content is not allowed under the RemArc license.

For commercial use, buy the sound effect from Pro Sound Effects which can be found in the More Detail section for each sound effect.

Evaluating pretrained CE, MoEE, and MMT models on multiple seeds and reproducing results

To reproduce results for the CE, MoEE, and MMT models in the tables below, multiple models trained with different seeds need to be downloaded and evaluated on the test sets.

The steps needed to reproduce results are:

1. Picking the experiment to be reproduced which is in the form - . Tables with experiment names and the corresponding form can be found in misc/exps-names.md.
2. Downloading the features and splits corresponding to the dataset for which the experiment is run. For example for AudioCaps run:

# fetch the pretrained experts for AudioCaps
python3 misc/sync_experts.py --dataset AudioCaps

Additional examples for the datasets used in this paper can be found in misc/exps-names.md.

3. Running the eval.py script.

For example, to reproduce the experiments for AudioCaps with complete visual and audio experts, run the following line:

python eval.py --experiment audiocaps-train-full-ce-r2p1d-inst-vggish-vggsound

If the --experiment flag is not provided, the eval.py script will download and evaluate all CE and MoEE models on the test set.

Training a new model

Training a new CE audio-text embedding requires:

1. The pretrained experts for the dataset used for training, which should be located in /data/ /symlinked-feats (this will be done automatically by the utility script, or can be done manually). Examples can be found in misc/exps-names.md.
2. A config.json file. You can define your own, or use one of the provided configs in the configs directory.

Training is then performed with the following command:

python3 train.py --config 
   
     --device 
    

    
   

where is the index of the GPU to train on. This option can be ommitted for training on the CPU.

For example, to train a new embedding for the CLOTHO dataset, run the following sequence of commands:

# fetch the pretrained experts for CLOTHO
python3 misc/sync_experts.py --dataset CLOTHO

# Train the model
python3 train.py --config configs/clotho/train-vggish-vggsound.json --device 0

To train MMT, use the following command:

python -m mmt/train.py --config 
   

   

For example, to train MMT on the CLOTHO dataset, run the following sequence of commands:

# fetch the pretrained experts for CLOTHO
python3 misc/sync_experts.py --dataset CLOTHO

# Train MMT on CLOTHO
python -m mmt/train --config mmt/configs/clotho/Clotho_mmt.json

AudioCaps

These are the retrieval results obtained for the AudioCaps dataset when using only audio experts:

Using only visual experts for AudioCaps:

Visual and audio experts for AudioCaps:

CLOTHO

SoundDescs

Pretraining on SoundDescs, finetuning on AudioCaps

Pretraining on AudioCaps, finetuning on CLOTHO

Pretraining on SoundDescs, finetuning on CLOTHO

Pretraining on AudioCaps, finetuning on SoundDescs

Visual centric datasets

More information can be found at our project page: https://www.robots.ox.ac.uk/~vgg/research/audio-retrieval/

References

If you find this code useful, please consider citing [1,2,3,4].

[1]

@inproceedings{Koepke2021,
    author    = {Koepke, A.S. and Oncescu, A.-M. and Henriques, J. and Akata, Z. and Albanie, S.},
    title     = {Audio Retrieval with Natural Language Queries: A Benchmark Study},
    booktitle = {arXiv preprint arXiv:2112.09418},
    year      = {2021}
}

[2]

@inproceedings{Oncescu21a,
    author    = {Oncescu, A.-M. and Koepke, A.S. and Henriques, J. and Akata, Z., Albanie, S.},
    title     = {Audio Retrieval with Natural Language Queries},
    booktitle = {INTERSPEECH},
    year      = {2021}
}

[3]

@inproceedings{Liu2019a,
    author    = {Liu, Y. and Albanie, S. and Nagrani, A. and Zisserman, A.},
    title     = {Use What You Have: Video retrieval using representations from collaborative experts},
    booktitle = {British Machine Vision Conference (BMVC)},
    year      = {2019},
}

[4]

@inproceedings{gabeur2020mmt,
    author    = {Gabeur, V. and Sun, C. and Alahari, K. and Schmid, C.},
    title     = {Multi-modal Transformer for Video Retrieval},
    booktitle = {European Conference on Computer Vision (ECCV)},
    year      = {2020}
}