Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

Last update: Dec 29, 2022

Related tags

Deep Learning wechsel

Overview

WECHSEL

Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

arXiv: https://arxiv.org/abs/2112.06598

Models from the paper are available on the HuggingFace Hub:

Installation

We distribute a Python Package via PyPI:

pip install wechsel

Alternatively, clone the repository, install requirements.txt and run the code in wechsel/.

Example usage

Transferring English roberta-base to Swahili:

import torch
from transformers import AutoModel, AutoTokenizer
from datasets import load_dataset
from wechsel import WECHSEL, load_embeddings

source_tokenizer = AutoTokenizer.from_pretrained("roberta-base")
model = AutoModel.from_pretrained("roberta-base")

target_tokenizer = source_tokenizer.train_new_from_iterator(
    load_dataset("oscar", "unshuffled_deduplicated_sw", split="train")["text"],
    vocab_size=len(source_tokenizer)
)

wechsel = WECHSEL(
    load_embeddings("en"),
    load_embeddings("sw"),
    bilingual_dictionary="swahili"
)

target_embeddings, info = wechsel.apply(
    source_tokenizer,
    target_tokenizer,
    model.get_input_embeddings().weight.detach().numpy(),
)

model.get_input_embeddings().weight.data = torch.from_numpy(target_embeddings)

# use `model` and `target_tokenizer` to continue training in Swahili!

Bilingual dictionaries

We distribute 3276 bilingual dictionaries from English to other languages for use with WECHSEL in dicts/.

Citation

Please cite WECHSEL as

@misc{minixhofer2021wechsel,
      title={WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models}, 
      author={Benjamin Minixhofer and Fabian Paischer and Navid Rekabsaz},
      year={2021},
      eprint={2112.06598},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

Related tags

Overview

WECHSEL

Installation

Example usage

Bilingual dictionaries

Citation

Owner

Institute of Computational Perception

Seach Losses of our paper 'Loss Function Discovery for Object Detection via Convergence-Simulation Driven Search', accepted by ICLR 2021.

For medical image segmentation

Awesome Artificial Intelligence, Machine Learning and Deep Learning as we learn it

Official code of the paper "Expanding Low-Density Latent Regions for Open-Set Object Detection" (CVPR 2022)

MolRep: A Deep Representation Learning Library for Molecular Property Prediction

learning and feeling SLAM together with hands-on-experiments

NumPy로 구현한 딥러닝 라이브러리입니다. (자동 미분 지원)

Static Features Classifier - A static features classifier for Point-Could clusters using an Attention-RNN model

Light-weight network, depth estimation, knowledge distillation, real-time depth estimation, auxiliary data.

Plenoxels: Radiance Fields without Neural Networks

PyTorch implementation of Super SloMo by Jiang et al.

StyleGAN2-ADA - Official PyTorch implementation

A PyTorch Library for Accelerating 3D Deep Learning Research

The implementation for paper Joint t-SNE for Comparable Projections of Multiple High-Dimensional Datasets.

Code accompanying the paper Shared Independent Component Analysis for Multi-subject Neuroimaging

Implementation of Neural Style Transfer in Pytorch

Scene-Text-Detection-and-Recognition (Pytorch)

Fuse radar and camera for detection

BridgeGAN - Tensorflow implementation of Bridging the Gap between Label- and Reference-based Synthesis in Multi-attribute Image-to-Image Translation.

Pytorch version of VidLanKD: Improving Language Understanding viaVideo-Distilled Knowledge Transfer