Optimizing Deeper Transformers on Small Datasets

Last update: Nov 14, 2022

Related tags

Deep Learning DT-Fixup

Overview

DT-Fixup

Optimizing Deeper Transformers on Small Datasets

Paper published in ACL 2021: arXiv

Detailed instructions to replicate our results in the paper can be found in the folders spider and reclor.

Cite

If you found this codebase or our work useful, please cite:

@InProceedings{xu2021optimizing,
  author = {Xu, Peng and Kumar, Dhruv and Yang, Wei and Zi, Wenjie and Tang, Keyi and Huang, Chenyang and Cheung, Jackie Chi Kit and Prince, Simon J.D. and Cao, Yanshuai},
  title = {Optimizing Deeper Transformers on Small Datasets}
  booktitle = {The 59th Annual Meeting of the Association for Computational Linguistics (ACL 2021)},
  month = {August},
  year = {2021},
  publisher = {ACL}
}

Owner

GitHub Repository

The implementation of PEMP in paper "Prior-Enhanced Few-Shot Segmentation with Meta-Prototypes"

Prior-Enhanced network with Meta-Prototypes (PEMP) This is the PyTorch implementation of PEMP. Overview of PEMP Meta-Prototypes & Adaptive Prototypes

8 Oct 14, 2021

Convolutional neural network that analyzes self-generated images in a variety of languages to find etymological similarities

This project is a convolutional neural network (CNN) that analyzes self-generated images in a variety of languages to find etymological similarities. Specifically, the goal is to prove that computer

1 Feb 03, 2022

Sub-tomogram-Detection - Deep learning based model for Cyro ET Sub-tomogram-Detection

Deep learning based model for Cyro ET Sub-tomogram-Detection High degree of stru

2 Feb 04, 2022

Official implementation of our paper "LLA: Loss-aware Label Assignment for Dense Pedestrian Detection" in Pytorch.

LLA: Loss-aware Label Assignment for Dense Pedestrian Detection This project provides an implementation for "LLA: Loss-aware Label Assignment for Dens

35 Dec 06, 2022

[NeurIPS 2021] Deceive D: Adaptive Pseudo Augmentation for GAN Training with Limited Data

Near-Duplicate Video Retrieval with Deep Metric Learning This repository contains the Tensorflow implementation of the paper Near-Duplicate Video Retr

238 Nov 25, 2022

.NET bindings for the Pytorch engine

TorchSharp TorchSharp is a .NET library that provides access to the library that powers PyTorch. It is a work in progress, but already provides a .NET

17 Aug 30, 2021

AEI: Actors-Environment Interaction with Adaptive Attention for Temporal Action Proposals Generation

AEI: Actors-Environment Interaction with Adaptive Attention for Temporal Action Proposals Generation A pytorch-version implementation codes of paper:

11 Dec 13, 2022

Beancount-mercury - Beancount importer for Mercury Startup Checking

beancount-mercury beancount-mercury provides an Importer for converting CSV expo

4 Oct 31, 2022

Pre-trained BERT Models for Ancient and Medieval Greek, and associated code for LaTeCH 2021 paper titled - "A Pilot Study for BERT Language Modelling and Morphological Analysis for Ancient and Medieval Greek"

Ancient Greek BERT The first and only available Ancient Greek sub-word BERT model! State-of-the-art post fine-tuning on Part-of-Speech Tagging and Mor

22 Dec 08, 2022

Tracking Progress in Question Answering over Knowledge Graphs

Tracking Progress in Question Answering over Knowledge Graphs Table of contents Question Answering Systems with Descriptions The QA Systems Table cont

47 Jan 02, 2023

🌾 PASTIS 🌾 Panoptic Agricultural Satellite TIme Series

🌾 PASTIS 🌾 Panoptic Agricultural Satellite TIme Series (optical and radar) The PASTIS Dataset Dataset presentation PASTIS is a benchmark dataset for

86 Jan 04, 2023

Mengzi Pretrained Models

中文 | English Mengzi 尽管预训练语言模型在 NLP 的各个领域里得到了广泛的应用，但是其高昂的时间和算力成本依然是一个亟需解决的问题。这要求我们在一定的算力约束下，研发出各项指标更优的模型。我们的目标不是追求更大的模型规模，而是轻量级但更强大，同时对部署和工业落地更友好的模型。

424 Jan 04, 2023

Code for IntraQ, PyTorch implementation of our paper under review

IntraQ: Learning Synthetic Images with Intra-Class Heterogeneity for Zero-Shot Network Quantization paper Requirements Python = 3.7.10 Pytorch == 1.7

1 Nov 19, 2021

Code for the paper "JANUS: Parallel Tempered Genetic Algorithm Guided by Deep Neural Networks for Inverse Molecular Design"

JANUS: Parallel Tempered Genetic Algorithm Guided by Deep Neural Networks for Inverse Molecular Design This repository contains code for the paper: JA

55 Nov 29, 2022

Optimizing Deeper Transformers on Small Datasets

Related tags

Overview

DT-Fixup

Cite

Owner

The implementation of PEMP in paper "Prior-Enhanced Few-Shot Segmentation with Meta-Prototypes"

Convolutional neural network that analyzes self-generated images in a variety of languages to find etymological similarities

Sub-tomogram-Detection - Deep learning based model for Cyro ET Sub-tomogram-Detection

Official implementation of our paper "LLA: Loss-aware Label Assignment for Dense Pedestrian Detection" in Pytorch.

[NeurIPS 2021] Deceive D: Adaptive Pseudo Augmentation for GAN Training with Limited Data

.NET bindings for the Pytorch engine

AEI: Actors-Environment Interaction with Adaptive Attention for Temporal Action Proposals Generation

Beancount-mercury - Beancount importer for Mercury Startup Checking

Pre-trained BERT Models for Ancient and Medieval Greek, and associated code for LaTeCH 2021 paper titled - "A Pilot Study for BERT Language Modelling and Morphological Analysis for Ancient and Medieval Greek"

Tracking Progress in Question Answering over Knowledge Graphs

🌾 PASTIS 🌾 Panoptic Agricultural Satellite TIme Series

Mengzi Pretrained Models

Code for IntraQ, PyTorch implementation of our paper under review

Code for the paper "JANUS: Parallel Tempered Genetic Algorithm Guided by Deep Neural Networks for Inverse Molecular Design"

Pytorch implementation of MaskGIT: Masked Generative Image Transformer

This is the accompanying toolbox for the paper "A Survey on GANs for Anomaly Detection"

Testing and Estimation of structural breaks in Stata

PyTorch implementation of "Image-to-Image Translation Using Conditional Adversarial Networks".

My course projects for the 2021 Spring Machine Learning course at the National Taiwan University (NTU)

MIMO-UNet - Official Pytorch Implementation