Fast Transformer - Pytorch
Implementation of Fast Transformer in Pytorch. This only work as an encoder.
Install
$ pip install fast-transformer-pytorch
Usage
import torch
from fast_transformer_pytorch import FastTransformer
model = FastTransformer(
num_tokens = 20000,
dim = 512,
depth = 2,
max_seq_len = 4096,
absolute_pos_emb = True # default uses relative positional encoding, but if that isn't working, then turn on absolute positional embedding by setting this to True
)
x = torch.randint(0, 20000, (1, 4096))
mask = torch.ones(1, 4096).bool()
logits = model(x, mask = mask) # (1, 4096, 20000)
Citations
@misc{wu2021fastformer,
title = {Fastformer: Additive Attention is All You Need},
author = {Chuhan Wu and Fangzhao Wu and Tao Qi and Yongfeng Huang},
year = {2021},
eprint = {2108.09084},
archivePrefix = {arXiv},
primaryClass = {cs.CL}
}
4 Jan 20, 2022
633 Dec 21, 2022
97 Nov 21, 2022
24.1k Jan 5, 2023
6.2k Dec 31, 2022
1.6k Dec 27, 2022
77 Jan 04, 2023
1.1k Dec 17, 2022
1.3k Dec 30, 2022
2 Mar 26, 2022
75 Dec 27, 2022
25.6k Dec 31, 2022
1 Nov 26, 2022
183 Dec 14, 2022
220 Dec 30, 2022
20 Apr 30, 2022
184 Dec 18, 2022
76 Dec 14, 2022
61 Nov 24, 2022
1.8k Jan 01, 2023
1 Feb 23, 2022
7 Nov 11, 2022
2.1k Dec 28, 2022
169 Jan 05, 2023
7 Aug 25, 2022