Implementation of a Transformer using ReLA (Rectified Linear Attention)

Last update: Oct 14, 2022

Related tags

Overview

ReLA (Rectified Linear Attention) Transformer

Implementation of a Transformer using ReLA (Rectified Linear Attention). It will also contain an attempt to combine the feedforward into the ReLA layer as memory key / values, as proposed in All Attention, suggestion made by Charles Foster.

Install

$ pip install rela-transformer

Usage

import torch
from rela_transformer.rela_transformer import ReLATransformer

model = ReLATransformer(
    num_tokens = 20000,
    dim = 512,
    depth = 8,
    max_seq_len = 1024,
    dim_head = 64,
    heads = 8,
    causal = True
)

x = torch.randint(0, 20000, (1, 1024))
logits = model(x) # (1, 1024, 20000)

Enwik8

$ python train.py

Citations

@misc{zhang2021sparse,
    title   = {Sparse Attention with Linear Units},
    author  = {Biao Zhang and Ivan Titov and Rico Sennrich},
    year    = {2021},
    eprint  = {2104.07012},
    archivePrefix = {arXiv},
    primaryClass = {cs.CL}
}

You might also like...

Attention for PyTorch with Linear Memory Footprint

Attention for PyTorch with Linear Memory Footprint Unofficially implements https://arxiv.org/abs/2112.05682 to get Linear Memory Cost on Attention (+

11 Jan 9, 2022

Episodic Transformer (E.T.) is a novel attention-based architecture for vision-and-language navigation. E.T. is based on a multimodal transformer that encodes language inputs and the full episode history of visual observations and actions.

Episodic Transformers (E.T.) Episodic Transformer for Vision-and-Language Navigation Alexander Pashevich, Cordelia Schmid, Chen Sun Episodic Transform

62 Dec 24, 2022

VSR-Transformer - This paper proposes a new Transformer for video super-resolution (called VSR-Transformer).

VSR-Transformer By Jiezhang Cao, Yawei Li, Kai Zhang, Luc Van Gool This paper proposes a new Transformer for video super-resolution (called VSR-Transf

225 Nov 13, 2022

Implementation of H-Transformer-1D, Hierarchical Attention for Sequence Learning using 🤗 transformers

hierarchical-transformer-1d Implementation of H-Transformer-1D, Hierarchical Attention for Sequence Learning using 🤗 transformers In Progress!! 2021.

7 Nov 6, 2022

Implementation of the 😇 Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones

HaloNet - Pytorch Implementation of the Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones. This re

189 Nov 22, 2022

Implementation of STAM (Space Time Attention Model), a pure and simple attention model that reaches SOTA for video classification

STAM - Pytorch Implementation of STAM (Space Time Attention Model), yet another pure and simple SOTA attention model that bests all previous models in

109 Dec 28, 2022

Official Pytorch Implementation of Relational Self-Attention: What's Missing in Attention for Video Understanding

Relational Self-Attention: What's Missing in Attention for Video Understanding This repository is the official implementation of "Relational Self-Atte

43 Dec 7, 2022

Implementation of a memory efficient multi-head attention as proposed in the paper, "Self-attention Does Not Need O(n²) Memory"

Memory Efficient Attention Pytorch Implementation of a memory efficient multi-head attention as proposed in the paper, Self-attention Does Not Need O(

180 Jan 5, 2023

Official implementation of cosformer-attention in cosFormer: Rethinking Softmax in Attention

cosFormer Official implementation of cosformer-attention in cosFormer: Rethinking Softmax in Attention Update log 2022/2/28 Add core code License This

120 Dec 15, 2022

Comments

LayerNorm/GatedRMS inconsistency
Hi! looking through pipeline it seems there are some inconsistencies with normalisation

# ReLA input to GRMSNorm # att code output: Linear(inner_dim, dim) + GRMSNorm # next in FF module input to LayerNorm

here we have problem with double norm since we have last layer GRMSNorm in att and first layer LayerNorm in FF.

looking at the paper it seems that in ReLA GRMSNorm is applied to result of mult(attn, v) before output projection not after projection like in this code. I also confused about usage of LayerNorm in FF should it be GRMSNorm instead? not clear from the paper as well
opened by inspirit 6

Releases(0.0.7)

0.0.7(Apr 6, 2022)

Source code(tar.gz)
Source code(zip)
0.0.6(Feb 22, 2022)

Source code(tar.gz)
Source code(zip)
0.0.5(Jan 13, 2022)

Source code(tar.gz)
Source code(zip)
0.0.4(Jan 11, 2022)

Source code(tar.gz)
Source code(zip)
0.0.3(Jan 10, 2022)

Source code(tar.gz)
Source code(zip)
0.0.2a(Jan 10, 2022)

Source code(tar.gz)
Source code(zip)
0.0.2(Jan 10, 2022)

Source code(tar.gz)
Source code(zip)
0.0.1(Jan 10, 2022)

Source code(tar.gz)
Source code(zip)

Owner

Phil Wang

Working with Attention. It's all we need

GitHub Repository

Implementation of a Transformer using ReLA (Rectified Linear Attention)

Related tags

Overview

ReLA (Rectified Linear Attention) Transformer

Install

Usage

Enwik8

Citations

You might also like...

Attention for PyTorch with Linear Memory Footprint

Episodic Transformer (E.T.) is a novel attention-based architecture for vision-and-language navigation. E.T. is based on a multimodal transformer that encodes language inputs and the full episode history of visual observations and actions.

VSR-Transformer - This paper proposes a new Transformer for video super-resolution (called VSR-Transformer).

Implementation of H-Transformer-1D, Hierarchical Attention for Sequence Learning using 🤗 transformers

Implementation of the 😇 Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones

Implementation of STAM (Space Time Attention Model), a pure and simple attention model that reaches SOTA for video classification

Official Pytorch Implementation of Relational Self-Attention: What's Missing in Attention for Video Understanding

Implementation of a memory efficient multi-head attention as proposed in the paper, "Self-attention Does Not Need O(n²) Memory"

Official implementation of cosformer-attention in cosFormer: Rethinking Softmax in Attention

Comments

LayerNorm/GatedRMS inconsistency

Releases(0.0.7)

0.0.7(Apr 6, 2022)

0.0.6(Feb 22, 2022)

0.0.5(Jan 13, 2022)

0.0.4(Jan 11, 2022)

0.0.3(Jan 10, 2022)

0.0.2a(Jan 10, 2022)

0.0.2(Jan 10, 2022)

0.0.1(Jan 10, 2022)

Owner

Phil Wang

ML powered analytics engine for outlier detection and root cause analysis.

A model that attempts to learn and benefit from data collected on card counting.

Code implementation of "Sparsity Probe: Analysis tool for Deep Learning Models"

A general python framework for single object tracking in LiDAR point clouds, based on PyTorch Lightning.

LogAvgExp - Pytorch Implementation of LogAvgExp

Storage-optimizer - Identify potintial optimizations on the cloud storage accounts

Multi-Objective Loss Balancing for Physics-Informed Deep Learning

A Large Scale Benchmark for Individual Treatment Effect Prediction and Uplift Modeling

Manipulation OpenAI Gym environments to simulate robots at the STARS lab

Experiments and examples converting Transformers to ONNX

Complete U-net Implementation with keras

A high-level Python library for Quantum Natural Language Processing

Pytorch code for our paper Beyond ImageNet Attack: Towards Crafting Adversarial Examples for Black-box Domains)

Code to train models from "Paraphrastic Representations at Scale".

L-Verse: Bidirectional Generation Between Image and Text

you can add any codes in any language by creating its respective folder (if already not available).

Training Structured Neural Networks Through Manifold Identification and Variance Reduction

Cobalt Strike teamserver detection.

A Simulation Environment to train Robots in Large Realistic Interactive Scenes

Code release for "MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound"