Pytorch GPT-X

My Own Pytorch GPT-X

1. Abstract

Train GPT-3 model on V100(16GB Mem) Using improved Transformer.

2. Model

Transformer

Additional Module

① Rezero

Rezero Is All You Need link

② Explicit Sparse Transformer

Explicit Sparse Transformer: Concentrated Attention Through Explicit Selection link

③ Macaron Architecture

Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View link

④ RealFormer, Residual Attention

RealFormer link

Train

DeepSpeed

TODO

• ReZero
• RealFormer, Residual Attention
• Macaron architectures
• Macaron architectures - layer Scale 0.5
• Explicit Sparse Transformer
• torch lightning
• Deepspeed train on single GPU
• Deepspeed parallel trainig on 2 V100 GPU with 16GB Memory

Parameter For Few-shot

The 175B parameter model is very large, but a large model is needed for Few-Shot Learning. So this repository try to use DeepSpeed for training extremely big model.

GPT-3 Config

References

Transformer

• lucidrains/x-transformers

DeepSpeed

• DeepSpeed
• DeepSpeed Core API Doc

ReZero

• /majumderb/rezero

Explicit Sparse Transformer

• x-transformer: explicit_sparse_transformer

Macaron Architecrue

• Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View