Code for CPM-2 Pre-Train

Last update: Dec 28, 2022

Related tags

Deep Learning CPM-2

Overview

CPM-2 Pre-Train

Pre-train CPM-2 此分支为110亿非 MoE 模型的预训练代码，MoE 模型的预训练代码请切换到 moe 分支

CPM-2技术报告请参考link。

0 模型下载

请在智源资源下载页面进行申请，文件介绍如下：

文件名	描述	参数大小
100000.tar	纯中文模型	110亿
36000.tar	中英文双语模型	110亿
300000.tar	中英文MoE模型	1980亿

1 安装

可以直接拉取我们提供的 Docker 环境：

docker pull gyxthu17/cpm-2:1.0

2 数据

scripts/gen_data.sh 中给出了生成数据文件的脚本示例。该脚本将一个多行的纯文本文件（一个 document 一行）转化为二进制文件（会输出三个 .bin 和三个 .idx 文件），方便模型读取。

3 训练

首先需要将 WORKING_DIR 变量换成 CPM-2 目录的所在路径。调整 NUM_WORKERS 和 NUM_GPUS_PER_WORKER 指定机器数量与每台机器的 GPU 设备数量。修改 ${WORKING_DIR}/src/configs/host_files/hostfile-cpm2 文件将其中的主机名称替换成每台机器的 IP 地址或者和 IP 地址相关联的主机名称。

运行命令：

cd src
bash scripts/pretrain_enc_dec.sh

4 引用

如果您使用了我们的代码，请您引用下面的文章。

@article{cpm-v2,
  title={CPM-2: Large-scale Cost-efficient Pre-trained Language Models},
  author={Zhang, Zhengyan and Gu, Yuxian and Han, Xu and Chen, Shengqi and Xiao, Chaojun and Sun, Zhenbo and Yao, Yuan and Qi, Fanchao and Guan, Jian and Ke, Pei and Cai, Yanzheng and Zeng, Guoyang and Tan, Zhixing and Liu, Zhiyuan and Huang, Minlie and Han, Wentao and Liu, Yang and Zhu, Xiaoyan and Sun, Maosong},
  year={2021}
}

Code for CPM-2 Pre-Train

Related tags

Overview

CPM-2 Pre-Train

0 模型下载

1 安装

2 数据

3 训练

4 引用

Owner

Tsinghua AI

Activity image-based video retrieval

NICE-GAN — Official PyTorch Implementation Reusing Discriminators for Encoding: Towards Unsupervised Image-to-Image Translation

[ICCV 2021] Code release for "Sub-bit Neural Networks: Learning to Compress and Accelerate Binary Neural Networks"

SMORE: Knowledge Graph Completion and Multi-hop Reasoning in Massive Knowledge Graphs

Toward Realistic Single-View 3D Object Reconstruction with Unsupervised Learning from Multiple Images (ICCV 2021)

IAUnet: Global Context-Aware Feature Learning for Person Re-Identification

Algorithmic Trading using RNN

A library of scripts that interact with the PythonTurtle module to create games, drawings, and more

An open-access benchmark and toolbox for electricity price forecasting

Code for the paper "Controllable Video Captioning with an Exemplar Sentence"

Confident Semantic Ranking Loss for Part Parsing

The official PyTorch implementation of the paper: Xili Dai, Xiaojun Yuan, Haigang Gong, Yi Ma. "Fully Convolutional Line Parsing." .

Physics-Aware Training (PAT) is a method to train real physical systems with backpropagation.

这个开源项目主要是对经典的时间序列预测算法论文进行复现，模型主要参考自GluonTS，框架主要参考自Informer

Official code repository for the publication "Latent Equilibrium: A unified learning theory for arbitrarily fast computation with arbitrarily slow neurons"

An implementation of the research paper "Retina Blood Vessel Segmentation Using A U-Net Based Convolutional Neural Network"

Codes to calculate solar-sensor zenith and azimuth angles directly from hyperspectral images collected by UAV. Works only for UAVs that have high resolution GNSS/IMU unit.

PyTorch implementation of Self-supervised Contrastive Regularization for DG (SelfReg)

Automatically measure the facial Width-To-Height ratio and get facial analysis results provided by Microsoft Azure

[CVPR 21] Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.

Code for CPM-2 Pre-Train

Related tags

Overview

CPM-2 Pre-Train

0 模型下载

1 安装

2 数据

3 训练

4 引用

Owner

Tsinghua AI

Activity image-based video retrieval

NICE-GAN — Official PyTorch Implementation Reusing Discriminators for Encoding: Towards Unsupervised Image-to-Image Translation

[ICCV 2021] Code release for "Sub-bit Neural Networks: Learning to Compress and Accelerate Binary Neural Networks"

SMORE: Knowledge Graph Completion and Multi-hop Reasoning in Massive Knowledge Graphs

Toward Realistic Single-View 3D Object Reconstruction with Unsupervised Learning from Multiple Images (ICCV 2021)

IAUnet: Global Context-Aware Feature Learning for Person Re-Identification

Algorithmic Trading using RNN

A library of scripts that interact with the PythonTurtle module to create games, drawings, and more

An open-access benchmark and toolbox for electricity price forecasting

Code for the paper "Controllable Video Captioning with an Exemplar Sentence"

Confident Semantic Ranking Loss for Part Parsing

The official PyTorch implementation of the paper: *Xili Dai, Xiaojun Yuan, Haigang Gong, Yi Ma. "Fully Convolutional Line Parsing." *.

Physics-Aware Training (PAT) is a method to train real physical systems with backpropagation.

这个开源项目主要是对经典的时间序列预测算法论文进行复现，模型主要参考自GluonTS，框架主要参考自Informer

Official code repository for the publication "Latent Equilibrium: A unified learning theory for arbitrarily fast computation with arbitrarily slow neurons"

An implementation of the research paper "Retina Blood Vessel Segmentation Using A U-Net Based Convolutional Neural Network"

Codes to calculate solar-sensor zenith and azimuth angles directly from hyperspectral images collected by UAV. Works only for UAVs that have high resolution GNSS/IMU unit.

PyTorch implementation of Self-supervised Contrastive Regularization for DG (SelfReg)

Automatically measure the facial Width-To-Height ratio and get facial analysis results provided by Microsoft Azure

[CVPR 21] Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.

The official PyTorch implementation of the paper: Xili Dai, Xiaojun Yuan, Haigang Gong, Yi Ma. "Fully Convolutional Line Parsing." .