ICE Tokenizer

Token id [0, 20000) are image tokens.
Token id [20000, 20100) are common tokens, mainly punctuations. E.g., icetk[20000] == ' ', icetk[20003] == ' ', icetk[20006] == ','.
Token id [20100, 83823) are English tokens.
Token id [83823, 145653) are Chinese tokens.
Token id [145653, 150000) are rare tokens. E.g., icetk[145803] == 'α'.

You can install the package via

pip install icetk

Tokenization

from icetk import icetk
tokens = icetk.tokenize('Hello World! I am icetk.')
# tokens == ['▁Hello', '▁World', '!', '▁I', '▁am', '▁ice', 'tk', '.']
ids = icetk.encode('Hello World! I am icetk.')
# ids == [39316, 20932, 20035, 20115, 20344, 22881, 35955, 20007]
en = icetk.decode(ids)
# en == 'Hello World! I am icetk.' # always perfectly recover (if without 
   
    )
   

ids = icetk.encode('你好世界！这里是 icetk。')
# ids == [20005, 94874, 84097, 20035, 94947, 22881, 35955, 83823]

ids = icetk.encode(image_path='test.jpeg', image_size=256, compress_rate=8)
# ids == tensor([[12738, 12430, 10398,  ...,  7236, 12844, 12386]], device='cuda:0')
# ids.shape == torch.Size([1, 1024])
img = icetk.decode(image_ids=ids, compress_rate=8)
# img.shape == torch.Size([1, 3, 256, 256])
from torchvision.utils import save_image
save_image(img, 'recover.jpg')

A unified tokenization tool for Images, Chinese and English.

Related tags

Overview

ICE Tokenizer

Tokenization

Owner

THUDM

Python3 to Crystal Translation using Python AST Walker

PIZZA - a task-oriented semantic parsing dataset

Yodatranslator is a simple translator English to Yoda-language

Code for the paper "BERT Loses Patience: Fast and Robust Inference with Early Exit".

Rank-One Model Editing for Locating and Editing Factual Knowledge in GPT

Examples of using sparse attention, as in "Generating Long Sequences with Sparse Transformers"

A library for end-to-end learning of embedding index and retrieval model

American Sign Language (ASL) to Text Converter

Neural network models for joint POS tagging and dependency parsing (CoNLL 2017-2018)

Yomichad - a Japanese pop-up dictionary that can display readings and English definitions of Japanese words

GraphNLI: A Graph-based Natural Language Inference Model for Polarity Prediction in Online Debates

KLUE-baseline contains the baseline code for the Korean Language Understanding Evaluation (KLUE) benchmark.

Spert NLP Relation Extraction API deployed with torchserve for inference

Stack based programming language that compiles to x86_64 assembly or can alternatively be interpreted in Python

This is the Alpha of Nutte language, she is not complete yet / Essa é a Alpha da Nutte language, não está completa ainda

Backend for the Autocomplete platform. An AI assisted coding platform.

Stanford CoreNLP provides a set of natural language analysis tools written in Java

CrossNER: Evaluating Cross-Domain Named Entity Recognition (AAAI-2021)

MEDIALpy: MEDIcal Abbreviations Lookup in Python

Fast, DB Backed pretrained word embeddings for natural language processing.