ASL-Skeleton3D and ASL-Phono Datasets Generator

The ASL-Skeleton3D contains a representation based on mapping into the three-dimensional space the coordinates of the signers in the ASLLVD dataset. The ASL-Phono, in turn, introduces a novel linguistics-based representation, which describes the signs in the ASLLVD dataset in terms of a set of attributes of the American Sign Language phonology.

This is the source code used to generate the ASL-Skeleton3D and ASL-Phono datasets, which are based on the American Sign Language Lexicon Video Dataset (ASLLVD).

Learn more about the datasets:

• Paper: "ASL-Skeleton3D and ASL-Phono: Two NovelDatasets for the American Sign Language" -> CIn

Download

Download the processed datasets by using the links below:

• ASL-Skeleton3D -> CIn | DOI

• ASL-Phono -> CIn | DOI

Generate

If you prefer generating the datasets by yourself, this section presents the requirements, setup and procedures to execute the code.

The generation is a process comprising the phases below, which start by the retrieval of the original ASLLVD samples for then computing additional properties, as follows:

• download: original samples (video sequences) are obtained from the ASLLVD.
• segment: signs are segmented from the original samples.
• skeleton: signer skeletons are estimated.
• normalize: the coordinates of the skeletons are normalized.
• phonology: the phonological attributes are extracted.

Requirements

To generate the datasets, your system will need the following software configured:

• Python 3.7 (or later)
• Poetry (latest)
• OpenPose
• asllvd-vid-reader (embedded to this project in the ./3rd_party/ folder or at the source-code repository)

OpenPose will require additional hardware and software configured which might include a NVIDIA GPU and related drivers and software. Please, check this link for the full list.

Recommended

If you prefer running a Docker container with the software requirements already configured, check out the link below -- just make sure to have a GPU available to your Docker environment:

• Docker / OpenPose

Installation

Once observed the requirements, checkout the source code and execute the following command, which will setup your virtual environment and dependencies:

$ poetry install

Configuration

There is a set of files in the folder ./config that will help you to configure the parameters for generating the datasets. A good starting point is to take a look into the ./config/template.yaml file, which contains a basic structure with all the properties documented.

You will also find other predefined configurations that might help you to generate the datasets. Just remember to always review the comments inside of the files to fine-tune the execution to your environment.

Learn about the configurations available in the ./config/template.yaml, which contains the properties documented.

Generation

ASL-Skeleton3D

The ASL-Skeleton3D is generated by using the configuration predefined in the file ./config/asl-skeleton3d.yaml. Thus, to start processing the dataset, execute the following command informing this file as the parameter -c (or --config):

$ poetry run python main.py -c ./config/asl-skeleton3d.yaml

The resulting dataset will be located in the folder configured as output for the phase normalize, which by default is set to ../work/dataset/normalized.

ASL-Phono

The ASL-Skeleton3D is generated by using the configuration predefined in the file ./config/asl-phono.yaml. Thus, to start processing the dataset, execute the following command informing this file as the parameter -c (or --config):

$ poetry run python main.py -c ./config/asl-phono.yaml

The resulting dataset will be located in the folder configured as output for the phase phonology, which by default is set to ../work/dataset/phonology.

Logs

The logs from the datasets processing will be recorded in the file ./output.log.

Deprecated datasets

Previously, we introduced the dataset ASLLVD-Skeleton, which is now being replaced by the ASL-Skeleton3D. Read more about the old dataset in the links:

• Paper: "Spatial-Temporal Graph Convolutional Networks for Sign Language Recognition" -> CIn | ICANN 2019 | Pre-print (arXiv)
• Dataset: ASLLVD-Skeleton (deprecated) -> CIn | Google Drive
• Dataset: ASLLVD-Skeleton-20 (deprecated) -> CIn | Google Drive

Citation

Please cite the following paper if you use this repository in your reseach.

@article{asl-datasets-2021,
  title     = {ASL-Skeleton3D and ASL-Phono: Two Novel Datasets for the American Sign Language},
  author    = {Cleison Correia de Amorim and Cleber Zanchettin},
  year      = {2021},
}

Contact

For any question, feel free to contact me at:

• Cleison Amorim : [email protected]