S3 Extract

Extract an archive file (zip file or tar file) stored on AWS S3.

Details

Downloads archive from S3 into memory, then extract and re-upload to given destination.

The following S3 information is expected to be given as Environment Variables:

AWS_ENDPOINT_URL
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
CERT_PATH (optional)
CERT_DL_URL (optional)
SIGNATURE_VERSION (optional, defaults to "s3v4")
REGION_NAME (optional, defaults to "us-east-1")

Additionally, these information needed for ClearML remote execution can also be given as Env Variable (optional, args will override env var):

DEFAULT_DOCKER_IMG
DEFAULT_QUEUE

For those not familiar, environment variables can be set through various ways, some being:

export = in terminal
can be set in ~/.bashrc as well for more permanence

Iteratively extracting a folder of zips/tars is supported as well, through --src-is-dir flag.

Usage

usage: run.py [-h] [--src-is-dir] [--dst-bucket DST_BUCKET] [--verbose] [--remote] [--clml-proj CLML_PROJ] [--clml-task-name CLML_TASK_NAME] [--clml-task-type CLML_TASK_TYPE]
              [--docker-img DOCKER_IMG] [--queue QUEUE]
              src_bucket src_path dst_path

positional arguments:
  src_bucket            Source bucket
  src_path              Source path
  dst_path              Destination path

optional arguments:
  -h, --help            show this help message and exit
  --src-is-dir          Flag to indicate that given src path is a directory. Will iteratively extract any files in it ending with .zip or .tar.
  --dst-bucket DST_BUCKET
                        Destination bucket (optional), will default to Source bucket.
  --verbose             print out current upload filename as it progresses
  --remote              use clearml to remotely run job
  --clml-proj CLML_PROJ
                        ClearML Project Name
  --clml-task-name CLML_TASK_NAME
                        ClearML Task Name
  --clml-task-type CLML_TASK_TYPE
                        ClearML Task Type, e.g. training, testing, inference, etc
  --docker-img DOCKER_IMG
                        Base docker image to pull for ClearML remote execution
  --queue QUEUE         ClearML remote execution queue

Example usage:

python run.py my-bucket dataset/coco/images.tar dataset/coco/ --verbose --remote --clml-proj coco --clml-task-name coco_extraction --docker-img ubuntu/20.04 --queue 1xGPU

Extract an archive file (zip file or tar file) stored on AWS S3

Related tags

Overview

S3 Extract

Details

Usage

Owner

Evan

Transforme rapidamente seu arquivo CSV (de qualquer tamanho) para SQL de forma rápida.

Search for files under the specified directory. Extract the file name and file path and import them as data.

Simple Python File Manager

Better directory iterator and faster os.walk(), now in the Python 3.5 stdlib

Media file renamer and organizion tool

Listreqs is a simple requirements.txt generator. It's an alternative to pipreqs

Python function to construct a ZIP archive with on the fly - without having to store the entire ZIP in memory or disk

Python's Filesystem abstraction layer

A tiny Configuration File Parser for Python Projects

This is a file deletion program that asks you for an extension of a file (.mp3, .pdf, .docx, etc.) to delete all of the files in a dir that have that extension.

Extract longest transcript or longest CDS transcript from GTF annotation file or gencode transcripts fasta file.

Fast Python reader and editor for ASAM MDF / MF4 (Measurement Data Format) files

A Python script to organize your files in a given directory.

Small Python script to generate a calendar (.ics) file from SIMASTER courses schedule.

A Python script to backup your favorite Discord gifs

Generates a clean .txt file of contents of a 3 lined csv file

Various converters to convert value sets from CSV to JSON, etc.

🧹 Create symlinks for .m2ts files and classify them into directories in yyyy-mm format.

Organize the files into the relevant sub-folders

pytiff is a lightweight library for reading chunks from a tiff file