Python function to construct an ODS spreadsheet on the fly - without having to store the entire file in memory or disk

Overview

stream-write-ods CircleCI Test Coverage

Python function to construct an ODS (OpenDocument Spreadsheet) on the fly - without having to store the entire file in memory or disk.

Can be used to convert CSV, SQLite, or JSON to ODS format.

Installation

pip install stream-write-ods

Usage

In general, pass a nested iterable to stream_write_ods and it will return an interable of bytes of the ODS file, as follows.

from stream_write_ods import stream_write_ods

def get_sheets():
    def get_rows_of_sheet_1():
        yield 'Value A', 'Value B'
        yield 'Value C', 'Value D'

    yield 'Sheet 1 name', ('col_1_name', 'col_2_name'), get_rows_of_sheet_1()

    def get_rows_of_sheet_2():
        yield 'col_1_value',

    yield 'Sheet 2 name', ('col_1_name',), get_rows_of_sheet_2()

ods_chunks = stream_write_ods(get_sheets())

Usage: Convert CSV to ODS

The following recipe converts a CSV to ODS.

import codecs
import csv
from stream_write_ods import stream_write_ods

# Any iterable that yields the bytes of a CSV file
# Hard coded for the purposes of this example
csv_bytes_iter = (
    b'col_1,col_2\n',
    b'1,"value"\n',
)

def get_sheets(sheet_name, csv_reader):
    yield sheet_name, next(csv_reader), csv_reader

csv_str_iter = codecs.iterdecode(csv_bytes_iter, 'utf-8')
csv_reader = csv.reader(csv_str_iter, csv.QUOTE_NONNUMERIC)
ods_chunks = stream_write_ods(get_sheets('Sheet 1', csv_reader))

Usage: Convert JSON to ODS

Using ijson to stream-parse a JSON file, it's possible to convert JSON data to ODS on the fly:

import ijson
import itertools
from stream_write_ods import stream_write_ods

# Any iterable that yields the bytes of a JSON file
# Hard coded for the purposes of this example
json_bytes_iter = (b'''{
  "data": [
      {"id": 1, "name": "Foo"},
      {"id": 2, "name": "Bar"}
  ]
}''',)

# ijson requires a file-like object
def to_file_like_obj(bytes_iter):
    chunk = b''
    offset = 0
    it = iter(bytes_iter)

    def up_to_iter(num):
        nonlocal chunk, offset

        while num:
            if offset == len(chunk):
                try:
                    chunk = next(it)
                except StopIteration:
                    break
                else:
                    offset = 0
            to_yield = min(num, len(chunk) - offset)
            offset = offset + to_yield
            num -= to_yield
            yield chunk[offset - to_yield:offset]

    class FileLikeObj:
        def read(self, n):
            return b''.join(up_to_iter(n))

    return FileLikeObj()

def get_sheets(json_file):
    columns = None

    def rows():
        nonlocal columns
        for item in ijson.items(json_file, 'data.item'):
            if columns is None:
                columns = list(item.keys())
            yield tuple(item[column] for column in columns)

    # Ensure columns populated
    rows_it = rows()
    first_row = next(rows_it)

    yield 'Sheet 1', columns, itertools.chain((first_row,), rows_it)

json_file = to_file_like_obj(json_bytes_iter)
ods_chunks = stream_write_ods(get_sheets(json_file))

Usage: Convert SQLite to ODS

SQLite isn't particularly streaming-friendly since typically you need random access to the file. But it's still possible to use stream-write-ods to convert SQLite to ODS.

import contextlib
import sqlite3
import tempfile
from stream_write_ods import stream_write_ods

@contextlib.contextmanager
def get_db():
    # Hard coded in memory database for the purposes of this example
    with sqlite3.connect(':memory:') as con:
        cur = con.cursor()
        cur.execute("CREATE TABLE my_table_a (my_col text);")
        cur.execute("CREATE TABLE my_table_b (my_col text);")
        cur.execute("INSERT INTO my_table_a VALUES ('Value A')")
        cur.execute("INSERT INTO my_table_b VALUES ('Value B')")
        yield con

def quote_identifier(value):
    return '"' + value.replace('"', '""') + '"'

def get_sheets(db):
    cur_table = db.cursor()
    cur_table.execute('''
        SELECT name FROM sqlite_master
        WHERE type = "table" AND name NOT LIKE 'sqlite\\_%' ESCAPE '\\'
    ''')
    cur_data = db.cursor()
    for table, in cur_table:
        cur_data.execute(f'SELECT * FROM {quote_identifier(table)} ORDER BY rowid')
        yield table, tuple(col[0] for col in cur_data.description), cur_data

with get_db() as db:
    ods_chunks = stream_write_ods(get_sheets(db))

Types

There are 8 possible data types in an Open Document Spreadsheet: boolean, currency, date, float, percentage, string, time, and void. 4 of these can be output by stream-write-ods, chosen automatically according to the following table.

Python type ODS type
boolean boolean
date date - without time component
datetime date - with time component
int float
float float
str string
NoneType string - as #NA

Limitations

ODS spreadsheets are essentially ZIP archives containing several member files. While in general ZIP archives can be up to 16EiB (exbibyte) in size using ZIP64, LibreOffice does not support ZIP64, and so ODS files are de-facto limited to 4GiB (gibibyte). This limit applies to the size of the entire compressed archive, the compressed size of each member file, and the uncompressed size of each member file.

Owner
Department for International Trade
Department for International Trade
Simple Instagram Login Link Generator

instagram-account-login Simple Instagram Login Link Generator Info Program generates instagram login links and you may get into someone´s thought the

Kevin 5 Dec 03, 2022
A python bot that stops muck chains

muck-chains-stopper-bot a bot that stops muck chains this is the source code of u/DaniDevChainBreaker (the main r/DaniDev muck chains breaker) guys th

24 Jan 04, 2023
A updated and improved version from the original Discord-Netflix from Nirewen.

Discord-Netflix A updated version from the original Discord-Netflix from nirewen A Netflix wrapper that uses Discord RPC to show what you're watching

Void 42 Jan 02, 2023
A simple python script to send files into your telegram Bot form your PC, Server etc.

telegramSend A simple python script to send files into your telegram Bot form your PC, Server etc. How to Use Install requirements.txt pip3 install -r

Ajay Kumar Tekam 1 Jul 19, 2022
Marketplace for self published books

Nile API API for the imaginary Nile marketplace for self published books. This is a project created to try out FastAPI as the post promising ASGI serv

Matt de Young 1 Jan 31, 2022
A Telegram Video Watermark Adder Bot in Pyrogram by @AbirHasan2005

Watermark-Bot A Telegram Video Watermark Adder Bot by @AbirHasan2005 Features: Save Custom Watermark Image. Auto Resize Watermark According to Video q

Abir Hasan 95 Nov 20, 2022
Its Is A Telegram Maths Basic Calculator Bot

Its Is A Telegram Maths Basic Calculator Bot

ANKIT KUMAR 1 Dec 26, 2021
A powerful application to automatically deploy GitHub Release.

A powerful application to automatically deploy GitHub Release.

Fentaniao 43 Sep 17, 2022
This repository contains unofficial code reproducing Agent57

Agent57 This repository contains unofficial code reproducing Agent57, which outp

19 Dec 29, 2022
Nowadays we don't have time to listen to each and every song that we come across in a playlist.

Nowadays we don't have time to listen to each and every song that we come across in a playlist. so, this project helps you. we used Spotify API for collecting the dataset information and able to do E

3 Apr 26, 2022
Fetch Flipkart product details including name, price, MRP and Stock details in general as well as specific to a pincode

Fetch Flipkart product details including name, price, MRP and Stock details in general as well as specific to a pincode

Vishal Das 6 Jul 11, 2022
LoL 台版10周年活動自動輸入邀請碼

LoLTW_10Year_88Event LoLTW 8.8 周年慶 邀請碼自動輸入 設定 在 LoLTW_10Year_88Evnet.exe 的位置建立一個檔案 .env,內容如下 Bahamut_Discussion = https://forum.gamer.com.tw/C.php?bsn

古丁丁 5 Dec 13, 2021
Wechat-file-cleaner - Clean files in PC WeChat FileStorage directory

Wechat-file-cleaner - Clean files in PC WeChat FileStorage directory

Xingjian Zhang 1 Feb 06, 2022
Telegram bot for searching videos in your PDisk account by @AbirHasan2005

PDisk-Videos-Search A Telegram bot for searching videos in your PDisk account by @AbirHasan2005. Configs API_ID - Get from @TeleORG_Bot API_HASH - Get

Abir Hasan 39 Oct 21, 2022
A cs:go cheat/hack made in Python3.

Atomic 💖 Cheat for cs:go written in Python. Features. Glow Esp No Flash Bunny Hop Third Person To-Do. It is prefered to start the cheat when you are

Sofia 6 Feb 12, 2022
Moon-TikTok-Checker - A TikTok Username checking tool that probably 3/4 people use to get rare usernames

Moon Checker (educational Purposes Only) What Is Moon Checker? This is a TikTok

glide 4 Nov 30, 2022
A reddit.com bot that will return reference links from official python documentation site for the standard library.

Python Docs Bot A reddit.com bot that will return documentation links for the library and language reference sections of the python docs website. The

Trevor Miller 2 Sep 14, 2021
TORNADO CASH Pancakeswap Sniper BOT 2022-V1 (MAC WINDOWS ANDROID LINUX)

TORNADO CASH Pancakeswap Sniper BOT 2022-V1 (MAC WINDOWS ANDROID LINUX)

Crypto Trader 1 Jan 06, 2022
A simple API wrapper for Discord written in Python.

AIOCord This project is work in progress not for production use A simple asynchronous API wrapper around Discord API written in Python. Inspiration Th

Izhar Ahmad 3 Dec 07, 2021
A Sublime Text plugin that displays inline images for single-line comments formatted like `// ![](example.png)`.

Inline Images Sometimes ASCII art is not enough. Sometimes an image says more than a thousand words. This Sublime Text plugin can display images inlin

Andreas Haferburg 8 Jul 01, 2022