Macro recording and metaprogramming in Python

Related tags

General Utilitiesmacro-kit
Overview

macro-kit

macro-kit is a package for efficient macro recording and metaprogramming in Python using abstract syntax tree (AST).

The design of AST in this package is strongly inspired by Julia metaprogramming. Similar methods are also implemented in builtin ast module but macro-kit is more focused on the macro generation and customization.

Installation

  • use pip
pip install macro-kit
  • from source
pip install git+https://github.com/hanjinliu/macro-kit

Examples

  1. Define a macro-recordable function
from macrokit import Macro, Expr, Symbol
macro = Macro()

@macro.record
def str_add(a, b):
    return str(a) + str(b)

val0 = str_add(1, 2)
val1 = str_add(val0, "xyz")
macro
[Out]
var0x24fdc2d1530 = str_add(1, 2)
var0x24fdc211df0 = str_add(var0x24fdc2d1530, 'xyz')

# substitute identifiers of variables
# var0x24fdc2d1530 -> x
macro.format([(val0, "x")]) 
[Out]
x = str_add(1, 2)
var0x24fdc211df0 = str_add(x, 'xyz')

# substitute to _dict["key"], or _dict.__getitem__("key")
expr = Expr(head="getitem", args=[Symbol("_dict"), "key"])
macro.format([(val0, expr)])
[Out]
_dict['key'] = str_add(1, 2)
var0x24fdc211df0 = str_add(_dict['key'], 'xyz')
  1. Record class
macro = Macro()

@macro.record
class C:
    def __init__(self, val: int):
        self.value = val
    
    @property
    def value(self):
        return self._value
    
    @value.setter
    def value(self, new_value: int):
        if not isinstance(new_value, int):
            raise TypeError("new_value must be an integer.")
        self._value = new_value
    
    def show(self):
        print(self._value)

c = C(1)
c.value = 5
c.value = -10
c.show()
[Out]
-10

macro.format([(c, "ins")])
[Out]
ins = C(1)
ins.value = -10     # setattr (and setitem) will not be recorded in duplicate
var0x7ffed09d2cd8 = ins.show()

macro.eval({"C": C})
[Out]
-10
  1. Record module
import numpy as np
macro = Macro()
np = macro.record(np) # macro-recordable numpy

arr = np.random.random(30)
mean = np.mean(arr)

macro
[Out]
var0x2a0a2864090 = numpy.random.random(30)
var0x2a0a40daef0 = numpy.mean(var0x2a0a2864090)

from dask import array as da
dask_macro = macro.format([(np, "da")])
dask_macro
[Out]
var0x2a0a2864090 = da.random.random(30)
var0x2a0a40daef0 = da.mean(var0x2a0a2864090)

output = {}
dask_macro.eval({"da": da}, output)
output
[Out]
{:da: 
   
    ,
 :var0x2a0a2864090: dask.array
    
     ,
 :var0x2a0a40daef0: dask.array
     
      }
You might also like...
Find dependent python scripts of a python script in a project directory.

Find dependent python scripts of a python script in a project directory.

null 2 Dec 5, 2021
SysInfo is an app developed in python which gives Basic System Info , and some detailed graphs of system performance .
SysInfo is an app developed in python which gives Basic System Info , and some detailed graphs of system performance .

SysInfo SysInfo is an app developed in python which gives Basic System Info , and some detailed graphs of system performance . Installation Download t

null 5 Nov 8, 2021
An OData v4 query parser and transpiler for Python

odata-query is a library that parses OData v4 filter strings, and can convert them to other forms such as Django Queries, SQLAlchemy Queries, or just plain SQL.

Gorilla 39 Jan 5, 2023
Python program to do with percentages and chances, random generation.

Chances and Percentages Python program to do with percentages and chances, random generation. What is this? This small program will generate a list wi

n0 3 Jul 15, 2021
A Python library for reading, writing and visualizing the OMEGA Format
A Python library for reading, writing and visualizing the OMEGA Format

A Python library for reading, writing and visualizing the OMEGA Format, targeted towards storing reference and perception data in the automotive context on an object list basis with a focus on an urban use case.

Institut für Kraftfahrzeuge, RWTH Aachen, ika 12 Sep 1, 2022
A simple and easy to use Spam Bot made in Python!

This is a simple spam bot made in python. You can use to to spam anyone with anything on any platform.

null 7 Sep 8, 2022
RapidFuzz is a fast string matching library for Python and C++

RapidFuzz is a fast string matching library for Python and C++, which is using the string similarity calculations from FuzzyWuzzy

Max Bachmann 1.7k Jan 4, 2023
pydsinternals - A Python native library containing necessary classes, functions and structures to interact with Windows Active Directory.
pydsinternals - A Python native library containing necessary classes, functions and structures to interact with Windows Active Directory.

pydsinternals - Directory Services Internals Library A Python native library containing necessary classes, functions and structures to interact with W

Podalirius 36 Dec 14, 2022
A Python package implementing various colour checker detection algorithms and related utilities.
A Python package implementing various colour checker detection algorithms and related utilities.

A Python package implementing various colour checker detection algorithms and related utilities.

colour-science 147 Dec 29, 2022
Releases(v0.3.8)

  • v0.3.8(Mar 16, 2022)

    Bug fixes

    • Length 1 tuple (1,) and length 0 set set() was not recorded correctly.
    • Import and __all__ did not match.
    • The __getitem__ was not correctly defined when a slice is used as indices.
    Source code(tar.gz)
    Source code(zip)

  • v0.3.7(Jan 22, 2022)

  • v0.3.6(Jan 13, 2022)

    Changes

    • Mock object for easier Expr construction. Instead of writing like Expr("getattr", [Expr("getattr", ...), b]), you can use m = Mock("m") and create Expr object by m.a(0, "s").
    • Full support of Python 3.7, and add tox test.

    Bug Fixes

    • The __eq__ method will return False instead of raise Exception to make macro recording safer.
    Source code(tar.gz)
    Source code(zip)

  • v0.3.4(Dec 12, 2021)

  • v0.3.3(Dec 10, 2021)

    New Features

    • New ast type supports: break, continue and raise.
    • Modules will be registered to Symbol whenever it is converted into a Symbol so that you'll not have to pass module object when you call eval. For instance, instead of macro.eval({"np", np}) you only have to call macro.eval().

    Changes

    • The "type" argument in Symbol is removed because it is never used.

    Bug Fixes

    • Hash of Symbol was not completely safe.
    • mModule did not return constant values correctly (such as np.pi).
    Source code(tar.gz)
    Source code(zip)

  • v0.3.2(Dec 8, 2021)

    New Features

    • register_type and Symbol.register_type can be used as a decorator.
    • Symbol.var("x") can create unique variable x.
    • Many missing AST types are supported, such as % (mod operation), += (in-place operations) and unary operations.
    • Validators are implemented in Expr to assert correct grammar for certain header (although they are not perfect). For instance, Expr("getattr", [Symbol("x"), "name"]) used to return :(x.'name') but now it returns :(x.name).
    • Many special methods, such as __int__ and __hash__, are correctly converted to int(X) in macro.
    • Macro can used as a field of a class. Unique Macro object will be created for different instances.

    Changes

    • Attribute _last_setval is moved to Macro.
    • Method type definition is changed to delayed definition.
    • Type mapping of symbol now uses dictionary as much as possible.
    • The type keyword argument in Symbol constructor is deleted because it was not used.

    Bug Fixes

    • Boolean operations (and and or) could not be parsed.
    Source code(tar.gz)
    Source code(zip)

  • v0.3.0(Dec 5, 2021)

    New Features

    • Support function definition, if, for, and many other expressions.
    • Macro.callbacks attribute for expression-added callbacks.
    • flags keyword argument to specify what kind of expressions will be recorded.

    Changes

    • Macro inherits Expr but head is fixed to Head.block to simplify module.
    Source code(tar.gz)
    Source code(zip)

  • v0.2.0(Oct 24, 2021)

    New Features

    • Support string parsing using ast.
    • optimize method of Macro can remove unused returned variables. It makes macro looks better.

    Changes

    • Head is different from that in v0.1.0, thus may be inconsistent.
    Source code(tar.gz)
    Source code(zip)

  • v0.1.0(Oct 23, 2021)

Owner
Department of Biological Sciences, School of Science, The University of Tokyo. My major is biophysics, especially microtubule and motor proteins.
GitHub Repository
A script to parse and display buy_tag and sell_reason for freqtrade backtesting trades

freqtrade-buyreasons A script to parse and display buy_tag and sell_reason for freqtrade backtesting trades Usage Copy the buy_reasons.py script into

Robert Davey 31 Jan 01, 2023
Script to decrypt / import chromium (edge/chrome) cookies

Cloonie Script to decrypt / import chromium (edge/chrome) cookies. Requirements Install the python dependencies via pip: pip install -r requirements.t

Lorenzo Bernardi 5 Sep 13, 2022
python script to generate color coded resistor images

Resistor image generator I got nerdsniped into making this. It's not finished at all, and the code is messy. The end goal it generate a whole E-series

MichD 1 Nov 12, 2021
Adding two matrix from scratch using python.

Adding-two-matrix-from-scratch-using-python. Here, I have take two matrix from user and add it without using any library. I made this program from scr

Sachin Vinayak Dabhade 4 Sep 24, 2021
[P]ython [w]rited [B]inary [C]onverter

pwbinaryc [P]ython [w]rited [Binary] [C]onverter You have rights to: Modify the code and use it private (friends are allowed too) Make a page and redi

0 Jun 21, 2022
Script to generate a massive volume of data in sql, csv, json or xml format

DataGenerator Made with Python Open for pull requests 1. Dependencies To install required dependencies run pip install -r requirements.txt 2. Executi

icrescenti 3 Sep 20, 2022
Astvuln is a simple AST scanner which recursively scans a directory, parses each file as AST and runs specified method.

Astvuln Astvuln is a simple AST scanner which recursively scans a directory, parses each file as AST and runs specified method. Some search methods ar

Bitstamp Security 7 May 29, 2022
Package that allows for validate and sanitize of string values.

py.validator A library of string validators and sanitizers Insipired by validator.js Strings only This library validates and sanitizes strings only. P

Sanel Hadzini 22 Nov 08, 2022
Display your calendar on the wallpaper.

wallCal Have your calendar appear as the wallpaper. disclaimer Use at your own risk. Don't blame me if you miss a meeting :-) Some parts of the script

7 Jun 14, 2022
A toolkit for writing and executing automation scripts for Final Fantasy XIV

XIV Scripter This is a tool for scripting out series of actions in FFXIV. It allows for custom actions to be defined in config.yaml as well as custom

Jacob Beel 1 Dec 09, 2021
Check username

Checker-Oukee Check username It checks the available usernames and creates a new account for them Doesn't need proxies Create a file with usernames an

4 Jun 05, 2022
A (very dirty) experiment to remove layers from a Docker image.

Surgically remove layers from a Docker image (with a chainsaw)

Jérôme Petazzoni 9 Jun 08, 2022
An online streamlit development platform

streamlit-playground An online streamlit development platform Run, Experiment and Play with streamlit Components Develop full-fledged apps online All

Akshansh Kumar 3 Nov 06, 2021
An awesome tool to save articles from RSS feed to Pocket automatically.

RSS2Pocket An awesome tool to save articles from RSS feed to Pocket automatically. About the Project I used to use IFTTT to save articles from RSS fee

Hank Liao 10 Nov 12, 2022
Shypan, a simple, easy to use, full-featured library written in Python.

Shypan, a simple, easy to use, full-featured library written in Python.

ShypanLib 4 Dec 08, 2021
Tools to connect to and interact with the Mila cluster

milatools The milatools package provides the mila command, which is meant to help with connecting to and interacting with the Mila cluster. Install Re

Mila 32 Dec 01, 2022
Similar looking domain detection using python fuzzywuzzy

Major cause of phishing and BEC incident is similar looking domain, if you detect it early, you can prevent incidents early, python fuzzywuzzy module let you do that

2 Nov 07, 2021
A tool written in python to generate basic repo files from github

A tool written in python to generate basic repo files from github

Riley 7 Dec 02, 2021
A script to check for common mistakes in LaTeX source files of scientific papers.

LaTeX Paper Linter This script checks for common mistakes in LaTeX source files of scientific papers. Usage python3 paperlint.py file.tex [-i/x inc

Michael Schwarz 12 Nov 16, 2022
Quickly edit your slack posts.

Lightning Edit Quickly edit your Slack posts. Heavily inspired by @KhushrajRathod's LightningDelete. Usage: Note: Before anything, be sure to head ove

14 Nov 19, 2021