A Python library for Deep Probabilistic Modeling

Abstract

DeeProb-kit is a Python library that implements deep probabilistic models such as various kinds of Sum-Product Networks, Normalizing Flows and their possible combinations for probabilistic inference. Some models are implemented using PyTorch for fast training and inference on GPUs.

Features

• Inference algorithms for SPNs. ^{1 4}
• Learning algorithms for SPNs structure. ^{1 2 3 4}
• Chow-Liu Trees (CLT) as SPN leaves. ^{11 12}
• Batch Expectation-Maximization (EM) for SPNs with arbitrarily leaves. ^{13 14}
• Structural marginalization and pruning algorithms for SPNs.
• High-order moments computation for SPNs.
• JSON I/O operations for SPNs and CLTs. ⁴
• Plotting operations based on NetworkX for SPNs and CLTs. ⁴
• Randomized And Tensorized SPNs (RAT-SPNs) using PyTorch. ⁵
• Masked Autoregressive Flows (MAFs) using PyTorch. ⁶
• Real Non-Volume-Preserving (RealNVP) and Non-linear Independent Component Estimation (NICE) flows. ^{7 8}
• Deep Generalized Convolutional SPNs (DGC-SPNs) using PyTorch. ¹⁰

The collection of implemented models is summarized in the following table. The supported data dimensionality for each model is showed in the Input Dimensionality column. Moreover, the Supervised column tells which model is suitable for a supervised learning task, other than density estimation task.

Installation & Documentation

The library can be installed either from PIP repository or by source code.

# Install from PIP repository
pip install deeprob-kit

# Install from `main` git branch
pip install -e git+https://github.com/deeprob-org/[email protected]#egg=deeprob-kit

The documentation is generated automatically by Sphinx (with Read-the-Docs theme), and it's hosted using GitHub Pages at deeprob-kit.

Datasets and Experiments

A collection of 29 binary datasets, which most of them are used in Probabilistic Circuits literature, can be found at UCLA-StarAI-Binary-Datasets.

Moreover, a collection of 5 continuous datasets, commonly present in works regarding Normalizing Flows, can be found at MAF-Continuous-Datasets.

After downloading them, the datasets must be stored in the experiments/datasets directory to be able to run the experiments (and Unit Tests). The experiments scripts are available in the experiments directory and can be launched using the command line by specifying the dataset and hyper-parameters.

Code Examples

A collection of code examples can be found in the examples directory. However, the examples are not intended to produce state-of-the-art results, but only to present the library.

The following table contains a description about them and a code complexity ranging from one to three stars. The Complexity column consists of a measure that roughly represents how many features of the library are used, as well as the expected time required to run the script.

Related Repositories

• SPFlow
• RAT-SPN
• Random-PC
• LibSPN-Keras
• MAF
• RealNVP

References

1. Peharz et al. On Theoretical Properties of Sum-Product Networks. AISTATS (2015).

2. Poon and Domingos. Sum-Product Networks: A New Deep Architecture. UAI (2011).

3. Molina, Vergari et al. Mixed Sum-Product Networks: A Deep Architecture for Hybrid Domains. AAAI (2018).

4. Molina, Vergari et al. SPFLOW : An easy and extensible library for deep probabilistic learning using Sum-Product Networks. CoRR (2019).

5. Peharz et al. Probabilistic Deep Learning using Random Sum-Product Networks. UAI (2020).

6. Papamakarios et al. Masked Autoregressive Flow for Density Estimation. NeurIPS (2017).

7. Dinh et al. Density Estimation using RealNVP. ICLR (2017).

8. Dinh et al. NICE: Non-linear Independent Components Estimation. ICLR (2015).

9. Papamakarios, Nalisnick et al. Normalizing Flows for Probabilistic Modeling and Inference. JMLR (2021).

10. Van de Wolfshaar and Pronobis. Deep Generalized Convolutional Sum-Product Networks for Probabilistic Image Representations. PGM (2020).

11. Rahman et al. Cutset Networks: A Simple, Tractable, and Scalable Approach for Improving the Accuracy of Chow-Liu Trees. ECML-PKDD (2014).

12. Di Mauro, Gala et al. Random Probabilistic Circuits. UAI (2021).

13. Desana and Schnörr. Learning Arbitrary Sum-Product Network Leaves with Expectation-Maximization. CoRR (2016).

14. Peharz et al. Einsum Networks: Fast and Scalable Learning of Tractable Probabilistic Circuits. ICML (2020).