Python Jupyter kernel using Poetry for reproducible notebooks

Last update: Jan 04, 2023

Related tags

Overview

Poetry Kernel

Use per-directory Poetry environments to run Jupyter kernels. No need to install a Jupyter kernel per Python virtual environment!

The idea behind this project is to allow you to capture the exact state of your environment. This means you can email your work to your peers, and they'll have exactly the same set of packages that you do! Reproducibility!

Why not virtual environments (venvs)?

Virtual environments were (and are) an important advancement to Python's package management story, but they have a few shortcomings:

They are not great for reproducibility. Usually, you'll create a new virtual environment using a requirements.txt which includes all the direct dependencies (numpy, pandas, etc.), but not transient dependencies (pandas depends on pytz for timezone support, for example). And usually, even the direct dependencies are specified only as minimum (or semver) ranges (e.g., numpy>=1.21) which can make it hard or impossible to accurately recreate the venv later.
With Jupyter, they usually require that the kernels be installed globally. This means you'll need need to have a separate kernelspec for every venv you want to use with Jupyter.

Poetry uses venvs transparently under the hood by constructing them from the pyproject.toml and poetry.lock files. The poetry.lock file records the exact state of dependencies (and transient dependencies) and can be used to more accurately reproduce the environment.

Additionally, Poetry Kernel means you only have to install one kernelspec. It then uses the pyproject.toml file from the directory of the notebook (or any parent directory) to choose which environment to run the notebook in.

Shameless plug

The reason we created this package was to make sure that the code environments created for running student code on Pathbird exactly match your development environment. Interested in developing interactive, engaging, inquiry-based lessons for your students? Check out Pathbird for more information!

Usage

Install Poetry if not yet installed.

Install this package:

# NOTE: Do **NOT** install this package in your Poetry project, it should be
# installed at the system or user level.
pip3 install --user poetry-kernel

Initialize a Poetry project (only required if you do not have an existing Poetry project ready to use):
```
poetry init -n
```

IMPORTANT: Add ipykernel to your project's dependencies:

# In the directory of your Poetry project
poetry add ipykernel

Start a "Poetry" Jupyter kernel and see it in action!

Troubleshooting

Kernel isn't starting ("No Kernel" message)

Pro-tip: Check the output of the terminal window where you launched Jupyter. It will usually explain why the kernel is failing to start.

Make sure that you are launching a notebook in a directory/folder that contains a Poetry project (pyproject.toml and poetry.lock files). You can turn a directory into a Poetry project by running:

poetry init -n

Make sure that you've installed ipykernel into your project:

poetry add ipykernel

Make sure the Poetry project is installed! This is especially important for projects that you have downloaded from others (warning: installing a Poetry project could run arbitrary code on your computer, make sure you trust your download first!):
```
poetry install
```
Still can't figure it out? Open an issue!

A package I added won't import properly

If you added the package after starting the kernel, you might need to restart the kernel for it to see the new package.

FAQ

See FAQ.md.

Comments

Windows Does not have SIGKILL

I believe the following line needs more attention to be compatible with Windows considering windows does not have SIGKILL: https://github.com/pathbird/poetry-kernel/blob/main/poetry_kernel/main.py#L39
bug good first issue prs accepted

opened by amirhessam88 4
FileNotFoundError: [Errno 2] No such file or directory: 'poetry'.
Hey, cool project!

I got the following error, not sure how informative it is for you. Happy to dig into logs etc. I'm using a Gitpod workspace with a public github repo too, so could potentially offer a completely reproducible clean environment.

Failed to start the Kernel. FileNotFoundError: [Errno 2] No such file or directory: 'poetry'.
opened by andrewcstewart 3
Check in parent_dirs for pyproject.toml

This now loops through parent directories to check for pyproject.toml to allow for kernel execution when running notebooks in sub directories. I used the same code from poetry itself to ensure compatibility

opened by nick-gorse 3
modified forward_signals to be compatible with windows

in regard to issue #3

removed signal.SIGTERM from forward_signals, as well as two others as they could not be used by signal.signal (ValueError: invalid signal value)

from https://docs.python.org/3/library/signal.html signal.CTRL_C_EVENT The signal corresponding to the Ctrl+C keystroke event. This signal can only be used with os.kill()

same for signal.CTRL_BREAK_EVENT

opened by gpfv 0
Check in parent_dirs for pyproject.toml

This now loops through parent directories to check for pyproject.toml to allow for kernel execution when running notebooks in sub directories. I used the same code as used poetry itself to ensure compatibility

opened by nick-gorse 0
poetry-kernel uses Jupyter env instead of notebook env

If a poetry pyproject.toml is used to create the environment for the Jupyter lab (adding poetry-kernel as a dependency), then no matter in which working directory the jupyter server is started, any notebook that is opened with the "poetry" kernel will have the Jupyter environment, not the environment of the notebook's project.

Minimal example: https://github.com/drakesiardxy/poetry-kernel-bug To replicate: Create the jupyer-base env and the kernel_a env separately, then start the jupyter server with the first environment and attempt to run kernel_a.ipynb using the "Poetry" kernel. pandas will be missing, because the notebook will have been launched with the environment of the server, not the environment belonging to the notebook's project.

opened by drakesiardxy 0

Is it possible to make poetry-kernel work with JupyterHub?

Hi!

We have a multi-user JupyterHub instance and we would like to use your library. So far we have only been able to see the poetry-kernel button in the kernel selection screen by installing poetry kernel at the user level (if we install poetry kernel from the user that launches JupyterHub, the root user, the button does not show). But the problem that we have is that when we press the poetry button and create a new notebook in a folder with a poetry project (and ipykernel installed), the kernel is never connected so no code can be executed. The generated logs are the following:

Apr 26 15:30:26 labs-ubuntu-20-04 python3[446728]: [I 2022-04-26 15:30:26.298 SingleUserLabApp restarter:66] AsyncIOLoopKernelRestarter: restarting kernel (2/5), new random ports
Apr 26 15:30:26 labs-ubuntu-20-04 python3[1028846]: Traceback (most recent call last):
Apr 26 15:30:26 labs-ubuntu-20-04 python3[1028846]:   File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
Apr 26 15:30:26 labs-ubuntu-20-04 python3[1028846]:     return _run_code(code, main_globals, None,
Apr 26 15:30:26 labs-ubuntu-20-04 python3[1028846]:   File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
Apr 26 15:30:26 labs-ubuntu-20-04 python3[1028846]:     exec(code, run_globals)
Apr 26 15:30:26 labs-ubuntu-20-04 python3[1028846]:   File "/home/marta/.local/lib/python3.8/site-packages/poetry_kernel/__main__.py", line 68, in <module>
Apr 26 15:30:26 labs-ubuntu-20-04 python3[1028846]:     main()
Apr 26 15:30:26 labs-ubuntu-20-04 python3[1028846]:   File "/home/marta/.local/lib/python3.8/site-packages/poetry_kernel/__main__.py", line 37, in main
Apr 26 15:30:26 labs-ubuntu-20-04 python3[1028846]:     proc = subprocess.Popen(cmd)
Apr 26 15:30:26 labs-ubuntu-20-04 python3[1028846]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Apr 26 15:30:26 labs-ubuntu-20-04 python3[1028846]:     self._execute_child(args, executable, preexec_fn, close_fds,
Apr 26 15:30:26 labs-ubuntu-20-04 python3[1028846]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Apr 26 15:30:26 labs-ubuntu-20-04 python3[1028846]:     raise child_exception_type(errno_num, err_msg, err_filename)
Apr 26 15:30:26 labs-ubuntu-20-04 python3[1028846]: FileNotFoundError: [Errno 2] No such file or directory: 'poetry'

This looks like poetry is not being found by the library although it is indeed installed both at the root and user level. Is there a way to solve this? Or is this case out of the scope of the library for now?

opened by MsLimon 2

Releases(v0.1.2)

v0.1.2(Mar 30, 2022)
What's Changed

modified forward_signals to be compatible with windows by @gpfv in https://github.com/pathbird/poetry-kernel/pull/4

New Contributors

@gpfv made their first contribution in https://github.com/pathbird/poetry-kernel/pull/4

Full Changelog: https://github.com/pathbird/poetry-kernel/compare/v0.1.1...v0.1.2
Source code(tar.gz)
Source code(zip)
v0.1.0(Aug 26, 2021)

Source code(tar.gz)
Source code(zip)

Owner

Pathbird

Pathbird is a platform for instructors to build interactive, engaging, inquiry-based lessons for computational courses.

GitHub Repository

Finite difference solution of 2D Poisson equation. Can handle Dirichlet, Neumann and mixed boundary conditions.

Poisson-solver-2D Finite difference solution of 2D Poisson equation Current version can handle Dirichlet, Neumann, and mixed (combination of Dirichlet

34 Dec 23, 2022

Adversarial-autoencoders - Tensorflow implementation of Adversarial Autoencoders

Adversarial Autoencoders (AAE) Tensorflow implementation of Adversarial Autoencoders (ICLR 2016) Similar to variational autoencoder (VAE), AAE imposes

236 Nov 13, 2022

Code release for NeuS

NeuS We present a novel neural surface reconstruction method, called NeuS, for reconstructing objects and scenes with high fidelity from 2D image inpu

813 Jan 04, 2023

A minimalist environment for decision-making in autonomous driving

highway-env A collection of environments for autonomous driving and tactical decision-making tasks An episode of one of the environments available in

1.6k Jan 07, 2023

Summary Explorer is a tool to visually explore the state-of-the-art in text summarization.

Summary Explorer Summary Explorer is a tool to visually inspect the summaries from several state-of-the-art neural summarization models across multipl

42 Aug 14, 2022

Pythonic particle-based (super-droplet) warm-rain/aqueous-chemistry cloud microphysics package with box, parcel & 1D/2D prescribed-flow examples in Python, Julia and Matlab

PySDM PySDM is a package for simulating the dynamics of population of particles. It is intended to serve as a building block for simulation systems mo

32 Oct 18, 2022

Coarse implement of the paper "A Simultaneous Denoising and Dereverberation Framework with Target Decoupling", On DNS-2020 dataset, the DNSMOS of first stage is 3.42 and second stage is 3.47.

SDDNet Coarse implement of the paper "A Simultaneous Denoising and Dereverberation Framework with Target Decoupling", On DNS-2020 dataset, the DNSMOS

43 Nov 21, 2022

Discord bot-CTFD-Thread-Parser - Discord bot CTFD-Thread-Parser

Discord bot CTFD-Thread-Parser Description: This tools is used to create automat

15 Mar 22, 2022

Source code for our Paper "Learning in High-Dimensional Feature Spaces Using ANOVA-Based Matrix-Vector Multiplication"

NFFT4ANOVA Source code for our Paper "Learning in High-Dimensional Feature Spaces Using ANOVA-Based Matrix-Vector Multiplication" This package uses th

1 Aug 10, 2022

NitroFE is a Python feature engineering engine which provides a variety of modules designed to internally save past dependent values for providing continuous calculation.

100 Sep 28, 2022

Annotate with anyone, anywhere.

h h is the web app that serves most of the https://hypothes.is/ website, including the web annotations API at https://hypothes.is/api/. The Hypothesis

2.6k Jan 08, 2023

Implementation of Feedback Transformer in Pytorch

Feedback Transformer - Pytorch Simple implementation of Feedback Transformer in Pytorch. They improve on Transformer-XL by having each token have acce

93 Oct 04, 2022

Official implementation of "Dynamic Anchor Learning for Arbitrary-Oriented Object Detection" (AAAI2021).

DAL This project hosts the official implementation for our AAAI 2021 paper: Dynamic Anchor Learning for Arbitrary-Oriented Object Detection [arxiv] [c

215 Nov 28, 2022

RGBD-Net - This repository contains a pytorch lightning implementation for the 3DV 2021 RGBD-Net paper.

[3DV 2021] We propose a new cascaded architecture for novel view synthesis, called RGBD-Net, which consists of two core components: a hierarchical depth regression network and a depth-aware generator

4 May 26, 2022

Codes for the compilation and visualization examples to the HIF vegetation dataset

High-impedance vegetation fault dataset This repository contains the codes that compile the "Vegetation Conduction Ignition Test Report" data, which a

1 Dec 12, 2021

Building Ellee — A GPT-3 and Computer Vision Powered Talking Robotic Teddy Bear With Human Level Conversation Intelligence

Using an object detection and facial recognition system built on MobileNetSSDV2 and Dlib and running on an NVIDIA Jetson Nano, a GPT-3 model, Google Speech Recognition, Amazon Polly and servo motors,

24 Oct 26, 2022

Official repository for "Orthogonal Projection Loss" (ICCV'21)

Orthogonal Projection Loss (ICCV'21) Kanchana Ranasinghe, Muzammal Naseer, Munawar Hayat, Salman Khan, & Fahad Shahbaz Khan Paper Link | Project Page

83 Dec 26, 2022

NeRF visualization library under construction

NeRF visualization library using PlenOctrees, under construction pip install nerfvis Docs will be at: https://nerfvis.readthedocs.org import nerfvis s

196 Jan 04, 2023

TorchDistiller - a collection of the open source pytorch code for knowledge distillation, especially for the perception tasks, including semantic segmentation, depth estimation, object detection and instance segmentation.

This project is a collection of the open source pytorch code for knowledge distillation, especially for the perception tasks, including semantic segmentation, depth estimation, object detection and i

147 Dec 03, 2022

Plenoxels: Radiance Fields without Neural Networks

Plenoxels: Radiance Fields without Neural Networks Alex Yu*, Sara Fridovich-Keil*, Matthew Tancik, Qinhong Chen, Benjamin Recht, Angjoo Kanazawa UC Be

81 Dec 25, 2022