Page Not Found

Page not found. Your pixels are in another canvas.

Jupyter notebook markdown generator

Kickstart your next Python project with with Poetry, pre-commit and GitHub Actions

12 minute read

Published: April 03, 2020

Kickstart your next Python project with Poetry, pre-commit, and GitHub Actions

When setting up a new Python project, you have probably experienced that it takes quite some time and effort to do so. You have to make a lot of decisions, and when you are not completely sure how to do it, it takes time and mental energy away from actually working on the project.

Wouldn’t it be great if you could do it right once and then profit from it in all the following projects? This is exactly what you are going to do in this tutorial. You will set up a reasonable Python template project suitable for most tasks.

These set up steps include:

Creating a sensible project structure: In this great article, Kenneth Reitz explains why a good project structure is so important and gives his recommendation for a sample repository.
Managing Python package dependencies: Most Python projects require additional packages to fulfill their purpose. Managing these dependencies is an essential task for every Python project.
Setting up code styling rules: Having consistent code styling rules improves the code’s readability and thus let’s other persons join a project easier. In the best case, the rules are automatically enforced, so you do not have to think about them once they are set up.
Setting up Continuous Integration (CI) to automatically test the code: With CI, you want to make sure that mistakes are discovered early and quickly. Using a CI tool that automatically runs the tests after every push to the repo also makes this step a piece of mind.

This list of steps is by no means inclusive, but a sensible starting point for most Python projects. In the next few paragraphs, you will go over each of these steps to create a Python template repo that gets your next projects up and running in minutes. For each step, you will make use of great open-source projects that help to achieve the goal.

Creating a Project Structure and Manage Dependencies with Poetry

As a first step, you create a default project structure and set up dependency management. Poetry is a tool for exactly that created by Sébastien Eustace. With Poetry, you can create deterministic builds, package your project, and publish it to PyPI using only a handful of easy commands. Besides its own nice documentation, there is also this great introduction to Poetry if you want to learn more about it.

Simply follow these steps to set up a default project structure and add first dependencies:

1. Create a new project with a reasonable default project structure using Poetry

Install Poetry by following the install instructions for your OS on their website. Then you can run the following command to create a default project structure:

poetry new python-template-repo

This creates the following directory structure:

python-template-repo
├── pyproject.toml
├── python_template_repo
│   └── __init__.py
├── README.md
└── tests
    └── __init__.py

The most important file here at the moment is pyproject.toml that contains general information about the project and its dependencies and is used by Poetry to manage the project:

[tool.poetry]
name = "python-template-repo"
version = "0.1.0"
description = ""
authors = ["Christoph Clement <christoph.clement@students.unibe.ch>"]
readme = "README.md"
packages = [{include = "python_template_repo"}]

[tool.poetry.dependencies]
python = "^3.9"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

For now, the only dependencies we have is Python 3.9. The [build-system] entry makes it compliant with PEP-517. You can find more information about what can be specified in the file in Poetry’s documentation here.

2. Install the first dependency

cd python-template-repo
poetry add fire

Adding a dependency to the project works with the poetry add command and the required package’s name. You can find more information about the command and how to specify package versions here. In this example, you add Python Fire, a Python library for automatically generating command line interfaces (CLIs) by Google.

After adding a new dependency, Poetry adds (if it is the first dependency) or updates a file called poetry.lock containing the exact versions of the downloaded packages. These are used when someone or something else, e.g., a colleague or a CI server, installs the dependencies using the poetry install command. This ensures that the project does not break because of different versions of dependencies.

Automatically Enforcing Code Formatting Rules with pre-commit

When writing new code, one of the most important things to keep in mind is the following:

“[…] code is read much more often than it is written.” - PEP 8

Apart from structuring your code well, a consistent style highly contributes to readable and understandable code. Apart from PEP 8 itself, which is a great read, I can recommend going over this great article by The Hitchhiker’s Guide to Python about “Pythonic” guidelines and idioms.

It helps a lot to have an understanding of these rules and guidelines. But actually, when you are working on a project, you do not want to waste mental energy by thinking about how to best format the code. Exactly for this reason, there are some helpful tools out there that do this job for you.

A great way to incorporate these tools into a project is to use Git Hooks to check files automatically before committing them. Here, you will use pre-commit, a framework for managing and maintaining pre-commit hooks.

With the following steps, you can quickly set up several pre-commit checks to enforce a consistent code style throughout the whole project.

1. Install and add pre-commit to the dev dependencies

poetry add pre-commit --group dev

2. Configure the hooks that we want to use

Create a file called .pre-commit-config.yaml in the root dir of the project. There, the hooks are configured. The following is the default configuration that I use for projects. It makes use of hooks that come directly with pre-commit, like checking the format YAML files or whether large files are added to Git. Additionally, black is used to format the code automatically.

# See https://pre-commit.com for more information
# See https://pre-commit.com/hooks.html for more hooks
repos:
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v3.2.0
    hooks:
    -   id: check-added-large-files
    -   id: check-ast
    -   id: check-merge-conflict
    -   id: check-yaml
    -   id: detect-private-key
    -   id: end-of-file-fixer
    -   id: trailing-whitespace

-   repo: https://github.com/asottile/reorder_python_imports
    rev: v3.9.0
    hooks:
    -   id: reorder-python-imports

-   repo: https://github.com/psf/black
    rev: 22.10.0
    hooks:
      - id: black
        language_version: python3.9

3. Install the hooks for the project

git init
poetry run pre-commit install

Now, you can manually run these hooks whenever you want your code to be checked and formatted by running poetry run pre-commit run --all-files. Also, every time you want to commit, these checks are run, and if there is a failure, you cannot commit unless you fix it. This is sometimes a little annoying when you quickly want to commit something, but it ensures high coding quality in the long run.

Setting up Continuous Integration (CI) to automatically test code with GitHub Actions

The final step is to create a CI pipeline with GitHub Actions, which is free for public projects and students. With this pipeline, you will check whether all pre-commit hooks pass without an error and run tests with pytest.

The setup is straight forward. Simply add the following ci-testing.yml file to the .github/workflows/ directory of your project.

name: CI Testing

on: [push, pull_request]

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v3
    - name: Set up Python 3.9
      uses: actions/setup-python@v4
      with:
        python-version: 3.9
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install poetry
        poetry install
        poetry run pre-commit install
    - name: Run pre-commit hooks
      run: |
        poetry run pre-commit run --all-files
    - name: Test with pytest
      run: |
        poetry run pytest

This file defines the action CI Testing that gets triggered on new pushes and pull requests. It sets up an Ubuntu server with Python 3.9, installs the dependencies, runs the pre-commit hooks and the tests with pytest. For a more detailed look at how to set up actions for a Python project, check out the GitHub help pages for that topic.

Start Coding

Now that you have set up a sensible project structure, dependency management, automatic formatting checks, and CI, it is time to start with the actual work.

To demonstrate the pipeline, create a small demo script with following these steps:

1. Add necessary dependencies

You have already installed pre-commit in the previous steps as a dev dependency. For the sake of this demonstration, additionally add tqdm to create progress bars, and loguru for easy logging.

poetry add tqdm loguru

2. Create a template script

In the next step, create a simple template script in the python_template_repo/ directory:

from time import sleep

import fire
from loguru import logger
from tqdm import tqdm


def main(
    len_loop: int = 100,
    sleep_seconds: float = 0.1,
):
    logger.info("Starting loop")

    for _ in tqdm(range(len_loop)):
        sleep(sleep_seconds)

    logger.info("Finished loop")


if __name__ == "__main__":
    fire.Fire(main)

3. Run the pre-commit hooks

After creating the script, check whether it passes the pre-commit hooks. Try to commit your work by running the following command:

git add .
git commit -m "Initial commit"

You should get the following output from pre-commit:

Check for added large files..............................................Passed
Check python ast.........................................................Passed
Check for merge conflicts................................................Passed
Check Yaml...............................................................Passed
Detect Private Key.......................................................Passed
Fix End of Files.........................................................Failed
- hook id: end-of-file-fixer
- exit code: 1
- files were modified by this hook

Fixing .pre-commit-config.yaml

Trim Trailing Whitespace.................................................Passed
Reorder python imports...................................................Passed
black....................................................................Passed

One check from pre-commit failed and it fixed the end-of-file for us. When staging the changed files and committing again, all hooks pass, and you are good to go:

git add .
git commit -m "Initial commit"

4. Run the script

Use the following command to run the script inside the project’s environment using Poetry:

poetry run python python_template_repo/main.py

To shorten the command, you can add the script to the pyproject.toml file:

...

[tool.poetry.scripts]
main = "python_template_repo.main:main"

[build-system]
...

Then, you can run the script like so:

poetry run main

5. Add test

First, let’s add pytest to our test dependencies:

poetry add pytest --group test

Then, let’s add a simple test that checks whether the template script runs without errors. Create a file called test_main.py in the tests directory:

from python_template_repo.main import main


def test_main():
    main()

You can run the test locally with:

poetry run pytest

To see how the CI testing works, you first need to create a new GitHub repository. After you have done that, you can then push your work so far by running:

git remote add origin git@github.com:[Your GitHub Username]/python-template-repo.git
git add .
git commit -m "Add test for main"
git push -u origin master

You can then head over to your newly created GitHub repo and check whether the CI tests passed under “Actions”.

Conclusion

With these steps, you are well prepared for your next Python projects. You are ready to use Poetry for managing dependencies, pre-commit to automatically check your code style, and GitHub Actions to automatically test your code.

Let me know how your Python project-setup steps look like!

Bonus: Cookiecutter Template

I provide this repo as a Cookiecutter template for the case you want to use exactly this Python project setup. Simply follow these steps to set up your project in minutes:

1. Install Cookiecutter

pip install cookiecutter

2. Clone my Cookiecutter template repo

git clone https://github.com/chris-clem/python-template-repo.git

3. Run Cookiecutter to create a new project from the template

cookiecutter python template-repo

You will be asked how you want to name the repo, the package, and the script. Simply fill in these variables, and you get a customized version of the template repo.

The template also comes with a README containing setup instructions.

Portfolio item number 1

Short description of portfolio item number 1

Deep Learning for Predicting Gamma-Ray Interaction Positions in LYSO Detector

Published in 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), 2021

Download paper here

Recommended citation: C. Clement et al., "Deep Learning for Predicting Gamma-Ray Interaction Positions in LYSO Detector," 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), 2021, pp. 3366-3369, doi: 10.1109/EMBC46164.2021.9630934. http://dx.doi.org/10.1109/embc46164.2021.9630934

Concept development of an on-chip PET system

Published in EJNMMI Physics, 2022