Julia Community 🟣: Wen Wei Tseng

Avoiding precompilation in Julia projects in GitHub actions

Wen Wei Tseng — Wed, 29 Mar 2023 16:32:35 +0000

Edit on 2023-05-11
This trick might be no longer needed. It seems like Julia 1.9 has resolved this invalidation issue in GitHub actions. Kudos to the Julia development team 🎉.

Notes on 2023-06-27

After trial and error, the most reliable way to avoid precompile cache rejection is to build a docker image with Julia dependencies and set JULIA_CPU_TARGET=generic.

TL;DR. Running Julia in a container may avoid precompiling for the same dependencies. Example here and the difference can be found in the Actions tab.

I accidentally found this "hack" when I was preparing the code examples for the class.

Setting up Julia with the regular way julia-actions/setup-julia@v1 and caching the full Julia depot ~/.julia (or using Julia's cache action ) still needs some precompilation desipite the vary same dependencies. This might be related to this issue.

However, Julia in the docker container seemed to fully reuse the precompiled code and would not precompile for the same dependencies.

For comparison, this is the first run. Caches were not populated.

And this is the second run after caches were populated.

The Code

The GitHub actions CI workflow:

name: CI

on:
  workflow_dispatch:
  push:
    branches: [main]
  pull_request:
    branches: [main]

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

env:
  JULIA_CACHE: '1'
  JULIA_NUM_THREADS: 'auto'

jobs:
  docker:
    runs-on: ubuntu-latest
    container:
      image: julia:1.8.5
    steps:
      - name: Checkout repository
        uses: actions/checkout@v3
      - name: Cache Julia dependencies
        uses: actions/cache@v3
        if: ${{ env.JULIA_CACHE != '0'}}
        with:
          path: ~/.julia
          key: ${{ runner.os }}-juliacontainer-${{ env.JULIA_CACHE }}-${{ hashFiles('**/Manifest.toml') }}
          # restore-keys: |
          #   ${{ runner.os }}-juliacontainer-${{ env.JULIA_CACHE }}
      - name: Install Julia dependencies
        env:
          JULIA_PROJECT: ${{ github.workspace }}
        run: julia --color=yes -e 'import Pkg; Pkg.instantiate(); Pkg.resolve(); Pkg.precompile()'
      - name: Julia execution
        env:
          JULIA_PROJECT: ${{ github.workspace }}
        run: julia --color=yes -e 'using DifferentialEquations; println("Hello")'
  regular:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v3
      - name: Setup Julia
        uses: julia-actions/setup-julia@v1
        with:
          version: '1.8.5'
      - name: Cache Julia dependencies
        uses: actions/cache@v3
        if: ${{ env.JULIA_CACHE != '0'}}
        with:
          path: ~/.julia
          key: ${{ runner.os }}-julia-${{ env.JULIA_CACHE }}-${{ hashFiles('**/Manifest.toml') }}
          # restore-keys: |
          #   ${{ runner.os }}-julia-${{ env.JULIA_CACHE }}
      - name: Install Julia dependencies
        env:
          JULIA_PROJECT: ${{ github.workspace }}
        run: julia --color=yes -e 'import Pkg; Pkg.instantiate(); Pkg.resolve(); Pkg.precompile()'
      - name: Julia execution
        env:
          JULIA_PROJECT: ${{ github.workspace }}
        run: julia --color=yes -e 'using DifferentialEquations; println("Hello")'

Right now, I'm using this trick to test run and publish my class material. Click here to see it in action.

As a side note, the repository uses Literate.jl to execute code and generate Jupyter notebooks from Julia scripts, with 2 worker processes to fully utilize both cores of the GitHub runner. This new workflow is more efficient than the previous one in terms of CPU time. The previous workflow built a docker image for the runtime environment. Then a runner was assigned for each notebook to load the environment and execute the code. Finally, a runner gathered and rendered the executed notebooks into a website.

Caveats

Manifest.toml is required here to fully capture Julia dependencies; therefore, this hack may not work for Julia packages, but may work for projects with a Manifest.toml to lock their dependencies.
I only test it on the Linux runner (ubuntu-latest). Other OSes (Mac and Windows) may or may not work.
Caching the precompiled code might not be a good idea. See this issue. Also, GitHub has a 10 GB size limit for the total size of all caches. Older ones will be deleted when the limit is exceeded.

Publishing Julia Notebooks with Docker and GitHub Actions (II: GitHub actions)

Wen Wei Tseng — Sun, 12 Feb 2023 16:10:57 +0000

This series will

Use Docker to build a Julia runtime environment for continuous integration (CI).
Use GitHub actions and the Docker container to execute notebooks in parallel.
Use GitHub actions and jupyter-book to publish notebooks automatically upon git push.

Main workflow file: https://github.com/sosiristseng/template-juliabook/blob/main/.github/workflows/ci-matrix.yml

The workflow includes 4 stages

setup: builds and caches the runtime docker container
execute: executes notebooks in parallel
jupyter-book: renders executed notebooks
Deployment: pushes rendered webpages to GitHub pages.

Build runtime Docker container

The Docker container for the runtime is built by setup-buildx-action and build-push-action.

jobs:
  setup:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.set-matrix.outputs.matrix }}
    steps:
      - name: Checkout repository
        uses: actions/checkout@v3
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
      - name: Build and cache Docker container
        uses: docker/build-push-action@v4
        with:
          context: .
          file: ${{ env.DFILE }}
          tags: ${{ env.IMAGE_NAME }}
          outputs: type=cacheonly
          cache-from: type=gha
          cache-to: type=gha,mode=max
      - name: List notebooks as a JSON array
        id: set-matrix
        run: echo "matrix=$(python -c 'import glob, json; print(json.dumps(glob.glob("docs/*.ipynb")))')" >> $GITHUB_OUTPUT

The setup-buildx-action uses buildx for more docker builder abilities. (e.g. image layer caching)
The build-push-action builds the docker image from our .github/Dockerfile, with GitHub actions caching.
We also list all the jupyter notebooks (*.ipynb) in the docs folder as a JSON array for the next step.

Execute notebooks

To decrease build time, I use a job matrix to execute notebooks in parallel. This stage uses the Docker container from the previous stage and execute notebooks from the output of the previous step. Finished notebooks are then uploaded as artifacts. The concurrency limit is 20 for GitHub free personal and organization accounts. That is, you can run up to 20 notebooks simultaneously.

  execute:
    needs: setup
    strategy:
      max-parallel: 20
      fail-fast: false
      matrix:
        notebook: ${{ fromJSON(needs.setup.outputs.matrix) }} # Notebooks need to be executed
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v3
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
      - name: Restore Docker container
        uses: docker/build-push-action@v4
        with:
          context: .
          file: ${{ env.DFILE }}
          load: true
          tags: ${{ env.IMAGE_NAME }}
          cache-from: type=gha
      - name: Execute Notebook
        run: >
          docker run --rm
          --workdir=/tmp -v ${{ github.workspace }}:/tmp
          ${{ env.IMAGE_NAME }}
          jupyter nbconvert --to notebook --execute --inplace
          --ExecutePreprocessor.timeout=${{ env.TIMEOUT }}
          --ExecutePreprocessor.kernel_name=julia-$(julia -e 'print(VERSION.major,".",VERSION.minor)')
          ${{ matrix.notebook }}
      - name: Upload Notebook
        uses: actions/upload-artifact@v3
        with:
          name: notebook
          path: ${{ matrix.notebook }}
          retention-days: 1

You can see the parallel matrix in action : https://github.com/ww-jl/dataframes/actions/runs/4131921732
(The repo is my modification on DataFrames.jl examples by Bogumił Kamiński)

Render the website using jupyter-book

jupyter-book is a static site generator (SSG) and builds publication-quality books and websites from Markdown documents(*.md) and Jupyter notebooks (*.ipynb).

Here, we collect executed notebooks from previous jobs using actions/download-artifact, use jupyter-book to render them into a website, and upload them as a website artifact.

  jupyter-book:
    needs: execute
    runs-on: ubuntu-latest
    # store success output flag for the ci job
    outputs:
      success: ${{ steps.setoutput.outputs.success }}
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Setup Python
        uses: actions/setup-python@v4
        id: python
        with:
          python-version: '3.x'
      - name: Install Jupyter Book
        run: pip install jupyter-book
      - name: Download notebooks
        uses: actions/download-artifact@v3
        with:
          name: notebook
          path: out/
      - name: Display structure of downloaded files
        run: ls -R
        working-directory: out
      - name: Move notebooks
        run: find out/ -type f -iname '*.ipynb' -exec mv -t docs/ {} +
      - name: Build website
        run: jupyter-book build docs/
      - name: Upload pages artifact
        if: ${{ github.ref == 'refs/heads/main' }}
        uses: actions/upload-pages-artifact@v1
        with:
          path: docs/_build/html
      - name: Set output flag
        id: setoutput
        run: echo "success=true" >> $GITHUB_OUTPUT

Deploy to GH pages

Finally, we deploy the rendered files to GitHub pages.

  # Deployment job
  deploy:
    needs: jupyter-book
    if: if{{ github.ref == 'refs/heads/main' }}
    # Grant GITHUB_TOKEN the permissions required to make a Pages deployment
    permissions:
      pages: write # to deploy to Pages
      id-token: write # to verify the deployment originates from an appropriate source
    environment:
      name: github-pages
      url: ${{ steps.deployment.outputs.page_url }}
    runs-on: ubuntu-latest
    steps:
      - name: Deploy

Caveat: CI status check

GitHub status check treats skipped workflows as passed. Thus, even if any of the notebooks went wrong, the jupyter-book step will be skipped and the overall status check will still pass, which is not ideal for CI depending on status checks. The blog post by Bruno Scheufler provides a workaround for this issue.

  # GitHub status check
  # https://brunoscheufler.com/blog/2022-04-09-the-required-github-status-check-that-wasnt
  ci:
    needs: jupyter-book
    runs-on: ubuntu-latest
    if: always() # always run, so we never skip the check
    steps:
      # pass step only when output of previous jupyter-book job is set
      # in case at least one of the execution fails, jupyter-book is skipped
      # and the output will not be set, which will then cause the ci job to fail
      - run: |
          passed="${{ needs.jupyter-book.outputs.success }}"
          if [[ $passed == "true" ]]; then
            echo "Tests passed"
            exit 0
          else
            echo "Tests failed"
            exit 1
          fi

Publishing Julia Notebooks with Docker and GitHub Actions (I: Dockerfile)

Wen Wei Tseng — Sat, 10 Sep 2022 09:11:29 +0000

This series will

Use Docker to build a Julia runtime environment for continuous integration (CI).
Use GitHub actions and the Docker container to execute notebooks in parallel.
Use GitHub actions and jupyter-book to publish notebooks automatically upon git push.

The one-click-to-copy repository is on: https://github.com/sosiristseng/template-juliabook-docker

I will try to explain major steps in my Dockerfile to build a runtime environment for Julia code running in Jupyter notebooks.

FROM python:3.11.2-slim

# Julia
ENV JULIA_CI true
ENV JULIA_NUM_THREADS "auto"
ENV JULIA_PATH /usr/local/julia/
ENV JULIA_DEPOT_PATH /srv/juliapkg/
ENV PATH ${JULIA_PATH}/bin:${PATH}
COPY --from=julia:1.8.5 ${JULIA_PATH} ${JULIA_PATH}

# Python dependencies. e.g. matplotlib
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt && pip install --no-cache-dir nbconvert

# Julia environment
COPY Project.toml Manifest.toml ./
COPY src/ src
RUN julia --color=yes --project="" -e 'import Pkg; Pkg.add("IJulia"); using IJulia; installkernel("Julia", "--project=@.")' && \
    julia --color=yes --project=@. -e 'import Pkg; Pkg.instantiate(); Pkg.resolve(); Pkg.precompile()'

CMD ["julia"]

Base docker image

Usually, Julia projects uses julia as the base image; however, we need a python package nbconvert to render Jupyter notebooks. Thus, we use python as our base image, which includes pip to install required Python packages.

FROM python:3.11.2-slim

We can copy the julia executable from the official julia docker container.

ENV JULIA_CI true
ENV JULIA_NUM_THREADS "auto"
ENV JULIA_PATH /usr/local/julia/
ENV JULIA_DEPOT_PATH /srv/juliapkg/
ENV PATH ${JULIA_PATH}/bin:${PATH}
COPY --from=julia:1.8.5 ${JULIA_PATH} ${JULIA_PATH}

System packages (optional)

One might need some system dependencies, installed by apt-get.

RUN apt-get update && apt-get install -y <pkgs> && rm -rf /var/lib/apt/lists/*

For example,

gnuplot is required by Gnuplot.jl or Gaston.jl.
parallel can run multiple notebooks in parallel in multi-core machines (e.g. cirrus CI provides 8-core CI machines for free)
clang or gcc if you want to use PacakgeCompiler.jl to compile a sysimage.

Python dependencies

Install Python dependencies e.g. matplotlib for PyPlot.jl via requirements.txt so that it is easier to maintain and update. You can leave requirements.txt blank if you have no such dependencies.

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt && pip install --no-cache-dir -U nbconvert

requirements.txt

matplotlib==3.5.3

Julia dependencies

IJulia.jl is install globally for the Jupyter kernel. Julia dependencies are defined in Project.toml and Manifest.toml into the container's working directory and Pkg.instantiate() installs the dependencies. Please make sure Manifest.toml is not in .gitignore to ensure Manifest.toml is tracked and the runtime docker image is reproducible.

# Julia environment
COPY Project.toml Manifest.toml ./
COPY src/ src
RUN julia --color=yes --project="" -e 'import Pkg; Pkg.add("IJulia"); using IJulia; installkernel("Julia", "--project=@.")' && \
    julia --color=yes --project=@. -e 'import Pkg; Pkg.instantiate(); Pkg.resolve(); Pkg.precompile()'

(Optiona) Building a sysimage

You can also build a sysimage to reduce package load time. I recommend Satoshi Terasaki's sysimage_creator for details.

You'll need to add a compiler to the docker container by adding this line to Dockerfile

RUN apt-get update && apt-get install -y gcc && rm -rf /var/lib/apt/lists/*

and in the Julia packages part in the Dockerfile, put julia command into script.

COPY Project.toml Manifest.toml build-kernel.jl ./
COPY src/ src
RUN julia --color=yes build-kernel.jl

Content of build-kernel.jl:

# Adapted from https://github.com/terasakisatoshi/sysimage_creator/
import Pkg

Pkg.add(["PackageCompiler", "IJulia"])

using PackageCompiler

sysimage_path = joinpath(@__DIR__, "sysimage.so")

@info "SysImage path: " sysimage_path

PackageCompiler.create_sysimage(
    ["Plots"]; # Packages of your choice
    project=".",
    sysimage_path=sysimage_path,
    cpu_target=PackageCompiler.default_app_cpu_target()
)

using IJulia
IJulia.installkernel("Julia-sys","--project=@.", "--sysimage=$(sysimage_path)")

Pkg.rm("PackageCompiler")
Pkg.gc()