Wen Wei Tseng

Posted on Sep 10, 2022 • Updated on Feb 12 • Originally published at sosiristseng.github.io

Publishing Julia Notebooks with Docker and GitHub Actions (I: Dockerfile)

#docker #julia #jupyter #gha

This series will

Use Docker to build a Julia runtime environment for continuous integration (CI).
Use GitHub actions and the Docker container to execute notebooks in parallel.
Use GitHub actions and jupyter-book to publish notebooks automatically upon git push.

The one-click-to-copy repository is on: https://github.com/sosiristseng/template-juliabook-docker

I will try to explain major steps in my Dockerfile to build a runtime environment for Julia code running in Jupyter notebooks.

FROM python:3.11.2-slim

# Julia
ENV JULIA_CI true
ENV JULIA_NUM_THREADS "auto"
ENV JULIA_PATH /usr/local/julia/
ENV JULIA_DEPOT_PATH /srv/juliapkg/
ENV PATH ${JULIA_PATH}/bin:${PATH}
COPY --from=julia:1.8.5 ${JULIA_PATH} ${JULIA_PATH}

# Python dependencies. e.g. matplotlib
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt && pip install --no-cache-dir nbconvert

# Julia environment
COPY Project.toml Manifest.toml ./
COPY src/ src
RUN julia --color=yes --project="" -e 'import Pkg; Pkg.add("IJulia"); using IJulia; installkernel("Julia", "--project=@.")' && \
    julia --color=yes --project=@. -e 'import Pkg; Pkg.instantiate(); Pkg.resolve(); Pkg.precompile()'

CMD ["julia"]

Base docker image

Usually, Julia projects uses julia as the base image; however, we need a python package nbconvert to render Jupyter notebooks. Thus, we use python as our base image, which includes pip to install required Python packages.

FROM python:3.11.2-slim

We can copy the julia executable from the official julia docker container.

ENV JULIA_CI true
ENV JULIA_NUM_THREADS "auto"
ENV JULIA_PATH /usr/local/julia/
ENV JULIA_DEPOT_PATH /srv/juliapkg/
ENV PATH ${JULIA_PATH}/bin:${PATH}
COPY --from=julia:1.8.5 ${JULIA_PATH} ${JULIA_PATH}

System packages (optional)

One might need some system dependencies, installed by apt-get.

RUN apt-get update && apt-get install -y <pkgs> && rm -rf /var/lib/apt/lists/*

For example,

gnuplot is required by Gnuplot.jl or Gaston.jl.
parallel can run multiple notebooks in parallel in multi-core machines (e.g. cirrus CI provides 8-core CI machines for free)
clang or gcc if you want to use PacakgeCompiler.jl to compile a sysimage.

Python dependencies

Install Python dependencies e.g. matplotlib for PyPlot.jl via requirements.txt so that it is easier to maintain and update. You can leave requirements.txt blank if you have no such dependencies.

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt && pip install --no-cache-dir -U nbconvert

requirements.txt

matplotlib==3.5.3

Julia dependencies

IJulia.jl is install globally for the Jupyter kernel. Julia dependencies are defined in Project.toml and Manifest.toml into the container's working directory and Pkg.instantiate() installs the dependencies. Please make sure Manifest.toml is not in .gitignore to ensure Manifest.toml is tracked and the runtime docker image is reproducible.

# Julia environment
COPY Project.toml Manifest.toml ./
COPY src/ src
RUN julia --color=yes --project="" -e 'import Pkg; Pkg.add("IJulia"); using IJulia; installkernel("Julia", "--project=@.")' && \
    julia --color=yes --project=@. -e 'import Pkg; Pkg.instantiate(); Pkg.resolve(); Pkg.precompile()'

(Optiona) Building a sysimage

You can also build a sysimage to reduce package load time. I recommend Satoshi Terasaki's sysimage_creator for details.

You'll need to add a compiler to the docker container by adding this line to Dockerfile

RUN apt-get update && apt-get install -y gcc && rm -rf /var/lib/apt/lists/*

and in the Julia packages part in the Dockerfile, put julia command into script.

COPY Project.toml Manifest.toml build-kernel.jl ./
COPY src/ src
RUN julia --color=yes build-kernel.jl

Content of build-kernel.jl:

# Adapted from https://github.com/terasakisatoshi/sysimage_creator/
import Pkg

Pkg.add(["PackageCompiler", "IJulia"])

using PackageCompiler

sysimage_path = joinpath(@__DIR__, "sysimage.so")

@info "SysImage path: " sysimage_path

PackageCompiler.create_sysimage(
    ["Plots"]; # Packages of your choice
    project=".",
    sysimage_path=sysimage_path,
    cpu_target=PackageCompiler.default_app_cpu_target()
)

using IJulia
IJulia.installkernel("Julia-sys","--project=@.", "--sysimage=$(sysimage_path)")

Pkg.rm("PackageCompiler")
Pkg.gc()