Julia Community 🟣

Wen Wei Tseng
Wen Wei Tseng

Posted on • Edited on • Originally published at sosiristseng.github.io

Publishing Julia Notebooks with Docker and GitHub Actions (I: Dockerfile)

#julia #gha #jupyter #docker

This series will

  • Use Docker to build a Julia runtime environment for continuous integration (CI).
  • Use GitHub actions and the Docker container to execute notebooks in parallel.
  • Use GitHub actions and jupyter-book to publish notebooks automatically upon git push.

The one-click-to-copy repository is on: https://github.com/sosiristseng/template-juliabook-docker

I will try to explain major steps in my Dockerfile to build a runtime environment for Julia code running in Jupyter notebooks.

FROM python:3.11.2-slim

# Julia
ENV JULIA_CI true
ENV JULIA_NUM_THREADS "auto"
ENV JULIA_PATH /usr/local/julia/
ENV JULIA_DEPOT_PATH /srv/juliapkg/
ENV PATH ${JULIA_PATH}/bin:${PATH}
COPY --from=julia:1.8.5 ${JULIA_PATH} ${JULIA_PATH}

# Python dependencies. e.g. matplotlib
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt && pip install --no-cache-dir nbconvert

# Julia environment
COPY Project.toml Manifest.toml ./
COPY src/ src
RUN julia --color=yes --project="" -e 'import Pkg; Pkg.add("IJulia"); using IJulia; installkernel("Julia", "--project=@.")' && \
    julia --color=yes --project=@. -e 'import Pkg; Pkg.instantiate(); Pkg.resolve(); Pkg.precompile()'

CMD ["julia"]
Enter fullscreen mode Exit fullscreen mode

Base docker image

Usually, Julia projects uses julia as the base image; however, we need a python package nbconvert to render Jupyter notebooks. Thus, we use python as our base image, which includes pip to install required Python packages.

FROM python:3.11.2-slim
Enter fullscreen mode Exit fullscreen mode

We can copy the julia executable from the official julia docker container.

ENV JULIA_CI true
ENV JULIA_NUM_THREADS "auto"
ENV JULIA_PATH /usr/local/julia/
ENV JULIA_DEPOT_PATH /srv/juliapkg/
ENV PATH ${JULIA_PATH}/bin:${PATH}
COPY --from=julia:1.8.5 ${JULIA_PATH} ${JULIA_PATH}
Enter fullscreen mode Exit fullscreen mode

System packages (optional)

One might need some system dependencies, installed by apt-get.

RUN apt-get update && apt-get install -y <pkgs> && rm -rf /var/lib/apt/lists/*
Enter fullscreen mode Exit fullscreen mode

For example,

  • gnuplot is required by Gnuplot.jl or Gaston.jl.
  • parallel can run multiple notebooks in parallel in multi-core machines (e.g. cirrus CI provides 8-core CI machines for free)
  • clang or gcc if you want to use PacakgeCompiler.jl to compile a sysimage.

Python dependencies

Install Python dependencies e.g. matplotlib for PyPlot.jl via requirements.txt so that it is easier to maintain and update. You can leave requirements.txt blank if you have no such dependencies.

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt && pip install --no-cache-dir -U nbconvert
Enter fullscreen mode Exit fullscreen mode

requirements.txt

matplotlib==3.5.3
Enter fullscreen mode Exit fullscreen mode

Julia dependencies

IJulia.jl is install globally for the Jupyter kernel. Julia dependencies are defined in Project.toml and Manifest.toml into the container's working directory and Pkg.instantiate() installs the dependencies. Please make sure Manifest.toml is not in .gitignore to ensure Manifest.toml is tracked and the runtime docker image is reproducible.

# Julia environment
COPY Project.toml Manifest.toml ./
COPY src/ src
RUN julia --color=yes --project="" -e 'import Pkg; Pkg.add("IJulia"); using IJulia; installkernel("Julia", "--project=@.")' && \
    julia --color=yes --project=@. -e 'import Pkg; Pkg.instantiate(); Pkg.resolve(); Pkg.precompile()'
Enter fullscreen mode Exit fullscreen mode

(Optiona) Building a sysimage

You can also build a sysimage to reduce package load time. I recommend Satoshi Terasaki's sysimage_creator for details.

You'll need to add a compiler to the docker container by adding this line to Dockerfile

RUN apt-get update && apt-get install -y gcc && rm -rf /var/lib/apt/lists/*
Enter fullscreen mode Exit fullscreen mode

and in the Julia packages part in the Dockerfile, put julia command into script.

COPY Project.toml Manifest.toml build-kernel.jl ./
COPY src/ src
RUN julia --color=yes build-kernel.jl
Enter fullscreen mode Exit fullscreen mode

Content of build-kernel.jl:

# Adapted from https://github.com/terasakisatoshi/sysimage_creator/
import Pkg

Pkg.add(["PackageCompiler", "IJulia"])

using PackageCompiler

sysimage_path = joinpath(@__DIR__, "sysimage.so")

@info "SysImage path: " sysimage_path

PackageCompiler.create_sysimage(
    ["Plots"]; # Packages of your choice
    project=".",
    sysimage_path=sysimage_path,
    cpu_target=PackageCompiler.default_app_cpu_target()
)

using IJulia
IJulia.installkernel("Julia-sys","--project=@.", "--sysimage=$(sysimage_path)")

Pkg.rm("PackageCompiler")
Pkg.gc()
Enter fullscreen mode Exit fullscreen mode

Oldest comments (2)

Collapse
 
rikhuijzer profile image
Rik Huijzer •

Very impressive setup!

Collapse
 
Sloan, the sloth mascot
Comment deleted