This series will
- Use Docker to build a Julia runtime environment for continuous integration (CI).
- Use GitHub actions and the Docker container to execute notebooks in parallel.
- Use GitHub actions and
jupyter-book
to publish notebooks automatically upongit push
.
The one-click-to-copy repository is on: https://github.com/sosiristseng/template-juliabook-docker
I will try to explain major steps in my Dockerfile to build a runtime environment for Julia code running in Jupyter notebooks.
FROM python:3.11.2-slim
# Julia
ENV JULIA_CI true
ENV JULIA_NUM_THREADS "auto"
ENV JULIA_PATH /usr/local/julia/
ENV JULIA_DEPOT_PATH /srv/juliapkg/
ENV PATH ${JULIA_PATH}/bin:${PATH}
COPY --from=julia:1.8.5 ${JULIA_PATH} ${JULIA_PATH}
# Python dependencies. e.g. matplotlib
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt && pip install --no-cache-dir nbconvert
# Julia environment
COPY Project.toml Manifest.toml ./
COPY src/ src
RUN julia --color=yes --project="" -e 'import Pkg; Pkg.add("IJulia"); using IJulia; installkernel("Julia", "--project=@.")' && \
julia --color=yes --project=@. -e 'import Pkg; Pkg.instantiate(); Pkg.resolve(); Pkg.precompile()'
CMD ["julia"]
Base docker image
Usually, Julia projects uses julia
as the base image; however, we need a python package nbconvert
to render Jupyter notebooks. Thus, we use python
as our base image, which includes pip
to install required Python packages.
FROM python:3.11.2-slim
We can copy the julia executable from the official julia docker container.
ENV JULIA_CI true
ENV JULIA_NUM_THREADS "auto"
ENV JULIA_PATH /usr/local/julia/
ENV JULIA_DEPOT_PATH /srv/juliapkg/
ENV PATH ${JULIA_PATH}/bin:${PATH}
COPY --from=julia:1.8.5 ${JULIA_PATH} ${JULIA_PATH}
System packages (optional)
One might need some system dependencies, installed by apt-get
.
RUN apt-get update && apt-get install -y <pkgs> && rm -rf /var/lib/apt/lists/*
For example,
-
gnuplot
is required byGnuplot.jl
orGaston.jl
. -
parallel
can run multiple notebooks in parallel in multi-core machines (e.g. cirrus CI provides 8-core CI machines for free) -
clang
orgcc
if you want to usePacakgeCompiler.jl
to compile a sysimage.
Python dependencies
Install Python dependencies e.g. matplotlib
for PyPlot.jl
via requirements.txt so that it is easier to maintain and update. You can leave requirements.txt blank if you have no such dependencies.
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt && pip install --no-cache-dir -U nbconvert
requirements.txt
matplotlib==3.5.3
Julia dependencies
IJulia.jl
is install globally for the Jupyter kernel. Julia dependencies are defined in Project.toml
and Manifest.toml
into the container's working directory and Pkg.instantiate()
installs the dependencies. Please make sure Manifest.toml
is not in .gitignore
to ensure Manifest.toml
is tracked and the runtime docker image is reproducible.
# Julia environment
COPY Project.toml Manifest.toml ./
COPY src/ src
RUN julia --color=yes --project="" -e 'import Pkg; Pkg.add("IJulia"); using IJulia; installkernel("Julia", "--project=@.")' && \
julia --color=yes --project=@. -e 'import Pkg; Pkg.instantiate(); Pkg.resolve(); Pkg.precompile()'
(Optiona) Building a sysimage
You can also build a sysimage to reduce package load time. I recommend Satoshi Terasaki's sysimage_creator for details.
You'll need to add a compiler to the docker container by adding this line to Dockerfile
RUN apt-get update && apt-get install -y gcc && rm -rf /var/lib/apt/lists/*
and in the Julia packages part in the Dockerfile, put julia command into script.
COPY Project.toml Manifest.toml build-kernel.jl ./
COPY src/ src
RUN julia --color=yes build-kernel.jl
Content of build-kernel.jl:
# Adapted from https://github.com/terasakisatoshi/sysimage_creator/
import Pkg
Pkg.add(["PackageCompiler", "IJulia"])
using PackageCompiler
sysimage_path = joinpath(@__DIR__, "sysimage.so")
@info "SysImage path: " sysimage_path
PackageCompiler.create_sysimage(
["Plots"]; # Packages of your choice
project=".",
sysimage_path=sysimage_path,
cpu_target=PackageCompiler.default_app_cpu_target()
)
using IJulia
IJulia.installkernel("Julia-sys","--project=@.", "--sysimage=$(sysimage_path)")
Pkg.rm("PackageCompiler")
Pkg.gc()
Top comments (2)
Very impressive setup!