Introduction
I am a large fan of tutorials to provide users an easy start for example when using a new Julia package. For best of cases, a tutorial contains code that can not only be run by the user, but that is also
- run in the example to illustrate which result to expect
- included in the documentation and matches in style
- consistent βΒ that is does not suddenly break
These points brought me to quarto, since it is easy to write, in markdown + Julia Code, getβs rendered into Markdown, which can be stored in the documentation folder easily to match the general style. One can even reference other functions in the documentation using just @ref
from Documenter.
My favourite feature of Quarto is that, since it is just Julia Code evaluated, it allowed to use local environments for consistency. For a package, I prefer to change one small thing from pure consistency, that is, load always the master branch of the package the documentation / tutorial is for. This way, one can easily check, that also the tutorials are not breaking.
Since this workflow means, that some markdown files of the documentation, it would be great to have the examples automatically run on GitHub CI when generating the documentation.
Of course, since some examples might take a bit to run, caching them would also be great. Luckily, Quarto already has a _freeze
mechanism.
Structure of this tutorial
After a short explanation of the setup we use in Manopt.jl
, this tutorial explains how to set up you Documentation CI to also include Quarto examples to be run, if they have been changed.
Setup
In the Manopt.jl
structure, the tutorials share a common environment in tutorials/
, with their own Project.toml
.
Each of the tutorials should be developed locally and tested with quarto render
(or quarto preview
).
This workflow was developed with Julia 1.9 and Quarto 1.3
Handling Python Dependencies
A main challenge of rendering quarto markdown files is, that this is done with Jupyter and hence requires python as a dependency.
So there is a necessity to handle python dependencies when/before running quarto. Since this happens within the documentation, the environment Project.toml for the docs/
folder takes care of this employing CondaPkg.jl, see also the CondaPkg.toml file, which is quite minimal.
Together with IJulia.jl
just calling
CondaPkg.withenv() do
[...]
end
allows to execute command line calls with the specified python packages (here only Jupyter) installed and having a Julia kernel available
Documentation: Modifying the make.jl
In order to make the tutorials only render when specified, the make.jl
is executable and has as a first line
#!/usr/bin/env Julia
This way a normal run of building the documentation, for example running include("make.jl")
on REPL would not trigger running all tutorials, but on terminal doing
./make.jl --quarto
does. The idea is that either quarto render
or the command line call would only be executed from time to time and usually the .md
files from the tutorials are already rendered and available in the documentation source tree.
Besides that, the make.jl
contains a block that, given --quarto
, renders all .qmd
files.
To be safe, this block not only runs in the CondaPkg
environment, but also resolves/instantiates the tutorial folder for once, but also makes sure, the new Jupyter (if not from cache) is aware of the IJulia
kernel by once building that package in the tutorials environment:
CondaPkg.withenv() do
@info "Rendering Quarto"
tutorials_folder = (@__DIR__) * "/../tutorials"
# instantiate the tutorials environment if necessary
Pkg.activate(tutorials_folder)
Pkg.resolve()
Pkg.instantiate()
Pkg.build("IJulia") # build IJulia to the right version.
Pkg.activate(@__DIR__) # but return to the docs one before
run(`quarto render $(tutorials_folder)`)
end
Since the quarto config is set up to store the results in the documentation docs/src
directory, they are rendered when running make.jl
into html. Here, the quarto
command is also run within the environment having the python packages installed, as discussed in the last section.
CI: Modifying the GitHub Action
The second part of this tutorial is concerned with the documenter.yml
workflow, which is modified to also allow caching of the results.
We first install Quarto with the corresponding action
- uses: quarto-dev/quarto-actions/setup@v2
with:
version: 1.3.353
But the main part is, to cache quite a few folders between CI runs. This is done by Hashing the corresponding folder and allowing to load old caches from this folder as well.
We cache
Julia
Which is possible with its own cache
- name: Julia Cache
uses: julia-actions/cache@v1
Quarto
Since quarto stores pre-rendered results in the _freeze
subfolder, we cache this folder. Note that the restored keys allow to also load previous versions, where the current hash does not hit, that way, if some tutorial is changed, the unchanged ones still obtain their old cached versions
- name: Cache Quarto
id: cache-quarto
uses: actions/cache@v3
env:
cache-name: cache-quarto
with:
path: tutorials/_freeze
key: ${{ runner.os }}-${{ env.cache-name }}-${{ hashFiles('tutorials/*.qmd') }}
restore-keys: |
${{ runner.os }}-${{ env.cache-name }}-
Documenter
Since we also want to cache the old .md
files that quarto
stores within the documenter tree, we also cache that folder
- name: Cache Documenter
id: cache-documenter
uses: actions/cache@v3
env:
cache-name: cache-documenter
with:
path: docs/src/tutorials
key: ${{ runner.os }}-${{ env.cache-name }}-${{ hashFiles('tutorials/*.qmd') }}
restore-keys: |
${{ runner.os }}-${{ env.cache-name }}-
CondaPkg
It might be useful to also cache the downloaded artefacts from CondaPkg
- name: Cache CondaPkg
id: cache-condaPkg
uses: actions/cache@v3
env:
cache-name: cache-condapkg
with:
path: docs/.CondaPkg
key: ${{ runner.os }}-${{ env.cache-name }}-${{ hashFiles('docs/CondaPkg.toml') }}
restore-keys: |
${{ runner.os }}-${{ env.cache-name }}-
Finally we modify the call to run the documentation rendering such that it always runs the rendering. This will only include changed tutorials, since the others are cached due to all the caches above
- name: "Documenter rendering (including Quarto)"
run: "docs/make.jl --quarto"
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
DOCUMENTER_KEY: ${{ secrets.DOCUMENTER_KEY }}
Summary
While a bit technical with all the caches, this modification of the documentation build run and the documentation GitHub Action allows to automatically build quarto notebooks within the documentation. This way there is no need to commit the (pre-)rendered markdown files in the documentation, which might have led to them being outdated. Since all the caches avoid running the tutorials on every run, the GitHub CI run is still reasonably fast.
The main files to look at are in summary
- the
tutorials/_quarto.yml
to configure Quarto, mainly to store the result directly in thedocs/src/tutorials/
folder - the
docs/make.jl
file to render the documentation including running the quarto rendering, but especially also the python dependencies. - the
.github/workflows/documenter.yml
especially with a bit of caches.
Open Problems
Two tutorials are currently still rendered locally, see the _quarto.yml
, one since it requires Asymptote for rendering, which is not so easy to get on CI, and the other one since benchmarking might be a bit too slow on CI.
Since the .md
of the tutorials are not committed to the GitHub repository, but only cached/regenerated on CI before running Documenter.jl
, Edit on GitHub
link is broken for these. It would be nice to add a link here to the .qmd
file in the tutorials/
folder instead.
Top comments (1)
Thanks, Ronny, for this amazing demo! Looking forward to adding that workflow to my packages π₯³