Edit on 2023-05-11
This trick might be no longer needed. It seems like Julia 1.9 has resolved this invalidation issue in GitHub actions. Kudos to the Julia development team 🎉.
Notes on 2023-06-27
After trial and error, the most reliable way to avoid precompile cache rejection is to build a docker image with Julia dependencies and set JULIA_CPU_TARGET=generic
.
TL;DR. Running Julia in a container may avoid precompiling for the same dependencies. Example here and the difference can be found in the Actions
tab.
I accidentally found this "hack" when I was preparing the code examples for the class.
Setting up Julia with the regular way julia-actions/setup-julia@v1
and caching the full Julia depot ~/.julia
(or using Julia's cache action ) still needs some precompilation desipite the vary same dependencies. This might be related to this issue.
However, Julia in the docker container seemed to fully reuse the precompiled code and would not precompile for the same dependencies.
For comparison, this is the first run. Caches were not populated.
And this is the second run after caches were populated.
The Code
The GitHub actions CI workflow:
name: CI
on:
workflow_dispatch:
push:
branches: [main]
pull_request:
branches: [main]
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
env:
JULIA_CACHE: '1'
JULIA_NUM_THREADS: 'auto'
jobs:
docker:
runs-on: ubuntu-latest
container:
image: julia:1.8.5
steps:
- name: Checkout repository
uses: actions/checkout@v3
- name: Cache Julia dependencies
uses: actions/cache@v3
if: ${{ env.JULIA_CACHE != '0'}}
with:
path: ~/.julia
key: ${{ runner.os }}-juliacontainer-${{ env.JULIA_CACHE }}-${{ hashFiles('**/Manifest.toml') }}
# restore-keys: |
# ${{ runner.os }}-juliacontainer-${{ env.JULIA_CACHE }}
- name: Install Julia dependencies
env:
JULIA_PROJECT: ${{ github.workspace }}
run: julia --color=yes -e 'import Pkg; Pkg.instantiate(); Pkg.resolve(); Pkg.precompile()'
- name: Julia execution
env:
JULIA_PROJECT: ${{ github.workspace }}
run: julia --color=yes -e 'using DifferentialEquations; println("Hello")'
regular:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v3
- name: Setup Julia
uses: julia-actions/setup-julia@v1
with:
version: '1.8.5'
- name: Cache Julia dependencies
uses: actions/cache@v3
if: ${{ env.JULIA_CACHE != '0'}}
with:
path: ~/.julia
key: ${{ runner.os }}-julia-${{ env.JULIA_CACHE }}-${{ hashFiles('**/Manifest.toml') }}
# restore-keys: |
# ${{ runner.os }}-julia-${{ env.JULIA_CACHE }}
- name: Install Julia dependencies
env:
JULIA_PROJECT: ${{ github.workspace }}
run: julia --color=yes -e 'import Pkg; Pkg.instantiate(); Pkg.resolve(); Pkg.precompile()'
- name: Julia execution
env:
JULIA_PROJECT: ${{ github.workspace }}
run: julia --color=yes -e 'using DifferentialEquations; println("Hello")'
Right now, I'm using this trick to test run and publish my class material. Click here to see it in action.
As a side note, the repository uses Literate.jl
to execute code and generate Jupyter notebooks from Julia scripts, with 2 worker processes to fully utilize both cores of the GitHub runner. This new workflow is more efficient than the previous one in terms of CPU time. The previous workflow built a docker image for the runtime environment. Then a runner was assigned for each notebook to load the environment and execute the code. Finally, a runner gathered and rendered the executed notebooks into a website.
Caveats
-
Manifest.toml
is required here to fully capture Julia dependencies; therefore, this hack may not work for Julia packages, but may work for projects with aManifest.toml
to lock their dependencies. - I only test it on the Linux runner (ubuntu-latest). Other OSes (Mac and Windows) may or may not work.
- Caching the precompiled code might not be a good idea. See this issue. Also, GitHub has a 10 GB size limit for the total size of all caches. Older ones will be deleted when the limit is exceeded.
Top comments (0)