Avoiding precompilation in Julia projects in GitHub actions

Edit on 2023-05-11
This trick might be no longer needed. It seems like Julia 1.9 has resolved this invalidation issue in GitHub actions. Kudos to the Julia development team 🎉.

• First run (uncached)
• Second run (cached)
• Third run (cached)

Notes on 2023-06-27

After trial and error, the most reliable way to avoid precompile cache rejection is to build a docker image with Julia dependencies and set JULIA_CPU_TARGET=generic.

TL;DR. Running Julia in a container may avoid precompiling for the same dependencies. Example here and the difference can be found in the Actions tab.

I accidentally found this "hack" when I was preparing the code examples for the class.

Setting up Julia with the regular way julia-actions/setup-julia@v1 and caching the full Julia depot ~/.julia (or using Julia's cache action ) still needs some precompilation desipite the vary same dependencies. This might be related to this issue.

However, Julia in the docker container seemed to fully reuse the precompiled code and would not precompile for the same dependencies.

For comparison, this is the first run. Caches were not populated.

And this is the second run after caches were populated.

The Code

The GitHub actions CI workflow:

name: CI

on:
  workflow_dispatch:
  push:
    branches: [main]
  pull_request:
    branches: [main]

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

env:
  JULIA_CACHE: '1'
  JULIA_NUM_THREADS: 'auto'

jobs:
  docker:
    runs-on: ubuntu-latest
    container:
      image: julia:1.8.5
    steps:
      - name: Checkout repository
        uses: actions/checkout@v3
      - name: Cache Julia dependencies
        uses: actions/cache@v3
        if: ${{ env.JULIA_CACHE != '0'}}
        with:
          path: ~/.julia
          key: ${{ runner.os }}-juliacontainer-${{ env.JULIA_CACHE }}-${{ hashFiles('**/Manifest.toml') }}
          # restore-keys: |
          #   ${{ runner.os }}-juliacontainer-${{ env.JULIA_CACHE }}
      - name: Install Julia dependencies
        env:
          JULIA_PROJECT: ${{ github.workspace }}
        run: julia --color=yes -e 'import Pkg; Pkg.instantiate(); Pkg.resolve(); Pkg.precompile()'
      - name: Julia execution
        env:
          JULIA_PROJECT: ${{ github.workspace }}
        run: julia --color=yes -e 'using DifferentialEquations; println("Hello")'
  regular:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v3
      - name: Setup Julia
        uses: julia-actions/setup-julia@v1
        with:
          version: '1.8.5'
      - name: Cache Julia dependencies
        uses: actions/cache@v3
        if: ${{ env.JULIA_CACHE != '0'}}
        with:
          path: ~/.julia
          key: ${{ runner.os }}-julia-${{ env.JULIA_CACHE }}-${{ hashFiles('**/Manifest.toml') }}
          # restore-keys: |
          #   ${{ runner.os }}-julia-${{ env.JULIA_CACHE }}
      - name: Install Julia dependencies
        env:
          JULIA_PROJECT: ${{ github.workspace }}
        run: julia --color=yes -e 'import Pkg; Pkg.instantiate(); Pkg.resolve(); Pkg.precompile()'
      - name: Julia execution
        env:
          JULIA_PROJECT: ${{ github.workspace }}
        run: julia --color=yes -e 'using DifferentialEquations; println("Hello")'

Right now, I'm using this trick to test run and publish my class material. Click here to see it in action.

As a side note, the repository uses Literate.jl to execute code and generate Jupyter notebooks from Julia scripts, with 2 worker processes to fully utilize both cores of the GitHub runner. This new workflow is more efficient than the previous one in terms of CPU time. The previous workflow built a docker image for the runtime environment. Then a runner was assigned for each notebook to load the environment and execute the code. Finally, a runner gathered and rendered the executed notebooks into a website.

Caveats

• Manifest.toml is required here to fully capture Julia dependencies; therefore, this hack may not work for Julia packages, but may work for projects with a Manifest.toml to lock their dependencies.
• I only test it on the Linux runner (ubuntu-latest). Other OSes (Mac and Windows) may or may not work.
• Caching the precompiled code might not be a good idea. See this issue. Also, GitHub has a 10 GB size limit for the total size of all caches. Older ones will be deleted when the limit is exceeded.