Julia Community 🟣

Patrick Altmeyer
Patrick Altmeyer

Posted on • Originally published at Medium on

A year of using Quarto with Julia

Earlier this year in July, I gave a short Experience Talk at JuliaCon. In a related blog post I explained how the introduction of Quarto made my transition from R to Julia painless: I would be able to start learning Julia without having to give up on all the benefits associated with R Markdown.

In November, 2022, I am presenting on this topic again at the 2nd JuliaLang Eindhoven meetup. In addition to the slides, I thought I’d share a small companion blog post that highlights some useful tips and tricks for anyone interested in using Quarto with Julia.

General things

We will start in this section with a few general recommendations.

Setup

I continue to recommend using VSCode for any work with Quarto and Julia. The Quarto docs explain how to get started by installing the necessary Quarto and IJulia extensions. Since most Julia users will regularly want to update their Julia version, I would additionally recommend to add IJulia.jl to your ~/.julia/config/startup.jl file:

# Setup OhMyREPL, Revise and Term
import Pkg
let
    pkgs = ["Revise", "OhMyREPL", "Term", "IJulia"]
    for pkg in pkgs
        if Base.find_package(pkg) === nothing
            Pkg.add(pkg)
        end
    end
end
Enter fullscreen mode Exit fullscreen mode

Additionally, you only need to remember that …

… if you install a new Julia binary […], you must update the IJulia installation […] by running Pkg.build("IJulia")

Source: IJulia docs

I guess this step can also be automated in ~/.julia/config/startup.jl, but haven't tried that yet.

Using .ipynb vs .qmd

I also continue to recommend working with Quarto notebooks as opposed to Jupyter notebooks (files ending in .qmd and .ipynb, respectively). This is partially just based on preference (from R Markdown I'm used to working with .Rmd files), but there is also a good reason to consider using .qmd, even if you're used to working with Jupyter: the code chunks in your Quarto notebook automatically link to the Julia REPL in VSCode. In other words, you can run code chunks in your notebook and then access any variable that you may have created in the REPL. I find this quite useful, cause it allows me to quickly test code. Perhaps there's a good way to do this with Jupyter notebooks as well, but when I last used them I would always have to insert new code cells to test stuff.

Either way switching between Jupyter and Quarto notebooks is straight-forward: quarto convert notebook.qmd will convert any Quarto notebook into a Jupyter notebook and vice versa. One potential benefit of Jupyter notebooks is their connection to Google Colab: it is possible to store Jupyter notebooks on Github and make them available on Colab, allowing users to quickly interact with your code without the need to clone anything. If this is important to you, you can still work with .qmd documents and simply specify keep-ipynb: true in the YAML header.

Dynamic Content

The world and the data that describes it is not static 📈. Why should scientific outputs be?

One of the things I have always really loved about R Markdown was the ability to use inline code: the Knitr engine allows you to call and render any object x that you have created in preceding R chunks like this: r x. This is very powerful, because it enables us to bridge the gap between computations and output. In other words, it allows us to easily produce reproducible and dynamic content.

Until recently I had not been aware that this is also possible for Julia. Consider the following example. The code below depends on remote data that is continuously updated:

using MarketData
snp = yahoo("^GSPC")

using Dates
last_trade_day = timestamp(snp[end])[1]
p_close = values(snp[end,:Close])[1]
last_trade_day_formatted = Dates.format(last_trade_day, "U d, yyyy")
Enter fullscreen mode Exit fullscreen mode

It loads the most recent publicly available data on equity prices from Yahoo finance. In an ideal world, we’d like any updates to these inputs to be reflected in our output. That way you can just re-render the Quarto notebook to get an updated report. To render Julia code inline, we use Markdown.jl like so:

using Markdown
Markdown.parse("""
When the S&P 500 last traded, on $(last_trade_day_formatted), it closed at $(p_close). 
""")
Enter fullscreen mode Exit fullscreen mode

When the S&P 500 last traded, on November 18, 2022, it closed at 3965.340088.

In practice, one would of course set #| echo: false in this case. Whatever content you publish, this approach will keep it up-to-date. This practice of simply re-rendering the source notebook also ensures that any other output remains up-to-date (e.g. Figure 1).

Figure 1: Price history of the S&P 500. Image by author.

Code Execution

Related to the previous point, I typically define the following execution options in my _quarto.yml or _metadata.yml. The freeze: auto option ensures that documents are only rerendered if the source changes. In cases where code should always be re-executed you whould want to set freeze: false, instead. I set output: false because typically I have a lot of code chunks that don't generate any output that is of immediate interest to readers.

Reproducibility

To ensure that your content can be reproduced easily, it may additionally be helpful to explicitly specify the Julia version you used ( jupyter: julia-1.8) and set up a global or local Julia environments. Inserting the following at the beginning of your Quarto notebook

using Pkg; Pkg.activate("<path>")
Enter fullscreen mode Exit fullscreen mode

ensures that the desired environment that lives in is actually activated and used.

Package Documentation

I have also continued to use Quarto in combination with Documenter.jl to document my Julia packages. This essentially boils down to writing up documentation using interactive .qmd notebooks and then rendering those to .md files as inputs for Documenter.jl. There are a few good reasons for this approach, especially if you're used to working with Quarto anyway:

  1. Re-rendering any docs with eval: true provides an additional layer of quality assurance: if any of the code chunks throws an error, you know that your documentation is outdated (perhaps due to an API change). It also offers a straight-forward way to test package functions that produce non-testable (e.g. stochastic) output. In such cases, the use of jldoctest is not always straight-forward (see here).
  2. You get some stuff for free, e.g. citation management. Unfortunately, as far as I’m aware there is still no support for cross-referencing.
  3. You can use Quarto execution options like execute-dir: project and resources: www/ to globally specify the working directory and a directory for external resources like images.

There are also a few peculiarities to be aware of. To avoid any issues with Documenter.jl, I've found it useful to ensure that the rendered .md files do not contain any raw HTML and to preserve text wrapping:

format: 
  commonmark:
    variant: -raw_html
    wrap: preserve
Enter fullscreen mode Exit fullscreen mode

When working with .qmd files you also need to use a slightly different syntax for admonitions. The following syntax inside the .qmd

| !!! note \"An optional title\"
|     Here is something that you should pay attention to.   
Enter fullscreen mode Exit fullscreen mode

will generate the desired output inside the rendered .md:

!!! note "An optional title"
    Here is something that you should pay attention to.   
Enter fullscreen mode Exit fullscreen mode

Any of my package repos — CounterfactualExplanations.jl, LaplaceRedux.jl, ConformalPrediction.jl — should provide additional colour on this topic.

Quarto for Academic Journal Articles

Quarto supports LaTeX templates/classes, which has helped me with paper submissions in the past (e.g. my pending JuliaCon Proceedings submissions). I’ve found that rticles still has an edge here, but the list of out-of-the-box templates for journal articles is growing. Should I find some time in the future, I will try to add a template for JuliaCon Proceedings. The beauty of this is that it should enable publishers to not only use traditional forms of publication (PDF), but also include more dynamic formats with ease (think distill, but more than that.)

Wrapping up

This short post has provided a bit of an update on using Quarto with Julia. From my own experience so far, things have been getting easier and better (thanks to the amazing work of Quarto dev team). I’m exicted to see things improve even further and still think that Quarto is a revolutionary new tool for scientific publishing. Let’s hope publishers eventually recognise this value 👀.

Originally published at https://www.paltmeyer.com on November 21, 2022.

Latest comments (14)

Collapse
 
kellertuer profile image
Ronny Bergmann

Quarto in documenter.jl looks super interesting – I could not see how you actually run quarto on these – do you do that manually before pushing to GitHub? Or is that done on CI?

Collapse
 
patalt profile image
Patrick Altmeyer

Hi @kellertuer - good question! So far i've relied on conventional Quarto workflows (so just quarto render the whole thing locally). Should be straightforward to use CI for that, but I actually like to do this locally and see if everything renders well. Hope this helps!

Collapse
 
kellertuer profile image
Ronny Bergmann

I see your point, especially for longer computations in a tutorial pre-rendering locally might be nice – but if one could store the _freeze as an artefact then that might still be quite fast. I might experiment with that once I get a little more familiar with Quarto (currently fighting my starting steps).

Collapse
 
rolandka profile image
Roland Schätzle

A question concerning the use of forem: How did you get the phrase “Originally published at Medium” below the author name?

Collapse
 
patalt profile image
Patrick Altmeyer

Hi Roland, this was auto-generated, since I imported the article from Medium. @logankilpatrick pinned post describes how to link your forem account to an existing Medium account or blog. Hope this helps!

Collapse
 
rolandka profile image
Roland Schätzle

Hi Patrick, thanks for your rapid answer!

I've already read the articles of @logankilpatrick on how to use forem, but I haven't found a description on how to import a Medium article. I've only read that you can do this, but not how. Can you give me a hint, on how to do that or on a article that explains the process?

Thread Thread
 
patalt profile image
Patrick Altmeyer

Oh I see! You can find this under Settings > Extensions: forem.julialang.org/settings/exten....

Thread Thread
 
rolandka profile image
Roland Schätzle

Ah, thank you very much! 👍

Collapse
 
svilupp profile image
Jan Siml

Nice overview! Me and my team have jumped to Quarto after reading your summer blog post - never looked back! A lot of html reports and revealjs presentations for our stakeholders.

A tip we learned: if you write a lot of code (our case for data analyses/EDA), it’s easier to stay in .jl and convert to .qmd with Literate.jl (PR: github.com/fredrikekre/Literate.jl...) with VSCode tasks

Collapse
 
patalt profile image
Patrick Altmeyer

Thanks, Jan - great to hear that this has worked out well for you and your team.

Very interesting you mention Literate.jl, actually. At the JuliaLang Eindhoven Meetup last night, I had the chance to discuss the potential for a Pluto.jl <-> Quarto conversion with the folks behind Pluto. The PR you linked might actually go a long way in terms of making that possible, since Pluto notebooks are just .jl files (with a bit of magic on top). Have a few other commitments in the coming weeks, but will give turn back to this.

Collapse
 
kellertuer profile image
Ronny Bergmann

I would be intereste in Pluto <-> Quarto as well :)

Collapse
 
fortunewalla profile image
Fortune Walla

One potential benefit of Jupyter notebooks is their connection to Google Colab: it is possible to store Jupyter notebooks on Github and make them available on Colab

Quite an informative article with many things to try for newcomers.

Did you try if it would be possible to install Quarto within Colab & try to run the notebook using Colab & generate the final document?

Collapse
 
patalt profile image
Patrick Altmeyer

Thanks! I don’t think there is an actual Quarto plugin for Colab, but you can of course always work with Jupyter notebooks in Colab and still use them with Quarto. In the article I recommend to use .qmd files, but Colab is actually one reason people might still also like to use .ipynb. Fortunately, converting .qmd to .ipynb and vice versa is seamless.

Collapse
 
fortunewalla profile image
Fortune Walla

I guess python tooling is a priority for Colab as of now. Perhaps someone will bridge the gap between Quarto & Colab soon.