Julia Community 🟣: Roland Schätzle

Statistical Plotting with Julia: VegaLite.jl

Roland Schätzle — Fri, 09 Dec 2022 16:17:37 +0000

How to create statistical plots using the VegaLite.jl package

This is the second of several articles where I compare different Julia graphics packages for creating statistical plots. I've started with the Gadfly package (Statistical Plotting with Julia: Gadfly.jl, [SPJ02]) and continue the series here with the VegaLite package.

In the introduction to the series (The Grammar of Graphics or how to do ggplot-style plotting in Julia, [SPJ01]), I've explained the Grammar of Graphics (GoG) which is the conceptual base for these graphics packages. There I've also introduced the data which will be used for the plotting examples.

The objective of this article (and the ones which will follow in the series) is to reproduce the visualizations from [SPJ02] using the exact same data, but each time of course with another graphics package in order to achieve a 1:1 comparison of all packages.

Therefore the descriptions of these visualizations in the following text will be identical to the ones in [SPJ02]. I.e. this article is self-contained (you can read and understand it, without having read [SPJ02]). It has also the same structure (headlines etc.) like [SPJ02], so that it easy to make a side-by-side comparison.

VegaLite

VegaLite.jl is (like Gadfly.jl) a very complete implementation of the Grammar of Graphics (GoG). It has been written by a group led by Prof. David Anthoff (University of Berkeley) consisting of more than 20 contributors. VegaLite is part of a larger ecosystem of data science packages (called Queryverse) which includes query languages (Query.jl), tools for file IO and UI Tools (ElectronDisplay.jl).

Technically VegaLite takes quite a different approach: Whereas Gadfly is completely written in Julia, VegaLite is more like a language interface for the Vega-Lite graphics package (note the dash in its name in contrast to VegaLite, which denotes the Julia package). Vega-Lite takes specifications of visualizations in JSON format as inputs which the Vega-Lite compiler transforms into the corresponding visualizations.

Vega-Lite is completely independent of the Julia ecosystem and apart from VegaLite there exist interfaces for other languages like JavaScript, Python, R or Scala (see "Vega-Lite Ecosystem" for a complete list).

As Vega-Lite uses JSON as its input format, these specifications have a rather declarative nature. VegaLite tries to mimic this format with the @vlplot-macro, which is the basis for all visualizations as we will see in the following examples. This makes it less Julian than e.g. Gadfly, but has on the other hand the advantage, that somebody who is familiar with Vega-Lite will easily learn how to use VegaLite. And if there is something missing in the VegaLite documentation it is often easy to find the corresponding part within the Vega-Lite docs.

A distinguishing feature of Vega-Lite (as well as VegaLite) is its interactivity. Its specifications may not only describe a visualization but also events, points of interest and rules about how to react to these events. But this feature is beyond the article at hand. For readers interested in this aspect, I recommend to have a look at the Vega-Lite home page or the paper "Vega-Lite: A Grammar of Interactive Graphics".

Example Plots

As in the preceding article, I will use for the comparison a few diagram types (or geometries as they are called by the GoG) which are commonly used in data science, namely:

bar plots
scatter plots
histograms
box plots
violin plots

VegaLite offers of course many types more as you can see in this gallery. But in order to obtain a 1:1-comparison between all packages, I stuck with the types listed above.

The data for the examples is assumed to be ready in the DataFrames structures countries, subregions_cum and regions_cum presented in the introducing article [SPJ01] to the series.

Most plots are first presented in a basic version, using the defaults of the graphics package and get then refined using customized attributes (for labels, background color etc.).

Bar Plots

Population by Region

As in [SPJ02] we start with a simple bar chart, that shows population size (in 2019) by region. This is done using the following @vlplot-command mapping data to aesthetics and using a bar-geometry as we learned in the introducing article about the Grammar of Graphics. Julia’s pipeline syntax is used (|>) to specify the regions_cum -DataFrame as being the input to @vlplot.

regions_cum |>
    @vlplot(
         width = 600, height = 300,
         :bar,
         x = :Region, y = :Pop2019, color = :Region
     )

This results in the following bar chart:

In a second version we don’t rely on defaults, but set axis labels, title and background color manually. Apart from that, we want the bar labels on the x-axis with a horizontal orientation for better readability. This leads to the following code, where title-attributes are used for the labels as well as the diagram title, an axis-attribute for changing the orientation of the bar labels and a config for general attributes like background color.

… creating the following beautified bar chart:

Population by Subregion

The next bar chart depicts population by subregion using the following @vlplot-command:

subregions_cum |>
    @vlplot(
        width = 600, height = 300,
        :bar,
        x = :Subregion, y = :Pop2019, color = :Region
    )

… resulting in the following bar chart:

We can see that there is room for improvement: As there are quite a few subregions and their names a relatively long, a horizontal bar diagram might be more readable. Apart from this, we adapt again labels, title, background color etc. leading to the following code, where we switch to a horizontal layout just by flipping the data attributes for the x- and the y-axis:

… resulting indeed in a more readable bar chart:

It get’s even more readable, if we sort the subregions by population size before rendering the diagram. We could sort the subregions_cum-DataFrame using Julia (as we did in the Gadfly-example), but VegaLite offers the possibility to sort the data in the graphics engine using the sort-attribute.

If we apply this code we finally get:

A word of caution at this point: While it is possible to sort data within the graphics engine I wouldn’t recommend it with larger data sets, because it is considerably slower than doing it directly using Julia.

Scatter Plots

In the next step we have a look at the population at the country level in relation to the growth rate. A scatter plot is a good way to visualize this relationship. We get one, using a point geometry as follows:

countries |>
    @vlplot(
        width = 600, height = 300,
        :point,
        x = :Pop2019, y = :PopChangePct, color = :Region
    )

… resulting in this scatter plot:

As we also mapped the region to the color aesthetics, we get a more differentiated picture involving region information in addition.

But the distribution of the data is quite skewed — most countries have a population below 200 Mio. So a logarithmic scale on the x-axis might give a better insight into the data. And again, we add some labels, background color etc. leading to the following code:

… giving us the following improved scatter plot:

Histograms

Bar plots and histograms have the same geometry (in the sense of the “Grammar of Graphics”). But in order to get categorical data on the x-axis, the data used for a histogram has to be mapped to (artificial) categories in a process called binning. In the GoG this is done using a so-called bin statistic.

VegaLite follows the GoG strictly. So we get a histogram that shows the distribution of GDP per capita among the different countries with the following @vlplot-command using a bar geometry with the parameter bin set to true:

countries |>
    @vlplot(
        width = 600, height = 300,
        :bar,
        x = {:GDPperCapita, bin = true}, y = “count()”
    )

… resulting in this histogram:

A reasonable bin size has been chosen by default (which wasn’t the case with Gadfly).

And again we can add labels etc. And in order to have exactly the same number of bins as in the Gadfly example, we set it explicitly to 20 using the following code:

… leading to the following improved histogram:

Box Plots and Violin Plots

To obtain an insight into the distribution of some numerical data, box plots or violin plots are typically used. Each of these diagram types has its specific virtues. So let’s visualize the distribution of the GDP per capita for each region using these plots.

Box Plot

Let’s immediately use the ‘beautified’ version based on a boxplot-geometry:

… giving us the following box plot:

Violin Plot

As VegaLite doesn’t support violin plots as a geometry on its own, they have to be constructed using density plots (one for each region) which are lined up horizontally. This leads to the following, rather complicated specification:

The basic geometry used to create the density plots is an area geometry. The data is then grouped by region and for each group the density is computed. This is done using a transform-operation. Assigning the density to the x-axis results in vertical density plots. In the next step all five density plots are lined up horizontally using the column-attribute.

The width and spacing attributes in the last line define each column (i.e. each density plot) to have a width of 120 pixels horizontally and to leave no space between these plots.

So we finally get the following violin plot:

Zooming in

As in the Gadfly-examples we note, that the really interesting part of the distributions lies in the range from 0 to 100,000$. Therefore we want to restrict the plot to that range on the y-axis, doing sort of a zoom-in.

In the Gadfly example we restricted the values on the y-axis to this range to achieve the desired effect. Such a restriction can also be specified in VegaLite using scale = {domain = [0, 100000]}. Unfortunately this doesn’t give us the result we want: The diagram will be plotted in this range but the plots themselves still use the whole range up to 200,000$, thus getting partly plotted outside the diagram:

The only way to get a roughly similar result in VegaLite would be to restrict the data to values in that range up to 100,000$ using a filter expression. But be aware: this is conceptually something different, giving us not exactly the same plots as if we would do it on the whole dataset. So we don’t have a real solution for this visualization.

This may be just a problem of the VegaLite documentation, where I couldn’t find any other solution (or my fault for not doing enough research and e.g. using also the extensive documentation of Vega-Lite).

Conclusions

As we can see, VegaLite follows most of the time quite closely the concepts of the Grammar of Graphics (even more closely than Gadfly does). That’s one of the reasons why the plot specifications are so consistent (same things are always specified in the same way independent of context) und thus easy to learn and to memorize.

But as we can see with the violin plot, if things are not predefined, the specifications can become quite complex. Together with the rather non-Julian syntax which needs some time to learn and to get used to, I wouldn’t recommend VegaLite to occasional users. It needs some learning and training. But if you invest that time and effort, you get a really powerful (and interactive) visualization tool.

An interesting add-on to VegaLite, which I would like to mention, is the interactive data explorer Voyager (see: DataVoyager.jl). It’s an application that allows to load data and create a variety of visualizations without any programming.

If you want to try out the examples from above by yourself you can get a Pluto notebook which is sort of an executable variant of this article from my GitHub repository.

Statistical Plotting with Julia: Gadfly.jl

Roland Schätzle — Mon, 05 Dec 2022 15:58:09 +0000

This article appeared in Towards Data Science on Aug 25th, 2022

How to create statistical plots using the Gadfly.jl package

This is the first of several articles where I compare different Julia graphics packages for creating statistical plots. I start the series here with the Gadfly-package.

In the introduction to the series (The Grammar of Graphics or how to do ggplot-style plotting in Julia), I’ve explained the Grammar of Graphics (GoG) which is the conceptual base for these graphics packages. In that article I’ve also introduced the data which will be used for the plotting examples.

Gadfly

Gadfly is a very complete implementation of the Grammar of Graphics. Its original author is Daniel C. Jones, but the package has currently more than 100 contributors listed on GitHub. The first versions appeared in 2014. In the meantime it is a very mature package with only a few new releases per year.

It’s completely written in Julia and plays well with rest of the Julia ecosystem. There is e.g. a tight integration with DataFrames.jl and via the IJulia package it can be directly used within Jupyter notebooks.

For the rendering of publication quality graphics it’s able to render SVG out of the box and using Cairo.jl and Fontconfig.jl it can also produce formats like PNG, PDF, PS and PGF.

The plots produced by Gadfly offer some interactivity like panning, zooming and toggling.

Example Plots

For the comparison I will use a few diagram types (or geometries as they are called by the GoG) which are commonly used in data science, namely:

bar plots
scatter plots
histograms
box plots
violin plots

Gadfly offers of course many types more as you can see in this gallery. But in order to obtain a 1:1-comparison between all packages, I stuck with the types listed above.

The data for the examples is assumed to be ready in the DataFrames structures countries, subregions_cum and regions_cum presented in the introducing article to the series.

Most plots are first presented in a basic version, using the defaults of the graphics package and get then refined using customized attributes (for labels, background color etc.).

Bar Plots

Population by Region

We start with a simple bar chart, that shows population size (in 2019) by region. This is done using the following plot-command mapping data to aesthetics and using a bar-geometry as we learned in the introducing article about the Grammar of Graphics:

plot(regions_cum, 
        x = :Region, y = :Pop2019, color = :Region, 
        Geom.bar)

… resulting in the following bar chart:

In a second version we don’t rely on defaults, but set axis labels, title and background color manually. Apart from that we don’t want the numbers on the y-axis in scientific format and there should be some space between the bars (to conform to the definition of a bar chart). This leads to the following code, where Guide-elements are used for the labels, a Scale for changing the numbers on the y-axis and a Theme for general attributes like background color or bar spacing.

… creating the following beautified bar chart:

Population by Subregion

The next bar chart depicts population by subregion using the following plot-command:

plot(subregions_cum, 
        x = :Subregion, y = :Pop2019, color = :Region, 
        Geom.bar)

… resulting in the following bar chart:

We can see that there is room for improvement: As there are quite a few subregions and their names a relatively long, a horizontal bar diagram might be more readable. Apart from this we adapt again labels, title, background color etc. leading to the following code, where we switch to a horizontal layout using the parameter orientation on the bar geometry:

… resulting indeed in a more readable bar chart:

It gets even more readable, if we sort the subregions subregions_cum by population size (Pop2019) before rendering the diagram using the following command:

subregions_cum_sorted = sort(subregions_cum, :Pop2019)

If we apply the plot command from above to the sorted data subregions_cum_sorted we finally get:

Scatter Plots

In the next step we have a look at the population at the country level in relation to the growth rate. A scatter plot is good way to visualize this relationship. We get one using a point geometry as follows:

plot(countries,
        x = :Pop2019, y = :PopChangePct, color = :Region,
        Geom.point)

… resulting in this scatter plot:

As we also mapped the region to the color aesthetics, we get a more differentiated picture involving region information in addition.

… giving us the following improved scatter plot:

The labels-parameter for the log scale needs a bit of an explanation: Without this specification we would get the logarithms (to base 10) on the x-axis, which is for many people hard to understand. Instead we want just population numbers (e.g. 100.0 instead of 2). So we pass a function to labels which calculates the ‘correct’ labels. The log value x is converted to $10^x$ to get a ‘readable’ number, then rounded to two digits and finally converted to a string (which is the expected type for a label).

Histograms

Bar plots and histograms have the same geometry (in the sense of the “Grammar of Graphics”). But in order to get categorical data on the x-axis the data used for a histogram has to be mapped to (artificial) categories in a process called ‘binning’. In the GoG this is done using a so-called bin statistic.

Gadfly doesn’t follow (or at least doesn’t show) the theory in this place. It introduces instead a separate geometry for histograms (which might be more practical for everyday use).

So we get a histogram that shows the distribution of GDP per capita among the different countries with the following plot-command using a histogram geometry:

plot(countries, x = :GDPperCapita, Geom.histogram)

… resulting in this histogram:

The number of bins used can be controlled by the bincount-parameter of the histogram geometry. And again we can add labels etc. resulting in the following code:

… leading to the following improved histogram:

Box Plots and Violin Plots

Box Plot

Let’s immediately use the ‘beautified’ version using a boxplot-geometry:

… giving us the following box plot:

Violin Plot

The code for a violin plot for this visualization looks quite similar. The only difference being the use of a violin-geometry (instead of a boxplot):

… leading to the following violin plot:

Here we note that the defaults for the scaling of the y-axis don’t work as good as with the box plot. Apart from that, the really interesting part of the distribution lies in the range from 0 to 100,000. Therefore we want to restrict the plot to that range on the y-axis, doing sort of a zoom-in.

Zooming in

This can easily be achieved by adding the following line to the list of plot-parameters:

Coord.cartesian(ymin = 0, ymax = 100000),

… leading to the following violin diagram:

The same restriction to the y-axis can be applied to the box plot:

Conclusions

As we can see, Gadfly follows most of the time quite closely the concepts of the Grammar of Graphics. That’s one of the reasons why the plot specifications are so consistent (same things are always specified in the same way independent of context) und thus easy to learn and to memorize.

You reach only some limits when it comes to edge cases. E.g. if you specify a scatter plot where there is only a mapping to the x-axis but not to the y-axis. According to the GoG you should get points distributed on a line (the x-axis). That doesn’t work with Gadfly. And there is e.g. no polar coordinate system implemented (but could be done in the future).

But if your visualization needs are centered around the (large) list of geometries which are implemented in Gadfly and you don’t need rather exotic customizations of these diagrams then you will be quite happy with Gadfly.

If you want to try out the examples by yourself you can get a Pluto notebook which is sort of an executable variant of this article from my GitHub repository.

The Grammar of Graphics or how to do ggplot-style plotting in Julia

Roland Schätzle — Thu, 01 Dec 2022 18:01:30 +0000

This article appeared in Towards Data Science on Aug 12th, 2022

Introduction to a comparison of Julia graphics packages for statistical plotting

The Grammar of Graphics (GoG) is a concept that has been developed by Leland Wilkinson (The Grammar of Graphics, Springer, 1999) and refined by Hadley Wickham (A Layered Grammar of Graphics, Journal of Computational and Graphical Statistics, vol. 19, no. 1, pp. 3–28, 2010; pdf).

Its main idea is that every statistical plot can be created by a combination of a few basic building blocks (or mechanisms). This allows

a simple and concise definition of a visualization
an easy adaptation of a visualization by exchanging only the building blocks which are affected in a modular way
reusable specifications (the same visualization can e.g. be applied to different data)

Wickham showed that this concept is not only a nice theory. He implemented it in the R-package ggplot2 which became quite popular. Several GoG-implementations are also available for the Julia programming language.

In this article I will first explain the basic concepts and ideas of the Grammar of Graphics. In follow-up articles I will then present the following four Julia graphics packages which are based (completely or partially) on the GoG:

In order to allow you a 1:1-comparison of these Julia packages, I will use the same example plots and the same underlying data for each article. In the second part of this article, I will present the data used for the examples, so I don’t have to repeat that in each of the follow-up articles.

Grammar of Graphics

In the next sections I will explain the basic ideas of “The Grammar of Graphics” by Wilkinson as well as “A Layered Grammar of Graphics” by Wickham. I won’t go into every detail and in aspects where both concepts differ, I will deliberately pick one and give a rather “unified” view.

For the code examples, I’m using Julia’s Gadfly-package (vers. 1.3.4 & Julia 1.7.3).

The main ingredients

The main building blocks for a visualization are

data
aesthetics
geometry

Data

The most familiar of these three concepts is probably data. We assume here, that data comes in tabular form (like a database table). For a visualization it’s important to distinguish between numerical and categorical data.

Here we have e.g. the inventory list of a fruit dealer:

Row │ quantity  fruit   price 
────────────────────────────────
 1  │    3     apples   2.5
 2  │   20     oranges  3.9
 3  │    8     bananas  1.9

It consists of the three variables quantity, fruit and price. fruit is a categorical variable whereas the other two variables are numerical.

Aesthetics

To visualize a data variable, it is mapped to one or more aesthetics.

Numerical variables can be mapped e.g. to a

position on the x-, y- or z-axis
size

Categorical variables can be mapped e.g. to a

color
shape
texture

Geometry

Apart from data variables and aesthetics we need at least a geometry to specify a complete visualization. The geometry tells us basically which type of diagram we want. Some examples are:

line (= line diagram)
point (= scatter plot)
bar (= bar plot)

Basic examples

Now we have enough information to build our first visualizations based on the Grammar of Graphics. For the code examples using the Gadfly-package we assume, that the inventory table above is in a variable named inventory of type DataFrame.

First we want to see how the quantities are distributed by price. Depending on the geometry chosen, we get either a scatter plot or a line diagram:

Map price to the x-axis, quantity to the y-axis using a point geometry

In Gadfly: plot(inventory, x = :price, y = :quantity, Geom.point)

Map price to the x-axis, quantity to the y-axis using a line geometry

In Gadfly: plot(inventory, x = :price, y = :quantity, Geom.line)`

In the next step we want additionally see, which fruits are involved. So we have to map fruit to a suitable aesthetic too. In the following two examples first a shape is used and then a color.

Map price to the x-axis, quantity to the y-axis, fruit to a shape using a point geometry

In Gadfly: plot(inventory, x = :price, y = :quantity, shape = :fruit, Geom.point)

Map price to the x-axis, quantity to the y-axis, fruit to a color using a point geometry

In Gadfly: plot(inventory, x = :price, y = :quantity, color = :fruit, Geom.point)`

It is also possible to map one variable to several aesthetics. We can e.g. map fruit to shape as well as color.

Map price to the x-axis, quantity to the y-axis, fruit to a shape, fruit to a color, using a point geometry

In Gadfly: plot(inventory, x = :price, y = :quantity, shape = :fruit, color = :fruit, Geom.point)`

Using a bar geometry we can plot a statistics of the quantities in stock. Here we map a categorical variable (fruit) to positions on the x-axis.

Map fruit to the x-axis, quantity to the y-axis using a bar geometry

In Gadfly: plot(inventory, x = :fruit, y = :quantity, Geom.bar)

These basic examples show nicely how a visualization can be specified using a few simple building blocks, thus making up a powerful visualization language.

They show also that these specifications enable a graphics package to derive meaningful defaults for a variety of aspects of a visualization which aren’t given explicitly.

All the examples had

meaningful scales for the x- and y-axis (typically using a slightly larger interval than that of the data variable given)
together with appropriate ticks and axis labeling
as well as a descriptive label (simply using the variable name)

Some examples even had an automatically generated legend. This is possible because a legend is simply the inverse function of a data mapping to an aesthetic. If we e.g. map the variable fruit to a color, then the corresponding legend is the reverse mapping from color to fruit.

More ingredients

To be honest, we need a few more elements than just data, aesthetics and a geometry for a complete visualization.

Scale

In order to map numerical variables e.g. to positional aesthetics (like the positions on the x- or y-axis), we need also a scale which maps the data units to physical units (e.g. of the screen, a window or a web page).

In the examples above, a linear scale was used by default. But we could also exchange it e.g. with a logarithmic scale.

It’s also possible to map a numerical variable to a color. Then a continuous color scale is used for that mapping as we can see in the following example:

Map price to the x-axis, quantity to the y-axis, quantity to a color using a point geometry

In Gadfly: plot(inventory, x = :price, y = :quantity, color = :quantity, Geom.point)

Coordinate system

Closely related to a scale is the concept of a coordinate system, which defines how positional values are mapped onto the plotting plane. In the examples above, the Cartesian coordinate system has been used by default. Other possibilities are polar or barycentric coordinate systems or the various systems which are used for map projections.

It is an interesting aspect that we can produce different types of diagrams from the same data and aesthetics mappings, just by changing the coordinate system: E.g. a bar plot is based on the Cartesian coordinate system. If we replace that with a polar system, we get a Coxcomb chart, as the following example from R for Data Science (by Hadley Wickham and Garret Grolemund, O’Reilly, 2017) shows.

Conclusions

With these two additional concepts we have now a complete picture of the basic GoG. In this short article I could of course only present a subset of all possible aesthetics and graphics and there are more elements to the GoG like statistics and facets. But what we have seen so far is the core of the Grammar of Graphics and should be enough to grasp the main ideas.

Comparison of Julia graphics packages

Let’s now switch to the comparison of different Julia graphics packages which I will present in several follow-up articles. As sort of a preparation I will now present the data used for different example plots (which are inspired by the YouTube tutorial Julia Analysis for Beginners from the channel julia for talented amateurs) within these follow-up articles and give an outlook on what sorts of diagrams I will use for the comparison.

Countries by GDP

The basis of the data used for the plotting examples is a list of all countries and their GDP and population size for the years 2018 and 2019. It’s from this Wikipedia-page (which got the data from a database of the IMF and the United Nations). The data is also available in CSV-format from my GitHub-repository.

The columns of the list have the following meaning:

ID: unique identifier
Region: the continent where the country is located
Subregion: each continent is divided into several subregions
Pop2018: population of the country in 2018 [million people]
Pop2019: population of the country in 2019 [million people]
PopChangeAbs: change in population from 2018 to 2019 in absolute numbers [million people]
PopChangePct: like PopChangeAbs but as a percentage [%]
GDP: gross domestic product of the country in 2019 [million USD]
GDPperCapita: GDP divided by the number of people living in the country [USD/person]; this column is not in the source file, but will be computed (see below)

The file is downloaded and converted to a DataFrame using the following Julia code:

Line 7 computes the new column GDPperCapita mentioned above and adds it to the countries-DataFrame.

Aggregated data

The detailed list which has one row per country (in 210 rows) will be grouped and aggregated on two levels (using DataFrame-functions):

Level 1 — Regions: The following code groups the list by Region (i.e. continent) omitting the columns Country and Subregion (using a nested select) in line 1 and then creates an aggregation summing up all numerical columns (lines 2–5).

Level 2 — Subregions: The same operations are applied on the subregion level in lines 7–11. First the countries are grouped by Subregion omitting column Country (line 7) and then an aggregation is created on that data; again summing up all numerical columns. Besides, the name of the region is picked from each subgroup (:Region => first)

This resulting DataFrames regions_cum and subregions_cum look as follows:

Summary

The DataFrames countries, subregions_cum and regions_cum are the basis for the plotting examples in the forthcoming articles about the different Julia graphics packages. In these articles we will see how to create

bar plots
scatter plots
histograms
box plots and violin plots

in each of these graphics packages.

The first article will present Gadfly. So stay tuned!