Julia Community 🟣: Daniel Pinyol

Test Data Builder in Julia

Daniel Pinyol — Mon, 27 Feb 2023 23:17:45 +0000

Short tests are better

Ideally, we'd like tests to have 3 lines of code:

Setup: aka Given or Arrange
Run: aka When or Act
Test: aka Then or Assert

But they often require setting up a lot of data before actually executing the function to test. This causes several drawbacks:

The tests get longer, less clear and more error prone.
Developers may feel lazy to create new tests if the required effort is too high.
The explosion in the fan-in of the setup code will slow down potential refactors.

How to reduce the setup phase

In this post I'll present some examples on using the Test Data Builder pattern to mitigate this issue.

The Model

Our toy model represents Circles and Squares which are fully enclosed in a Canvas.

Base.@kwdef struct Canvas
  x::Int
  y::Int
end

Base.@kwdef struct Position
  x::Int
  y::Int
end

Base.@kwdef struct Circle
    center::Position
    radius::Int
end

Base.@kwdef struct Square
    center::Position
    side::Int
end

The Builder pattern

Using Julia's Base.@kwdef, you can quickly create a Builder for Position values.
The builder should ensure that the non-explicitly set fields will get consistent values. In this simple case, you could actually enforce inconsistent instances to define unhappy path tests. However, when available, I prefer Builders to transparently invoke the model integrity tests to ensure that test setups start from consistent data.

const TEST_CANVAS = Canvas(640, 480)
randX(c::Canvas) = rand(1:c.x)
randY(c::Canvas) = rand(1:c.y)

Base.@kwdef mutable struct PositionBuilder 
    canvas::Canvas = TEST_CANVAS
    x::Int = randX(canvas)
    y::Int = randY(canvas)
end

Base.rand(pb::PositionBuilder) = Position(; pb.x, pb.y)

Since all data takes default values, the intent of the setup code becomes evident

# testing a position at first column
rand(PositionBuilder(;x=0))
# testing a position at a 1-pixel-sized Canvas
rand(PositionBuilder(;canvas=Canvas(1,1))

Builders are composable

You can define your CircleBuilder & SquareBuilder to compose PositionBuilder to create the shape center.

Base.@kwdef mutable struct CircleBuilder
    canvas::Canvas = TEST_CANVAS
    centerB::PositionBuilder = PositionBuilder(; canvas)
    radius::Int = randRadius(centerB, canvas)
end

function randRadius(pb::PositionBuilder, canvas::Canvas)
    maxRadius = min(pb.x, canvas.x - pb.x, pb.y, canvas.y - pb.y)
    return rand(1:maxRadius)
end

function Base.rand(cb::CircleBuilder)
    radius = randRadius(cb.centerB, cb.canvas)
    return Circle(; center=rand(cb.centerB), radius)
end

Going farther

However, we suffer several limitations in the current design:

If we do CircleBuilder().radius = X, the assigned value will get lost when creating a Circle.
We need to invoke radius = randRadius(...) twice.

To avoid these shortcomings, I tend to use the following design:

Base.@kwdef struct SquareBuilder
    canvas::Optional{Canvas} = nothing
    center::Optional{PositionBuilder} = nothing
    side::Optional{Int} = Nothing
end
canvas!(sb::SquareBuilder) = @something sb.canvas TEST_CANVAS
center!(sb::SquareBuilder) = sb.centerB = @something sb.centerB PositionBuilder(; canvas=canvas!(sb))
side!(sb::SquareBuilder) = sb.side = @something sb.side calculateSide(canvas!(sb), center!(sb))

function Base.rand(sb::SquareBuilder)
    center = rand(center!(sb))
    side = side!(sb)
    return Square(; center, side)
end

Now, all fields are nothing until the tests assign a specific value, or they are automatically assigned during the construction of the struct.

Regression tests gotchas in Julia

Daniel Pinyol — Sun, 14 Aug 2022 09:59:00 +0000

Imagine that you have a function which computes a result from a large vector of numbers:

result = myfun([...])

In your battery of tests, you may want to ensure that the performance of your function does not degrade due to future updates of your code. Since we developers are lazy, we may be tempted to try to always run the test with the same fixed input and then check the time and memory performance.

You may end up with something like:

using Random
using Test

Random.seed!(42)
const inputlen = rand(1000:2000)
const input = rand(inputlen)

result = myfun(input)
@test result == EXPECTED

perf = @timed myfun(input)
@test perf.time < TIME_THRESHOLD
@test perf.bytes < BYTES_THRESHOLD

Since you're clever, you fixed the random seed. So, what can go wrong?

1. Random engine does not generate the same values accross Julia versions

This happened from 1.6 to 1.7. It's possible that it won't happen again in a long time, but if you have many of these tests it may be a pain to change the expected results. Tp ensure stable results, you should use StableRNGs

2. Relying on the global Random engine is error prone

Once I was bitten by this. At one point, I realized that sometimes my tested function was not behaving as expected.

After some digging, I found that:

The random seed was set at the beginning of the file.
Then I was creating a struct S in order to call myfun(s).

I realized that S's constructor was doing this:

    function myfinalizer(s::S)
        @async begin
            try
                Base.close(s)
            catch e
                for (exc, bt) in current_exceptions()
                    showerror(stderr, exc, bt)
                end
            end
        end
    end

    function S()
        s = new(....)
        finalizer(myfinalizer, s)
    end

What is this doing?

It registers a finalizer so that the S object automatically closes its acquired resources when it's destroyed.
Since finalizers are called from GC thread, you cannot directly call IO stuff. So you need @async to close your resources from a different thead.
Last but not least, exceptions must be caught and logged since Julia will silently swallow them in this case.

So what?

What happened was that @async internally uses Random (maybe to generate the Task id), which screwed the reproducibility of the random function input.

But why was did the input only changed sometimes?

I guess that, as the execution of GC is not controlled by my code, depending on how the finalizer is called, the input may be either randomly generated with the fresh seed, or after @async's usage of Random.

Conclusions

finalizer's are dodgy in all GC languages. If possible use do blocks instead.
In general, your code should be very explicit on its dependencies (Random here). They should be injected to avoid side effects.
Don't be (so) lazy, and avoid randomly generated inputs for regression tests. You can easily seralize input data with Julia serialization, or with libraries such as JLD2. JLD2 is supposed to be more stable accross Julia versions, but in any case it's wise to serialize simple data types as Dict's or Vector only.

Update 2022/8/30

Thank you @chrisrackauckas for point me out to StableRNGs
Asserting on wall clock duration of tests is very fragile. There are plenty of tools which measure instead the number of executed instruction. Eg GFlops.jl counts the number of floating point operations, LinuxPerf.jl counts linux events, or LIKWID.jl which wraps linux likwid tool.

Detecting @test @allocated gotchas

Daniel Pinyol — Wed, 03 Aug 2022 08:34:18 +0000

Hi,
if you care about performance, you probably end up having a lot of these on your unit tests, to ensure that your functions do not allocate when they don't need to:

@test @allocated testedfunction(...) === 0

So far so good. But, if you need assignments or comparisons, you'll need some brackets.

@test (@allocated var=yourexpression) === 0))

This is where you start thinking on DRY: should you create a macro to enable more expressive tests?

@testnoallocations var=yourexpression

macro testnoallocations(expressions...)
    return esc(
        quote
            @test iszero(@allocated $(expressions...))
        end
    )
end

I confess I'm a encapsulation junkie, so I'll try to justify why I think it's a good idea.

1. Global non-consts

foo = "foo"
@testset "myset" begin
    @test @allocated startswith(foo, "foo")
end

If your tests are complex or you're a Julia noob, it's not difficult to miss that this test will fail because foo is a non-const global. Why not adding the check within your macro? You just need to recursively get all the symbols, and find those which are global (isdefined(__module__, arg)) but !isconst.


macro testnoallocations(expressions...)
    _failIfNonConstGlobalsInExpressions(__module__, expressions...)
    return esc(....)
end

function _failIfNonConstGlobalsInExpressions(mod::Module, expressions...)
    for e in expressions
        nonConstGlobals = (
            arg for arg in expressionsymbols(e) if isdefined(mod, arg) && !isconst(mod, arg)
        )
        if !isempty(nonConstGlobals)
            error(
                "testnoallocations called with expression containing non const global symbols $(collect(
            nonConstGlobals
        ))",
            )
        end
    end
end

" Return all the symbols that make up an expression (or itself if a symbol is passed)"
function expressionsymbols(e::Union{Expr, Symbol, Number})
    !isa(e, Expr) && return (ex for ex in (e,) if isa(e, Symbol))
    topSymbols = (arg for arg in e.args if isa(arg, Symbol))
    subExpressions = (expressionsymbols(arg) for arg in e.args if isa(arg, Expr))
    return (topSymbols..., Iterators.flatten(subExpressions)...)
end

2. `@testset`s are not functions

I don't have a MWE for this, but a more difficult gotcha is when your function allocates because you're calling it from a function. Indeed, @testset bodies run from the global scope. You can detect this by exploiting the fact that requesting the current function with nameof(var"#self#") will fail when called from the global scope.

macro testnoallocations(expressions...)
    return esc(
        quote
            !@isCalledFromFunction() &&
                @warn "Since not called from a function @allocated could be imprecise"
            @test (@allocated $(expressions...)) === 0
        end,
    )
end

macro isCalledFromFunction()
    expr = esc(:(
        try
            currentFunctionName = nameof(var"#self#")
            true
        catch
            false
        end
    ))
    return expr
end

3. Running with --compile=min

A trick to run short tests very quickly is running them with --compile=min. Unfortunately, the interpreter may then allocate extra memory.

macro testnoallocations(expressions...)
    if !iscompileenabled()
        @warn "Allocations measures are not precise because executed with --compile=min"
    end
    ...
end
iscompileenabled() = Base.JLOptions().compile_enabled == 1

That's all

I hope the post is also useful to show how powerful are Julia reflection capabilities. You comments are more than welcome, specially if you know similar tricks to ease your testing experience. Happy testing!