Julia Community 🟣

DearDiary.jl: A lightweight but powerful machine learning experiment tracking tool for Julia

Jose Esparza — Tue, 04 Nov 2025 22:53:25 +0000

After months of planning, and some weeks of development, the package is finally here and ready for use! As a solution for tracking machine learning experiments in Julia, DearDiary.jl aims to be lightweight, easy to use, and flexible enough to adapt to different workflows.

Motivation

After the unpleasant experience trying to maintain an interface for the REST API from Python's MLFlow (MLFlowClient.jl, MLJFlow.jl), after finding out that it is poorly documented, incomplete, and has some abandoned or partially implemented features (and they are still adding new ones...); an idea came to my mind: why not write the same API but well designed and documented but in Julia? This package is that idea.

Core concepts

Architecture-first

Unlike the common monolithic architecture found in many Julia packages, my goal was to implement something that can be easily maintained and extended over time, focusing on developer experience and code readability (inspired by Alan Edelman's TED talk and MLJ.jl "micro-package" architecture).

DearDiary.jl consists of an N-layered architecture, which bring us the possibility of encapsulating different functionalities to allow better collaboration and separation of concerns.
Now it is composed of the following layers:

Repository layer: responsible for data storage and retrieval.
Service layer: handles package logic and data processing.
Route layer: manages RESTful API endpoints and HTTP requests.

And having the idea of implementing a frontend layer in the future.

Simple types

One of the problems I found while working on the integration project was the overuse of complex types. Imagine a type that has a field that is another type with field that is another type that has a field with an integer. Well, that's real and you can find it if you are curious enough.
DearDiary.jl tries to avoid that by keeping types simple and flat, totally immutable, and clear as possible. Never search for complexity when you don't need it.

Flexible by design

DearDiary.jl is flexible enough to adapt to different workflows. You can use it as a standalone package, or integrate it with other tools in your ML pipeline, or call it from the "outside world" via its RESTful API.
In the case something is not implemented in the way you want it, you can always modify or extend it, thanks to its modularity.

Portability

DearDiary.jl is designed to be portable. You can run it locally, on a server, or in the cloud. Thanks to SQLite as the default storage backend, you can easily move your projects between different environments without worrying about compatibility issues.

Note: one of the main goals for next releases is to support more storage backends, coming from SQL and NoSQL databases, or cloud storage solutions.

Getting started

A Tutorial is available in the documentation. It covers installation, and a workflow example with MLJ.jl.

Contributing

Contributions are welcome! If you find a bug or have a feature request, please open an issue on the GitHub repository. Pull requests are also encouraged. Please make sure to follow the existing code style and include tests for any new features.

StaticCompiler - Generating small binaries

Thomas R — Thu, 03 Jul 2025 14:26:54 +0000

Static compilation with help of StaticCompiler

StaticCompiler is a tool for generating small binaries from Julia source code. Here I'd like to share some experiences of using StaticCompiler.

So far StaticCompiler works for julia 1.10.x. Unfortunately it does not work for julia 1.11 and above so far: it seems to force to use an old version of GPUCompiler. It would be great, if one could fix that.

Static Compilation

Static compilation imposes some limitations to the julia code: GC allocations are not allowed. Additionally, the code has to be type-stable. Global variables are only allowed if they are constants. Additionally these constants should be non-composite; e.g. arrays as constants are not possible. Because allocations are also forbidden for return values of functions, these values must be of native types or unmodified structs, unless the respective functions or the function calls are inlined.

StaticTools provides allocation-free datatypes

StaticTools is a package that procides statically compilable datatypes for StaticCompiler. For example, strings are implemented as MallocString or StaticString, and can be coded using the string macros m"" and c"", respectively. Both types ensure 0-terminated strings, just like C strings. A StaticString object has a fixed length. For a representing a MallocString object, memory is being allocated. After declaring a MallocString, the user itself is responsible for freeing the allocated memory.

function f() a = c"anne" b = m"mike" free(b) 0 end

Arrays are represented as MallocVector, MallocArray, MallocMatrix, StackVector or StackArray. The functions mfill and sfill can be used to initialize a Malloc- or StackVector, respectively. You can also use the constructors themselves, and have an undefined initialization.

a = mfill(m"", 2) a[1] = m"anne" a[2] = m"mike"

b = MallocVector{MallocString}(undef, 2) b[1] = m"paul" b[2] = m"john"

c = mfill(m"", 2, 3) c[1, 1] = m"a" c[1, 2] = m"b"

d = MallocMatrix{MallocString}(undef, 2, 3) d[1, 1] = m"a" d[1, 2] = m"b"

free(a) free(b) free(c) free(d)

e = sfill(0,2) e[1] = 1 e[2] = 2

f = StackVector{MallocString}(undef, 2) f[1] = m"a" f[2] = m"b"

g = StackMatrix{MallocString}(undef, 2, 3) g[1, 1] = m"a" g[1, 2] = m"b"

A composite type of a StaticString, e.g. StackVector{StaticString}(undef, 2) does not to seem possible.

Tuples, NamedTuples and Structs can be used almost in the usual way.

function f() t = (2, 3.3, m"a") a, b, c = t printf(c"a: %d\n", a) printf(c"b: %g\n", b) printf(c"c: %s\n", c) n = (x = 1, y = 1) printf(c"n.x: %d\n", n.x) printf(c"n.y: %d\n", n.y) d = NTuple{3, MallocString}((m"a",m"b",m"c")) for i in 1:3 printf(c"%s\n", d[i]) end 0 end

struct A s :: MallocString n :: Int64 end

function h() a = A(m"a", 1) 0 end

When using mutable structs and calling a function that assignes a MallocString, you have to inline the called function or the function call.

mutable struct A s :: MallocString n :: Int64 end

function f() a = A(m"a", 1) a.s = m"b" a.n = 2 g(a) end

@inline function g(a::A) a.s = m"c" a.n = 3 b = m"a" end

Dictionaries and Unions need heap allocation. Enabling static compilation for them is an open, promising project.

Input and Output

Printing to stdout can be done with help of print, println and printf, in C matter. Flushing (and a newline) is supplied by putchar().

a = m"a" print(c"hello") println(c"hello") printf(c"a: %s\n", a) putchar('\n')

Here are examples for reading and writing files. The function readline was adapted to Windows (I did a PR for StaticTools).

fp = fopen(file_name, m"r") str = readline(fp) fclose(fp)

fp = fopen(file_name, m"w") printf(fp, c"hello") fclose(fp)

StaticTools also supplies functions to run system functions, convert strings to numbers, measure time, sleep (adapted for Windows), allocate memory, and load dll’s and apply functions of them.

StaticTools.system(c"dir") float = StaticTools.strtod(c"12.34")[1] int = StaticTools.strtol(c"12")[1] t_now = StaticTools.time()

StaticTools also provides the symbolcall-macro for calling a symbol via LLVM, and can be used to call further C functions. For example, the system function and the time function mentioned above, would be done with

function sys(s) @symbolcall system(pointer(s)::Ptr{UInt8})::Int end

function time() @symbolcall time(C_NULL::Ptr{Nothing})::Int end

If you want the time in milliseconds, you can use

function time_ms() :: Int64 @symbolcall clock()::Int end

StaticTools provides the function usleep. However, it does not seem to work in Windows. With help of symbolcall it can be adapted.

function sleep_win(secs::Real) millisecs = round(Int, secs * 1000) @symbolcall Sleep(millisecs :: Int) :: Int end

Debugging - some guide lines

When there is an error during compiling, the following error messages are typical:
`f() did not infer to a concrete type. Got Union{}, an error when linking (ld.lld: error: undefined symbol), orclang: error: unable to execute command: program not executable`, or a huge error message regarding the LLVM IR.

From those error message it is hardly possible to distinguish, whether the error is due to some restriction of the static compilation, or simply due to an program error in the Julia code, much less to debug this code.

Thus it is a good practice, to:

Firstly, let run the Julia code without compiling.
Here you can use your favourite debug and trace tools, e.g. @info and @show. Writing in file seems to disturb the printing on stdout. So you would have to leave out any writing to a file.

Second, check whether any GC allocations occur.

You can do this, e.g. with help of the macro @time, or some tool like Allocations. If allocations occur, identify the code segments that cause allocations, and find allocation-free alternatives.

Third, compile using compile_executable
When here an error message occurs, then locate the code part, leading to the compilation error. After that, find altenatives for the error-causing lines of code.

In general, it is of course helpful to start compiling smaller code pieces, and then getting more and more code to compile.

Conclusion

StaticCompiler is a great tool to compile StaticCompiler small, stand-alone, executable binaries. The generated binaries are in the range usually between around 90 and 300 kB (e.g. for 1500 lines Julia code I got a binary of 210 KB). In contrast to StaticCompiler, PackageCompiler would produce a system image, taking hundreds of MB.

StaticTools provides a lot of allocation-free functions and data structures. A few have to be adapted to the respective operating system. Dictionaries and unions cannot be compiled so far. It would be a promising project, to extend StaticTools, implementing allocation-free alternatives of these data structures.

Profile a short-running function

Neven — Mon, 02 Jun 2025 19:21:59 +0000

Recently I went to profile a short-running function. Several Julia packages, ProfileCanvas.jl being one of them, provide functionality for getting a CPU execution profile, often via a macro called like @profview expr_to_profile.

One problem that can occur is the run time of one invocation of expr_to_profile being so short that the profile ends up being useless because not enough samples are collected. Here's a quick-and-dirty hack solution to this problem. Criticize it in the comments below!

Example: profiling `cos(::Float64)`

"""
    run_in_a_loop(run_workload, get_argument, repeat_count)::Nothing
"""
function run_in_a_loop(run_workload::F, get_argument::G, repeat_count::Int) where {F, G}
    for i ∈ 1:repeat_count
        x = get_argument(i)
        r = @noinline run_workload(x)
        Base.donotdelete(r)
    end
end

# Example: profile `cos(::Float64)` execution

# Use a global to prevent constant folding, but precompute everything
# to allow the intended workload to dominate the profiling results.
precomputed_arguments::NTuple{128, Float64} = (rand(Float64, 128)...,);

function get_argument(i::Int)
    axs = Base.OneTo(length(precomputed_arguments))
    j = mod(i, axs)
    precomputed_arguments[j]
end

# Use a global to prevent unexpected compiler optimization.
repeat_count::Int = 1000000

using ProfileCanvas: @profview

# Get everything compiled. The profile may (?) include compilation
# here, so we're not interested in it yet.
run_in_a_loop(cos, get_argument, repeat_count)
@profview run_in_a_loop(cos, get_argument, repeat_count)

# Run again to see run-time performance.
repeat_count *= 1000
@profview run_in_a_loop(cos, get_argument, repeat_count)

Here's the profile flame graph I ended up with:

So about 85% of the samples do end up in cos, even though a single call of cos(::Float64) is over very quickly. Not bad!

`donotdelete`

The secret sauce above is donotdelete. This is its doc string:

help?> Base.donotdelete
  │ Warning
  │
  │  The following bindings may be internal; they may change or be removed in future versions:
  │
  │    •  Base.donotdelete

  Base.donotdelete(args...)

  This function prevents dead-code elimination (DCE) of itself and any arguments passed to it, but is
  otherwise the lightest barrier possible. In particular, it is not a GC safepoint, does not model an
  observable heap effect, does not expand to any code itself and may be re-ordered with respect to
  other side effects (though the total number of executions may not change).

  A useful model for this function is that it hashes all memory reachable from args and escapes this
  information through some observable side-channel that does not otherwise impact program behavior. Of
  course that's just a model. The function does nothing and returns nothing.

  This is intended for use in benchmarks that want to guarantee that args are actually computed.
  (Otherwise DCE may see that the result of the benchmark is unused and delete the entire benchmark
  code).

  │ Note
  │
  │  donotdelete does not affect constant folding. For example, in donotdelete(1+1), no add
  │  instruction needs to be executed at runtime and the code is semantically equivalent to
  │  donotdelete(2).

  │ Note
  │
  │  This intrinsic does not affect the semantics of code that is dead because it is
  │  unreachable. For example, the body of the function f(x) = false && donotdelete(x) may be
  │  deleted in its entirety. The semantics of this intrinsic only guarantee that if the
  │  intrinsic is semantically executed, then there is some program state at which the value of
  │  the arguments of this intrinsic were available (in a register, in memory, etc.).

  │ Julia 1.8
  │
  │  This method was added in Julia 1.8.

  Examples
  ≡≡≡≡≡≡≡≡

  function loop()
      for i = 1:1000
          # The compiler must guarantee that there are 1000 program points (in the correct
          # order) at which the value of `i` is in a register, but has otherwise
          # total control over the program.
          donotdelete(i)
      end
  end

Similar functionality in Rust:

https://doc.rust-lang.org/beta/std/hint/fn.black_box.html

Some C++ libraries offer similar functionality, too. For example, the folly library has doNotOptimizeAway.

EDIT: Base.donotdelete is now a public interface:

https://github.com/JuliaLang/julia/pull/55774

Julia expertise needed at the University of Glasgow

Eric Silverman — Fri, 16 May 2025 09:57:03 +0000

Hi everyone! I'm running a 5-year research project on computational modelling for public health, during which I'll be using an agent-based modelling framework developed previously in Julia. I'm looking for someone to join me as a Research Associate for the duration, to work alongside me to build out this framework and use it to help Scotland develop improved public health policies.

Please note that Julia experience and a PhD are required for this position. You can apply here: https://www.jobs.gla.ac.uk/job/research-associate-46?source=gla.ac.uk

If you have any questions, please contact me anytime: https://www.gla.ac.uk/schools/healthwellbeing/staff/ericsilverman/

Convex hull of a Cartesian product

WalterMadelim — Fri, 16 May 2025 02:43:45 +0000

I find that this proposition is hardly found in textbooks. And its proof is not easily found on the internet (including asking ChatGPT). Therefore I write it down here as a reference.

function co(X::Set) return convex_hull(X) end
[Proposition1]   co(X) × co(Y) == co(X × Y) # "×" is the times---Cartesian Product symbol
[Corollary]  ⨉_i co(S_i) = co(⨉_i S_i), # "⨉" is the bigtimes---Cartesian Product symbol

Proof:
Assume there is a point denoted by (x, y) ∈ co(X) × co(Y), where
x ∈ co(X) with x = _p * _x + p_ * x_ for some
_x, x_ ∈ X and _p, p_ ≥ 0 and _p + p_ == 1;
y ∈ co(Y) with y = _q * _y + q_ * y_ for some
_y, y_ ∈ Y and _q, q_ ≥ 0 and _q + q_ == 1;
We need to prove that (x, y) ∈ co(X × Y).
Indeed, there are 4 points in X × Y as follows
# point => its coefficient in a convex combination
(_x, _y) => _p * _q 
(_x, y_) => _p * q_
(x_, _y) => p_ * _q
(x_, y_) => p_ * q_
such that their convex combination is precisely (x, y).
Therefore we've proved (x, y) ∈ co(X × Y).
The reverse direction is easier, hence omitted.
Therefore [Proposition1] holds.
Furthermore, co(X) × co(Y) × co(Z)
= co(X × Y) × co(Z)
= co(X × Y × Z).
Likewise, we conclude that [Corollary] holds.

NeoVim config to use as an IDE for Julia

Arash — Wed, 12 Feb 2025 19:12:20 +0000

Hello Julia community. i modified my nVim to use as my IDE because i though vscode is too boring and zed is still under development. I shared it on Github so i hope you like it. Just if you have some nVim config before take care and get backup before changing. It is working on Arch linux right now. so I would like to give me some response if its working fluently.

https://github.com/arashshakeri/NeoJulia

Generate Prime Pairs for Goldbach's Conjecture

Jabari Zakiya — Fri, 07 Feb 2025 23:48:50 +0000

Hi.

This is my first program of any substance done in Julia.
It was told Julia is easy to program in, and fast, so I gave it a try.

The code works (gives correct results) but I'm looking for help to improve it.
It generates all the prime pairs that sum to any even integer n.

If you want an explanation of hows|whys it works read my just released paper.

Proof of Goldbach's Conjecture and Bertrand's Postulate Using Prime Genrator Theory(GPT)

https://www.academia.edu/127404211/Proof_of_Goldbachs_Conjecture_and_Bertrands_Postulate_Using_Prime_Generator_Theory_PGT_

Here's the Ruby code from the paper, which is much faster than my Julia code.

# Enable YJIT if using CRuby >= 3.3"
RubyVM::YJIT.enable if RUBY_ENGINE == "ruby" and RUBY_VERSION.to_f >= 3.3

def prime_pairs_lohi(n)
  return puts "Input not even n > 2" unless n.even? && n > 2
  return (pp [n, 1]; pp [n/2, n/2]; pp [n/2, n/2]) if n <= 6

  # generate the low-half-residues (lhr) r < n/2
  lhr = 3.step(n/2, 2).select { |r| r if r.gcd(n) == 1 }
  ndiv2, rhi = n/2, n-2            # lhr:hhr midpoint, max residue limit
  lhr_mults = []                   # for lhr values not part of a pcp

  # store all the powers of the lhr members < n-2
  lhr.each do |r|                  # step thru the lhr members
    r_pwr = r                      # set to first power of r
    break if r > rhi / r_pwr       # exit if r^2 > n-2, as all others are too
    while    r < rhi / r_pwr       # while r^e < n-2
      lhr_mults << (r_pwr *= r)    # store its current power of r
    end
  end

  # store all the cross-products of the lhr members < n-2
  lhr_dup = lhr.dup                # make copy of the lhr members list
  while (r = lhr_dup.shift) && !lhr_dup.empty? # do mults of 1st list r w/others
    ri_max = rhi / r               # ri can't multiply r with values > this
    break if lhr_dup[0] > ri_max   # exit if product of consecutive r’s > n-2
    lhr_dup.each do |ri|           # for each residue in reduced list
      break if ri > ri_max         # exit for r if cross-product with ri > n-2
      lhr_mults << r * ri          # store value if < n-2
    end                            # check cross-products of next lhr member
  end

  # convert lhr_mults vals > n/2 to their lhr complements n-r,
  # store them, those < n/2, in lhr_del; it now holds non-pcp lhr vals
  lhr_del = lhr_mults.map { |r_del| (r_del > ndiv2 ? n - r_del : r_del) }

  pp [n, (lhr -= lhr_del).size]    # show n and pcp prime pairs count
  pp [lhr.first, n-lhr.first]      # show first pcp prime pair of n
  pp [lhr.last,  n-lhr.last]       # show last  pcp prime pair of n
end

def tm; t = Time.now; yield; Time.now - t end  # to time runtime execution

n = ARGV[0].to_i                   # get n value from terminal
puts tm { prime_pairs_lohi(n) }    # show execution runtime as last output

Here's my Julia (functional) translation.
I know there's gotta be a faster, more elegant, idomatic way to do it.

function prime_pairs_lohi(n)
  if (n&1 == 1 || n < 4) return println("Input not even n > 2") end
  if (n <= 6) println([n, 1]); println([div(n,2),div(n,2)]); println([div(n,2),div(n,2)]); return end

  # generate the low-half-residues (lhr) r < n/2
  ndiv2, rhi = div(n,2), n-2         # lhr:hhr midpoint, max residue limit
  lhr = []                           # array for low half residues
  for r in 3:2:ndiv2; if (gcd(r, n) == 1) append!(lhr, r) end end
  lhr_mults = []                     # for lhr values not part of a pcp

  # store all the powers of the lhr members < n-2
  for r in lhr                       # step thru the lhr members
    r_pwr = r                        # set to first power of r
    if (r > div(rhi, r_pwr)) break end # exit if r^2 > n-2, as all others are too
    while (r < div(rhi, r_pwr))      # while r^e < n-2
      r_pwr *= r                     # increase power of r
      append!(lhr_mults, r_pwr)      # store its current power of r
    end
  end

  # store all the cross-products of the lhr members < n-2
  i = 1; ln = length(lhr)
  while (i < ln)
    r = lhr[i]
    rmax = div(rhi, r)               # ri can't multiply r with values > this
    if (lhr[i+1] > rmax) break end   # exit if product of consecutive r’s > n-2
    j = i+1
    while (j < ln)                   # for each residue in reduced list
      ri = lhr[j]
      if (ri > rmax) break end       # exit for r if cross-product with ri > n-2
      append!(lhr_mults, r * ri)     # store value if < n-2
      j += 1
    end
    i += 1
  end

  # convert lhr_mults vals > n/2 to their lhr complements n-r,
  # store them, those < n/2, in lhr_del; it now holds non-pcp lhr vals
  lhr_del = []
  for r in lhr_mults
    r_del = (r > ndiv2) ? n - r : r
    append!(lhr_del, r_del)
  end

  lhr = setdiff(lhr,lhr_del)         # remove lhr_del residues from lhr

  println([n,  length(lhr)])         # show n and pcp prime pairs count
  println([first(lhr),n-first(lhr)]) # show first pcp prime pair of n
  println([last(lhr), n-last(lhr)])  # show last  pcp prime pair of n
end


num = readline()                     # get n value from terminal
n = parse(Int64, num)
prime_pairs_lohi(n)                  # show execution runtime as last output
println()

It computes correct answers but:
1) It's slower than the Ruby version.
2) Taking input directly from the cli isn't the same as Ruby
3) It doesn't take input numbers using underscores: 123_456_780
4) It needs internal timer like Ruby

Here a timing difference example.

➜  ~ ruby prime_pairs_lohi.rb 100_000_000
[100000000, 291400]
[11, 99999989]
[49999757, 50000243]
24.21507651

➜  ~ time julia prime_pairs_lohi.jl
100000000
[100000000, 291400]
[11, 99999989]
[49999757, 50000243]

julia prime_pairs_lohi.jl  39.54s user 3.68s system 87% cpu 49.275 total

So I'm asking help to improve performance and make entering numbers the same as Ruby.
I used the latest Julia 1.11.3 and Ruby 3.4.1 w/YJIT

🌟 Excited to Share: Introducing AXE Lang! 🌟

Sahith Karra — Tue, 07 Jan 2025 21:13:26 +0000

Hi everyone! I’m thrilled to announce my latest project: AXE Lang, an expressive functional programming language hosted in Julia, designed for rapid development and educational use

This journey began in CSC 430: Programming Languages, where I explored the fascinating world of parsers and interpreters, ultimately implementing my first programming language. That class ignited a passion for understanding how languages work at their core.
Recently, I took this knowledge further, synthesizing key concepts from the course and building my own language from scratch. By hosting AXE Lang in Julia, I was able to combine Julia’s high-performance features with the flexibility and elegance of functional programming. Here are some of my key takeaways from developing AXE.

💻 The Power of Function Programming
Working with various function programming languages while designing AXE deepened my appreciation for functional paradigms. I found these languages expressive and incredibly fun to work with and urge every programming student to explore this side of programming.

♻️ Iterative Design
Building a language taught me that iteration was key. It took my multiple iterations in hosting languages before I landed on Julia as the best bet to host my programming language. Additionally, iterating on the syntax and grammar of my language taught me to balance expressiveness with simplicity. For example, I decided to remove the concept of named functions and opted for anonymous functions (lambdas) in my language to promote simplicity and expressiveness, reducing the syntactic overhead.

🏔️ Development Challenges
Some of the hardest parts of this project was developing the interpreter for a function application where concepts such as lexical scoping and environment management became paramount. These concerns were only heightened after the implementation of mutation in the language, which would introduce side effects to my language. You can read more the strategies while developing this language in the GitHub linked below.

⏭️ Next Steps with AXE
https://github.com/sahkar/axe-lang
If you are interested in AXE Lang, please clone this GitHub repository which provides access to AXE's source code as well as a local interactive REPL to learn AXE syntax and features.

Please reach out if you have any questions about AXE, Julia, or anything else. Enjoy!

Categorical Encoding in Julia

Essam — Fri, 23 Aug 2024 23:38:46 +0000

In this post, I will introduce the Google Summer of Code project that I have been involved in with my mentor @ablaom for the past three months. It's a new package MLJTransforms.jl which makes five major contributions:

➊ Introducing Contrast Categorical Methods to MLJ

These include Dummy Coding, Sum Coding, Backward/Forward Difference Coding, Helmert Coding as well as generic Contrast/Hypothesis Coding. It's even easily possible to apply different encoding techniques to different columns of an input table!

These are provided via the ContrastEncoder construct implemented in the MLJTransforms.jl package. The motivation behind grouping them is that they can all be viewed as a special case of Hypothesis Coding; the difference from one encoding method to the other is in the hypothesis that the method could be viewed to be testing.

➋ Introducing Other Well Known Categorical Encoding Methods

This includes simpler ones such as Ordinal and Frequency Encoding as well as more sophisticated methods such as Target Encoding. Target encoding supports both binary and multiclass targets and allows regularization to avoid overfitting. It's one of the most renowned and effective methods for categorical encoding.

These are provided via the OrdinalEncoder, FrequencyEncoder and TargetEncoder constructs implemented in the MLJTransformers.jl package.

Prior to this, MLJ only natively supported the OneHotEncoder for categorical encoding.

➌ Introducing Utility Transformers and Encoders

Many machine learning settings can benefit from treating missing values as an extra category for categorical variables. For this, I implemented a Missingness Encoder that can help fill the missing values for the three major data types of categorical variables. It's even quite useful to cascade this with other categorical encoders that may not be able to deal with missingness.

Another issue that often comes up when dealing with categorical features is that when their cardinality is high, a classification model may easily overfit; if 100% of the three examples having category "A" belong to class "X" have then might as well predict "X" whenever the category "A". For this, a CardinalityReducer was implemented to group categories that can be regarded as infrequent.

These are implemented in the MissingnessEncoder and CardinalityReducer constructs in the MLJTransforms.jl package.

➍ Porting Encoders and Transformers from MLJModels.jl

MLJModels.jl already housed a number of transformers as well as a nicely implemented OneHotEncoder. These were ported to MLJTransforms.jl so that all encoder/transformers are in the same package.

➎ Introducing the EntityEmbedder

The scope of the summer project originally required only implementing this. Indeed, it proved to be more challenging than other models I implemented in this project. Entity embedding is newer deep learning approach for categorical encoding introduced in 2016 by Cheng Guo and Felix Berkhahn. It employs a set of embedding layers to map each categorical feature into a dense continuous vector in a similar fashion to how they are employed in NLP architectures.

Consequently, the NeuralNetworkClassifier, NeuralNetworkRegressor and the MultitargetNeuralNetworkRegressor can be trained and evaluated with heterogenous data (i.e., containing categorical features). Moreover, they now offer a transform which encode the categorical features with the learnt embeddings to be used by an upstream machine learning model.

To see what it did take to implement this, see the corresponding PR where I also mentioned the implementation plan I followed.

➕📕 Documentation !

Although my work extended beyond the original scope of the summer project, I considered it worthwhile to contribute further, knowing it could benefit others in the future. With this in mind, I carefully built and organized the documentation page
, creating a taxonomy for the methods I implemented, adding four tutorials to help users learn about categorical encoding techniques, and streamlining the process for other developers to contribute to the codebase.
Do you like the work? See the Imbalance project I worked on last year as well here. Thank you!

🎁 Final Bonus

In the community bonding period, I took some time to expose and revamp documentation for MLJFlux.jl. This included preparing seven workflow examples to present various features of the package as well as a novel tutorial on how to use RNNs for sequence classification via the package. See that and more here.

Why you might avoid `deepcopy` in Julia

Eric Hanson — Sun, 21 Jul 2024 11:45:20 +0000

Why use `deepcopy`?

In Julia, copy is a function which creates a shallow copy. For example:

julia> a = [1] # vector with one element, namely 1
1-element Vector{Int64}:
 1

julia> b = [a] # vector with one element, `a`
1-element Vector{Vector{Int64}}:
 [1]

julia> b2 = copy(b) # new vector, also with one element which is `a`
1-element Vector{Vector{Int64}}:
 [1]

julia> push!(a, 2) # mutate `a` so it contains 1 and 2
2-element Vector{Int64}:
 1
 2

julia> b # since `b` contains `a`, we can see its (nested) contents have changed
1-element Vector{Vector{Int64}}:
 [1, 2]

julia> b2 # same for `b2`!
1-element Vector{Vector{Int64}}:
 [1, 2]

Since copy is shallow, b2 still contains the same vector a (whose contents we modified to be [1,2]), just like b, even though they are independent vectors which do not share memory:

julia> push!(b, [10]) # mutate `b`
2-element Vector{Vector{Int64}}:
 [1, 2]
 [10]

julia> b2 # not mutated
1-element Vector{Vector{Int64}}:
 [1, 2]

In contrast, deepcopy is a function which recursively copies objects:

julia> a = [1] # vector with one element, namely 1
1-element Vector{Int64}:
 1

julia> b = [a] # vector with one element, `a`
1-element Vector{Vector{Int64}}:
 [1]

julia> b2 = deepcopy(b) # new vector, with new contents
1-element Vector{Vector{Int64}}:
 [1]

julia> push!(a, 2) # mutate `a` so it contains 1 and 2
2-element Vector{Int64}:
 1
 2

julia> b
1-element Vector{Vector{Int64}}:
 [1, 2]

julia> b2 # contents are unchanged
1-element Vector{Vector{Int64}}:
 [1]

We can see that b2 still just contains [1].

It's easy to see why deepcopy might be appealing: it could be surprising that modifying a affects b2.

Note also that deepcopy also has some other nice properties. For example, note here that since b contains a twice, modifying a has the following effect on b:

julia> a = [1]
1-element Vector{Int64}:
 1

julia> b = [a, a]
2-element Vector{Vector{Int64}}:
 [1]
 [1]

julia> push!(a, 2)
2-element Vector{Int64}:
 1
 2

julia> b
2-element Vector{Vector{Int64}}:
 [1, 2]
 [1, 2]

deepcopy preserves this internal structure. Continuing the example:

julia> b2 = deepcopy(b)
2-element Vector{Vector{Int64}}:
 [1, 2]
 [1, 2]

julia> a2 = b2[1]
2-element Vector{Int64}:
 1
 2

julia> push!(a2, 3)
3-element Vector{Int64}:
 1
 2
 3

julia> b2
2-element Vector{Vector{Int64}}:
 [1, 2, 3]
 [1, 2, 3]

Pretty nice! Semantically, deepcopy should be the same as composing deserialize and serialize.

Why not use `deepcopy`?

deepcopy is reaching into the internals of the object, rather than relying on the API of the object (namely, its method for copy).

It's easy to construct cases in which this is semantically incorrect. For example, lets say we are constructing our own vector type which stores its memory elsewhere, and stores a token to use to lookup the memory. Here is a quick implementation:

# mutable so each instance has its own identity
mutable struct Token end

const STORAGE = Dict{Token, Vector{Float64}}()

struct MyVectorType <: AbstractVector{Float64}
    token::Token
end
# construction from a `vector`
function MyVectorType(v::Vector)
    token = Token()
    STORAGE[token] = v
    return MyVectorType(token)
end
Base.getindex(m::MyVectorType, i::Int) = STORAGE[m.token][i]
Base.setindex!(m::MyVectorType, v, i::Int) = STORAGE[m.token][i] = v
Base.size(m::MyVectorType) = size(STORAGE[m.token])
function Base.copy(m::MyVectorType)
    return MyVectorType(copy(STORAGE[m.token]))
end

For example:

julia> v = MyVectorType(rand(2))
2-element MyVectorType:
 0.49321258978106763
 0.6022070713363459

Then copy works as expected, since we defined a method for it:

julia> v2 = copy(v)
2-element MyVectorType:
 0.49321258978106763
 0.6022070713363459

julia> v[1] = 2.0
2.0

julia> v
2-element MyVectorType:
 2.0
 0.6022070713363459

julia> v2
2-element MyVectorType:
 0.49321258978106763
 0.6022070713363459

But deepcopy fails:

julia> deepcopy(v)
Error showing value of type MyVectorType:
ERROR: KeyError: key Token() not found
Stacktrace:
  [1] getindex
    @ ./dict.jl:477 [inlined]
  [2] length
    @ ./REPL[6]:1 [inlined]

It has constructed new Token instance which does not have a corresponding entry in STORAGE.

Using deepcopy, we have made assumptions about the implementation details of how MyVectorType works and constructed an invalid instance!

One can run into similar problems when the object contains references to memory allocated in another language, see e.g. JuMP's rationale for disabling deepcopy on its models.

Revisiting reasons to use `deepcopy`

Sometimes, feeling a need to use deepcopy is actually a hint that something else is wrong. I think the main one is missing copy methods, but I believe there might be several other reasons; if you think of one, let me know and I might add it here.

"Missing" `copy` method

I think one of the most common reasons to reach for deepcopy is that one of the objects you are working with is missing a copy method, or it is not copying quite what it should¹.

While copy is defined to be a "shallow" copy, it is not always totally clear exactly how deep or shallow that should be. Adding semantics around whether the object represents a nested or flat datastructure can clarify this.

For example, a DataFrame is semantically a 2D object, which is implemented with a vector-of-vectors similar to our b in the first example. DataFrame defines a copy method which by default does copy the nested inner vectors, but not any of their contents. I believe this is semantically correct, because the vector-of-vectors construction is simply an implementation detail of DataFrame, and the object itself is a flat 2D object (as shown by e.g. size), and therefore copy should not behave like it is a vector-of-vectors.

Sometimes deepcopy usage hints at both a "missing" struct definition and copy method. For example, when using nested dictionaries to store configuration state, one might start with a "default configuration", then deepcopy it to pass to a user to modify. This seems practical and totally fine, but it might be nicer to wrap up the configuration in a struct and define a copy method for it.

Should you use `deepcopy`?

Maybe! It depends on the situation. To me it is often a sign that some abstraction is not working as intended, or a copy method is missing somewhere. But it is a good workaround, and sometimes a useful shortcut in some scenarios like writing tests, and there are situations in which it seems like the correct tool for the job.

Note: in this case, using deepcopy may be the best way to stay unblocked in this situation! I don't think there's anything wrong with using it as a workaround, but it may signal there is an upstream issue somewhere to be filed or fixed. ↩

AIHelpMe.jl Pt.2: Instant Expertise, Infinite Solutions for Julia Developers

Jan Siml — Tue, 30 Apr 2024 08:36:09 +0000

TL;DR

AIHelpMe.jl is a Julia package that harnesses the power of AI models to provide tailored coding guidance, integrating seamlessly with PromptingTools.jl to offer a unique approach to answering coding queries directly in Julia's environment.

The Frustration of Searching for Julia Answers

We've all been there. You're stuck on a problem, and you need some guidance on how to implement a specific functionality in Julia. You head to your trusty search engine, type in your query, and... wait, why are all these results in Python? You tweak your query, adding "Julia" this and "Julia language" that, but still, the results are scattered and unclear.

After 5 minutes of searching, you finally stumble upon a Discourse post from 2018. But wait, is this even relevant to Julia 1.0? Should you bother opening it? You take a deep breath and dive in, hoping that the answer lies within.

Another 5 minutes pass, and you finally find what you're looking for. You copy out the necessary snippet, and after a few more minutes of customizing it to your needs, you have what you needed. The process is tedious, to say the least.

Introduction to AIHelpMe's Official Release

AIHelpMe, a Julia package designed to enhance coding assistance with AI, has been officially registered and released. This milestone makes it easier to install directly from the Julia registry and introduces new functionalities and new knowledge packs! Now, developers can access sophisticated, AI-powered coding guidance more efficiently than ever.

How AIHelpMe Works

AIHelpMe uses a Retrieval Augment Generation (RAG) pattern to provide accurate and relevant answers to your coding questions. It preprocesses the provided documentation, converting text snippets into numerical embeddings. When you ask a question, AIHelpMe looks up the most relevant documentation snippets, feeds them into the AI model, and generates an answer tailored to Julia's ecosystem and best practices.

Get started with:

using Pkg
Pkg.add("AIHelpMe")
using AIHelpMe
aihelp"How do I implement quicksort in Julia?"

Note: It requires at least the OpenAI API key (ENV["OPENAI_API_KEY"]), but I would strongly recommend getting a FREE Cohere API key for re-ranking (silver pipeline).

How AIHelpMe Differs from Other Solutions

Compared to chatbots, AIHelpMe offers several advantages:

Grounded in actual resources: AIHelpMe's answers are based on actual, up-to-date Julia resources, not outdated training data.
Customizable knowledge: You choose what knowledge to include, allowing you to improve precision and recall.
Flexible trade-offs: You choose the trade-off between cost, performance, and time, giving you greater control over your coding experience.
State-of-the-art RAG methods: AIHelpMe leverages the latest RAG methods, and full customization is possible, ensuring that you get the most accurate and relevant answers.

Note: Only a limited number of packages have been pre-processed so far: Julia docs, Tidier ecosystem, and Makie ecosystem. It's still experimental, but it works! Load them with:

Starter Example

using AIHelpMe
using AIHelpMe: pprint, last_result

# ideally, switch to better pipeline for proper results, requires setting up Cohere API key
AIHelpMe.update_pipeline!(:silver)

# load tidier index, others available: :julia, :makie
AIHelpMe.load_index!(:tidier);

# Ask a question
aihelp"How do you add a regression line to a plot in TidierPlots?"

Or show the highlighted answer (you can customize to add the actual source docs, or remove the scores/highlights):

# See highlighted answer (optional)
pprint(last_result())

If you're an infrequent user of AlgebraOfGraphics like me, you'll certainly appreciate the :makie knowledge pack -- it's much faster now to find the right keywords to customize your plot!

# Load all knowledge packs
AIHelpMe.load_index!([:julia,:makie,:tidier]);
aihelp"How to set the label of a y-axis in Makie?"gpt3t # testing a weak model
pprint(last_result())

Note: There can still be some issues with the quality of the answers (it's GenAI!), especially for the bronze pipeline and weaker models, but, hopefully, it's already good enough to create value for you!

Advanced Usage of AIHelpMe

AIHelpMe offers several advanced features for experienced users wanting more control and deeper insights:

Response Insights:

Use pprint to highlight potential hallucinations, show sources, their scores, and context snippets.
Example: aihelp("Explain Julia's multiple dispatch system", return_all=true)|>AIHelpMe.pprint

Note: You'll always get better responses with better pipelines - see below.

Customizing the AI Pipeline:

Adjust the complexity of AI responses with update_pipeline! choosing from bronze, silver, or gold levels.
Specify AI models, including local options like Ollama.
Example: AIHelpMe.update_pipeline!(:silver; model="gllama370") # gllama370 is Groq.com-hosted Llama 3 70b that you can access for free!

Safe Code Execution:

Execute AI-generated code safely with PromptingTools.AICode struct.
Example: aihelp("How to create a named tuple from a dictionary?")|>PromptingTools.AICode

For more details on these advanced features, please refer to the AIHelpMe Advanced Documentation.

What's Next

Over the summer, we hope to optimize the performance (in terms of quality) and add more knowledge packs.

We're also working on making it super easy for you to develop your own knowledge packs for the packages you use, regardless of whether they're public or private. This will enable you to tailor AIHelpMe to your specific needs and workflow.

Conclusion

AIHelpMe is the solution to your Julia coding woes. With its AI-powered assistance, flexible querying system, and cost-effective approach, AIHelpMe is poised to revolutionize the way you code in Julia. Try it out today and experience the future of coding assistance!

Credit for the title image: DALL-E 3.

ProToPortal: The Portal to the Magic of PromptingTools

Jan Siml — Sun, 28 Apr 2024 16:02:47 +0000

TL;DR

ProToPortal streamlines your interaction with any LLM tasks on the go, offering customizable templates for automatic replies, direct code evaluation, and an intuitive, multi-device interface. Explore its full capabilities and enhance your productivity on GitHub.

Unveiling ProToPortal: A Julia-First LLM GUI built with Stipple.jl!

Hello, fellow GenAI enthusiasts! Today, I'm thrilled to introduce ProToPortal, a nifty tool born from my own need to simplify my daily interactions with Julia and AI models. It's a small but mighty project aimed at boosting productivity and minimizing those pesky prompting hassles.

Why ProToPortal?

Ever found yourself on a leisurely walk with your dog, struck by a sudden coding inspiration that just couldn't wait? That's exactly where ProToPortal comes into play. This tool isn't just another coding interface; it's your on-the-go, in-your-pocket coding companion, ready to tackle tasks from simple prompts to complex code evaluations—all before you've even finished your walk!

Cool Features to Check Out:

Accessible Anywhere: Whether you're on a train or in your comfy home office, ProToPortal is there. It works seamlessly across all devices, ensuring that your brilliant ideas never slip away just because you're not at your desk.
Code Evaluation and Fixing: Forget about flipping between screens to debug. ProToPortal lets you directly evaluate and fix Julia code within the GUI. Imagine tweaking your code with just a few clicks—yes, it's that easy!
Automatic Replies: Streamline your workflow even further with automated responses. Set up ProToPortal to handle repetitive tasks, leaving you more time to focus on the creative aspects of coding.

More Handy Features:

Editing Code Cells: Quickly edit any of the messages right within the chat tab—just click and modify.
Deleting Code Cells: Made a mistake? No worries! Easily remove any unwanted messages.
Saving Conversations: Keep a history of your sessions for future reference or continued experimentation.

For a complete list of features, including detailed explanations and how-tos, be sure to check out ProToPortal's Documentation.

See It in Action:

Curious to see how it works in real time?

Check out this video where it auto-fixes the generated code: Code fixing recording (webm format so not all browsers might play it).

And here's another cool feature—watch how ProToPortal handles editing a conversation (gif, easy to play): Editing a conversation.

Give It a Try!

Ready to dive in? Visit ProToPortal's GitHub to get started. It's open source, and I'm eager to hear your thoughts or see your contributions!

Thanks for checking out ProToPortal. Happy coding, and let the code be with you!

Big thank you to the Genie.jl team! This tool wouldn't exist without their amazing packages!

Credit for the title image goes to DALL-E 3.

Julia Community 🟣

DearDiary.jl: A lightweight but powerful machine learning experiment tracking tool for Julia

Motivation

Core concepts

Architecture-first

Simple types

Flexible by design

Portability

Getting started

Contributing

StaticCompiler - Generating small binaries

Profile a short-running function

Example: profiling cos(::Float64)

donotdelete

Julia expertise needed at the University of Glasgow

Convex hull of a Cartesian product

NeoVim config to use as an IDE for Julia

Generate Prime Pairs for Goldbach's Conjecture

🌟 Excited to Share: Introducing AXE Lang! 🌟

Categorical Encoding in Julia

➊ Introducing Contrast Categorical Methods to MLJ

➋ Introducing Other Well Known Categorical Encoding Methods

➌ Introducing Utility Transformers and Encoders

➍ Porting Encoders and Transformers from MLJModels.jl

➎ Introducing the EntityEmbedder

➕📕 Documentation !

🎁 Final Bonus

Why you might avoid `deepcopy` in Julia

Why use deepcopy?

Why not use deepcopy?

Revisiting reasons to use deepcopy

"Missing" copy method

Should you use deepcopy?

AIHelpMe.jl Pt.2: Instant Expertise, Infinite Solutions for Julia Developers

TL;DR

The Frustration of Searching for Julia Answers

Introduction to AIHelpMe's Official Release

How AIHelpMe Works

How AIHelpMe Differs from Other Solutions

Starter Example

Advanced Usage of AIHelpMe

What's Next

Conclusion

ProToPortal: The Portal to the Magic of PromptingTools

TL;DR

Unveiling ProToPortal: A Julia-First LLM GUI built with Stipple.jl!

Why ProToPortal?

Cool Features to Check Out:

More Handy Features:

See It in Action:

Give It a Try!

Example: profiling `cos(::Float64)`

`donotdelete`

Why use `deepcopy`?

Why not use `deepcopy`?

Revisiting reasons to use `deepcopy`

"Missing" `copy` method

Should you use `deepcopy`?