Julia Community 🟣: Saransh

My experience working as a Technical Writer with FluxML - Part 2

Saransh — Sun, 06 Nov 2022 14:28:40 +0000

“One thing that open-source can’t get enough of is documentation”

— Anonymous

I have finally completed my official period as a Technical Writer at FluxML, and I have enjoyed every second of it! This blog serves as a summary of the work done during the second half of my time at FluxML. You'd notice that the contributions listed in this blog would cover the entire FluxML ecosystem and not just Flux.jl. Furthermore, I will also list some additional documentation contributions made to the much larger Julia ecosystem during this period!

A quick introduction to FluxML, quoting their documentation -

Flux is a 100% pure-Julia stack and provides lightweight abstractions on top of Julia's native GPU and AD support. It makes the easy things easy while remaining fully hackable.

Flux is a library for machine learning geared towards high-performance production pipelines. It comes "batteries-included" with many useful tools built in, but also lets you use the full power of the Julia language where you need it.

Honestly, the nature of my work at FluxML diverged so quickly that summarising it would be a difficult task, but I will try my best! Buckle up! Psst, you can find more about the first half of the time spent at FluxML here.

Getting started section

Let's start from where I left the last blog - the new "Getting Started" section. The good news is that Flux has a better structured "Getting Started" section now! The bad news is that I did not work on it, but the second good news is that it turned out better than the one I had in mind!

Flux had a lot of tutorials and examples, but they were scattered around and very hard to navigate through. The old “Getting Started” section, “Overview” section, and “Basics” section had valuable information for beginners, but the information was scattered among these three sections. Now, all of these sections have been combined (a huge thanks to @mcabbott) in a single "Getting Started" section in Flux's docs! We still have a Getting Started page on Flux's website, but it will be either migrated or scrapped very soon!

Have a look at the new "Getting Started" section here.

Tutorials section

The new (and different than the one I imagined) "Getting Started" prompted a discussion on revamping the existing "Tutorials" section present on FluxML's website. This section is currently heavily outdated and is not updated very regularly, which often misguides or scares newcomers.

We are planning to migrate these tutorials to Flux's documentation to keep them up-to-date using doctests. The contents I initially planned for the new "Getting Started" section are now planned to go into the revamped "Tutorials" section.

Have a look at the new "Tutorials" section here.

Independent docs for NNlib.jl

NNlib now has standalone docs! NNlib's documentation used to live as an independent page under Flux's documentation, which was not helpful, as NNlib is an independent Julia package. This was not convenient for both the users and the developers of NNlib, as one had to constantly refer to the section in Flux's documentation to get NNlib working.

My work here started with migrating the existing page from Flux's documentation to NNlib, which was then iterated through helpful reviews! This page was then included in a brand new documentation infrastructure, which was merged way back, but the documentation did not appear live until a few days back. The delay in the deployment of the documentation was caused by a few minor Documenter.jl and DOCUMENTER_KEY related bugs which have been resolved.

Have a look at NNlib's updated and independent documentation here.

Revamped docs for Metalhead.jl

Metalhead.jl saw a great amount of development this summer, thanks to a GSoC project led by @theabhirath and @darsnack. With all of this development, it was required to revamp the old Publish.jl infrastructure to a modern Documenter.jl infrastructure. This invited numerous discussions from FluxML maintainers, given that both the packages have their ups and downs, but in the end, everyone agreed to migrate Metalhead's docs to Documenter.jl.

I aimed for a 1:1 port, not adding any information and not removing any information in the porting process, and the new documentation looks beautiful!

Have a look at Metalhead's revamped documentation here.

Better community health

I have noticed that Julia's ecosystem can be a bit daunting for new contributors because of the missing "Community Health" files. Most of the repositories under FluxML lacked issue and PR templates, making the review and triage process harder than it should be. FluxML also lacked an organization-wide README, recently introduced by GitHub.

FluxML now has default issues and PR templates, which can be overridden by every repository! The organization-wide README is also in its place, guiding newcomers better than ever!

Have a look at the default issue templates here.

Have a look at the organization-wide README here.

Create a PR in a repository under FluxML to see the default PR template 😉

Better docs for OneHotArrays.jl

Flux.jl maintainers recently decided to move the "One Hot Encoding" functionalities of Flux into a separate independent package. As you might have guessed, this called for some documentation audits!

I have updated and added the docstrings of the OneHot* structs to OneHotArray's manual, making them accessible to the users. I also debugged and helped to build the missing v0.1.0 documentation of OneHotArrays.jl!

Have a look at OneHotArray's v0.1.0 documentation here.

Migrating FluxML's website to Franklin.jl

Franklin.jl is a modern static site generator written in Julia! Quoting their documentation -

Franklin is a simple, customisable static site generator oriented towards technical blogging and light, fast-loading pages.

FluxML maintainers decided to port the existing FluxML website (built using Jekyll) to Franklin, and this was primarily carried out by @logankilpatrick and @darsnack. Where do I come in? I helped set up the infrastructure of the new site! I primarily worked on enabling PR previews using GitHub Actions, something not documented in Franklin's documentation (and has not been done before). I also fixed some minor bugs revolving around relative paths and production deployment!

Have a look at the migrated website here.

Adding a new section to Franklin.jl's docs 😉

Referring to the section above, I decided to update Franklin.jl's documentation to include a section on deploying PR previews using GitHub Actions. This eliminated the need for extra infrastructure and also lead to restructuring the original "deploy" documentation of Franklin. The revamped "deploy" documentation provides much more options to a user and is easier to navigate through.

I also took the liberty to update outdated documentation of Franklin, including outdated GitHub Actions and variable definitions!

Have a look at the revamped deployment page here.

Misc

I cannot possibly think of writing this section without missing out on something. I worked extensively on miscellaneous issues, not limited only to Flux.jl, but also related to FluxML as a whole. These miscellaneous issues ranged from refining existing FluxML's logos to developing and fixing CI/CD pipelines of FluxML packages.

Additionally, while working with Franklin.jl, I discovered that Julia's website is open-source and written in Franklin! I took this opportunity to update some outdated Actions and syntax on the website. Currently, I am also looking at migrating Julia's website to the new GitHub Pages infrastructure, or the new PR preview infrastructure (mentioned in the section above).

Refer to my GitHub for all the "misc" work carried out by me during the past six months 🙂

Minor code contributions 😉

Surprisingly, I also contributed to Flux's code. The code contributions weren't in the form of feature additions, instead, I worked on minor bugs and refined the public API.

For instance, I deprecated rng_from_array() in the favor of default_rng_value(), and then marked it as @non_differentiable. These contributions weren't too big, but adding code to Flux's repository did feel good. I look forward to writing more code for FluxML in the future!

Final words

I will be forever grateful to the whole FluxML community for giving me such a wonderful time! I do feel like I have accomplished a bit in the last 6 months, and the documentation of FluxML is at a better place than the point when I started.

A special thanks to @DhairyaLGandhi, @ToucheSir, @mcabbott, and @darsnack for bearing with me through my untidy PRs and helping me out as always!

Oh, I also joined FluxML's GitHub organisation a few weeks back! 🥳

Tweets

https://twitter.com/saranshchopra7/status/1581569050555330560

Appendix

PRs and issues

Either I created these PRs/issues, or I had some significant involvement in them 😄

Flux.jl

Metalhead.jl

Migrate docs to Documenter.jl

NNlib.jl

OneHotArrays.jl

fluxml.github.io

Franklin.jl

www.julialang.org

.github

My experience working as a Technical Writer with FluxML - Part 1

Saransh — Sun, 06 Nov 2022 14:26:51 +0000

“One thing that open-source can’t get enough of is documentation”

— Anonymous

This summer, I started working as a technical writer with FluxML under Julia Season of Contributions, and as expected, this experience was very different from writing code.

During the beginning of the summer, I decided to take up a technical writer’s job that involved writing documentation and tutorials for Machine Learning. At the same time, I was learning Julia, and the FluxML ecosystem sounded like a perfect place for me. I applied for the position through Google Season of Docs but unfortunately couldn’t get in because of limited openings. Fortunately, the Julia Language decided to fund me for the next few months to work on FluxML under Julia Season of Contributions!

In the following blog, I will share my experience and the work I have done so far as a part of Julia Season of Contributions!

Doctests

Flux’s documentation lacked doctests, making its code examples stale after each release. Further, many current examples thought to be covered by doctests, were already outdated.

Some of the doctests were straightforward to add, but most of them require in-depth discussions with the mentors and the community members. In addition, the documentation written in markdown format also required periodic testing to ensure that the changing API does not break the examples.

Flux now has doctests written for every single public API, and most of the markdown example are also covered under these doctests. Some of these changes haven’t been merged yet, but they are under review!

Missing docstrings

Some of Flux’s public API was undocumented, or had a relatively less clear documentation. For example, most of the neural network layers provided by Flux have two constructors — one for initializing layers with pre-defined weights and biases and another for generating weights and biases from a distribution. In most cases, only one constructor was documented, and the other was not.

Other such instance of missing docstrings was Flux‘s manual. Here the docstrings were present in the codebase, but they were not present in the manual. Such cases were solved by adding the docstrings to the manual or creating a new section for such missing docstrings.

In addition to Flux, other packages under FluxML also had a similar problem. Some of these packages like NNLib.jl, Zygote.jl, Optimisers.jl, Functors.jl, and MLUtils.jl (under JuliaML) were also referenced in Flux‘s documentation. The missing docstrings in these packages were transmitted to Flux‘s docs, resulting in even more missing docstrings.

All the missing docstrings have now been added to the respective manuals and functions, including that of other FluxML packages!

Broken documentation

Documentation is incomplete without cross-references, and Julia’s wonderful documentation package makes this super smooth. Flux had these cross-references in place, but some of them lead to nowhere.

Additionally, some of the docstrings were not rendered correctly in the manual, making the newcomers stray off from the ecosystem. These instances of broken documentation have been fixed, and once the PR is merged, users should not see 404 pages or un-rendered docstrings anymore!

CI/CD

The user-facing documentation of Flux is facilitated by a CI/CD service, which keeps this documentation deployed and available to users at all times. It is common for open-source projects to open-source their deployment recipes and work on it collaboratively.

FluxML‘s ecosystem had some minor issues in this CI service. For instance - Zygote.jl had doctests running twice, using twice the CI time and resources. On the other hand, Flux specified different versions of Julia in its CI and make.jl file, making the CI environment very ambiguous.

Further, Documenter.jl allows a user to generate documentation previews for pull requests, and Flux had a bot set up to facilitate this, but it was down for a long time. These documentation previews are collected in the gh-pages branch, which was getting very bulky and had to be cleaned up with an automated workflow.

I have restarted this bot and have added a workflow for cleaning these generated previews periodically! I have also fixed some issues in Optimisers.jl‘s documentation that were causing its CI to fail.

FluxML’s ecosystem

The FluxML ecosystem, in addition to Flux.jl, also had some documentation problems. For instance, Optimisers.jl and Zygote.jl had shortcomings in their CI suite, which have been discussed above. Similarly, Functors.jl and MLUtils.jl had broken documentation, missing docstrings, and an incomplete manual, which has also been discussed in great detail above!

Lastly, the “ecosystem” pages present on Flux‘s website and documentation were out of sync and had redundant information. The “ecosystem” page was completely revamped and updated with the latest advancements related to Machine Learning in Julia!

Minor code changes

Yes! The documentation changes were also accompanied by minor code changes!

Flux‘s public and internal API is very hard to differentiate if you are new to the codebase; hence, newcomers used to find it hard to navigate through this. The API was made clear by removing internal instances from the documentation and prepending an underscore to the internal API.

Furthermore, I also stumbled upon a bug while writing doctests for tversky loss. The tversky loss has two parameters, α, and β, and Flux internally calculates the value of α as 1-β. The loss mathematically is defined as 1-tversky index, and the tversky index mathematically is defined as:

S(P, G; α, β) = |P G| / (|P G| + α|P \ G| + β|G \ P|)

where α and β control the magnitude of penalties for FPs and FNs, respectively.

Flux implements the loss in the following way -

1 — sum(|y .* ŷ| + 1) / (sum(y .* ŷ + β*(1 .- y) .* ŷ + (1 — β)*y .* (1 .- ŷ)) + 1)

with the following code -

"""
     tversky_loss(ŷ, y; β = 0.7)

 Return the [Tversky loss](https://arxiv.org/abs/1706.05721).
 Used with imbalanced data to give more weight to false negatives.
 Larger β weigh recall more than precision (by placing more emphasis on false negatives)
 Calculated as:
     1 - sum(|y .* ŷ| + 1) / (sum(y .* ŷ + β*(1 .- y) .* ŷ + (1 - β)*y .* (1 .- ŷ)) + 1)
 """
 function tversky_loss(ŷ, y; β = ofeltype(ŷ, 0.7))
     _check_sizes(ŷ, y)
     #TODO add agg
     num = sum(y .* ŷ) + 1
     den = sum(y .* ŷ + β * (1 .- y) .* ŷ + (1 - β) * y .* (1 .- ŷ)) + 1
     1 - num / den
 end

Notice how the term (1 .- y) .* ŷ (False Positives) is multiplied by β, whereas it should be multiplied with α (which is 1-β). Similarly, the term y .* (1 .- ŷ) is multiplied with α (that is 1-β), whereas it should be multiplied with β.

This detail makes the loss function behave in a manner opposite to its documentation. For example -

julia> y = [0, 1, 0, 1, 1, 1];

julia> ŷ_fp = [1, 1, 1, 1, 1, 1]; # 2 false positive -> 2 wrong predictions

julia> ŷ_fnp = [1, 1, 0, 1, 1, 0]; # 1 false negative, 1 false positive -> 2 wrong predictions

julia> Flux.tversky_loss(ŷ_fnp, y)
0.19999999999999996

julia> Flux.tversky_loss(ŷ_fp, y) # should be smaller than tversky_loss(ŷ_fnp, y), as FN is given more weight
0.21875

Here the loss for ŷ_fnp, y should have been larger than the loss for ŷ_fp, y as the loss should give more weight or penalize the False Negatives (default β is 0.7; hence it should give more weight to FN), but the exact opposite happens.

Changing the implementation of the loss yields the following results -

julia> y = [0, 1, 0, 1, 1, 1];

julia> ŷ_fp = [1, 1, 1, 1, 1, 1]; # 2 false positive -> 2 wrong predictions

julia> ŷ_fnp = [1, 1, 0, 1, 1, 0]; # 1 false negative, 1 false positive -> 2 wrong predictions

julia> Flux.tversky_loss(ŷ_fnp, y)
0.19999999999999996

julia> Flux.tversky_loss(ŷ_fp, y) # should be smaller than tversky_loss(ŷ_fnp, y), as FN is given more weight
0.1071428571428571

which looks right!

(This bug has not been confirmed by someone other than me yet, and the fix is still under review.)

Getting started section

Flux has a lot of tutorials and examples, but they are scattered around and are very hard to navigate through. The current “Getting Started” section, “Overview” section, and “Basics” section have valuable information for beginners, but the information is scattered among these three sections.

Additionally, one of these three sections is on Flux’s website, and two are available on the documentation website, making it difficult for newcomers to navigate between these hackable yet basic examples. Instead, these three tutorials could be moved out of their current places and combined under a single section named “Getting Started”, which could then be added to the documentation and linked on the website.

I have started working on this section, and two extensive tutorials have already been added as PRs: one on linear regression (with and without Flux) and one on logistic regression (with and without Flux). These PRs are currently under review and should be merged in the upcoming weeks.

I have had an incredible time contributing to Flux and its neighbor repositories, and I hope to continue these contributions with the same momentum. I have also learned a lot, including that documentation additions require more discussions than code additions.

This work wouldn’t have been possible without my mentor @DhairyaLGandhi and a lot of other FluxML’s maintainers (@ToucheSir, @mcabbott, @CarloLucibello, @darsnack). They have been very patient with my questions and messy PRs 😆

Tweets

https://twitter.com/saranshchopra7/status/1532365616610672640

Appendix

Pull requests

Flux.jl

Issues / discussions

Flux.jl

Zygote.jl

Doctests running twice

Julia Community 🟣: Saransh

My experience working as a Technical Writer with FluxML - Part 2

Getting started section

Tutorials section

Independent docs for NNlib.jl

Revamped docs for Metalhead.jl

Better community health

Better docs for OneHotArrays.jl

Migrating FluxML's website to Franklin.jl

Adding a new section to Franklin.jl's docs 😉

Misc

Minor code contributions 😉

Final words

Tweets

Appendix

PRs and issues

Flux.jl

Metalhead.jl

NNlib.jl

OneHotArrays.jl

fluxml.github.io

Franklin.jl

www.julialang.org

.github

My experience working as a Technical Writer with FluxML - Part 1

Doctests

Missing docstrings

Broken documentation

CI/CD

FluxML’s ecosystem

Minor code changes

Getting started section

Tweets

Appendix

Pull requests

Flux.jl

Zygote.jl

Optimisers.jl

Functors.jl

MLUtils.jl

Issues / discussions

Flux.jl

Zygote.jl

Bonus