Managing Julia packages on HPC

Overview

Installing a package in Julia includes two steps (1) downloading (2) pre-compiling. In a laptop, both are done in one shot using

]
activate experiment_repo/julia_env
add NameOfPackage

In an HPC setup, step (2) requires computational effort hence is best done in compute node. However in some HPC setups such as Sockeye, compute nodes do not have internet access, so the process needs to be split. We cover here some utilities that facilitate this.

Julia environment

A Julia environment is a specification of all package dependencies and their versions.

By convention, we will store it in a directory called experiment_repo/julia_env.

Adding a package

There are two syntaxes in Julia to install packages: the interactive Julia package manager or using programmatic Pkg syntax. We cover both below.

In both syntaxes, we first need to tell Julia which environment to use. This is called “activating” an environment.

Interactive Julia package manager

The most common method is to use the interactive Julia package manager. To start it, type ] followed by enter.

Activating is done with the activate keyword.

You can then add a package using the add command. The argument of add can be either a registered Julia package, for example we show here how to add the Pigeons package for MCMC sampling:

ENV["JULIA_PKG_PRECOMPILE_AUTO"]=0 # Hold off precompile since we are in login node
]
activate experiment_repo/julia_env
add Pigeons

Alternatively, the package can be a git repository, optionally with a specific commit/tag:

add Example@0.5
add Example#master
add Example#c37b675
add https://github.com/JuliaLang/Example.jl#master
add git@github.com:JuliaLang/Example.jl.git
add "git@github.com:JuliaLang/Example.jl.git"#master
add https://github.com/Company/MonoRepo:juliapkgs/Package.jl

To exit the interactive Julia package manager, type control-C.

Programmatic interface

Alternatively, a programmatic interface is also available for scripting:

ENV["JULIA_PKG_PRECOMPILE_AUTO"]=0 # Hold off precompile since we are in login node
using Pkg 
Pkg.activate("experiment_repo/julia_env")
Pkg.add("Pigeons")

Precompilation on HPC

Once we have downloaded the packages in the login node, we now turn to the task of performing pre-compilation.

cd experiment_repo 
./nextflow run nf-nest/pkg.nf -profile cluster

N E X T F L O W  ~  version 24.10.0
Launching `nf-nest/pkg.nf` [ridiculous_volta] DSL2 - revision: fc0374e695
[a4/e6cda6] Submitted process > instantiate_process
[42/9bb2de] Submitted process > precompile

Optionally, you can append an argument to specify the number of threads to request and use during pre-compilation: e.g. add --nPrecompileThreads 20 to request 20 threads instead of the default of 10.

Testing your Julia environment interactively

In the code above, we have added the package Pigeons as an example.

To interactively test an installed package, simply activate the environment from a Julia session:

using Pkg
Pkg.activate("experiment_repo/julia_env")

using Pigeons
pigeons(target = toy_mvn_target(1000))

More information

See Pkg.jl documentation.