// we use utilities in the nf-nest submodule
// in user scripts, path would be './nf-nest/cross.nf'
{ crossProduct; filed; deliverables } from '../cross.nf'
include { instantiate; precompile; activate } from '../pkg.nf'
include
def variables = [
: 1..3,
first: 1..3,
second: ["+", "*"]
operation]
// specifies the order of operations
{
workflow // look at all combinations of variables
= crossProduct(variables)
configs // run Julia on 18 nodes!
run_julia(configs)
// equivalent syntax:
// crossProduct(variables) | run_julia
}
{
process run_julia true // by default, standard out is not shown, use this to show it
debug
// information used when submitting job to queue
2.min
time 1
cpus 5.GB
memory
:
input
val config """
${activate()}
# ^ this is just a shortcut for:
#!/usr/bin/env julia --threads=1
@show ${config.first} ${config.operation} ${config.second}
"""
}
Job cross products
Overview
Suppose you are interested in running a piece of code with many different inputs, with each execution performed on a different compute node of a cluster.
This page shows a streamlined way to do so.
Example
As a toy example, suppose we want to compute all additions of the form a + b
where a
and b
are integers from 1 to 3. In addition, we also want a * b
. This means we will need \(3 \times 3 \times 2\) calculations.
We can characterize the inputs a the cross-product denoted \(\{1, 2, 3\} \times \{1, 2, 3\} \times \{+, *\}\).
Nextflow script
The script below will perform the following operations.
For more information:
- Read on the nextflow scripting language
- Read on nextflow’s
process
block configs
in the above is an example of a nextflowChannel
- More on nextflow’s
workflow
block
Running the script
Running it with the -profile cluster
option will:
- build a cross-product from
variables
- for each one, automatically create submission scripts
- run these Julia processes and show the standard out.
From the command line, running the script is done as follows:
cd experiment_repo
./nextflow run nf-nest/examples/many_jobs.nf -profile cluster
N E X T F L O W ~ version 25.04.6
Launching `nf-nest/examples/many_jobs.nf` [tiny_lagrange] DSL2 - revision: aa082b1978
[77/ad410f] Submitted process > run_julia (9)
[0e/773f47] Submitted process > run_julia (13)
[a7/945d98] Submitted process > run_julia (3)
[ef/ba4162] Submitted process > run_julia (6)
[25/747142] Submitted process > run_julia (10)
[cc/2f94b3] Submitted process > run_julia (7)
[12/c46b9a] Submitted process > run_julia (4)
[46/348baf] Submitted process > run_julia (8)
[60/df2e03] Submitted process > run_julia (12)
[8b/5792ce] Submitted process > run_julia (11)
[5e/f5689d] Submitted process > run_julia (2)
[13/625db4] Submitted process > run_julia (5)
[06/cdbf65] Submitted process > run_julia (1)
[9e/31e73e] Submitted process > run_julia (14)
[ad/25ff98] Submitted process > run_julia (15)
[36/382458] Submitted process > run_julia (16)
[f0/634114] Submitted process > run_julia (17)
[db/921b35] Submitted process > run_julia (18)
2 + 2 = 4
1 + 2 = 3
3 + 1 = 4
1 * 3 = 3
2 * 2 = 4
2 + 1 = 3
1 * 2 = 2
2 * 1 = 2
2 * 3 = 6
2 + 3 = 5
1 * 1 = 1
1 + 3 = 4
1 + 1 = 2
3 * 1 = 3
3 + 2 = 5
3 * 2 = 6
3 + 3 = 6
3 * 3 = 9
Filtering
In some case we want to run only a subset of the cross product. For example, suppose we want only the runs of the form a * a
and a + a
. This can be done using the filter()
function in nextflow:
// we use utilities in the nf-nest submodule
// in user scripts, path would be './nf-nest/cross.nf'
{ crossProduct; filed; deliverables } from '../cross.nf'
include { instantiate; precompile; activate } from '../pkg.nf'
include
def variables = [
: 1..3,
first: 1..3,
second: ["+", "*"]
operation]
// specifies the order of operations
{
workflow = crossProduct(variables).filter{ config -> config.first == config.second }
configs run_julia(configs)
// equivalent pipe syntax:
// crossProduct(variables) | filter{ config -> config.first == config.second } | run_julia
}
{
process run_julia true // by default, standard out is not shown, use this to show it
debug :
input
val config """
${activate()}
@show ${config.first} ${config.operation} ${config.second}
"""
}
Running this
cd experiment_repo
./nextflow run nf-nest/examples/filter.nf
N E X T F L O W ~ version 25.04.6
Launching `nf-nest/examples/filter.nf` [gigantic_heisenberg] DSL2 - revision: d9de661ecc
[9c/40cdd5] Submitted process > run_julia (3)
[4c/c74cef] Submitted process > run_julia (2)
[14/ba5583] Submitted process > run_julia (4)
[9e/741be8] Submitted process > run_julia (6)
[c8/236bfe] Submitted process > run_julia (1)
[84/fa064c] Submitted process > run_julia (5)
2 + 2 = 4
1 * 1 = 1
3 * 3 = 9
2 * 2 = 4
1 + 1 = 2
3 + 3 = 6
Log scales
To create a list of parameters in log scale, use:
(0..2).collect{ i -> Math.pow(2.0, i)}
This will return 2.0^0, 2.0^1, 2.0^2
.