Job cross products

Overview

Suppose you are interested in running a piece of code with many different inputs, with each execution performed on a different compute node of a cluster.

This page shows a streamlined way to do so.

Example

As a toy example, suppose we want to compute all additions of the form a + b where a and b are integers from 1 to 3. In addition, we also want a * b. This means we will need \(3 \times 3 \times 2\) calculations.

We can characterize the inputs a the cross-product denoted \(\{1, 2, 3\} \times \{1, 2, 3\} \times \{+, *\}\).

Nextflow script

The script below will perform the following operations.

// we use utilities in the nf-nest submodule
// in user scripts, path would be './nf-nest/cross.nf' 
include { crossProduct; filed; deliverables } from '../cross.nf'
include { instantiate; precompile; activate } from '../pkg.nf'

def variables = [
    first: 1..3,
    second: 1..3,
    operation: ["+", "*"]
]

// specifies the order of operations
workflow {
    // look at all combinations of variables
    configs = crossProduct(variables)
    // run Julia on 18 nodes!
    run_julia(configs)

    // equivalent syntax:
    // crossProduct(variables) | run_julia
}

process run_julia {
    debug true // by default, standard out is not shown, use this to show it
    
    // information used when submitting job to queue
    time 2.min
    cpus 1 
    memory 5.GB

    input:
        val config 
    """
    ${activate()}
    # ^ this is just a shortcut for:
    #!/usr/bin/env julia --threads=1

    @show ${config.first} ${config.operation} ${config.second}
    """
}

For more information:

Running the script

Running it with the -profile cluster option will:

  • build a cross-product from variables
  • for each one, automatically create submission scripts
  • run these Julia processes and show the standard out.

From the command line, running the script is done as follows:

cd experiment_repo
./nextflow run nf-nest/examples/many_jobs.nf -profile cluster
N E X T F L O W  ~  version 24.10.0
Launching `nf-nest/examples/many_jobs.nf` [elegant_fermi] DSL2 - revision: aa082b1978
[4d/7492f2] Submitted process > run_julia (7)
[6c/41c384] Submitted process > run_julia (6)
[dd/7a5a36] Submitted process > run_julia (14)
[e8/b7de0b] Submitted process > run_julia (9)
[f0/6df027] Submitted process > run_julia (1)
[a3/ef9de9] Submitted process > run_julia (4)
[a3/f36c6f] Submitted process > run_julia (2)
[eb/83e088] Submitted process > run_julia (16)
[73/038281] Submitted process > run_julia (3)
[2c/689c09] Submitted process > run_julia (15)
[14/7503f2] Submitted process > run_julia (8)
[d8/14676b] Submitted process > run_julia (10)
[6e/7edb12] Submitted process > run_julia (5)
[ae/f4128b] Submitted process > run_julia (13)
[9c/092e9c] Submitted process > run_julia (11)
[60/531ea0] Submitted process > run_julia (12)
[d4/d42221] Submitted process > run_julia (18)
[0b/f6b1c6] Submitted process > run_julia (17)
2 + 1 = 3
1 * 3 = 3
3 * 1 = 3
2 + 2 = 4
1 + 1 = 2
1 * 2 = 2
1 * 1 = 1
3 * 2 = 6
2 * 2 = 4
1 + 3 = 4
3 + 1 = 4
3 * 3 = 9
3 + 3 = 6
1 + 2 = 3
3 + 2 = 5
2 * 1 = 2
2 + 3 = 5
2 * 3 = 6

Filtering

In some case we want to run only a subset of the cross product. For example, suppose we want only the runs of the form a * a and a + a. This can be done using the filter() function in nextflow:

// we use utilities in the nf-nest submodule
// in user scripts, path would be './nf-nest/cross.nf' 
include { crossProduct; filed; deliverables } from '../cross.nf'
include { instantiate; precompile; activate } from '../pkg.nf'

def variables = [
    first: 1..3,
    second: 1..3,
    operation: ["+", "*"]
]

// specifies the order of operations
workflow {
    configs = crossProduct(variables).filter{ config -> config.first == config.second }
    run_julia(configs)

    // equivalent pipe syntax:
    // crossProduct(variables) | filter{ config -> config.first == config.second } | run_julia
}

process run_julia {
    debug true // by default, standard out is not shown, use this to show it
    input:
        val config 
    """
    ${activate()}

    @show ${config.first} ${config.operation} ${config.second}
    """
}

Running this

cd experiment_repo
./nextflow run nf-nest/examples/filter.nf  
N E X T F L O W  ~  version 24.10.0
Launching `nf-nest/examples/filter.nf` [tiny_elion] DSL2 - revision: d9de661ecc
[c0/6ac1a6] Submitted process > run_julia (4)
[b5/180362] Submitted process > run_julia (1)
[e9/7403a5] Submitted process > run_julia (3)
[30/748169] Submitted process > run_julia (6)
[52/c579d6] Submitted process > run_julia (2)
[c1/141e36] Submitted process > run_julia (5)
2 * 2 = 4
2 + 2 = 4
1 * 1 = 1
1 + 1 = 2
3 * 3 = 9
3 + 3 = 6