Creating an experiment repo

Overview

We use the term “experiment repo” for a git repository that contains the plumbing required for a numerical experiment.

It consists in a git repository with:

  • the nextflow executable and its configuration, nextflow.config;
  • a git submodule, nf-nest containing useful tools;
  • your .nf files and other supporting files.

This pages explains how to setup an experiment repo.

Location

In an HPC context, the experiment repo should be in a read/write location accessible to all nodes.

For example, on Sockeye the first choice if you allocation has it would be to use burst storage, i.e. cd to a path of the form

export JULIA_DEPOT_PATH=/arc/burst/[allocation_code]/[username_in_alloc]/

A second choice would be to use the scratch space, i.e. cd to a path of the form

export JULIA_DEPOT_PATH=/scratch/[allocation_code]/[username_in_alloc]/

Instructions

For quick setup, run the following commands:

# create a directory and cd into it
mkdir experiment_repo && cd $_

git init

# setup nextflow
curl -s https://get.nextflow.io | bash

# add nf-nest utilities
git submodule add https://github.com/UBC-Stat-ML/nf-nest.git

# copy template for nextflow configurations
cp nf-nest/nextflow.config .