gnuparallel

About

GNU parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU parallel can then split the input and pipe it into commands in parallel.

Versions and Availability

Module Names for gnuparallel on qb2
Machine Version Module Name
None Available N/A N/A
▶ **FIX-ME** FAQ?

Usage

Parallel typical serial and MPI-based applications.

(1) Parallel serial jobs

Example of a blast job on Mike:

#!/bin/bash

#PBS -A hpc_smictest3
#PBS -l nodes=2:ppn=16
#PBS -l walltime=1:00:00
#PBS -q workq

cd $PBS_O_WORKDIR
export JOBS_PER_NODE=16
export WDIR=$PBS_O_WORKDIR

parallel --progress \				# shows progres
         --joblog logfile \			# job logfile
         -j $JOBS_PER_NODE \		# jobs per node
         --slf $PBS_NODEFILE \			# nodes assigned to your job
         --workdir $WDIR \			
         ./cmd_blast.sh {} {/.} :::: input.lst  #script_to_parallize input output joblist

where: input.lst contains job input list:

/work/$USER/blast/data/input1.faa
/work/$USER/blast/data/input2.faa
....
/work/$USER/blast/data/input200.faa

where: cmd_blast.sh is the script for running a serial blast job

e.g.: ./cmd_blast.sh input1.faa input1 -- how to run single serial job

#!/bin/bash

export WDIR=/xxx/xxx
cd $WDIR
blastp -query $1 -db db/img_v400_PROT.00 -out output/$2.out -outfmt 7 -max_target_seqs 100 -num_threads 2
	

(2) Parallel MPI jobs

Use "mpirun" to run a laplace

#!/bin/bash

#PBS -A your_allocation_name
#PBS -l walltime=2:00:00
#PBS -l nodes=4:ppn=16
#PBS -q checkpt

export JOBS_PER_NODE=8
export NPROCS=2
export WDIR=$PBS_O_WORKDIR
cd $WDIR
parallel --progress \
         -j $JOBS_PER_NODE \
         --slf $PBS_NODEFILE \
         --workdir $WDIR \
         ./cmd_mpi.sh {} $NPROCS :::: input.lst
	

where: cmd_mpi.sh is the script to run one MPI job

#!/bin/bash

export WDIR=$PBS_O_WORKDIR
FILE=$(eval echo $1)
param=`cat ${FILE}`
mpirun -ppn $2 $WDIR/lap_mpi $param

where: input.lst contains job input list:

/work/$USER/laplace/data/input1
/work/$USER/laplace/data/input2
....
/work/$USER/laplace/data/input200

cat input1:
4096 4096 2 2 0.08 20000 0 0

Resources

Last modified: January 30 2017 12:37:10.