HemeLB

Introduction

HemeLB is a high performance lattice-Boltzmann solver optimized for simulating blood flow through sparse geometries, such as those found in the human vasculature. It is routinely deployed on powerful supercomputers, scaling to hundreds of thousands of cores even for complex geometries . HemeLB has traditionally been used to model cerebral bloodflow and vascular remodelling in retinas , but is now being applied to simulating the fully coupled human arterial and venous trees.

The guide shows how to

  • run HemeLB using a pre-compiled module;

  • build a version of HemeLB;

  • submit a test HemeLB job.

To use HemeLB you can either use the pre-installed modules or build your own verion in your project $DATA area.

Using the pre-installed versions of HemeLB:

module spider HemeLB

     HemeLB/0.8

And load the appropriate version. For example:

module load HemeLB/0.8

The above will load HemeLB into your environment, and the package will be available to run via mpirun.

Building your own version of HemeLB from source

First ensure you are using an interactive session on a compute node:

srun -p interactive --time=02:00:00 --pty /bin/bash

We have a recipe which you can use to build HemeLB (this has been tested with version 0.8):

cd $DATA

export HEME_HOME=$DATA/HemeLB/0.8

module load foss/2020a
module load CMake/3.16.4-GCCcore-9.3.0
module load ParMETIS/4.0.3-gompi-2020a
module load Boost/1.72.0-gompi-2020a
module load Python/2.7.18-GCCcore-9.3.0

mkdir -p $HEME_HOME/bin
mkdir -p $HEME_HOME/lib

wget http://hemelb.org.s3-website.eu-west-2.amazonaws.com/files/hemelb-v0.8.tar.gz

tar xvf hemelb-v0.8.tar.gz
cd hemelb-v0.8

rm -rf CMakeCache.txt

mkdir bootstrap_build && cd bootstrap_build
export CXXFLAGS="-I/usr/include/tirpc"
export LDFLAGS="-ltirpc"

cmake -DCMAKE_INSTALL_PREFIX=$HEME_HOME ..
make -j8

cd ../dependencies/lib
cp -r * $HEME_HOME/lib/

To use the executable you will need to set the following environment variables:

export HEME_HOME=$DATA/HemeLB/0.8
export PATH=$HEME_HOME/bin:$PATH
export LD_LIBRARY_PATH=$HEME_HOME:$LD_LIBRARY_PATH

Example submission script

This example uses the input data found in HemeLB’s examples/bifurcation/bifurcation_hires directory, this is a version using the ARC module:

#! /bin/bash

#SBATCH --nodes=2
#SBATCH --ntasks-per-node=48
#SBATCH --partition=devel
#SBATCH --time=00:10:00

module load HemeLB/0.8

mpirun hemelb -in  input.xml -out output

The following is a version for a locally built version:

#! /bin/bash

#SBATCH --nodes=2
#SBATCH --ntasks-per-node=48
#SBATCH --partition=devel
#SBATCH --time=00:10:00

module load foss/2020a
module load ParMETIS/4.0.3-gompi-2020a
module load Boost/1.72.0-gompi-2020a
module load Python/2.7.18-GCCcore-9.3.0

export HEME_HOME=$DATA/HemeLB/0.8
export PATH=$HEME_HOME/bin:$PATH
export LD_LIBRARY_PATH=$HEME_HOME:$LD_LIBRARY_PATH

mpirun hemelb -in  input.xml -out output

Note

You need to explicitly include the modules that HemeLB requires in this script (the ARC build module loads these automatically)