HemeLB
Introduction
HemeLB is a high performance lattice-Boltzmann solver optimized for simulating blood flow through sparse geometries, such as those found in the human vasculature. It is routinely deployed on powerful supercomputers, scaling to hundreds of thousands of cores even for complex geometries . HemeLB has traditionally been used to model cerebral bloodflow and vascular remodelling in retinas , but is now being applied to simulating the fully coupled human arterial and venous trees.
The guide shows how to
run HemeLB using a pre-compiled module;
build a version of HemeLB;
submit a test HemeLB job.
To use HemeLB you can either use the pre-installed modules or build your own verion in your project $DATA
area.
Using the pre-installed versions of HemeLB:
module spider HemeLB
HemeLB/0.8
And load the appropriate version. For example:
module load HemeLB/0.8
The above will load HemeLB into your environment, and the package will be available to run via mpirun
.
Building your own version of HemeLB from source
First ensure you are using an interactive session on a compute node:
srun -p interactive --time=02:00:00 --pty /bin/bash
We have a recipe which you can use to build HemeLB (this has been tested with version 0.8):
cd $DATA
export HEME_HOME=$DATA/HemeLB/0.8
module load foss/2020a
module load CMake/3.16.4-GCCcore-9.3.0
module load ParMETIS/4.0.3-gompi-2020a
module load Boost/1.72.0-gompi-2020a
module load Python/2.7.18-GCCcore-9.3.0
mkdir -p $HEME_HOME/bin
mkdir -p $HEME_HOME/lib
wget http://hemelb.org.s3-website.eu-west-2.amazonaws.com/files/hemelb-v0.8.tar.gz
tar xvf hemelb-v0.8.tar.gz
cd hemelb-v0.8
rm -rf CMakeCache.txt
mkdir bootstrap_build && cd bootstrap_build
export CXXFLAGS="-I/usr/include/tirpc"
export LDFLAGS="-ltirpc"
cmake -DCMAKE_INSTALL_PREFIX=$HEME_HOME ..
make -j8
cd ../dependencies/lib
cp -r * $HEME_HOME/lib/
To use the executable you will need to set the following environment variables:
export HEME_HOME=$DATA/HemeLB/0.8
export PATH=$HEME_HOME/bin:$PATH
export LD_LIBRARY_PATH=$HEME_HOME:$LD_LIBRARY_PATH
Example submission script
This example uses the input data found in HemeLB’s examples/bifurcation/bifurcation_hires
directory, this is a version using the ARC module:
#! /bin/bash
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=48
#SBATCH --partition=devel
#SBATCH --time=00:10:00
module load HemeLB/0.8
mpirun hemelb -in input.xml -out output
The following is a version for a locally built version:
#! /bin/bash
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=48
#SBATCH --partition=devel
#SBATCH --time=00:10:00
module load foss/2020a
module load ParMETIS/4.0.3-gompi-2020a
module load Boost/1.72.0-gompi-2020a
module load Python/2.7.18-GCCcore-9.3.0
export HEME_HOME=$DATA/HemeLB/0.8
export PATH=$HEME_HOME/bin:$PATH
export LD_LIBRARY_PATH=$HEME_HOME:$LD_LIBRARY_PATH
mpirun hemelb -in input.xml -out output
Note
You need to explicitly include the modules that HemeLB requires in this script (the ARC build module loads these automatically)