Broadband Platform Utilities
Notes configuring BBP utilities to run on SDSC Gordon.
Motivation
SDSC Gordon has solid state disks. These may speed up BBP processing, because BBP uses many small files. Initial tests, running the UnitTests and Acceptance Tests suggest a speedup of 1.8x, which is enough it make this worthwhile investigating further.
Approach
We have information how long it takes to run BBP on SCEC HPC cluster. We will reconfigure to run on Gordon, and will compare time-to-solution using Gordon against use of HPC.
Guidance
Fabio pointed at bbp_hpcc_validation.py - This script will create a series of PBS files, that contain appropriate PBS parameters (e.g. number of nodes), and path names. These should be changed to point to directories on Gordon.
[ux454496@gordon-ln2 batch]$ ./bbp_hpcc_validation.py -h Usage: bbp_hpcc_validation.py [options]
Options:
-h, --help show this help message and exit
-c CODEBASE, --codebase=CODEBASE
Codebase for the simulation: ['gp', 'ucsb', 'sdsu',
'exsim', 'csm']
-e EVENT, --event=EVENT
Validation event (should be configured in BBP)
-d SIMDIR, --dir=SIMDIR
Simulation directory
--skip-rupgen Skip the rupture generator, run only 1 simulation
--hypo-rand Enables hypocenter randomization
--no-hypo-rand Disables hypocenter randomization
-n NUMSIM, --num-simulations=NUMSIM
Number of simulations to run
--email=EMAIL Email for job notifications
--new-nodes Schedule the job in the new HPCC nodes
--save-tmpdata Save the contents of the tmpdata directory
--only-rup Only runs the rupture generator
-g GMPE_GROUP_NAME, --gmpe-group=GMPE_GROUP_NAME
GMPE group: ['nga-west2', 'nga-west1', 'cena group 1']
An example on Gordon:
./bbp_hpcc_validation.py -c GP -e Landers --no-hypo-rand -d /oasis/scratch/ux454496/temp_project/bbp_home/part_a/Landers -n 50 --email maechlin@usc.edu
Fabio also pointed at the setup_bbp_env.sh script. This will configure the compute node run-time environment. This needs to be edited away from HPC to Gordon. He made special reference to the $TMP_DIR/BBP_DATA_DIR definition at two places in this script. These are used at HPC to point to a on-compute node disk. Once results are written there, they are moved to another spot for storage. We should be able to point this to solid-state disk on Gordon, rather than compute node disk storage on HPC.
