Outward on the Spiral: Petascale Inference in Earthquake System Science (SCEC PetaShake Project)

The SCEC Enabling Earthquake System Science Through Petascale Calculations (PetaShake) (NSF OCI-749313) Project was funded by the National Science Foundation (NSF) and started work on October 1, 2007. The PetaShake Project seeks to advance seismic hazard research through the use of Petascale computing facilities as they become available to the NSF research community.

Earthquake system science seeks a basic understanding of how matter and energy interact within the lithosphere to produce seismic phenomena; its practical mission is to provide society with better predictions of earthquake hazards. Dynamic simulations of fault ruptures and seismic wave propagation are proving to be important tools, but they cannot yet achieve the physical scale range needed to explore important domains of earthquake behavior. Our goal is to take earthquake system science to a new level using petascale computational resources. In this 2-year project, we will employ the most capable NSF computers to:

1. Improve the resolution of dynamic rupture simulations by an order of magnitude to investigate realistic friction laws, near-fault stress states, and off-fault plasticity;
2. Investigate the upper frequency limit of deterministic ground-motion prediction by simulating strong motions up to 3 Hz using realistic 3D structural models for Southern California; and
3. Improve the Southern California structural models using full 3D waveform tomography. The high-performance computing objectives are to improve the scalability of earthquake simulations, enhance fault detection and tolerance, and develop a verification and validation framework for petascale code development and usage.

We propose to achieve our science objectives by developing PetaShake, an advanced computational research platform designed to support high-resolution earthquake simulations on a regional (< 1000 km) scale. PetaShake will extend two high-performance, open-source scientific modeling codes, the finite-difference Olsen code and the finite-element Hercules code, toward petascale capability.

These operational codes scale efficiently on thousands of processors, and they are being widely applied to wave propagation simulations, dynamic fault rupture studies, physics-based seismic hazard analysis, and full 3D tomography. We will improve their single-processor performance through better cache usage, data localization, and platform-dependent optimizations; their parallel performance to scale onto 100,000+ cores through a higher degree of parallelization and overlapping between communication and computation; and their I/O performance to support high-resolution input meshes and time-varying output volumes by parallelizing all I/O (initialization, output, and check-pointing) and exploring the use of asynchronous I/O.  We will also improve their fault tolerance and fault detection capabilities, and we will incorporate an on-demand verification and validation capability into the PetaShake platform to support rapid development and enhanced flexibility while maintaining scientific validity.

We will move PetaShake up in scale and capability through a graduated series of milestone calculations tied to a timeline with clear scientific objectives and quantitative measures of success. The milestones will measure progress on two tracks, computational efficiency and I/O efficiency. Although the focus of PetaShake will be on capability computing, the research to optimize this platform will enable the petascale capacity-computing and data-intensive goals of three other CME platforms, CyberShake, DynaShake, and F3DT, which are supported by other means, primarily the PetaSHA cyberfacility grant from NSF/EAR. The proposed NSF/OCI project will thus leverage on GEO Directorate funding, as well as the NSF-USGS-funded SCEC base program. Early coding and testing on petascale hardware will accelerate the transformation of seismic hazard analysis to a physics-based science—our main goal.