The Intel Hackathon to explore the Intel Optane memory was organized at Durham University in June 2019. The workshop consisted of a great balance of talks as well as hands-on experiences. A series of talks by experts from Intel introduced the new Optane memory system. The hands-on sessions were very intensive with a lot of help available from Intel and Durham staff.
Our team from CCFE (Culham Centre for Fusion Energy) consisted of three members – S.J Pamela, L. Anton and D. Samaddar. Our laboratory specializes in fusion research and with a goal of having fusion energy on a commercial grid, we work with a very wide range of complex simulation codes. Our codes differ from one another in terms of algorithms, data volume and data structures as well as the physics they solve. Our team therefore used a number of codes as test-beds at the hackathon. Jorek is a 3-D nonlinear MHD code solving very large sparse matrices. OpenMC is a Monte-Carlo code that can be very memory intensive when sampling a large number of particles. BOUT++ on the other hand is a code package that studies turbulence involving strongly coupled non-linearities and multiscale physics. GS2 is a widely used code solving gyrokinetic equations in fusion plasma.
The Hackathon provided the right setting with great networking opportunities to generate the initial tests and lead the way for further explorations. For example, the Optane memory mode was found to be very beneficial for JOREK when the matrix size was increased. All the applications from CCFE used at the Hackathon are representative of other codes used in fusion research – so the results should benefit a wider range of HPC users within the community.
A low resolution 2D FEM grid for the JOREK code, with the third dimension represented by Fourier harmonics.
A snapshot of plasma instabilities simulated by JOREK.
Intel agreed to sponsor a 3-day Optane hackathon at Durham. It’s aim was to learn how Intel® Optane™ memory and intelligent system acceleration work to deliver higher performance. This event provided the DiRAC community an opportunity to explore the potential for Intel’s new Optane memory in supporting their science.
The Hackathon was open to all DiRAC HPC users. There were 5 teams of between 2 and 3 people each. Over the three days, several major DiRAC science codes were ported to make efficient use of this and standard memory and the teams who attend gained the skills to assist other DiRAC researchers to port additional codes in the future.
Teams
GRChombo
lead by Dr Kacper Kornet from the University of Cambridge
GRChombo is a new open-source code for numerical general relativity simulations. It is developed and maintained by a collaboration of numerical relativists with a wide range of research interests, from early universe cosmology to astrophysics and mathematical general relativity, and has been used in many papers since its first release in 2015. GRChombo is written entirely in C++14, using hybrid MPI/OpenMP parallelism and vector intrinsics to achieve good performance on the latest architectures. Furthermore, it makes use of the Chombo library for adaptive mesh refinement to allow automatic increasing and decreasing of the grid resolution in regions of arbitrary shape and topology.
TROVE
lead by Sergey Yurchenko from University College London
TROVE is a variational method with an associated Fortran 2003 program to construct and solve the ro-vibrational Schrödinger equation for a general polyatomic molecule of arbitrary structure. The energies and eigenfunctions obtained via a variational approach can be used to model absorption/emission intensities (molecular line lists) for a given temperature as well as to compute temperature independent line strengths and Einstein coefficients. A typical TROVE pipe line requires a construction and diagonalisation of about 200-300 double-real, dense symmetric matrices of sizes varying from 10,000×10,000 to 500,000×500,000. For the line list production it is important to compute almost all eigenvalues (or at least 50-80%) together with eigenvectors. The diagonalisations are efficiently done using external libraries, for example the DSYEV-family for smaller matrices (N< 200,000) or the PDSYEV-family for large matrices (N > 200,000). The TROVE program is highly optimized for the mass production of line lists for medium-size molecules applicable for high temperatures. TROVE has been extensively used to produce line lists for a number of key polyatomic molecules, including NH3 , CH4 , H2CO, H2CS, PH3 , SbH3 , HOOH, HSOH, CH3Cl, BiH3 , SO3 , SiH4 , CH3 , C2H4 , CH3F (about 80 papers in peer-reviewed journals). TROVE is a well-recognized method of the modern theoretical spectroscopy with the TROVE paper being highly cited.
Swift
lead by Prof Richard Bower from Durham University
Is a fine-grained task-parallel approach to cosmological simulations. The code solves gravity and hydro equations, with additional sub-grid physics sources terms that encapsulate star formation, black holes, chemical enrichment, cooling etc.
The novelty of the code lies in its approach to parallelising this challenging problem. It delivers a factor 10 speed improvement over the current state of the art code, gadget.
The code is primarily used by the EAGLE simualtions team and at JPL for planetary science. It is a new code and we expect widespread adoption as the results are published.
lead by Arjen Tamerus from the University of Cambridge
MODAL_LSS is an astrophysics code used for estimation of the bispectrum of simulation data or observational data, using the MODAL methodology. It is currently in active development and not (yet) open source.
This team was using the open source code RAMSES https://bitbucket.org/rteyssie/ramses written by Romain Teyssier. While mainly designed to study structure formation in an expanding Universe, the code has been applied to a range of problems dealing with self-gravitating MHD fluids (turbulence, planet formation) and is also equipped with a module to solve the radiative transfer equations. There also exists versions of the code with a modified Poisson solver to study alternatives to Einstein’s gravity. The RAMSES community consists of about ~150 users around the world, with ~15 in the UK.
SLH (Seven-League Hydro) Code
lead by Phillipp Edelmann from Newcastle University
SLH is a finite-volume hydrodynamics code, mainly intended for use in stellar astrophysics. Its most distinguishing feature is the fully implicit time discretization, which allows it to run efficiently for problems at low Mach numbers, such as the deep interiors of stars. The resulting large non-linear system is very challenging to solve and needs Newton-Krylov methods to be efficient.
It is designed with flexibility in mind, supporting arbitrary structured curvilinear grids, different equations of state and a selection of other physics modules (nuclear reactions, neutrino losses, …).
More information about the code at: https://slh-code.org/features.html The code is currently not open source, but our group is open to pass the code on in collaborations.
JBCA
lead by Rohini Joshi of the University of Manchester
They were using WSCLEAN (https://sourceforge.net/p/wsclean/wiki/Home/), a C++ code designed to perform Fast Fourier Transforms (FFTs) of large datasets produced by connected arrays of radio telescopes (interferometers), and then perform iterative beam-deconvolution (following the CLEAN algorithm; Hogbom, 1974) to produce sensitive, high-resolution, wide-field images for astrophysical interpretation. eMERLIN, the UK National Facility for high-resolution radio astronomy, has proved a productive instrument for smaller (few nights, yielding ~100GB datasets) PI-led studies, imaging narrow fields of view (4 mega-pixels), however the flagship “legacy” programs, which address key STFC science goals, entail several thousand hours of on-sky data aquisition and have produced datasets of the order >>1-10TB over fields of view orders of magnitude larger (~2 giga-pixels). Producing images from these legacy programmes has thus presented a unique “big data” challenge.
In recent years an experienced corps of radio astronomers have pushed the boundaries of radio imaging techniques to deliver eMERLIN legacy science using machines with ~512GB–1.5TB RAM, however to circumvent bottlenecks in our existing infrastructure and fully deliver on the potential of these datasets in the coming years we envisage even higher memory requirements.
These same challenges have been encountered by other modern radio telescopes (e.g. LOFAR, ASKAP, MeerKAT, and will be critical to science delivery from the Square Kilometer Array in the next decate). WSCLEAN is a modern imaging package designed to meet the needs of current and forthcoming facilities. It is rapidly becoming the standard wide-field imaging package, replacing older packages like AIPS and CASA. The source code is publicly available.
