We’ve helped pave the way for scalable, high-performance systems that will power radio telescopes for decades to come.
Sean Stansill, HPC Software Engineer, nAG

Challenge: Turning an Ambitious Concept into an Operational Reality

While the SKA telescopes have been envisioned since late 1980s, the computing infrastructure required to realise them only recently became feasible. Described as “software telescopes,” the SKA project does not rely on a single giant dish. Instead, it combines data from hundreds of smaller dishes and thousands of antennas using complex software pipelines. The computing demand is unprecedented — the two systems must sustain read/write speeds around 8 terabytes per second, non-stop.

The real bottleneck? I/O performance. Unlike embarrassingly parallel tasks, radio astronomy workflows require tightly synchronised data movement across many nodes. This choreography is dictated by the physics of how radio waves are processed. Efficiently managing this scale and complexity is a frontier problem in modern HPC.

Solution: Engineering Scalable, Future-Proof HPC Software Pipelines 

nAG HPC Services are playing a significant role in the Science Data Processing (SDP) phase where raw telescope data becomes scientifically useful. Sean Stansill, nAG HPC Engineer working on the SKAO project has contributed to:

• Development and integration of MSv4, a next-generation data format poised to become the global standard for radio telescopes.
• Enhancement of DP3, a calibration tool critical to ensuring scientific accuracy, now with distributed processing support for greater scalability.
• Leading input on I/O architecture and performance optimisation during an international review consisting of 24 institutions.

He has also championed object storage — a mainstay in cloud computing — as a novel approach within HPC. Although unconventional in this domain, it offers the flexibility and scalability needed for SKAO’s evolving data landscape, reflecting a broader convergence of HPC and big data technologies.

Collaboration and Global Reach

This effort is deeply collaborative. Sean works not only within the SKAO team but also alongside counterparts at the US National Radio Astronomy Observatory (NRAO) and the South African Radio Astronomy Observatory (SARAO). Weekly working groups ensure technical alignment and smooth software integration across continents.

Looking Ahead

As the SKAO prepares for full operations towards the end of the decade, the innovations taking place today are setting the standard for scientific computing tomorrow. nAG’s work is paving the way for resilient, high-performance data systems that will serve the radio astronomy community for decades.

We’ve helped pave the way for scalable, high-performance systems that will power radio telescopes for decades to come.
Sean Stansill, HPC Software Engineer, nAG

Key Takeaways

• 8 TB/s I/O is needed – the SKAO’s HPC systems must continuously sustain these rates.
• Global standards: MSv4 is becoming the benchmark format for radio astronomy worldwide.
• Innovative architecture: Adoption of object storage marks a shift in HPC design for data-intensive workloads.
• Scientific impact: From studying the cosmic down to understanding galaxy evolution, the SKAO’s discoveries depend on the Science Data Processor software that nAG’s experts are helping to develop.
• Cost-effective performance: Hardware tuning yielded up to 4× efficiency gains with minimal investment.

nAG are renowned experts in High Performance Computing (HPC), providing experience in system architecture, implementation and engineering applications. This combination aligned perfectly to develop and build infrastructure to migrate Mahindra’s workloads to the Google Cloud Platform (GCP).

The Challenge

Mahindra runs multiple on-premise HPC systems for complex engineering workloads, needed to design and verify their automotive products. As part of Mahindra’s digital strategy to move resources to Google Cloud, these HPC systems were to be consolidated using GCP compute, integrating their existing workflows, whilst ensuring a production-level environment.

The Result

Mahindra now have a production HPC capability to run their workloads in GCP, improving the engineering simulation turnaround, with the advantages of being able to scale their resources on-demand and make use new technology sooner. In addition, a test system environment was established, enabling trials of new software, configurations, etc., without affecting production workloads.

“The Open Front End (OFE) was a key objective of our HPC transformation to Cloud as it allows us to manage our cluster(s) from an intuitive portal and enables us to scale and provision our cluster quickly. nAG worked with Mahindra and GCP to enable OFE for us, add customizations, provide key interventions & knowledge transfer so we can be self-sufficient & derive the expected benefits from our HPC.”

Gaurav Sarda, Business Engagement & Adoption Lead – Cloud, Mahindra

Oxford Professor uses nAG components to aid the design of new magnetic materials

Magnetic materials come in many different forms. The most familiar are ferromagnets, such as iron and nickel, which become magnetized in an external magnetic field and retain their magnetization after the field has been switched off. Ferromagnets are used in compasses, electronic recording devices, transformers, speakers, and many other everyday devices. Other, more subtle, forms of magnetism also exist and are particularly interesting because of their role in creating new states of matter with potentially useful properties, such as high temperature superconductivity or quantum information processing devices.

Magnetism arises from the existence of magnetic dipole moments, like tiny bar magnets, on some or all of the constituent atoms of the substance. The size of the magnetic moment is governed by the electronic state of the atom, which in turn depends on the local surroundings of the atom and the way the atom is bonded to its neighbours. Experiments that determine the electronic state of atoms in magnetic substances are therefore extremely valuable for understanding the origin of magnetic behaviour and for engineering materials with new or improved magnetic properties.

The original impetus for developing the software to analyze the experiments described here came from the availability of new experimental probes for studying magnetism. Neutron spectroscopy, in particular, has become a very powerful probe of the magnetic state of atoms thanks largely to advances in instrumentation at spallation neutron sources like the ISIS Facility at the Rutherford Appleton Laboratory. This technique makes it possible to obtain accurate measurements of level splittings in atoms caused by interactions with the crystalline environment. Detailed information on the many-electron states of atoms can be obtained from these level splittings and the corresponding spectral intensities.

Armed with the experimental data, Professor Andrew Boothroyd, at the University of Oxford Physics Department, needed the analytical tools to work out the state of the electrons and hence understand their magnetic behaviour. The creation of SPECTRE – the system designed to undertake the data analysis – and its subsequent evolution took place over many years as specific problems arose during the course of the research.

Having used numerical routines from nAG in previous projects, Professor Boothroyd turned to nAG to provide the mathematical code required by SPECTRE. SPECTRE uses the most up-to-date atomic models, which makes the results very accurate and quantitative. In brief, SPECTRE calculates the neutron spectra in terms of a small number of unknown parameters, and determines these parameters by least-squares fitting to the data. The core of the calculation is a series of matrix diagonalizations carried out by nAG routines designed to handle Hermitian matrices. The least-squares fitting is also performed by nAG routines. The lowest energy eigenfunctions found by the program are used to calculate other experimentally accessible physical properties such as the magnetic susceptibility and specific heat capacity.

SPECTRE will be made available to other magnetic materials groups in the hope that it will make it easier for scientists to interpret neutron spectroscopy data and thereby contribute towards the improvement of magnet materials.

Professor Andrew Boothroyd said “nAG’s reputation for accuracy, flexibility and robustness made it the first choice for the calculations in SPECTRE. I am delighted to be working with nAG to make the fruits of one person’s labour freely available to the whole scientific community.”

Mick Pont, Principal Technical Consultant at nAG commented “Software such as that developed by Professor Boothroyd is of fundamental importance in advancing human knowledge in the field of magnetism. We at nAG are very proud that our numerical components have proved useful in this endeavour. Since nAG was founded on collaborative principles we are particularly pleased that SPECTRE will be available for the use of others.”

For more information about the availability of SPECTRE please contact Professor Boothroyd.

Contact details:
Professor Andrew Boothroyd, Department of Physics, The University of Oxford
(a.boothroyd@physics.ox.ac.uk)
http://xray.physics.ox.ac.uk/Boothroyd/

Project Background

The Centre for Environment, Fisheries and Aquaculture Science (Cefas) is a world leader in marine science and technology, providing innovative solutions for the aquatic environment, biodiversity and food security. The centre uses behaviour and transport models to address a range of marine management questions.

They use an off-line particle tracking model that requires velocity fields from a hydrodynamic model, which is known as the Individual Behaviour Model (IBM) GITM (General Individuals Transport Model) code. It includes physical particle advection and diffusion, and biological development and behaviour. The code is implemented in Fortran 90. It was originally sequential then parallelized with OpenMP.

Objective

The objective of this work was to investigate how to improve the parallel performance of the GITM application using OpenMP and to quantify the effort for it to run on distributed memory HPC systems using MPI.

Method

Audit – The GITM audit took two months. A set of performance metrics were used to assess the quality of performance and identify any limiting issues, these metrics relate to computational scaling and load balance. The audit identified that the application could further benefit from using multiple threads. The underlying cause was found to be related to load balance caused by inefficient cache usage and inefficient array alignments for vectorization. As the code was parallelized using OpenMP, the I/O using NetCDF was done sequentially.

Example of load balance test report

Report – Recommendations were made to improve the vectorization and computational performance of the application. Specifically, a change of compiler would assist in vectorization of masked Fortran 90 array operations. Improved array alignment would also enable more efficient vectorization and a reduction in the use of floating point division would enable more efficient computation. A long term recommendation was made for the use of MPI so that the I/O can be performed via the parallel NetCDF Library, rather than sequential access methods, to make it more scalable.

Code improvements – The code improvements are being carried out by the existing development with advice available from nAG when requested.

On-going – Work is currently progressing to improve I/O and vectorize further parts of the code.

Long term recommendation of the use of MPI is being considered so that the parallel NetCDF Library could be used for parallel I/O.

Note: This work was carried out by nAG staff working under the remit of the Performance Optimisation and Productivity Centre of Excellence in Computing Applications (POP)

Real-world success stories of nAG HPC Consulting

Project Background

nAG components add statistical advantage to global retailers

In today’s highly competitive retail market retailers need the most advanced and up to date software capabilities to achieve advantage over competitors and maximize profits. DemandTec provides the retail and consumer products industries with Consumer-Centric Merchandising software, which enables the planning, optimization and execution of merchandising and marketing programs based on a holistic understanding of consumer demand. No other company has as broad a merchandising planning and optimization software suite as DemandTec, nor does any other vendor have the large number of successful deployments with blue chip companies around the world.

A Different World

Retailers today face a different world than even five years ago. Fierce competition and the availability of massive amounts of data has driven even regional retailers to optimize supply chains, pricing and other critical aspects of their business. To meet these needs, DemandTec, Inc., the retail industry’s leading Consumer-Centric Merchandising software provider, recognized the need to deliver, to retail clients, modelling and analytical capabilities that would have seemed like science fiction just a few years earlier.

To realize its vision for a software architecture that could meet these demands, DemandTec knew that it needed an advanced analytics partner that could meet its current requirements and work with its science team to rapidly create new functionality as customer requirements evolved. After an intensive evaluation process DemandTec turned to nAG to supply the mathematical and statistical components needed in their software because of nAG’s longstanding reputation for quality and a track record of supplying code for dozens of ISV applications.

Code + Expertise = Results

In addition to nAG’s deserved reputation for accurate, robust and thoroughly documented code, DemandTec was particularly attracted to nAG’s extensive statistical functionality (300+ routines) including techniques such as Mixed Effect Regression, Correlation and Regression Analysis, Multivariate Methods and Time Series Analysis to name a few. Equally important for DemandTec was nAG’s staff of algorithmic experts and software engineers who were not only available to help the DemandTec team with integration issues but were also capable of creating entirely new methods for incorporation into DemandTec’s code.

Demand Management in Action

B&Q, the premier DIY retail chain in Europe utilizes DemandTec PriceTM, a web based software application, to more effectively manage their product pricing strategy. The software helps B&Q quickly understand how customers react to price changes to provide the best value to consumers while still meeting B&Q’s business goals.

Another blue chip company reaping the rewards of DemandTec’s Consumer-Centric Merchandising software is Fortune 50 food and drug retailer Safeway. DemandTec PriceTM and DemandTec PromotionTM support Safeway’s sales and marketing objectives and goals by enabling the company to determine and plan pricing and promotions based on the needs and wants of Safeway customers.

Partnership for Success

When it comes to creating cutting edge software to create retail and consumer products success, it is virtually impossible to assemble the customer knowledge, modelling science and algorithmic expertise within a single firm. DemandTec recognized this early on in its relationship with nAG. In the words of DemandTec CEO, Dan Fishback, “Partnering with nAG brought us world class software and algorithmic expertise that was a perfect complement to our customer, data and modelling knowledge and experience. nAG software and experts gave us the confidence to translate our ideas into market-leading software for our customers.“

Rob Meyer, CEO of nAG, was involved in discussions with DemandTec from the beginning, adding “DemandTec quickly recognized how the market for demand management software was evolving and how their and our unique capabilities could combine to create unique solutions for customers. They have been an outstanding partner.“

Learn More about DemandTec

DemandTec, an IBM Company, is a network of cloud apps and insights for more than 500 retailers and consumer products companies, providing common solutions to transact, interact, and collaborate on core merchandising and marketing activities. DemandTec’s services enable customers to achieve their sales volume, revenue, shopper loyalty, and profitability objectives. Approximately 16,000 retailer and manufacturer end-users on DemandTec collaborated on more than five million trade deals to date. DemandTec software and analytical services utilize a science-based platform to model and understand consumer behavior. DemandTec customers include leading retailers and consumer products companies such as Ahold USA, Best Buy, ConAgra Foods, Delhaize America, General Mills, The Home Depot, Monoprix, PETCO, Safeway, Sara Lee, Target, Walmart, and WHSmith.

The UK’s previous national supercomputing facility, HECToR, has been heavily used by scientists needing capability supercomputing resources since its official launch in October 2007. A substantial part of the Research Councils’ six-year funding for this facility is devoted to the Computational Science and Engineering (CSE) support provided by nAG. An important part of the CSE Service is the distributed CSE (dCSE) programme which, through a lightweight peer review process, delivers dedicated multi-month performance and scalability development projects in response to proposals from the user community. The first dCSE project to complete has proven to be an excellent example of what can be achieved through dedicated CSE effort, with dramatic improvements in code performance and scalability which could potentially save millions of pounds and allow significant new science to be undertaken for the UK Car-Parrinello Consortium (UKCP).

Project Background

The objective of the dCSE project was to develop an improved, more scalable version of CASTEP – a software package which uses density functional theory with a plane wave basis set to calculate electronic properties of solids from first principles. The key task of the project was to implement band-parallelism in order to allow the code to scale to more than 1000 cores on HECToR. CASTEP is used on HECToR to model a range of materials or molecules at the atomic level. In particular scientists run CASTEP to obtain information about total energies, forces and stresses on an atomic system, as well as calculating optimum geometries, band structures, optical spectra, phonon spectra as well as molecular dynamics simulations. Dr Keith Refson from the Computational Materials Science Group at the Rutherford Appleton Laboratory was the Principal Investigator on the project, and Dr Martin Plummer from Daresbury Laboratory and Dr Matt Probert of The University of York were the Co-Investigators. nAG contracted Dr Phil Hasnip of the Department of Physics at the University of York to carry out the code development work in collaboration with both the wider CASTEP team and the nAG CSE team.

Project Results

The results of this work were excellent. The improved code has a speed-up factor of between 2 and 4 times the original and now scales to over 1000 cores against 256 previously.

The UKCP Chairman, Dr Matt Probert of the Department of Physics at the University York estimated that the CASTEP consortium was using around 10m Allocation Units (AU)s per annum on HECToR – a nominal cost of around £640k. Making the code 2-4 times more efficient could result in a saving of £320k-£480k per annum (a saving of around say £1.6m-£2.4m over the remaining life of HECToR); all for around 8 person months of effort!

Commenting on the massive return on investment, Dr Probert said 

“I guess it goes to show the value of centrally supporting key software packages, and that there is a considerable saving to be made due to scale of usage. Also the HECToR dCSE scheme is well worth supporting and continuing – and that the dCSE postdoc (Phil Hasnip in this case) was very good value for money!”

CASTEP user and PI on the project, Dr Keith Refson added 

“The performance and scaling gains achieved by the band-parallel CASTEP represent a very substantial advance in the efficiency of utilisation of CPU cycles on HECToR. This will not only result in a lower time and cost and more rapid turnaround for jobs already planned, but as intended will permit larger and more complex simulations, using more processors, which were not previously feasible.”

Dr Probert predicts that due to the speed and scaling gains resulting from CASTEP’s improvements the software can now be utilized for larger scale scientific work, i.e. bigger atomic systems for less wall-clock time and/or more simulations– within their existing budgets. In fact, several research projects have been waiting in anticipation of the planned release of the improved CASTEP in 2009.

Update

Since the writing of this case study, ARCHER has replaced HECToR as the UK’s national supercomputer.

About UKCP

The UKCP is an association of academic research groups collaborating on the first-principles computer simulation of condensed matter. Their aim is to use the creativity and strength of collaboration and shared expertise to produce outstanding science and simulation software.

About CASTEP

CASTEP is a software package which uses density functional theory with a plane wave basis set to calculate electronic properties of solids from first principles. CASTEP is a fully featured first principles code and as such its capabilities are numerous. Aiming to calculate any physical property of the system from first principles, the basic quantity is the total energy from which many other quantities are derived. For example the derivative of total energy with respect to atomic positions results in the forces and the derivative with respect to cell parameters gives stresses. These are then used to perform full geometry optimizations and possibly finite temperature molecular dynamics. Furthermore, symmetry and constraints (both internal and external to the cell) can be imposed in the calculations, either as defined by the user, or automatically using in-built symmetry detection.

Code Background

Project Background