The Next Generation of High Performance Computing

High-Performance Computing (HPC), a once niche use-case, has been growing in terms of applicability over the last 5-10 years. We are seeing large scale, tightly integrated HPC resources beginning to run non-traditional HPC workloads in what we commonly refer to as “the long tail of scientific computing”. As that long tail is growing, it is becoming increasingly common for new applications, use-cases, and science domains to be making use of traditional high-performance computing infrastructures. Additionally, HPC environments are not being used only in science, adoption is growing into the enterprise via the advent of artificial intelligence training as well as compute and data-driven analytics.

Today when we refer to HPC, we no longer are referring only to those tightly coupled parallel MPI based applications. We are referring to any applications, or series of applications, which are designed to run a given workload as fast as the hardware will allow. This usually means that the performance-critical application will utilize a given subsystem of a hardware stack completely, whether that be CPU, memory, storage, network, GPU, PCI, etc. This could be on a single compute node, a massive GPU workhorse, CPU farm, or a tightly coupled parallel processing infrastructure. For the purpose of this charter, HPC inclusively refers to all of these types of workloads as HPC.

So while “HPC” is becoming increasingly diverse, the architecture to which it is built has not changed considerably over the past two and a half decades. This base architecture, commonly referred to as the “Beowulf” design, is predisposed to being flat, isolated and discrete which has imposed difficulties with integration of non-HPC specific capabilities.

This divide has been increasingly causing separation as enterprise architectures, like micro-services, containers, orchestration, etc., have been readily innovating over the past decade and creating a lot of opportunities for optimization of development, operations, security, automation, reliability, and scale. But these new capabilities are not easily transcribed into the HPC sector due to misalignments of base architectures and thus cross-pollination is not occurring. This leads to the replication of technology and a lack of benefit of experience and capabilities for everyone. HPCng strives to build a community of diverse brilliant people to consider these challenges and work together on solutions.