HPCng is an open community of people and organizations interested in the broad modernization of HPC capabilities across a wide range of use-cases ranging from traditional HPC to enterprise and hyper-scale workloads.
High-Performance Computing (HPC), a once niche use-case, has been growing in terms of applicability over the last 5-10 years. We are seeing large scale, tightly integrated HPC resources beginning to run non-traditional HPC workloads in what we commonly refer to as “the long tail of scientific computing”. As that long tail is growing, it is becoming increasingly common for new applications, use-cases, and science domains to be making use of traditional high-performance computing infrastructures. Additionally, HPC environments are not being used only in science, adoption is growing into the enterprise via the advent of artificial intelligence training as well as compute and data-driven analytics.
Today when we refer to HPC, we no longer are referring only to those tightly coupled parallel MPI based applications. We are referring to any applications, or series of applications, which are designed to run a given workload as fast as the hardware will allow. This usually means that the performance-critical application will utilize a given subsystem of a hardware stack completely, whether that be CPU, memory, storage, network, GPU, PCI, etc. This could be on a single compute node, a massive GPU workhorse, CPU farm, or a tightly coupled parallel processing infrastructure. For the purpose of this charter, HPC inclusively refers to all of these types of workloads as HPC.
So while “HPC” is becoming increasingly diverse, the architecture to which it is built has not changed considerably over the past two and a half decades. This base architecture, commonly referred to as the “Beowulf” design, is predisposed to being flat, isolated and discrete which has imposed difficulties with integration of non-HPC specific capabilities.
This divide has been increasingly causing separation as enterprise architectures, like micro-services, containers, orchestration, etc., have been readily innovating over the past decade and creating a lot of opportunities for optimization of development, operations, security, automation, reliability, and scale. But these new capabilities are not easily transcribed into the HPC sector due to misalignments of base architectures and thus cross-pollination is not occurring. This leads to the replication of technology and a lack of benefit of experience and capabilities for everyone. HPCng strives to build a community of diverse brilliant people to consider these challenges and work together on solutions.
We, the members of the HPCng community, strive to create a diverse community of backgrounds, ideas, skill sets, and perspectives. Such diverse perspectives are needed to unite the currently disparate capabilities of HPC and enterprise to allow for running advanced computational and data analytics at scale.
Engaging with these respective communities will enable a greater understanding of requirements as well as existing technologies that could be used as part of this vision. This will facilitate cross-pollination to create technologies and solve problems across the ecosystem.
“The convergence of AI, data analytics and traditional simulation will result in systems with broader capabilities and configurability as well as cross pollination.” -- Dr. Al Gara, Intel