SI2-SSI: FAMII: High Performance and Scalable Fabric Analysis, Monitoring and Introspection Infrastructure for HPC and Big Data

Ohio State University, The, Columbus OH

Investigators

Abstract

As the computing, networking, heterogeneous hardware, and storage technologies continue to evolve in High-End Computing (HEC) platforms, it becomes increasingly essential and challenging to understand the interactions between time-critical High-Performance Computing (HPC) and Big Data applications, the software infrastructures upon which they rely for achieving high-performing portable solutions, the underlying communication fabric these high-performance middlewares depend on and the schedulers that manage HPC clusters. Such understanding will enable all involved parties (application developers/users, system administrators, and middleware developers) to maximize the efficiency and performance of the individual components that comprise a modern HPC system and solve different grand challenge problems. There is a clear need and unfortunate lack of a high-performance and scalable tool that is capable of analyzing and correlating the communication on the fabric with the behavior of HPC/Big Data applications, underlying middleware and the job scheduler on existing large HPC systems. The proposed synergistic and collaborative effort, undertaken by a team of computer and computational scientists from OSU and OSC, aims to create an integrated software infrastructure for high-performance and scalable Fabric Analysis, Monitoring and Introspection for HPC and Big Data. This tool will achieve the following objectives: 1) be portable, easy to use and easy to understand, 2) have high performance and scalable rendering and storage techniques and, 3) be applicable to the different communication fabrics and programming models that are likely to be used on existing large HPC systems and emerging exascale systems. The transformative impact of the proposed research and development effort is to design a comprehensive analysis and performance monitoring tool for applications of current and next generation multi petascale/exascale systems to harness the maximum performance and scalability. The proposed research and the associated infrastructure will have a significant impact on enabling optimizations of HPC and Big Data applications that have previously been difficult to provide. These potential outcomes will be demonstrated by using the proposed framework to validate a variety of HPC and Big Data benchmarks and applications under multiple scenarios. The integrated middleware and tools will be made publicly available to the community through public repositories and publications in the top forums, enabling other MPI and Big Data stacks to adopt the designs. Research results will also be disseminated to the collaborating organizations of the investigators to impact their HPC software products and applications. The proposed research directions and their solutions will be used in the curriculum of the PIs to train undergraduate and graduate students, including under-represented minorities and female students. The technical challenges addressed by the proposal include: 1) Scalable visualization of large and complex HEC networks so as to provide a near instant rendering to end users, 2) A generalized data gathering scheme which is easily portable to multiple communication fabrics, novel compute architectures and high-performance middleware, 3) Enhanced data storage performance through optimized database schemas and the use of memory-backed key value stores/databases, 4) Support in MPI, PGAS, and Big Data libraries to enable the proposed monitoring, analysis, and introspection framework, and 5) Enabling deeper introspection of particular regions of application. The research will also be driven by a set of HPC and Big Data applications. The transformative impact of the proposed research and development effort is to design a comprehensive analysis and performance monitoring tool for applications of current and next generation multi petascale/exascale systems to harness the maximum performance and scalability.