MPI Storage Windows for High-Performance Parallel I/O

Traditional scientific applications and emerging data analytics and machine learning workloads running on supercomputers require high-performance I/O to read and store their data. As the concurrency of next-generation systems is expected to increase 100-1000x while the I/O bandwidth is expected to remain roughly constant, the I/O subsystem will become a serious bottleneck. Thus, it is of paramount importance to achieve high-performance I/O operations to support I/O workloads for both high-performance computing applications and data analytics on next-generation supercomputers.

In this talk, I will propose a novel use of MPI windows, a part of the MPI process memory that is exposed to other MPI remote processes, to simplify the programming interface and to support high-performance parallel I/O without requiring the use of MPI I/O. Files on storage devices appear to users as MPI windows (MPI storage windows) and can be seamlessly accessed through familiar "put" and "get" operations. Moreover, as memory and storage systems on supercomputers are becoming more heterogeneous, the proposed approach provides a unified programming interface to move data across the memory and storage hierarchies of large-scale supercomputers, without requiring any change in the MPI standard. I will illustrate that using MPI storage windows is intuitive and does not require drastic modifications of the existing applications.

Finally, I will conclude my presentation by showing performance results using MPI storage windows in three different mini-applications, demonstrating that it only incurs in a small additional overhead compared to MPI windows in memory in most cases. The performance benefit of using MPI storage windows with respect to traditional explicit MPI I/O read and write operations will be presented as well.