Investigating the scalability of applications on exascale supercomputers

Exascale computers that are capable of delivering 10¹⁸ floating-point operations per second are estimated to arrive by 2023. In fact, three pre-exascale supercomputers from the CORAL initiative in US have been scheduled for delivery either late this year or in 2018. Improving the scalability of applications running on these billion-way parallelism machines is pressed for time and also a challenging topic. 

First, we show the impact of process imbalance to applications with synchronization points. Using a Monte-Carlo simulator, we quantify the imbalance-absorption property in various synchronization algorithms. Second, we identify that self-synchronization can exist on the processes of applications with only point-to-point communication. By reconstructing the propagation of idle periods among processes, we show that idle periods on newer generation supercomputers are of the same order of the one on the older generation, indicating that process imbalance will remains as a critical issue on future supercomputers. Finally, we propose an asynchronous decoupling model to address the process imbalance on large-scale platforms. We develop a data-streaming model on top of MPI and enable different groups of processes to carry out different operations while connected by data streams. Our results demonstrate the effective of the decoupling model for applications with imbalanced processes. In the end of the talk, recent work on the emerging hybrid memory systems will also be briefly introduced.