Online data analysis and reduction: An important Co-design motif for extreme-scale computers

Date

2021-11-01

Author

Foster, Ian
Ainsworth, Mark
Bessac, Julie
Cappello, Franck
Choi, Jong
Di, Sheng
Di, Zichao
Gok, Ali M.
Guo, Hanqi
Huck, Kevin A.
Kelly, Christopher
Klasky, Scott
van Dam, Kerstin Kleese
Liang, Xin
Mehta, Kshitij
Parashar, Manish
Peterka, Tom
Pouchard, Line
Shu, Tong
Tuğluk, Ozan
van Dam, Hubertus
Wan, Lipeng
Wolf, Matthew
Wozniak, Justin M.
Xu, Wei
Yakushin, Igor
Yoo, Shinjae
Munson, Todd

Metadata

Show full item record

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Item Usage Stats

44
views

0
downloads

Cite This

A growing disparity between supercomputer computation speeds and I/O rates means that it is rapidly becoming infeasible to analyze supercomputer application output only after that output has been written to a file system. Instead, data-generating applications must run concurrently with data reduction and/or analysis operations, with which they exchange information via high-speed methods such as interprocess communications. The resulting parallel computing motif, online data analysis and reduction (ODAR), has important implications for both application and HPC systems design. Here we introduce the ODAR motif and its co-design concerns, describe a co-design process for identifying and addressing those concerns, present tools that assist in the co-design process, and present case studies to illustrate the use of the process and tools in practical settings.

URI

https://hdl.handle.net/11511/116556

Collections

Graduate School of Applied Mathematics, Article