Exploiting Data Locality in Swift T workflows
The ever-increasing power of supercomputer systems is both driving and enabling the emergence of new problem-solving methods that require the efficient execution of many concurrent and interacting tasks. Swift/T is a language and runtime for dynamically creating and executing workflows of tasks, varying in granularity, on high-component-count platforms. Swift/T uses Asynchronous Dynamic Load Balancer (ADLB) to dynamically distribute the tasks among the nodes and relies on a shared file system for data communication among tasks.
The objective of this work is to expose and exploit data locality in Swift/T through Hercules, a distributed memory store based on Memcached and to explore tradeoffs between data locality and load balance in distributed workflow executions. In this talk I will present our approach to enable locality-based optimizations in Swift/T by guiding ADLB to schedule computation jobs in the nodes containing the required data and an analysis of the interaction between locality and load balance. Initial measurements based on various workflow patterns (including a MapReduce benchmark) show promising results. Finally, I will discuss current work of integrating novel locality-aware and adaptive load-aware data distribution policies in Hercules.
About the Presenter:
Francisco Rodrigo Duro is a Ph.D. student working on distributed caching systems at the University Carlos III of Madrid, Spain. He obtained a M.S. in computer science from the same university. His current research interests include parallel I/O and distributed systems.