CCLTechnicalReportMPIWorkflows.pdf (600.19 kB)
Understanding Dramatic Performance Differences in Workflow/Middleware/Site Combinations: CCL Technical Report October 15th, 2018
journal contribution
posted on 2018-11-05, 00:00 authored by Douglas Thain, Kyle SweeneyScientists using workflows often have access to both High Performance Computing and High-Throughput Computing sites, but HPC sites' architecture is less conducive to HTC paradigms. The choices of middleware and site can have drastic performance differences on different workflows. To explore these differences, we created tools to expand Makeflow and Work Queue's capabilities. We then performed four speed of light tests, testing job dispatch rate, data delivery from the master to worker, system bandwidth, and meta-data operations. We then conducted three synthetic workflow tests, a pure data consumptive workflow, a data selectivity workflow, and a data-generating workflow. Finally, we tested our middleware with three real world workflows, BWA-GATK, BLAST, and Lifemapper. We created a short guide which helps guide users in matching site, workflow, and middleware.
History
Date Modified
2018-11-05Language
- English
Usage metrics
Categories
No categories selectedLicence
Exports
RefWorks
BibTeX
Ref. manager
Endnote
DataCite
NLM
DC