Data Locality Techniques in an Active Cluster File System Designed for Scientific Workflows

Doctoral Dissertation

Abstract

The continued exponential growth of storage capacity has catalyzed the broad acquisition of scientific data which must be processed. While today’s large data analysis systems are highly effective at establishing data locality and eliminating inter-dependencies, they are not so easily incorporated into scientific workflows that are often complex and irregular graphs of sequential programs with multiple dependencies. To address the needs of scientific computing, I propose the design of an active storage cluster file system which allows for execution of regular unmodified applications with full data locality.

This dissertation analyzes the potential benefits of exploiting the structural information already available in scientific workflows – the explicit dependencies – to achieve a scalable and stable system. I begin with an outline of the design of the Confuga active storage cluster file system and its applicability to scientific computing. The remainder of the dissertation examines the techniques used to achieve a scalable and stable system. First, file system access by jobs is scoped to explicitly defined dependencies resolved at job dispatch. Second, workflow’s structural information is harnessed to direct and control necessary file transfers to enforce cluster stability and maintain performance. Third, control of transfers is selectively relaxed to improve performance by limiting any negative effects of centralized transfer management.

This work benefits users by providing a complete batch execution platform joined with a cluster file system. The user does not need to redesign their workflow or provide additional consideration to the management of data dependencies. System stability and performance is managed by the cluster file system while providing jobs with complete data locality.

Attributes

Attribute NameValues
Author Patrick Joseph Donnelly
Contributor Douglas Thain, Research Director
Contributor Scott Emrich, Committee Member
Contributor Collin McMillan, Committee Member
Contributor Christian Poellabauer, Committee Member
Degree Level Doctoral Dissertation
Degree Discipline Computer Science and Engineering
Degree Name Doctor of Philosophy
Defense Date
  • 2016-04-06

Submission Date 2016-04-13
Access Rights Open Access
Content License
  • All rights reserved

Departments and Units

Files

Please Note: You may encounter a delay before a download begins. Large or infrequently accessed files can take several minutes to retrieve from our archival storage system.