Makeflow and Work Queue: Scaling Up from Laptops to Supercomputers

Presenter: Nate Kremer-Herman, Graduate Research Assistant, Cooperative Computing Lab, University of Notre Dame

Date: March 22, 2017

Slides: https://uofi.box.com/s/cc43do38j4pvnyzs4chwzp0izhav0kht

Video: https://www.youtube.com/watch?v=s4NitYI6BXg&list=PLO8UWE9gZTlAMRvvVfS7-6q3x1DrXKmkR&index=5

Abstract

This seminar presentation will introduce attendees to the workflow management system Makeflow along with an associated execution engine called Work Queue. These tools are designed to provide a generic, user-friendly, and scalable interface for workflow management. During this presentation, we will cover the steps of writing a makeflow, executing a makeflow locally, and running a makeflow via batch system using Work Queue. We will make use of a brief, hands-on tutorial. By the end of the seminar, an attendee will be able to get started writing their own makeflows and use Work Queue to effectively scale their research up to their campus- or national-scale infrastructure.

Target Audience: Research scientists, graduate students, upperclass undergraduate students of any domain.

Prerequisites: Basic Linux knowledge.?

User Base: We currently have users in bioinformatics, high energy physics, computational chemistry. We are always looking to help users in other domains get started.?

Software Availability: Our software is open source and available in its totality on GitHub.?

Software Requirements: Linux or Mac OS X operating system.?

Use cases: We present a generic platform, I will be presenting a fairly generic tutorial. When considering examples in the presentation portion of the seminar, I will cover applications in bioinformatics, high energy physics, and molecular dynamics.?

Training Materials: Our most recent tutorial can be found here.

Biography

Nate Kremer-Herman is a second year PhD student working in the Cooperative Computing Lab at the University of Notre Dame. Nate is interested in exploring storage constraints and scaling issues in parallel applications. He is currently in charge of outreach in his research group, and he helps introduce new users to the Makeflow workflow specification language and the Work Queue execution engine. These tools are designed to help scientists scale up their research, and making sure that experience is user-friendly and approachable is one of Nate’s top priorities.