Jupyter as the Interface to High Performance Computing

Presenter: Matthias Bussonnier, Research Facilitator, UC Merced; Shreyas Cholia, Group Leader (Usable Software Systems), Computational Research Division / National Energy Research Scientific Computing Center, Lawrence Berkeley National Laboratory; and Rollin Thomas, Big Data Architect, National Energy Research Scientific Computing Center, Lawrence Berkeley National Laboratory

Date: April 24, 2019

Slides: https://uofi.box.com/s/gxf2i23zl3kfoc1kjbxajeeq551eswpq

Jupyter at NERSC slides: https://uofi.box.com/s/0nuv3zghjchvu8zi4uxtq9ipkt4aavv0

Video: https://youtu.be/P1hIVZL_7jA


During the last few years the Jupyter notebook has become one of the tools of choice for the data science and high-performance computing (HPC) communities. This webinar will provide an overview of why Jupyter is gaining traction in education, data science, and HPC, with emphasis on how notebooks can be used as interactive documents for exploration and reporting. We will present an overview of how Jupyter works and how the network protocol can be leveraged for both a local single machine and remote-cluster work. We will discuss the nuts and bolts of how Jupyter has been deployed at NERSC as a case study in implementation of Jupyter in an HPC environment. This work implies learning the Jupyter ecosystem to take advantage of its powerful abstractions to develop custom infrastructure to satisfy policies and user needs.

Prerequisites: None.
Target Audience: Jupyter users, Python enthusiasts, HPC practitioners, HPC facility staff and management.


Matthias Bussonnier is a Research Facilitator at UC Merced and one of the founding members and core contributors to the Jupyter project.

Shreyas Cholia leads the Usable Software Systems Group in the Computational Research Division at Lawrence Berkeley National Laboratory. His group focuses on the usability aspects of computational and data analysis systems. He is involved in various Jupyter related efforts at NERSC and LBL. Projects include the NERSC JupyterHub deployment and seamless integration of Jupyter with remote distributed resources at scale.

Rollin Thomas is a data architect in the Data and Analytics Services group at NERSC. He directs Python support on NERSC’s Cray XC30 (Edison) and XC40 (Cori) systems, oversees the NERSC Exascale Science Application Program for Data, and is part of the Spin containers-as-a-service team. He also leads efforts to enable interactive supercomputing at NERSC through Jupyter notebooks.