Skip to content

Commit 487417d

Browse files
committed
2 parents 55fa6b9 + 8514083 commit 487417d

File tree

4 files changed

+66
-1
lines changed

4 files changed

+66
-1
lines changed

_data/speakers.yml

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -138,4 +138,15 @@
138138
talk_num: 9
139139
photo: souza.jpg
140140
bio: "Renan Souza earned his Ph.D., M.Sc., and B.Sc. in Computer Science (2009-2019) from the Federal University of Rio de Janeiro (UFRJ). Since 2022, he has been a researcher and software engineer at Oak Ridge National Laboratory, after spending seven years at IBM. He was a visiting scientist at INRIA, France, during his Ph.D. and, during his B.Sc., studied abroad at Missouri State University and interned at SLAC National Laboratory. Active in engineering, research, and technical leadership since 2010, he has authored 50+ peer-reviewed papers in leading venues and holds 10+ USPTO patents. His current focus is on designing and building scalable systems to support responsible and trustworthy AI workflows."
141-
141+
142+
- name: Jan Janssen
143+
role: Group Leader for Materials Informatics
144+
institution:
145+
- name: Max Planck Institute for Sustainable Materials
146+
link: https://www.mpie.de/en
147+
image: max_plank.png
148+
country: de
149+
link: https://www.mpie.de/4910750/Janssen
150+
talk_num: 10
151+
photo: janssen.jpg
152+
bio: "Jan Janssen is the group leader for Materials Informatics at the Max Planck Institute for Sustainable Materials. His group focuses on applying methods from computer science including machine learning to discover novel sustainable materials with applications ranging from machine-learned interatomic potentials to large language model agents for atomistic simulation. Previously, Jan was a director’s postdoctoral fellow in the T-division at Los Alamos National Laboratory as part of the Exascale Computing Project as well as an invited postdoctoral fellow at the University of Chicago and the University of California Los Angeles. Besides his research work, Jan is the lead developer of the pyiron atomistic simulation suite, maintains over 1000 open-source materials informatics software packages for the conda-forge community and is a regular contributor to open-source software on Github."

_talks/2026_01_14.html

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
---
2+
layout: talk
3+
title: "Up-scaling Python functions for HPC with executorlib"
4+
authors: Jan Janssen (Max Planck Institute for Sustainable Materials)
5+
event_date: January 14, 2026
6+
times: 11:00-11:30 PST / 14:00-14:30 EST / 20:00-20:30 CEST
7+
talk_number: 10
8+
given: false
9+
<!-- image: /images/talks/janssen-banner.png -->
10+
<!-- presentation: -->
11+
<!-- video: -->
12+
---
13+
14+
The up-scaling of Python workflows from the execution on a local workstation to
15+
the parallel execution on an HPC typically faces three challenges: (1) the
16+
management of inter-process communication, (2) the data storage and (3) the
17+
management of task dependencies during the execution. These challenges commonly
18+
lead to a rewrite of major parts of the reference serial Python workflow to
19+
improve computational efficiency. Executorlib addresses these challenges by
20+
extending Python's ProcessPoolExecutor interface to distribute Python functions
21+
on HPC systems. It interfaces with the job scheduler directly without the need
22+
for a database or daemon process, leading to seamless up-scaling.
23+
24+
<br /><br />
25+
26+
The presentation introduces the challenge of up-scaling Python workflows. It
27+
highlights how executorlib extends the ProcessPoolExecutor interface of the
28+
Python standard library to provide the user with a familiar interface, while
29+
the executorlib backend directly connects to the HPC job scheduler to distribute
30+
Python functions either from the login node to individual compute nodes or
31+
within an HPC allocation of a number of compute nodes, which is enabled by
32+
supporting both file-based and socket-based communication.
33+
34+
<br /><br />
35+
36+
The setup of executorlib on different HPC systems is introduced, based on the
37+
current support for the SLURM job scheduler as well as the Flux framework to
38+
enable hierarchical scheduling within large HPC job allocations as commonly
39+
used on Exascale computers. Application examples are then given to demonstrate
40+
how executorlib supports the assignment of computational resources like CPU
41+
cores, number of threads and GPU resources on a per-function basis, including
42+
support for MPI, which drastically simplifies the process of up-scaling Python
43+
workflows.
44+
45+
<br /><br />
46+
47+
In this context, the focus of this presentation is the user journey during the
48+
up-scaling of a Python workflow and how features like caching or the integrated
49+
debugging capabilities for the distributed execution of Python functions
50+
accelerate the development cycle. The presentation concludes by returning to
51+
challenges identified as part of DOE Exascale Computing Project's EXAALT effort
52+
to demonstrate how the development process was drastically simplified by using
53+
executorlib, with a specific focus on dynamic dependencies which are only
54+
resolved during run time of the Python workflow.

images/institutions/max_plank.png

45.5 KB
Loading

images/talks/janssen.jpg

119 KB
Loading

0 commit comments

Comments
 (0)