Getting Started#
executorlib extends the Executor interface from the Python standard library’s concurrent.futures to distribute Python functions as jobs on an HPC cluster, without having to rewrite the functions themselves. This notebook walks through submitting a serial function, an MPI-parallel function, and nested executors via SLURM.
Based on the cmti and cmmg clusters hosted at the MPCDF for the MPI for Sustainable Materials.
import executorlib
executorlib wraps Python functions as SLURM jobs. SlurmClusterExecutor submits each function call as a separate sbatch job, while SlurmJobExecutor (used later for nested tasks) runs steps inside an already-running job allocation.
executorlib.__version__
'0.0.1'
executorlib.__path__
['/u/janj/projects/executorlib/src/executorlib']
Submit Python Function to SLURM#
This is just like using sbatch to submit shell scripts. See the SlurmClusterExecutor documentation for the full set of options.
def get_slurm_env():
import os
from time import sleep
sleep(5)
return {
k:v for k, v in os.environ.items() if "SLURM_" in k
}
get_slurm_env is the function that will be sent to the cluster. It collects all SLURM_* environment variables from inside the running job, so we can verify which resources SLURM actually assigned. The sleep(5) gives the job a moment to settle before sampling the environment.
submission_template = """\
#!/bin/bash
#SBATCH --output=time.out
#SBATCH --job-name={{job_name}}
#SBATCH --chdir={{working_directory}}
#SBATCH --get-user-env=L
#SBATCH --partition={{partition}}
{%- if run_time_max %}
#SBATCH --time={{ [1, run_time_max // 60]|max }}
{%- endif %}
{%- if dependency %}
#SBATCH --dependency=afterok:{{ dependency | join(',') }}
{%- endif %}
{%- if memory_max %}
#SBATCH --mem={{memory_max}}G
{%- endif %}
#SBATCH --ntasks={{cores}}
{{command}}
"""
The submission template is a Jinja2 template for the sbatch script. The line #SBATCH --ntasks={{cores}} is the key mapping: the cores value in executorlib’s resource_dict becomes the number of SLURM tasks (i.e. processes). For a serial job cores=1 requests a single process. For an MPI job cores=N requests N processes — one per MPI rank. The --mem, --time, and --dependency lines are only emitted when those keys are present in resource_dict.
with executorlib.SlurmClusterExecutor() as exe:
f1 = exe.submit(
get_slurm_env,
resource_dict={
"submission_template": submission_template,
"run_time_max": 180, # in seconds
# "partition": "s.cmfe",
"partition": "s.cmmg",
"cores": 1,
"threads_per_core": 4,
})
result = f1.result()
print(result)
/u/janj/projects/executorlib/src/executorlib/task_scheduler/base.py:156: UserWarning: The following keys are not recognized and cannot be validated: ['partition']
self._validator(resource_dict=resource_dict)
{'SLURM_MEM_PER_CPU': '1500', 'SLURM_NODEID': '0', 'SLURM_TASK_PID': '272848', 'SLURM_PRIO_PROCESS': '0', 'SLURM_SUBMIT_DIR': '/cmmc/u/janj/notebooks/2026/2026-06-03-prabhath-exe/executorlib_cache/get_slurm_envccfcb90850bb2803c9b37c6ecde017ee', 'SLURM_CELL': 'CMMC', 'SLURM_PROCID': '0', 'SLURM_JOB_GID': '12500', 'SLURM_JOB_END_TIME': '1781017883', 'SLURM_TASKS_PER_NODE': '4', 'SLURM_NNODES': '1', 'SLURM_GET_USER_ENV': '1', 'SLURM_JOB_START_TIME': '1781017703', 'SLURM_JOB_NODELIST': 'cmmg001', 'SLURM_CLUSTER_NAME': 'cmmc', 'SLURM_NODELIST': 'cmmg001', 'SLURM_NTASKS': '4', 'SLURM_JOB_CPUS_PER_NODE': '8', 'SLURM_TOPOLOGY_ADDR': 'xxx-xxx-xxx.cmmg001', 'SLURM_JOB_NAME': 'executorlib_cache', 'SLURM_JOBID': '20332320', 'SLURM_CONF': '/etc/slurm/slurm.conf', 'SLURM_JOB_QOS': 'normal', 'SLURM_TOPOLOGY_ADDR_PATTERN': 'switch.node', 'SLURM_CPUS_ON_NODE': '8', 'SLURM_JOB_NUM_NODES': '1', 'SLURM_JOB_UID': '35052', 'SLURM_JOB_PARTITION': 's.cmmg', 'SLURM_SCRIPT_CONTEXT': 'prolog_task', 'SLURM_JOB_USER': 'janj', 'SLURM_NPROCS': '4', 'SLURM_SUBMIT_HOST': 'cmti001', 'SLURM_JOB_ACCOUNT': 'mpd', 'SLURM_GTIDS': '0', 'SLURM_JOB_ID': '20332320', 'SLURM_OOM_KILL_STEP': '1', 'SLURM_LOCALID': '0'}
Here we submit the serial function. cores=1 requests one process (--ntasks=1). threads_per_core=4 tells executorlib how many CPU threads are available to that process — this maps to --cpus-per-task in SLURM terminology. The returned Future object f1 is resolved by calling f1.result(), which blocks until the SLURM job completes and returns the function’s return value.
from executorlib import get_cache_data
import subprocess
cache_dir = "executorlib_cache"
for d in get_cache_data(cache_dir):
if "get_slurm_env" in str(d["function"]):
print(d['queue_id'])
shell_output = subprocess.check_output(["sacct", "-j", str(d['queue_id'])], shell=False, universal_newlines=True)
20332320
get_cache_data reads the on-disk cache that executorlib writes for every submitted function. The queue_id field is the SLURM job ID, which we pass to sacct to query the scheduler’s accounting database. sacct confirms that the job ran, which partition it used, and how many CPUs were allocated.
shell_output.split("\n")
['JobID JobName Partition Account AllocCPUS State ExitCode ',
'------------ ---------- ---------- ---------- ---------- ---------- -------- ',
'20332320 executorl+ s.cmmg mpd 8 COMPLETED 0:0 ',
'20332320.ba+ batch mpd 8 COMPLETED 0:0 ',
'20332320.ex+ extern mpd 8 COMPLETED 0:0 ',
'']
Submit MPI parallel Python function to SLURM#
One of the key features of executorlib is supporting mpi4py parallel Python functions. See the MPI support section of the installation guide for the required dependencies.
def get_slurm_parallel_env():
import os
from mpi4py import MPI
comm = MPI.COMM_WORLD
size = comm.Get_size()
rank = comm.Get_rank()
env_dict = os.environ
data = {
k:v for k, v in os.environ.items() if "SLURM_" in k
}
data = comm.gather(data, root=0)
return data
get_slurm_parallel_env is an MPI-parallel function. Each MPI rank calls os.environ independently, then comm.gather collects all per-rank dictionaries onto rank 0, which returns the combined list. Because this function uses mpi4py, executorlib must launch it with srun rather than a plain Python call — that is why pmi_mode="pmix" is passed to the executor.
The important part here is the pmi_mode="pmix" to use the right backend:
with executorlib.SlurmClusterExecutor(pmi_mode="pmix") as exe:
f1 = exe.submit(
get_slurm_parallel_env,
resource_dict={
"submission_template": submission_template,
"run_time_max": 180, # in seconds
# "partition": "s.cmfe",
"partition": "s.cmmg",
"cores": 4,
"threads_per_core": 4,
})
result = f1.result()
print(result)
/u/janj/projects/executorlib/src/executorlib/task_scheduler/base.py:156: UserWarning: The following keys are not recognized and cannot be validated: ['partition']
self._validator(resource_dict=resource_dict)
[[{'SLURM_MEM_PER_CPU': '1500', 'SLURM_NODEID': '0', 'SLURM_TASK_PID': '272999', 'SLURM_PRIO_PROCESS': '0', 'SLURM_SUBMIT_DIR': '/cmmc/u/janj/notebooks/2026/2026-06-03-prabhath-exe/executorlib_cache/get_slurm_parallel_env24890a8e967e7328abd85429d1ce0371', 'SLURM_CELL': 'CMMC', 'SLURM_PROCID': '0', 'SLURM_JOB_GID': '12500', 'SLURM_JOB_END_TIME': '1781017894', 'SLURM_TASKS_PER_NODE': '4', 'SLURM_NNODES': '1', 'SLURM_GET_USER_ENV': '1', 'SLURM_JOB_START_TIME': '1781017714', 'SLURM_JOB_NODELIST': 'cmmg001', 'SLURM_CLUSTER_NAME': 'cmmc', 'SLURM_NODELIST': 'cmmg001', 'SLURM_NTASKS': '4', 'SLURM_JOB_CPUS_PER_NODE': '32', 'SLURM_TOPOLOGY_ADDR': 'xxx-xxx-xxx.cmmg001', 'SLURM_JOB_NAME': 'executorlib_cache', 'SLURM_JOBID': '20332321', 'SLURM_CONF': '/etc/slurm/slurm.conf', 'SLURM_JOB_QOS': 'normal', 'SLURM_TOPOLOGY_ADDR_PATTERN': 'switch.node', 'SLURM_CPUS_ON_NODE': '32', 'SLURM_JOB_NUM_NODES': '1', 'SLURM_JOB_UID': '35052', 'SLURM_JOB_PARTITION': 's.cmmg', 'SLURM_SCRIPT_CONTEXT': 'prolog_task', 'SLURM_JOB_USER': 'janj', 'SLURM_NPROCS': '4', 'SLURM_SUBMIT_HOST': 'cmti001', 'SLURM_JOB_ACCOUNT': 'mpd', 'SLURM_GTIDS': '0,1,2,3', 'SLURM_JOB_ID': '20332321', 'SLURM_OOM_KILL_STEP': '1', 'SLURM_LOCALID': '0', 'SLURM_MPI_TYPE': 'pmix', 'SLURM_UMASK': '0022', 'SLURM_DISTRIBUTION': 'block', 'SLURM_STEP_ID': '0', 'SLURM_STEPID': '0', 'SLURM_STEP_NODELIST': 'cmmg001', 'SLURM_STEP_NUM_NODES': '1', 'SLURM_STEP_NUM_TASKS': '4', 'SLURM_STEP_TASKS_PER_NODE': '4', 'SLURM_STEP_LAUNCHER_PORT': '36151', 'SLURM_SRUN_COMM_PORT': '36151', 'SLURM_PMIXP_ABORT_AGENT_PORT': '36473', 'SLURM_PMIX_MAPPING_SERV': '(vector,(0,1,4))', 'SLURM_SRUN_COMM_HOST': '130.183.186.16', 'SLURM_CPU_BIND': 'quiet,mask_cpu:0x00000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000001,0x00000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000002,0x00000000000000000000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000000004,0x00000000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000008', 'SLURM_CPU_BIND_LIST': '0x00000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000001,0x00000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000002,0x00000000000000000000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000000004,0x00000000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000008', 'SLURM_CPU_BIND_TYPE': 'mask_cpu:', 'SLURM_CPU_BIND_VERBOSE': 'quiet', 'SLURM_LAUNCH_NODE_IPADDR': '130.183.186.16'}, {'SLURM_MEM_PER_CPU': '1500', 'SLURM_NODEID': '0', 'SLURM_TASK_PID': '273000', 'SLURM_PRIO_PROCESS': '0', 'SLURM_SUBMIT_DIR': '/cmmc/u/janj/notebooks/2026/2026-06-03-prabhath-exe/executorlib_cache/get_slurm_parallel_env24890a8e967e7328abd85429d1ce0371', 'SLURM_CELL': 'CMMC', 'SLURM_PROCID': '1', 'SLURM_JOB_GID': '12500', 'SLURM_JOB_END_TIME': '1781017894', 'SLURM_TASKS_PER_NODE': '4', 'SLURM_NNODES': '1', 'SLURM_GET_USER_ENV': '1', 'SLURM_JOB_START_TIME': '1781017714', 'SLURM_JOB_NODELIST': 'cmmg001', 'SLURM_CLUSTER_NAME': 'cmmc', 'SLURM_NODELIST': 'cmmg001', 'SLURM_NTASKS': '4', 'SLURM_JOB_CPUS_PER_NODE': '32', 'SLURM_TOPOLOGY_ADDR': 'xxx-xxx-xxx.cmmg001', 'SLURM_JOB_NAME': 'executorlib_cache', 'SLURM_JOBID': '20332321', 'SLURM_CONF': '/etc/slurm/slurm.conf', 'SLURM_JOB_QOS': 'normal', 'SLURM_TOPOLOGY_ADDR_PATTERN': 'switch.node', 'SLURM_CPUS_ON_NODE': '32', 'SLURM_JOB_NUM_NODES': '1', 'SLURM_JOB_UID': '35052', 'SLURM_JOB_PARTITION': 's.cmmg', 'SLURM_SCRIPT_CONTEXT': 'prolog_task', 'SLURM_JOB_USER': 'janj', 'SLURM_NPROCS': '4', 'SLURM_SUBMIT_HOST': 'cmti001', 'SLURM_JOB_ACCOUNT': 'mpd', 'SLURM_GTIDS': '0,1,2,3', 'SLURM_JOB_ID': '20332321', 'SLURM_OOM_KILL_STEP': '1', 'SLURM_LOCALID': '1', 'SLURM_MPI_TYPE': 'pmix', 'SLURM_UMASK': '0022', 'SLURM_DISTRIBUTION': 'block', 'SLURM_STEP_ID': '0', 'SLURM_STEPID': '0', 'SLURM_STEP_NODELIST': 'cmmg001', 'SLURM_STEP_NUM_NODES': '1', 'SLURM_STEP_NUM_TASKS': '4', 'SLURM_STEP_TASKS_PER_NODE': '4', 'SLURM_STEP_LAUNCHER_PORT': '36151', 'SLURM_SRUN_COMM_PORT': '36151', 'SLURM_PMIXP_ABORT_AGENT_PORT': '36473', 'SLURM_PMIX_MAPPING_SERV': '(vector,(0,1,4))', 'SLURM_SRUN_COMM_HOST': '130.183.186.16', 'SLURM_CPU_BIND': 'quiet,mask_cpu:0x00000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000001,0x00000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000002,0x00000000000000000000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000000004,0x00000000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000008', 'SLURM_CPU_BIND_LIST': '0x00000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000001,0x00000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000002,0x00000000000000000000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000000004,0x00000000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000008', 'SLURM_CPU_BIND_TYPE': 'mask_cpu:', 'SLURM_CPU_BIND_VERBOSE': 'quiet', 'SLURM_LAUNCH_NODE_IPADDR': '130.183.186.16'}, {'SLURM_MEM_PER_CPU': '1500', 'SLURM_NODEID': '0', 'SLURM_TASK_PID': '273001', 'SLURM_PRIO_PROCESS': '0', 'SLURM_SUBMIT_DIR': '/cmmc/u/janj/notebooks/2026/2026-06-03-prabhath-exe/executorlib_cache/get_slurm_parallel_env24890a8e967e7328abd85429d1ce0371', 'SLURM_CELL': 'CMMC', 'SLURM_PROCID': '2', 'SLURM_JOB_GID': '12500', 'SLURM_JOB_END_TIME': '1781017894', 'SLURM_TASKS_PER_NODE': '4', 'SLURM_NNODES': '1', 'SLURM_GET_USER_ENV': '1', 'SLURM_JOB_START_TIME': '1781017714', 'SLURM_JOB_NODELIST': 'cmmg001', 'SLURM_CLUSTER_NAME': 'cmmc', 'SLURM_NODELIST': 'cmmg001', 'SLURM_NTASKS': '4', 'SLURM_JOB_CPUS_PER_NODE': '32', 'SLURM_TOPOLOGY_ADDR': 'xxx-xxx-xxx.cmmg001', 'SLURM_JOB_NAME': 'executorlib_cache', 'SLURM_JOBID': '20332321', 'SLURM_CONF': '/etc/slurm/slurm.conf', 'SLURM_JOB_QOS': 'normal', 'SLURM_TOPOLOGY_ADDR_PATTERN': 'switch.node', 'SLURM_CPUS_ON_NODE': '32', 'SLURM_JOB_NUM_NODES': '1', 'SLURM_JOB_UID': '35052', 'SLURM_JOB_PARTITION': 's.cmmg', 'SLURM_SCRIPT_CONTEXT': 'prolog_task', 'SLURM_JOB_USER': 'janj', 'SLURM_NPROCS': '4', 'SLURM_SUBMIT_HOST': 'cmti001', 'SLURM_JOB_ACCOUNT': 'mpd', 'SLURM_GTIDS': '0,1,2,3', 'SLURM_JOB_ID': '20332321', 'SLURM_OOM_KILL_STEP': '1', 'SLURM_LOCALID': '2', 'SLURM_MPI_TYPE': 'pmix', 'SLURM_UMASK': '0022', 'SLURM_DISTRIBUTION': 'block', 'SLURM_STEP_ID': '0', 'SLURM_STEPID': '0', 'SLURM_STEP_NODELIST': 'cmmg001', 'SLURM_STEP_NUM_NODES': '1', 'SLURM_STEP_NUM_TASKS': '4', 'SLURM_STEP_TASKS_PER_NODE': '4', 'SLURM_STEP_LAUNCHER_PORT': '36151', 'SLURM_SRUN_COMM_PORT': '36151', 'SLURM_PMIXP_ABORT_AGENT_PORT': '36473', 'SLURM_PMIX_MAPPING_SERV': '(vector,(0,1,4))', 'SLURM_SRUN_COMM_HOST': '130.183.186.16', 'SLURM_CPU_BIND': 'quiet,mask_cpu:0x00000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000001,0x00000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000002,0x00000000000000000000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000000004,0x00000000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000008', 'SLURM_CPU_BIND_LIST': '0x00000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000001,0x00000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000002,0x00000000000000000000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000000004,0x00000000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000008', 'SLURM_CPU_BIND_TYPE': 'mask_cpu:', 'SLURM_CPU_BIND_VERBOSE': 'quiet', 'SLURM_LAUNCH_NODE_IPADDR': '130.183.186.16'}, {'SLURM_MEM_PER_CPU': '1500', 'SLURM_NODEID': '0', 'SLURM_TASK_PID': '273002', 'SLURM_PRIO_PROCESS': '0', 'SLURM_SUBMIT_DIR': '/cmmc/u/janj/notebooks/2026/2026-06-03-prabhath-exe/executorlib_cache/get_slurm_parallel_env24890a8e967e7328abd85429d1ce0371', 'SLURM_CELL': 'CMMC', 'SLURM_PROCID': '3', 'SLURM_JOB_GID': '12500', 'SLURM_JOB_END_TIME': '1781017894', 'SLURM_TASKS_PER_NODE': '4', 'SLURM_NNODES': '1', 'SLURM_GET_USER_ENV': '1', 'SLURM_JOB_START_TIME': '1781017714', 'SLURM_JOB_NODELIST': 'cmmg001', 'SLURM_CLUSTER_NAME': 'cmmc', 'SLURM_NODELIST': 'cmmg001', 'SLURM_NTASKS': '4', 'SLURM_JOB_CPUS_PER_NODE': '32', 'SLURM_TOPOLOGY_ADDR': 'xxx-xxx-xxx.cmmg001', 'SLURM_JOB_NAME': 'executorlib_cache', 'SLURM_JOBID': '20332321', 'SLURM_CONF': '/etc/slurm/slurm.conf', 'SLURM_JOB_QOS': 'normal', 'SLURM_TOPOLOGY_ADDR_PATTERN': 'switch.node', 'SLURM_CPUS_ON_NODE': '32', 'SLURM_JOB_NUM_NODES': '1', 'SLURM_JOB_UID': '35052', 'SLURM_JOB_PARTITION': 's.cmmg', 'SLURM_SCRIPT_CONTEXT': 'prolog_task', 'SLURM_JOB_USER': 'janj', 'SLURM_NPROCS': '4', 'SLURM_SUBMIT_HOST': 'cmti001', 'SLURM_JOB_ACCOUNT': 'mpd', 'SLURM_GTIDS': '0,1,2,3', 'SLURM_JOB_ID': '20332321', 'SLURM_OOM_KILL_STEP': '1', 'SLURM_LOCALID': '3', 'SLURM_MPI_TYPE': 'pmix', 'SLURM_UMASK': '0022', 'SLURM_DISTRIBUTION': 'block', 'SLURM_STEP_ID': '0', 'SLURM_STEPID': '0', 'SLURM_STEP_NODELIST': 'cmmg001', 'SLURM_STEP_NUM_NODES': '1', 'SLURM_STEP_NUM_TASKS': '4', 'SLURM_STEP_TASKS_PER_NODE': '4', 'SLURM_STEP_LAUNCHER_PORT': '36151', 'SLURM_SRUN_COMM_PORT': '36151', 'SLURM_PMIXP_ABORT_AGENT_PORT': '36473', 'SLURM_PMIX_MAPPING_SERV': '(vector,(0,1,4))', 'SLURM_SRUN_COMM_HOST': '130.183.186.16', 'SLURM_CPU_BIND': 'quiet,mask_cpu:0x00000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000001,0x00000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000002,0x00000000000000000000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000000004,0x00000000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000008', 'SLURM_CPU_BIND_LIST': '0x00000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000001,0x00000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000002,0x00000000000000000000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000000004,0x00000000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000008', 'SLURM_CPU_BIND_TYPE': 'mask_cpu:', 'SLURM_CPU_BIND_VERBOSE': 'quiet', 'SLURM_LAUNCH_NODE_IPADDR': '130.183.186.16'}], None, None, None]
cores=4 requests four tasks (--ntasks=4), one per MPI rank. With pmi_mode="pmix" the executor wraps the function call in srun --mpi=pmix -n 4, so all four ranks start simultaneously inside the single SLURM job allocation. The sacct output will show AllocCPUS matching cores × threads_per_core.
for d in get_cache_data(cache_dir):
if "get_slurm_parallel_env" in str(d["function"]):
print(d['queue_id'])
shell_output = subprocess.check_output(["sacct", "-j", str(d['queue_id'])], shell=False, universal_newlines=True)
20332321
shell_output.split("\n")
['JobID JobName Partition Account AllocCPUS State ExitCode ',
'------------ ---------- ---------- ---------- ---------- ---------- -------- ',
'20332321 executorl+ s.cmmg mpd 32 RUNNING 0:0 ',
'20332321.ba+ batch mpd 32 RUNNING 0:0 ',
'20332321.ex+ extern mpd 32 RUNNING 0:0 ',
'20332321.0 python mpd 32 RUNNING 0:0 ',
'']
Nested Executors#
Finally, to gain the most of the flexibility executorlib offers the option to nest executors within each other. See the HPC Job Executor documentation for more details on SlurmJobExecutor and running nested workloads inside an existing allocation.
def get_nested_env():
with executorlib.SlurmJobExecutor(max_workers=4, pmi_mode="pmix") as exe:
future_lst = []
for i in range(4):
future_lst.append(exe.submit(
get_slurm_env,
resource_dict={"cores": 1, "threads_per_core": 1},
))
return [f.result() for f in future_lst]
get_nested_env runs inside a SLURM job and itself creates a second-level executor. SlurmJobExecutor re-uses the existing allocation instead of submitting new sbatch jobs: it dispatches each of the four get_slurm_env calls as an srun step within the parent job. The outer job is allocated cores=1, threads_per_core=4 (one task slot with 4 CPUs). Each inner step requests cores=1, threads_per_core=1, so four steps share the available CPUs — this is valid because the outer allocation has enough CPU resources to run them.
from time import sleep
with executorlib.SlurmClusterExecutor(pmi_mode="pmix") as exe:
f1 = exe.submit(
get_nested_env,
resource_dict={
"submission_template": submission_template,
"run_time_max": 180, # in seconds
# "partition": "s.cmfe",
"partition": "s.cmmg",
"cores": 1,
"threads_per_core": 4,
})
sleep(10)
df = get_cache_data(cache_dir)
for d in df:
if "get_nested_env" in str(d["function"]):
print(d['queue_id'])
shell_output = subprocess.check_output(["sacct", "-j", str(d['queue_id'])], shell=False, universal_newlines=True)
result = f1.result()
print(result)
/u/janj/projects/executorlib/src/executorlib/task_scheduler/base.py:156: UserWarning: The following keys are not recognized and cannot be validated: ['partition']
self._validator(resource_dict=resource_dict)
20332322
[{'SLURM_TRES_PER_TASK': 'cpu=1', 'SLURM_MEM_PER_CPU': '1500', 'SLURM_NODEID': '0', 'SLURM_TASK_PID': '273267', 'SLURM_PRIO_PROCESS': '0', 'SLURM_SUBMIT_DIR': '/cmmc/u/janj/notebooks/2026/2026-06-03-prabhath-exe/executorlib_cache/get_nested_envfa45bec89a84ca0b105107cb298c9a0d', 'SLURM_CELL': 'CMMC', 'SLURM_PROCID': '0', 'SLURM_JOB_GID': '12500', 'SLURM_JOB_END_TIME': '1781017904', 'SLURM_TASKS_PER_NODE': '1', 'SLURM_NNODES': '1', 'SLURM_GET_USER_ENV': '1', 'SLURM_JOB_START_TIME': '1781017724', 'SLURM_JOB_NODELIST': 'cmmg001', 'SLURM_CLUSTER_NAME': 'cmmc', 'SLURM_NODELIST': 'cmmg001', 'SLURM_NTASKS': '1', 'SLURM_JOB_CPUS_PER_NODE': '8', 'SLURM_TOPOLOGY_ADDR': 'xxx-xxx-xxx.cmmg001', 'SLURM_JOB_NAME': 'executorlib_cache', 'SLURM_JOBID': '20332322', 'SLURM_CONF': '/etc/slurm/slurm.conf', 'SLURM_JOB_QOS': 'normal', 'SLURM_TOPOLOGY_ADDR_PATTERN': 'switch.node', 'SLURM_CPUS_ON_NODE': '2', 'SLURM_JOB_NUM_NODES': '1', 'SLURM_JOB_UID': '35052', 'SLURM_JOB_PARTITION': 's.cmmg', 'SLURM_SCRIPT_CONTEXT': 'prolog_task', 'SLURM_JOB_USER': 'janj', 'SLURM_NPROCS': '1', 'SLURM_SUBMIT_HOST': 'cmti001', 'SLURM_JOB_ACCOUNT': 'mpd', 'SLURM_GTIDS': '0', 'SLURM_JOB_ID': '20332322', 'SLURM_OOM_KILL_STEP': '1', 'SLURM_LOCALID': '0', 'SLURM_MPI_TYPE': 'pmix', 'SLURM_UMASK': '0022', 'SLURM_CPUS_PER_TASK': '1', 'SLURM_DISTRIBUTION': 'cyclic', 'SLURM_STEP_ID': '0', 'SLURM_STEPID': '0', 'SLURM_STEP_NODELIST': 'cmmg001', 'SLURM_STEP_NUM_NODES': '1', 'SLURM_STEP_NUM_TASKS': '1', 'SLURM_STEP_TASKS_PER_NODE': '1', 'SLURM_STEP_LAUNCHER_PORT': '37523', 'SLURM_SRUN_COMM_PORT': '37523', 'SLURM_PMIXP_ABORT_AGENT_PORT': '32831', 'SLURM_PMIX_MAPPING_SERV': '(vector,(0,1,1))', 'SLURM_SRUN_COMM_HOST': '130.183.186.16', 'SLURM_CPU_BIND': 'quiet,mask_cpu:0x00000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000001', 'SLURM_CPU_BIND_LIST': '0x00000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000001', 'SLURM_CPU_BIND_TYPE': 'mask_cpu:', 'SLURM_CPU_BIND_VERBOSE': 'quiet', 'SLURM_LAUNCH_NODE_IPADDR': '130.183.186.16'}, {'SLURM_TRES_PER_TASK': 'cpu=1', 'SLURM_MEM_PER_CPU': '1500', 'SLURM_NODEID': '0', 'SLURM_TASK_PID': '273273', 'SLURM_PRIO_PROCESS': '0', 'SLURM_SUBMIT_DIR': '/cmmc/u/janj/notebooks/2026/2026-06-03-prabhath-exe/executorlib_cache/get_nested_envfa45bec89a84ca0b105107cb298c9a0d', 'SLURM_CELL': 'CMMC', 'SLURM_PROCID': '0', 'SLURM_JOB_GID': '12500', 'SLURM_JOB_END_TIME': '1781017904', 'SLURM_TASKS_PER_NODE': '1', 'SLURM_NNODES': '1', 'SLURM_GET_USER_ENV': '1', 'SLURM_JOB_START_TIME': '1781017724', 'SLURM_JOB_NODELIST': 'cmmg001', 'SLURM_CLUSTER_NAME': 'cmmc', 'SLURM_NODELIST': 'cmmg001', 'SLURM_NTASKS': '1', 'SLURM_JOB_CPUS_PER_NODE': '8', 'SLURM_TOPOLOGY_ADDR': 'xxx-xxx-xxx.cmmg001', 'SLURM_JOB_NAME': 'executorlib_cache', 'SLURM_JOBID': '20332322', 'SLURM_CONF': '/etc/slurm/slurm.conf', 'SLURM_JOB_QOS': 'normal', 'SLURM_TOPOLOGY_ADDR_PATTERN': 'switch.node', 'SLURM_CPUS_ON_NODE': '2', 'SLURM_JOB_NUM_NODES': '1', 'SLURM_JOB_UID': '35052', 'SLURM_JOB_PARTITION': 's.cmmg', 'SLURM_SCRIPT_CONTEXT': 'prolog_task', 'SLURM_JOB_USER': 'janj', 'SLURM_NPROCS': '1', 'SLURM_SUBMIT_HOST': 'cmti001', 'SLURM_JOB_ACCOUNT': 'mpd', 'SLURM_GTIDS': '0', 'SLURM_JOB_ID': '20332322', 'SLURM_OOM_KILL_STEP': '1', 'SLURM_LOCALID': '0', 'SLURM_MPI_TYPE': 'pmix', 'SLURM_UMASK': '0022', 'SLURM_CPUS_PER_TASK': '1', 'SLURM_DISTRIBUTION': 'cyclic', 'SLURM_STEP_ID': '3', 'SLURM_STEPID': '3', 'SLURM_STEP_NODELIST': 'cmmg001', 'SLURM_STEP_NUM_NODES': '1', 'SLURM_STEP_NUM_TASKS': '1', 'SLURM_STEP_TASKS_PER_NODE': '1', 'SLURM_STEP_LAUNCHER_PORT': '37645', 'SLURM_SRUN_COMM_PORT': '37645', 'SLURM_PMIXP_ABORT_AGENT_PORT': '42193', 'SLURM_PMIX_MAPPING_SERV': '(vector,(0,1,1))', 'SLURM_SRUN_COMM_HOST': '130.183.186.16', 'SLURM_CPU_BIND': 'quiet,mask_cpu:0x00000000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000008', 'SLURM_CPU_BIND_LIST': '0x00000000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000008', 'SLURM_CPU_BIND_TYPE': 'mask_cpu:', 'SLURM_CPU_BIND_VERBOSE': 'quiet', 'SLURM_LAUNCH_NODE_IPADDR': '130.183.186.16'}, {'SLURM_TRES_PER_TASK': 'cpu=1', 'SLURM_MEM_PER_CPU': '1500', 'SLURM_NODEID': '0', 'SLURM_TASK_PID': '273270', 'SLURM_PRIO_PROCESS': '0', 'SLURM_SUBMIT_DIR': '/cmmc/u/janj/notebooks/2026/2026-06-03-prabhath-exe/executorlib_cache/get_nested_envfa45bec89a84ca0b105107cb298c9a0d', 'SLURM_CELL': 'CMMC', 'SLURM_PROCID': '0', 'SLURM_JOB_GID': '12500', 'SLURM_JOB_END_TIME': '1781017904', 'SLURM_TASKS_PER_NODE': '1', 'SLURM_NNODES': '1', 'SLURM_GET_USER_ENV': '1', 'SLURM_JOB_START_TIME': '1781017724', 'SLURM_JOB_NODELIST': 'cmmg001', 'SLURM_CLUSTER_NAME': 'cmmc', 'SLURM_NODELIST': 'cmmg001', 'SLURM_NTASKS': '1', 'SLURM_JOB_CPUS_PER_NODE': '8', 'SLURM_TOPOLOGY_ADDR': 'xxx-xxx-xxx.cmmg001', 'SLURM_JOB_NAME': 'executorlib_cache', 'SLURM_JOBID': '20332322', 'SLURM_CONF': '/etc/slurm/slurm.conf', 'SLURM_JOB_QOS': 'normal', 'SLURM_TOPOLOGY_ADDR_PATTERN': 'switch.node', 'SLURM_CPUS_ON_NODE': '2', 'SLURM_JOB_NUM_NODES': '1', 'SLURM_JOB_UID': '35052', 'SLURM_JOB_PARTITION': 's.cmmg', 'SLURM_SCRIPT_CONTEXT': 'prolog_task', 'SLURM_JOB_USER': 'janj', 'SLURM_NPROCS': '1', 'SLURM_SUBMIT_HOST': 'cmti001', 'SLURM_JOB_ACCOUNT': 'mpd', 'SLURM_GTIDS': '0', 'SLURM_JOB_ID': '20332322', 'SLURM_OOM_KILL_STEP': '1', 'SLURM_LOCALID': '0', 'SLURM_MPI_TYPE': 'pmix', 'SLURM_UMASK': '0022', 'SLURM_CPUS_PER_TASK': '1', 'SLURM_DISTRIBUTION': 'cyclic', 'SLURM_STEP_ID': '1', 'SLURM_STEPID': '1', 'SLURM_STEP_NODELIST': 'cmmg001', 'SLURM_STEP_NUM_NODES': '1', 'SLURM_STEP_NUM_TASKS': '1', 'SLURM_STEP_TASKS_PER_NODE': '1', 'SLURM_STEP_LAUNCHER_PORT': '39559', 'SLURM_SRUN_COMM_PORT': '39559', 'SLURM_PMIXP_ABORT_AGENT_PORT': '39135', 'SLURM_PMIX_MAPPING_SERV': '(vector,(0,1,1))', 'SLURM_SRUN_COMM_HOST': '130.183.186.16', 'SLURM_CPU_BIND': 'quiet,mask_cpu:0x00000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000002', 'SLURM_CPU_BIND_LIST': '0x00000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000002', 'SLURM_CPU_BIND_TYPE': 'mask_cpu:', 'SLURM_CPU_BIND_VERBOSE': 'quiet', 'SLURM_LAUNCH_NODE_IPADDR': '130.183.186.16'}, {'SLURM_TRES_PER_TASK': 'cpu=1', 'SLURM_MEM_PER_CPU': '1500', 'SLURM_NODEID': '0', 'SLURM_TASK_PID': '273271', 'SLURM_PRIO_PROCESS': '0', 'SLURM_SUBMIT_DIR': '/cmmc/u/janj/notebooks/2026/2026-06-03-prabhath-exe/executorlib_cache/get_nested_envfa45bec89a84ca0b105107cb298c9a0d', 'SLURM_CELL': 'CMMC', 'SLURM_PROCID': '0', 'SLURM_JOB_GID': '12500', 'SLURM_JOB_END_TIME': '1781017904', 'SLURM_TASKS_PER_NODE': '1', 'SLURM_NNODES': '1', 'SLURM_GET_USER_ENV': '1', 'SLURM_JOB_START_TIME': '1781017724', 'SLURM_JOB_NODELIST': 'cmmg001', 'SLURM_CLUSTER_NAME': 'cmmc', 'SLURM_NODELIST': 'cmmg001', 'SLURM_NTASKS': '1', 'SLURM_JOB_CPUS_PER_NODE': '8', 'SLURM_TOPOLOGY_ADDR': 'xxx-xxx-xxx.cmmg001', 'SLURM_JOB_NAME': 'executorlib_cache', 'SLURM_JOBID': '20332322', 'SLURM_CONF': '/etc/slurm/slurm.conf', 'SLURM_JOB_QOS': 'normal', 'SLURM_TOPOLOGY_ADDR_PATTERN': 'switch.node', 'SLURM_CPUS_ON_NODE': '2', 'SLURM_JOB_NUM_NODES': '1', 'SLURM_JOB_UID': '35052', 'SLURM_JOB_PARTITION': 's.cmmg', 'SLURM_SCRIPT_CONTEXT': 'prolog_task', 'SLURM_JOB_USER': 'janj', 'SLURM_NPROCS': '1', 'SLURM_SUBMIT_HOST': 'cmti001', 'SLURM_JOB_ACCOUNT': 'mpd', 'SLURM_GTIDS': '0', 'SLURM_JOB_ID': '20332322', 'SLURM_OOM_KILL_STEP': '1', 'SLURM_LOCALID': '0', 'SLURM_MPI_TYPE': 'pmix', 'SLURM_UMASK': '0022', 'SLURM_CPUS_PER_TASK': '1', 'SLURM_DISTRIBUTION': 'cyclic', 'SLURM_STEP_ID': '2', 'SLURM_STEPID': '2', 'SLURM_STEP_NODELIST': 'cmmg001', 'SLURM_STEP_NUM_NODES': '1', 'SLURM_STEP_NUM_TASKS': '1', 'SLURM_STEP_TASKS_PER_NODE': '1', 'SLURM_STEP_LAUNCHER_PORT': '36917', 'SLURM_SRUN_COMM_PORT': '36917', 'SLURM_PMIXP_ABORT_AGENT_PORT': '42277', 'SLURM_PMIX_MAPPING_SERV': '(vector,(0,1,1))', 'SLURM_SRUN_COMM_HOST': '130.183.186.16', 'SLURM_CPU_BIND': 'quiet,mask_cpu:0x00000000000000000000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000000004', 'SLURM_CPU_BIND_LIST': '0x00000000000000000000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000000004', 'SLURM_CPU_BIND_TYPE': 'mask_cpu:', 'SLURM_CPU_BIND_VERBOSE': 'quiet', 'SLURM_LAUNCH_NODE_IPADDR': '130.183.186.16'}]
shell_output.split("\n")
['JobID JobName Partition Account AllocCPUS State ExitCode ',
'------------ ---------- ---------- ---------- ---------- ---------- -------- ',
'20332322 executorl+ s.cmmg mpd 8 RUNNING 0:0 ',
'20332322.ba+ batch mpd 8 RUNNING 0:0 ',
'20332322.ex+ extern mpd 8 RUNNING 0:0 ',
'20332322.0 python mpd 1 RUNNING 0:0 ',
'20332322.1 python mpd 1 RUNNING 0:0 ',
'20332322.2 python mpd 1 RUNNING 0:0 ',
'20332322.3 python mpd 1 RUNNING 0:0 ',
'']
The sacct output for the nested job shows the outer job allocation at the top followed by one python step per inner task (20332322.0 through 20332322.3). Each step uses AllocCPUS=1, confirming that the four tasks ran as separate srun steps within the same job rather than as independent sbatch submissions.
import pandas
pandas.DataFrame(df)
| function | input_args | input_kwargs | output | resource_dict | runtime | queue_id | error_log_file | filename | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | <function get_slurm_env at 0x14d04fd31260> | [] | {} | {'SLURM_MEM_PER_CPU': '1500', 'SLURM_NODEID': ... | {'submission_template': '#!/bin/bash #SBATCH -... | 5.015562 | 20332320 | None | /cmmc/u/janj/notebooks/2026/2026-06-03-prabhat... |
| 1 | <function get_slurm_parallel_env at 0x14d04fd3... | [] | {} | [[{'SLURM_MEM_PER_CPU': '1500', 'SLURM_NODEID'... | {'submission_template': '#!/bin/bash #SBATCH -... | 0.011143 | 20332321 | None | /cmmc/u/janj/notebooks/2026/2026-06-03-prabhat... |
| 2 | <function get_nested_env at 0x14d04fd311c0> | [] | {} | NaN | {'submission_template': '#!/bin/bash #SBATCH -... | NaN | 20332322 | None | /cmmc/u/janj/notebooks/2026/2026-06-03-prabhat... |
The cache DataFrame gives a complete audit trail: the function object, its arguments, the full resource_dict used for submission, the wall-clock runtime, the SLURM queue_id, and the path to the HDF5 file storing the serialised return value. The output column is NaN for the nested job while it is still running; once complete, it holds the deserialized Python return value.
for d in get_cache_data(cache_dir):
if "get_nested_env" in str(d["function"]):
print(d['queue_id'])
shell_output = subprocess.check_output(["sacct", "-j", str(d['queue_id'])], shell=False, universal_newlines=True)
20332322
shell_output.split("\n")
['JobID JobName Partition Account AllocCPUS State ExitCode ',
'------------ ---------- ---------- ---------- ---------- ---------- -------- ',
'20332322 executorl+ s.cmmg mpd 8 COMPLETED 0:0 ',
'20332322.ba+ batch mpd 8 COMPLETED 0:0 ',
'20332322.ex+ extern mpd 8 COMPLETED 0:0 ',
'20332322.0 python mpd 1 COMPLETED 0:0 ',
'20332322.1 python mpd 1 COMPLETED 0:0 ',
'20332322.2 python mpd 1 COMPLETED 0:0 ',
'20332322.3 python mpd 1 COMPLETED 0:0 ',
'']
Clean up#
import os
import shutil
cache_dir = "executorlib_cache"
os.listdir(cache_dir)
['get_slurm_envccfcb90850bb2803c9b37c6ecde017ee',
'get_nested_envfa45bec89a84ca0b105107cb298c9a0d_o.h5',
'get_nested_envfa45bec89a84ca0b105107cb298c9a0d',
'get_slurm_envccfcb90850bb2803c9b37c6ecde017ee_o.h5',
'get_slurm_parallel_env24890a8e967e7328abd85429d1ce0371',
'get_slurm_parallel_env24890a8e967e7328abd85429d1ce0371_o.h5']
shutil.rmtree(cache_dir)