Submission Errors

executorlib extends the Executor interface from the Python standard library’s concurrent.futures to distribute Python functions as jobs on an HPC cluster. This notebook deliberately submits a job with an incomplete resource_dict to show how SlurmClusterExecutor surfaces sbatch submission failures through the returned Future. See the resource dictionary troubleshooting page for the required keys.

Based on the cmti and cmmg clusters hosted at the MPCDF for the MPI for Sustainable Materials.

import executorlib

executorlib.__version__

'0.0.1'

executorlib.__path__

['/u/janj/projects/executorlib/src/executorlib']

Submit Python Function to SLURM

This is just like using sbatch to submit shell scripts. See the SlurmClusterExecutor documentation for the full set of options.

def echo(i):
    return i

echo is a trivial placeholder function — the point of this notebook is not the function itself but the empty resource_dict={} passed to submit() below. Required keys such as partition are intentionally omitted to trigger a submission failure.

with executorlib.SlurmClusterExecutor() as exe:
    f1 = exe.submit(
        echo,
        resource_dict={})
    result = f1.result()
    print(result)

---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
Cell In[5], line 5
      1 with executorlib.SlurmClusterExecutor() as exe:
      2     f1 = exe.submit(
      3         echo,
      4         resource_dict={})
----> 5     result = f1.result()
      6     print(result)

File /cmmc/ptmp/pyironhb/mambaforge/envs/pyiron_mpie_cmti_2026-06-08/lib/python3.12/concurrent/futures/_base.py:456, in Future.result(self, timeout)
    454     raise CancelledError()
    455 elif self._state == FINISHED:
--> 456     return self.__get_result()
    457 else:
    458     raise TimeoutError()

File /cmmc/ptmp/pyironhb/mambaforge/envs/pyiron_mpie_cmti_2026-06-08/lib/python3.12/concurrent/futures/_base.py:401, in Future.__get_result(self)
    399 if self._exception:
    400     try:
--> 401         raise self._exception
    402     finally:
    403         # Break a reference cycle with the exception in self._exception
    404         self = None

CalledProcessError: Command '['sbatch', '--parsable', '/cmmc/u/janj/notebooks/2026/2026-06-03-prabhath-exe/executorlib_cache/echoe5873521b7330f831bed941744079e83/run_queue.sh']' returned non-zero exit status 1.

Because resource_dict does not specify a partition, the generated sbatch script has no --partition line, and SLURM rejects it with invalid partition specified: (null). executorlib runs sbatch as a subprocess and raises the resulting CalledProcessError from within f1.result() — exactly as if sbatch had been called directly on the command line without --partition.

Rather than letting .result() re-raise the exception, Future.exception() returns it directly so it can be inspected without a try/except block. For a CalledProcessError raised by sbatch, the .output attribute holds the captured stderr — here it confirms the missing-partition error. See the resource dictionary troubleshooting page for the list of keys resource_dict accepts, including partition.

excep = f1.exception()
excep.output

'sbatch: error: invalid partition specified: (null)\nsbatch: error: Batch job submission failed: Invalid partition name specified\n'

Clean up

import os
import shutil

cache_dir = "executorlib_cache"
os.listdir(cache_dir)

['echoe5873521b7330f831bed941744079e83_o.h5',
 'echoe5873521b7330f831bed941744079e83']

shutil.rmtree(cache_dir)