Is your feature request related to a problem? Please describe.
We noticed few scalability issues with Parsl beyond 2048 nodes on Aurora, with 102 workers per node. Additionally, executing heterogeneous scheduling of MPI+serial tasks involves creating multiple executors. This can cause resource fragmentation.
Describe the solution you'd like
At ALCF, we developed an executor to fill these gaps. Additionally, we also provide a programmable scheduling interface, where one can design custom scheduling policies. We have identified few use cases, where this can be useful. We would like to contribute this executor to the Parsl as a subclass of BlockProviderExecutor. Please find our preprint here [1].
Is your feature request related to a problem? Please describe.
We noticed few scalability issues with Parsl beyond 2048 nodes on Aurora, with 102 workers per node. Additionally, executing heterogeneous scheduling of MPI+serial tasks involves creating multiple executors. This can cause resource fragmentation.
Describe the solution you'd like
At ALCF, we developed an executor to fill these gaps. Additionally, we also provide a programmable scheduling interface, where one can design custom scheduling policies. We have identified few use cases, where this can be useful. We would like to contribute this executor to the Parsl as a subclass of BlockProviderExecutor. Please find our preprint here [1].