Backends

gwf supports multiple backends for running workflows. If you don’t find a backend that suits your needs here, it’s easy to write your own backend.

By default, gwf comes with the local, slurm, and sge backends.

Local

class gwf.backends.local.LocalBackend[source]

Backend that runs targets on a local cluster.

To use this backend you must activate the local backend and start a local cluster (with one or more workers) that the backend can submit targets to. To start a cluster with two workers run the command:

gwf -b local workers -n 2

in the working directory of your project. The workflow file must be accessible to gwf. Thus, if your workflow file is not called workflow.py or the workflow object is not called gwf, you must specify this so that gwf can locate the workflow:

gwf -f myworkflow.py:wf1 -b local workers -n 2

If the local backend is your default backend you can of course omit the -b local option.

If the -n option is omitted, gwf will detect the number of cores available and use all of them.

To run your workflow, open another terminal and then type:

gwf -b local run

To stop the pool of workers press Control-c.

Backend options:

  • local.host (str): Set the host that the workers are running on (default: localhost).
  • local.port (int): Set the port used to connect to the workers (default: 12345).

Target options:

None available.

Slurm

class gwf.backends.slurm.SlurmBackend[source]

Backend for the Slurm workload manager.

To use this backend you must activate the slurm backend.

Backend options:

  • backend.slurm.log_mode (str): Must be either full, merged or none. If full, two log files will be stored for each target, one for standard output and one for standard error. If merged, only one log file will be written containing the combined streams. If none, no logs will be stored. (default: full).

Target options:

  • cores (int): Number of cores allocated to this target (default: 1).
  • memory (str): Memory allocated to this target (default: 1).
  • walltime (str): Time limit for this target (default: 01:00:00).
  • queue (str): Queue to submit the target to. To specify multiple queues, specify a comma-separated list of queue names.
  • account (str): Account to be used when running the target.
  • constraint (str): Constraint string. Equivalent to setting the –constraint flag on sbatch.
  • qos (str): Quality-of-service strring. Equivalent to setting the –qos flog on sbatch.

Sun Grid Engine (SGE)

class gwf.backends.sge.SGEBackend[source]

Backend for Sun Grid Engine (SGE).

To use this backend you must activate the sge backend. The backend currently assumes that a SGE parallel environment called “smp” is available. You can check which parallel environments are available on your system by running qconf -spl.

Backend options:

None.

Target options:

  • cores (int): Number of cores allocated to this target (default: 1).
  • memory (str): Memory allocated to this target (default: 1).
  • walltime (str): Time limit for this target (default: 01:00:00).
  • queue (str): Queue to submit the target to. To specify multiple queues, specify a comma-separated list of queue names.
  • account (str): Account to be used when running the target. Corresponds to the SGE project.