gridjug package

Submodules

gridjug.grid_jug module

gridjug.grid_jug.grid_jug(jugfile, jugdir=None, jug_args=None, jug_nworkers=4, name='gridjug', keep_going=False, verbose=False, capture_jug_stdout=False, **kwargs)[source]

A light-weight wrapper to run Jug with GridMap on a Grid Engine cluster

From their own description, GridMap is a package that allows to easily create jobs on a Grid Engine powered cluster directly from Python. This wrapper lets GridMap simply spawn several jug-execute workers on a Grid Engine cluster. Thus we have the benefit of programmatic (reproducible) execution of Jug processes. Furthermore, GridMap adds a convenient monitoring and reporting layer. Under the hood, of course, Jug keeps doing the actual work.

Parameters:
  • jugfile (path) – Path to the jugfile
  • jugdir (path) – Where to save intermediate results
  • jug_args (list) –

    Other jug command-line arguments. Note that 'execute' is already included. The command line is roughly equivalent to:

    ‘jug execute {jugfile} ‘ + ‘ ‘.join(jug_args)
  • jug_nworkers (int, optional) – number of Grid Engine tasks to start (i.e. number of times ‘jug execute’ is run)
  • name (str, optional) – base name of the Grid Engine task
  • keep_going (bool, optional) – Strongly recommended! Defaults to False: if a single Jug task fails, GridMap will cancel all jobs! If True, Jug does not raise an exception but keeps retrying the task.
  • verbose (bool, optional) – If True, Jug logs INFO events
  • capture_jug_stdout (bool, optional) – Defaults to False. If True, captures Jug’s task summary printed to stdout.
  • **kwargs –

    additional options passed through to gridmap.grid_map

See also

gridmap.grid_map : The map function

Jug subcommands

Module contents

Notes

For some reason, pickle does not work on some cluster environments if the submodule has the same name as the package. Hence, we call the submodule grid_jug instead of gridjug.

See also

GridMap
Easily map Python functions onto a cluster using a DRMAA-compatible grid engine like Sun Grid Engine (SGE).
Jug
A Task-Based Parallelization Framework