Welcome to gridjug!¶

gridjug is a light-weight wrapper to run Jug with GridMap on a Grid Engine cluster.

It stands on the shoulder of the giants Jug and GridMap. Please credit their authors accordingly.

August 26, 2015 (Version 0.4.0)

Andreas Sorge <gridjug@asorge.de>

Max Planck Institute for Dynamics & Self-Organization, Göttingen, Germany
Organization for Research on Complex Adaptive Systems, Göttingen, Germany

Citation¶

Please cite as: Andreas Sorge. (2015). gridjug 0.4.0. Zenodo.

Please also cite Jug and GridMap.

Documentation license¶

This documentation is licensed under a Creative Commons Attribution 4.0 International License.

Table of Contents¶

About gridjug¶

License¶

gridjug is free software; you can redistribute it and/or modify it under the terms of the ISC License:

Permission to use, copy, modify, and/or distribute this software for any purpose with or without fee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies.

THE SOFTWARE IS PROVIDED “AS IS” AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

Changelog¶

0.4.0 2015-08-24

[Feature] #8: Add capture_jug_stdout option

0.3.0 2015-08-20

[Feature] #1: Implement keep_going option
[Feature] #3: Implement Jug logging (verbose option)

0.2.0 2015-08-17

[Feature]: Implement tests for NLD cluster
[Feature]: Rename gridjug module to grid_jug

0.1.0 2015-08-17

[Feature]: Basic functionality

Setup¶

Requirements¶

gridjug runs under Python 2.7 and Python 3.4 (and later). It requires the following Python packages:

future
gridmap
jug

Installation from PyPI¶

To install the most recent stable version from the Python Package Index (PyPI), including the dependencies, run

$ pip install gridjug

Installation from source¶

To install the latest version from the source repository, run

$ pip install git+https://github.com/andsor/gridjug.git

Developer Guide¶

Repository: github.com/andsor/gridjug

Development environment¶

Use tox to prepare virtual environments for development.

To set up a Python 2.7 environment in .devenv27, run:

$ tox -e devenv27

To set up a Python 3.4 environment in .devenv34, run:

$ tox -e devenv34

Add requirements for the development environments to the requirements-dev.txt file.

Packaging¶

This package uses setuptools.

Run

$ python setup.py sdist

or

$ python setup.py bdist

or

$ python setup.py bdist_wheel

to build a source, binary or wheel distribution.

Complete Git Integration¶

The package is maintained in a git repository. The setuptools script setup.py uses the information of tags to infer the version of your project with the help of setuptools_scm. To use this feature you need to tag with the format MAJOR.MINOR[.PATCH] , e.g. 0.0.1 or 0.1.

Run

$ python setup.py --version

to retrieve the current PEP440-compliant version. This version will be used when building a package and is also accessible through gridjug.__version__.

Unleash the power of Git by using its pre-commit hooks. Make sure pre-commit is installed, e.g. pip install pre-commit, then just run pre-commit install.

Sphinx Documentation¶

This project follows the NumPy documentation style.

Build the documentation with:

$ tox -e docs

Add further options separated from tox options by a double dash --:

$ tox -e docs -- --help

Add requirements for building the documentation to the requirements-doc.txt file.

Continuous documentation building¶

For continuously building the documentation during development, run:

$ tox -e cdocs

Unittest & Coverage¶

Run

$ tox -e py27

or:

$ tox -e py34

to run all unittests defined in the subfolder test with the help of tox and py.test.

The py.test plugin pytest-cov is used to automatically generate a coverage report.

Continuous testing¶

For continuous testing in a Python 2.7 environment, run:

$ tox -e c27

For continuous testing in a Python 3.4 environment, run:

$ tox -e c34

Requirements Management¶

Add requirements to the requirements.txt file which will be automatically used by setup.py.

Continuous Integration¶

gridjug uses Travis to run the tests on each commit. Travis also reports the test coverage to Coveralls. If further deploys each tagged commit as a release to the Python Package Index (PyPI).

Landscape.io continuously measures “Code Health”.

ReadTheDocs builds and hosts this documentation.

gridjug package¶

Submodules¶

gridjug.grid_jug module¶

gridjug.grid_jug.grid_jug(jugfile, jugdir=None, jug_args=None, jug_nworkers=4, name='gridjug', keep_going=False, verbose=False, capture_jug_stdout=False, **kwargs)[source]¶

A light-weight wrapper to run Jug with GridMap on a Grid Engine cluster

From their own description, GridMap is a package that allows to easily create jobs on a Grid Engine powered cluster directly from Python. This wrapper lets GridMap simply spawn several jug-execute workers on a Grid Engine cluster. Thus we have the benefit of programmatic (reproducible) execution of Jug processes. Furthermore, GridMap adds a convenient monitoring and reporting layer. Under the hood, of course, Jug keeps doing the actual work.

Parameters:

jugfile (path) – Path to the jugfile
jugdir (path) – Where to save intermediate results
jug_args (list) –
Other jug command-line arguments. Note that 'execute' is already included. The command line is roughly equivalent to:

‘jug execute {jugfile} ‘ + ‘ ‘.join(jug_args)
jug_nworkers (int, optional) – number of Grid Engine tasks to start (i.e. number of times ‘jug execute’ is run)
name (str, optional) – base name of the Grid Engine task
keep_going (bool, optional) – Strongly recommended! Defaults to False: if a single Jug task fails, GridMap will cancel all jobs! If True, Jug does not raise an exception but keeps retrying the task.
verbose (bool, optional) – If True, Jug logs INFO events
capture_jug_stdout (bool, optional) – Defaults to False. If True, captures Jug’s task summary printed to stdout.
**kwargs –
additional options passed through to gridmap.grid_map

Module contents¶

Notes

For some reason, pickle does not work on some cluster environments if the submodule has the same name as the package. Hence, we call the submodule grid_jug instead of gridjug.