Contributing
Welcome!
This document is fairly extensive and you are not really expected
to study this in detail for small contributions;
The most important rule is that contributing must be easy
and that the community is friendly and not nitpicking on details
such as coding style.
If you’re reporting a bug you should read the Reporting bugs section
below to ensure that your bug report contains enough information
to successfully diagnose the issue, and if you’re contributing code
you should try to mimic the conventions you see surrounding the code
you are working on, but in the end all patches will be cleaned up by
the person merging the changes so don’t worry too much.
You must never report security related issues, vulnerabilities or bugs
including sensitive information to the bug tracker, or elsewhere in public.
Instead sensitive bugs must be sent by email to security@celeryproject.org.
If you’d like to submit the information encrypted our PGP key is:
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.4.15 (Darwin)
mQENBFJpWDkBCADFIc9/Fpgse4owLNvsTC7GYfnJL19XO0hnL99sPx+DPbfr+cSE
9wiU+Wp2TfUX7pCLEGrODiEP6ZCZbgtiPgId+JYvMxpP6GXbjiIlHRw1EQNH8RlX
cVxy3rQfVv8PGGiJuyBBjxzvETHW25htVAZ5TI1+CkxmuyyEYqgZN2fNd0wEU19D
+c10G1gSECbCQTCbacLSzdpngAt1Gkrc96r7wGHBBSvDaGDD2pFSkVuTLMbIRrVp
lnKOPMsUijiip2EMr2DvfuXiUIUvaqInTPNWkDynLoh69ib5xC19CSVLONjkKBsr
Pe+qAY29liBatatpXsydY7GIUzyBT3MzgMJlABEBAAG0MUNlbGVyeSBTZWN1cml0
eSBUZWFtIDxzZWN1cml0eUBjZWxlcnlwcm9qZWN0Lm9yZz6JATgEEwECACIFAlJp
WDkCGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJEOArFOUDCicIw1IH/26f
CViDC7/P13jr+srRdjAsWvQztia9HmTlY8cUnbmkR9w6b6j3F2ayw8VhkyFWgYEJ
wtPBv8mHKADiVSFARS+0yGsfCkia5wDSQuIv6XqRlIrXUyqJbmF4NUFTyCZYoh+C
ZiQpN9xGhFPr5QDlMx2izWg1rvWlG1jY2Es1v/xED3AeCOB1eUGvRe/uJHKjGv7J
rj0pFcptZX+WDF22AN235WYwgJM6TrNfSu8sv8vNAQOVnsKcgsqhuwomSGsOfMQj
LFzIn95MKBBU1G5wOs7JtwiV9jefGqJGBO2FAvOVbvPdK/saSnB+7K36dQcIHqms
5hU4Xj0RIJiod5idlRC5AQ0EUmlYOQEIAJs8OwHMkrdcvy9kk2HBVbdqhgAREMKy
gmphDp7prRL9FqSY/dKpCbG0u82zyJypdb7QiaQ5pfPzPpQcd2dIcohkkh7G3E+e
hS2L9AXHpwR26/PzMBXyr2iNnNc4vTksHvGVDxzFnRpka6vbI/hrrZmYNYh9EAiv
uhE54b3/XhXwFgHjZXb9i8hgJ3nsO0pRwvUAM1bRGMbvf8e9F+kqgV0yWYNnh6QL
4Vpl1+epqp2RKPHyNQftbQyrAHXT9kQF9pPlx013MKYaFTADscuAp4T3dy7xmiwS
crqMbZLzfrxfFOsNxTUGE5vmJCcm+mybAtRo4aV6ACohAO9NevMx8pUAEQEAAYkB
HwQYAQIACQUCUmlYOQIbDAAKCRDgKxTlAwonCNFbB/9esir/f7TufE+isNqErzR/
aZKZo2WzZR9c75kbqo6J6DYuUHe6xI0OZ2qZ60iABDEZAiNXGulysFLCiPdatQ8x
8zt3DF9BMkEck54ZvAjpNSern6zfZb1jPYWZq3TKxlTs/GuCgBAuV4i5vDTZ7xK/
aF+OFY5zN7ciZHkqLgMiTZ+RhqRcK6FhVBP/Y7d9NlBOcDBTxxE1ZO1ute6n7guJ
ciw4hfoRk8qNN19szZuq3UU64zpkM2sBsIFM9tGF2FADRxiOaOWZHmIyVZriPFqW
RUwjSjs7jBVNq0Vy4fCu/5+e+XLOUBOoqtM5W7ELt0t1w9tXebtPEetV86in8fU2
=0chn
-----END PGP PUBLIC KEY BLOCK-----
Bugs can always be described to the Mailing list, but the best
way to report an issue and to ensure a timely response is to use the
issue tracker.
- Create a GitHub account.
You need to create a GitHub account to be able to create new issues
and participate in the discussion.
- Determine if your bug is really a bug.
You should not file a bug if you are requesting support. For that you can use
the Mailing list, or IRC.
- Make sure your bug hasn’t already been reported.
Search through the appropriate Issue tracker. If a bug like yours was found,
check if you have new information that could be reported to help
the developers fix the bug.
- Check if you’re using the latest version.
A bug could be fixed by some other improvements and fixes - it might not have an
existing report in the bug tracker. Make sure you’re using the latest releases of
celery, billiard and kombu.
- Collect information about the bug.
To have the best chance of having a bug fixed, we need to be able to easily
reproduce the conditions that caused it. Most of the time this information
will be from a Python traceback message, though some bugs might be in design,
spelling or other errors on the website/docs/code.
If the error is from a Python traceback, include it in the bug report.
We also need to know what platform you’re running (Windows, OS X, Linux,
etc.), the version of your Python interpreter, and the version of Celery,
and related packages that you were running when the bug occurred.
If you are reporting a race condition or a deadlock, tracebacks can be
hard to get or might not be that useful. Try to inspect the process to
get more diagnostic data. Some ideas:
- Enable celery’s breakpoint signal and use it
to inspect the process’s state. This will allow you to open a
pdb session.
- Collect tracing data using strace_(Linux), dtruss (OSX) and ktrace(BSD),
ltrace and lsof.
Include the output from the celery report command:
This will also include your configuration settings and it try to
remove values for keys known to be sensitive, but make sure you also
verify the information before submitting so that it doesn’t contain
confidential information like API tokens and authentication
credentials.
- Submit the bug.
By default GitHub will email you to let you know when new comments have
been made on your bug. In the event you’ve turned this feature off, you
should check back on occasion to ensure you don’t miss any questions a
developer trying to fix the bug might ask.
Bugs for a package in the Celery ecosystem should be reported to the relevant
issue tracker.
If you are unsure of the origin of the bug you can ask the
Mailing list, or just use the Celery issue tracker.
There’s a separate section for internal details,
including details about the codebase and a style guide.
Read Contributors Guide to the Code for more!
Version numbers consists of a major version, minor version and a release number.
Since version 2.1.0 we use the versioning semantics described by
semver: http://semver.org.
Stable releases are published at PyPI
while development releases are only available in the GitHub git repository as tags.
All version tags starts with “v”, so version 0.8.0 is the tag v0.8.0.
Current active version branches:
You can see the state of any branch by looking at the Changelog:
If the branch is in active development the topmost version info should
contain metadata like:
2.4.0
======
:release-date: TBA
:status: DEVELOPMENT
:branch: master
The status field can be one of:
PLANNING
The branch is currently experimental and in the planning stage.
DEVELOPMENT
The branch is in active development, but the test suite should
be passing and the product should be working and possible for users to test.
FROZEN
The branch is frozen, and no more features will be accepted.
When a branch is frozen the focus is on testing the version as much
as possible before it is released.
The master branch is where development of the next version happens.
Maintenance branches are named after the version, e.g. the maintenance branch
for the 2.2.x series is named 2.2. Previously these were named
releaseXX-maint.
The versions we currently maintain is:
Archived branches are kept for preserving history only,
and theoretically someone could provide patches for these if they depend
on a series that is no longer officially supported.
An archived version is named X.Y-archived.
Our currently archived branches are:
- 2.5-archived
- 2.4-archived
- 2.3-archived
- 2.1-archived
- 2.0-archived
- 1.0-archived
Major new features are worked on in dedicated branches.
There is no strict naming requirement for these branches.
Feature branches are removed once they have been merged into a release branch.
Note
Contributing to Celery should be as simple as possible,
so none of these steps should be considered mandatory.
You can even send in patches by email if that is your preferred
work method. We won’t like you any less, any contribution you make
is always appreciated!
However following these steps may make maintainers life easier,
and may mean that your changes will be accepted sooner.
First you need to fork the Celery repository, a good introduction to this
is in the Github Guide: Fork a Repo.
After you have cloned the repository you should checkout your copy
to a directory on your machine:
$ git clone git@github.com:username/celery.git
When the repository is cloned enter the directory to set up easy access
to upstream changes:
$ cd celery
$ git remote add upstream git://github.com/celery/celery.git
$ git fetch upstream
If you need to pull in new changes from upstream you should
always use the --rebase option to git pull:
git pull --rebase upstream master
With this option you don’t clutter the history with merging
commit notes. See Rebasing merge commits in git.
If you want to learn more about rebasing see the Rebase
section in the Github guides.
If you need to work on a different branch than master you can
fetch and checkout a remote branch like this:
git checkout --track -b 3.0-devel origin/3.0-devel
To run the Celery test suite you need to install a few dependencies.
A complete list of the dependencies needed are located in
requirements/test.txt.
Installing the test requirements:
$ pip install -U -r requirements/test.txt
When installation of dependencies is complete you can execute
the test suite by calling nosetests:
Some useful options to nosetests are:
-x
Stop running the tests at the first test that fails.
-s
--nologcapture
Don’t capture log output.
-v
If you want to run the tests for a single test file only
you can do so like this:
$ nosetests celery.tests.test_worker.test_worker_job
When your feature/bugfix is complete you may want to submit
a pull requests so that it can be reviewed by the maintainers.
Creating pull requests is easy, and also let you track the progress
of your contribution. Read the Pull Requests section in the Github
Guide to learn how this is done.
You can also attach pull requests to existing issues by following
the steps outlined here: http://bit.ly/koJoso
To calculate test coverage you must first install the coverage module.
Installing the coverage module:
$ pip install -U coverage
Code coverage in HTML:
$ nosetests --with-coverage --cover-html
The coverage output will then be located at
celery/tests/cover/index.html.
Code coverage in XML (Cobertura-style):
$ nosetests --with-coverage --cover-xml --cover-xml-file=coverage.xml
The coverage XML output will then be located at coverage.xml
There is a tox configuration file in the top directory of the
distribution.
To run the tests for all supported Python versions simply execute:
If you only want to test specific Python versions use the -e
option:
To build the documentation you need to install the dependencies
listed in requirements/docs.txt:
$ pip install -U -r requirements/docs.txt
After these dependencies are installed you should be able to
build the docs by running:
$ cd docs
$ rm -rf .build
$ make html
Make sure there are no errors or warnings in the build output.
After building succeeds the documentation is available at .build/html.
To use these tools you need to install a few dependencies. These dependencies
can be found in requirements/pkgutils.txt.
Installing the dependencies:
$ pip install -U -r requirements/pkgutils.txt
To ensure that your changes conform to PEP8 and to run pyflakes
execute:
To not return a negative exit code when this command fails use the
-E option, this can be convenient while developing:
To make sure that all modules have a corresponding section in the API
reference please execute:
$ paver autodoc
$ paver verifyindex
If files are missing you can add them by copying an existing reference file.
If the module is internal it should be part of the internal reference
located in docs/internals/reference/. If the module is public
it should be located in docs/reference/.
For example if reference is missing for the module celery.worker.awesome
and this module is considered part of the public API, use the following steps:
Use an existing file as a template:
$ cd docs/reference/
$ cp celery.schedules.rst celery.worker.awesome.rst
Edit the file using your favorite editor:
$ vim celery.worker.awesome.rst
# change every occurance of ``celery.schedules`` to
# ``celery.worker.awesome``
Edit the index using your favorite editor:
$ vim index.rst
# Add ``celery.worker.awesome`` to the index.
Commit your changes:
# Add the file to git
$ git add celery.worker.awesome.rst
$ git add index.rst
$ git commit celery.worker.awesome.rst index.rst \
-m "Adds reference for celery.worker.awesome"
You should probably be able to pick up the coding style
from surrounding code, but it is a good idea to be aware of the
following conventions.
- All Python code must follow the PEP-8 guidelines.
pep8.py is an utility you can use to verify that your code
is following the conventions.
Lines should not exceed 78 columns.
You can enforce this in vim by setting the textwidth option:
If adhering to this limit makes the code less readable, you have one more
character to go on, which means 78 is a soft limit, and 79 is the hard
limit :)
Import order
- Python standard library (import xxx)
- Python standard library (‘from xxx import`)
- Third party packages.
- Other modules from the current package.
or in case of code using Django:
- Python standard library (import xxx)
- Python standard library (‘from xxx import`)
- Third party packages.
- Django packages.
- Other modules from the current package.
Within these sections the imports should be sorted by module name.
Example:
import threading
import time
from collections import deque
from Queue import Queue, Empty
from .datastructures import TokenBucket
from .five import zip_longest, items, range
from .utils import timeutils
Wildcard imports must not be used (from xxx import *).
For distributions where Python 2.5 is the oldest support version
additional rules apply:
Absolute imports must be enabled at the top of every module:
from __future__ import absolute_import
If the module uses the with statement and must be compatible
with Python 2.5 (celery is not) then it must also enable that:
from __future__ import with_statement
Every future import must be on its own line, as older Python 2.5
releases did not support importing multiple features on the
same future import line:
# Good
from __future__ import absolute_import
from __future__ import with_statement
# Bad
from __future__ import absolute_import, with_statement
(Note that this rule does not apply if the package does not include
support for Python 2.5)
Note that we use “new-style` relative imports when the distribution
does not support Python versions below 2.5
This requires Python 2.5 or later:
Some features like a new result backend may require additional libraries
that the user must install.
We use setuptools extra_requires for this, and all new optional features
that require 3rd party libraries must be added.
Add a new requirements file in requirements/extras
E.g. for the Cassandra backend this is
requirements/extras/cassandra.txt, and the file looks like this:
These are pip requirement files so you can have version specifiers and
multiple packages are separated by newline. A more complex example could
be:
# pycassa 2.0 breaks Foo
pycassa>=1.0,<2.0
thrift
Modify setup.py
After the requirements file is added you need to add it as an option
to setup.py in the extras_require section:
extra['extras_require'] = {
# ...
'cassandra': extras('cassandra.txt'),
}
Document the new feature in docs/includes/installation.txt
You must add your feature to the list in the Bundles section
of docs/includes/installation.txt.
After you’ve made changes to this file you need to render
the distro README file:
$ pip install -U requirements/pkgutils.txt
$ paver readme
That’s all that needs to be done, but remember that if your feature
adds additional configuration options then these needs to be documented
in docs/configuration.rst. Also all settings need to be added to the
celery/app/defaults.py module.
Result backends require a separate section in the docs/configuration.rst
file.
Python AMQP 0.9.1 client.
Fork of multiprocessing containing improvements
that will eventually be merged into the Python stdlib.
Very fast Python AMQP client written in C.
Celery monitor web-service.
Django <-> Celery Integration.
Distributed Celery Instance manager.
Old name for librabbitmq.
The version number must be updated two places:
- celery/__init__.py
- docs/include/introduction.txt
After you have changed these files you must render
the README files. There is a script to convert sphinx syntax
to generic reStructured Text syntax, and the paver task readme
does this for you:
Now commit the changes:
$ git commit -a -m "Bumps version to X.Y.Z"
and make a new version tag:
$ git tag vX.Y.Z
$ git push --tags
Commands to make a new public stable release:
$ paver releaseok # checks pep8, autodoc index, runs tests and more
$ paver removepyc # Remove .pyc files
$ git clean -xdn # Check that there's no left-over files in the repo
$ python setup.py sdist upload # Upload package to PyPI
If this is a new release series then you also need to do the
following: