The “Python Distribute” guide (was at python-distribute.org, but that registration has lapsed) tells me to include doc/txt files and .py files are excluded in MANIFEST.in file

The sourcedist documentation tells me only sdist uses MANIFEST.in and only includes file you specify and to include .py files. It also tells me to use: python setup.py sdist --manifest-only to generate a MANIFEST, but python tells me this doesn’t exist

I appreciate these are from different versions of python and the distribution system is in a complete mess, but assuming I am using python 3 and setuptools (the new one that includes distribute but now called setuptools, not the old setuptools that was deprecated for distribute tools only to be brought back into distribute and distribute renamed to setuptools…..)

and I’m following the ‘standard’ folder structure and setup.py file,

1. Do I need a MANIFEST.in ?
2. What should be in it ?
3. When will all these different package systems and methods be made into one single simple process ?

Re: “Do I need a MANIFEST.in?

No, you do not have to use MANIFEST.in. Both, distutils and setuptools are including in source distribution package all the files mentioned in setup.py – modules, package python files, README.txt and test/test*.py. If this is all you want to have in distribution package, you do not have to use MANIFEST.in.

If you want to manipulate (add or remove) default files to include, you have to use MANIFEST.in.

Re: What should be in it?

The procedure is simple:

1. Make sure, in your setup.py you include (by means of setup arguments) all the files you feel important for the program to run (modules, packages, scripts …)

2. Clarify, if there are some files to add or some files to exclude. If neither is needed, then there is no need for using MANIFEST.in.

3. If MANIFEST.in is needed, create it. Usually, you add there tests*/*.py files, README.rst if you do not use README.txt, docs files and possibly some data files for test suite, if necessary.

For example:

include README.rst
include COPYING.txt

To test it, run python setup.py sdist, and examine the tarball created under dist/.

When will all these different package systems …

Comparing the situation today and 2 years ago – the situation is much much better – setuptools is the way to go. You can ignore the fact, distutils is a bit broken and is low level base for setuptools as setuptools shall take care of hiding these things from you.

EDIT: Last few projects I use pbr for building distribution packages with three line setup.py and rest being in setup.cfg and requirements.txt. No need to care about MANIFEST.in and other strange stuff. Even though the package would deserve a bit more documentation. See http://docs.openstack.org/developer/pbr/

Old question, new answer:

No, you don’t need MANIFEST.in. However, to get setuptools to do what you (usually) mean, you do need to use the setuptools_scm, which takes the role of MANIFEST.in in 2 key places:

• It ensures all relevant files are packaged when running the sdist command (where all relevant files is defined as “all files under source control”)
• When using include_package_data to include package data as part of the build or bdist_wheel. (again: files under source control)

The historical understanding of MANIFEST.in is: when you don’t have a source control system, you need some other mechanism to distinguish between “source files” and “files that happen to be in your working directory”. However, your project is under source control (right??) so there’s no need for MANIFEST.in. More info in this article.

How could I get the version defined in setup.py from my package (for --version, or other purposes)?

Interrogate version string of already-installed distribution

To retrieve the version from inside your package at runtime (what your question appears to actually be asking), you can use:

import pkg_resources  # part of setuptools
version = pkg_resources.require("MyProject")[0].version

Store version string for use during install

If you want to go the other way ’round (which appears to be what other answer authors here appear to have thought you were asking), put the version string in a separate file and read that file’s contents in setup.py.

You could make a version.py in your package with a __version__ line, then read it from setup.py using execfile('mypackage/version.py'), so that it sets __version__ in the setup.py namespace.

If you want a much simpler way that will work with all Python versions and even non-Python languages that may need access to the version string:

Store the version string as the sole contents of a plain text file, named e.g. VERSION, and read that file during setup.py.

version_file = open(os.path.join(mypackage_root_dir, 'VERSION'))
version = version_file.read().strip()

The same VERSION file will then work exactly as well in any other program, even non-Python ones, and you only need to change the version string in one place for all programs.

Warning about race condition during install

By the way, DO NOT import your package from your setup.py as suggested in another answer here: it will seem to work for you (because you already have your package’s dependencies installed), but it will wreak havoc upon new users of your package, as they will not be able to install your package without manually installing the dependencies first.

example study: mymodule

Imagine this configuration:

setup.py
mymodule/
        / __init__.py
        / version.py
        / myclasses.py

Then imagine some usual scenario where you have dependencies and setup.py looks like:

setup(...
    install_requires=['dep1','dep2', ...]
    ...)

And an example __init__.py:

from mymodule.myclasses import *
from mymodule.version import __version__

And for example myclasses.py:

# these are not installed on your system.
# importing mymodule.myclasses would give ImportError
import dep1
import dep2

problem #1: importing mymodule during setup

If your setup.py imports mymodule then during setup you would most likely get an ImportError. This is a very common error when your package has dependencies. If your package does not have other dependencies than the builtins, you may be safe; however this isn’t a good practice. The reason for that is that it is not future-proof; say tomorrow your code needs to consume some other dependency.

problem #2: where’s my __version__ ?

If you hardcode __version__ in setup.py then it may not match the version that you would ship in your module. To be consistent, you would put it in one place and read it from the same place when you need it. Using import you may get the problem #1.

solution: à la setuptools

You would use a combination of open, exec and provide a dict for exec to add variables:

# setup.py
from setuptools import setup, find_packages
from distutils.util import convert_path

main_ns = {}
ver_path = convert_path('mymodule/version.py')
with open(ver_path) as ver_file:
    exec(ver_file.read(), main_ns)

setup(...,
    version=main_ns['__version__'],
    ...)

And in mymodule/version.py expose the version:

__version__ = 'some.semantic.version'

This way, the version is shipped with the module, and you do not have issues during setup trying to import a module that has missing dependencies (yet to be installed).

The best technique is to define __version__ in your product code, then import it into setup.py from there. This gives you a value you can read in your running module, and have only one place to define it.

The values in setup.py are not installed, and setup.py doesn’t stick around after installation.

What I did (for example) in coverage.py:

# coverage/__init__.py
__version__ = "3.2"


# setup.py
from coverage import __version__

setup(
    name = 'coverage',
    version = __version__,
    ...
    )

UPDATE (2017): coverage.py no longer imports itself to get the version. Importing your own code can make it uninstallable, because you product code will try to import dependencies, which aren’t installed yet, because setup.py is what installs them.

Your question is a little vague, but I think what you are asking is how to specify it.

You need to define __version__ like so:

__version__ = '1.4.4'

And then you can confirm that setup.py knows about the version you just specified:

% ./setup.py --version
1.4.4

I wasn’t happy with these answers… didn’t want to require setuptools, nor make a whole separate module for a single variable, so I came up with these.

For when you are sure the main module is in pep8 style and will stay that way:

version = '0.30.unknown'
with file('mypkg/mymod.py') as f:
    for line in f:
        if line.startswith('__version__'):
            _, _, version = line.replace("'", '').split()
            break

If you’d like to be extra careful and use a real parser:

import ast
version = '0.30.unknown2'
with file('mypkg/mymod.py') as f:
    for line in f:
        if line.startswith('__version__'):
            version = ast.parse(line).body[0].value.s
            break

setup.py is somewhat of a throwaway module so not an issue if it is a bit ugly.

Update: funny enough I’ve moved away from this in recent years and started using a separate file in the package called meta.py. I put lots of meta data in there that I might want to change frequently. So, not just for one value.

Create a file in your source tree, e.g. in yourbasedir/yourpackage/_version.py . Let that file contain only a single line of code, like this:

__version__ = "1.1.0-r4704"

Then in your setup.py, open that file and parse out the version number like this:

verstr = "unknown"
try:
    verstrline = open('yourpackage/_version.py', "rt").read()
except EnvironmentError:
    pass # Okay, there is no version file.
else:
    VSRE = r"^__version__ = ['\"]([^'\"]*)['\"]"
    mo = re.search(VSRE, verstrline, re.M)
    if mo:
        verstr = mo.group(1)
    else:
        raise RuntimeError("unable to find version in yourpackage/_version.py")

Finally, in yourbasedir/yourpackage/__init__.py import _version like this:

__version__ = "unknown"
try:
    from _version import __version__
except ImportError:
    # We're running in a tree that doesn't have a _version.py, so we don't know what our version is.
    pass

An example of code that does this is the “pyutil” package that I maintain. (See PyPI or google search — stackoverflow is disallowing me from including a hyperlink to it in this answer.)

@pjeby is right that you shouldn’t import your package from its own setup.py. That will work when you test it by creating a new Python interpreter and executing setup.py in it first thing: python setup.py, but there are cases when it won’t work. That’s because import youpackage doesn’t mean to read the current working directory for a directory named “yourpackage”, it means to look in the current sys.modules for a key “yourpackage” and then to do various things if it isn’t there. So it always works when you do python setup.py because you have a fresh, empty sys.modules, but this doesn’t work in general.

For example, what if py2exe is executing your setup.py as part of the process of packaging up an application? I’ve seen a case like this where py2exe would put the wrong version number on a package because the package was getting its version number from import myownthing in its setup.py, but a different version of that package had previously been imported during the py2exe run. Likewise, what if setuptools, easy_install, distribute, or distutils2 is trying to build your package as part of a process of installing a different package that depends on yours? Then whether your package is importable at the time that its setup.py is being evaluated, or whether there is already a version of your package that has been imported during this Python interpreter’s life, or whether importing your package requires other packages to be installed first, or has side-effects, can change the results. I’ve had several struggles with trying to re-use Python packages which caused problems for tools like py2exe and setuptools because their setup.py imports the package itself in order to find its version number.

By the way, this technique plays nicely with tools to automatically create the yourpackage/_version.py file for you, for example by reading your revision control history and writing out a version number based on the most recent tag in revision control history. Here is a tool that does that for darcs: http://tahoe-lafs.org/trac/darcsver/browser/trunk/README.rst and here is a code snippet which does the same thing for git: http://github.com/warner/python-ecdsa/blob/0ed702a9d4057ecf33eea969b8cf280eaccd89a1/setup.py#L34

This should also work, using regular expressions and depending on the metadata fields to have a format like this:

__fieldname__ = 'value'

Use the following at the beginning of your setup.py:

import re
main_py = open('yourmodule.py').read()
metadata = dict(re.findall("__([a-z]+)__ = '([^']+)'", main_py))

After that, you can use the metadata in your script like this:

print 'Author is:', metadata['author']
print 'Version is:', metadata['version']

With a structure like this:

setup.py
mymodule/
        / __init__.py
        / version.py
        / myclasses.py

where version.py contains:

__version__ = 'version_string'

You can do this in setup.py:

import sys

sys.path[0:0] = ['mymodule']

from version import __version__

This won’t cause any problem with whatever dependencies you have in your mymodule/__init__.py

To avoid importing a file (and thus executing its code) one could parse it and recover the version attribute from the syntax tree:

# assuming 'path' holds the path to the file

import ast

with open(path, 'rU') as file:
    t = compile(file.read(), path, 'exec', ast.PyCF_ONLY_AST)
    for node in (n for n in t.body if isinstance(n, ast.Assign)):
        if len(node.targets) == 1:
            name = node.targets[0]
            if isinstance(name, ast.Name) and \
                    name.id in ('__version__', '__version_info__', 'VERSION'):
                v = node.value
                if isinstance(v, ast.Str):
                    version = v.s
                    break
                if isinstance(v, ast.Tuple):
                    r = []
                    for e in v.elts:
                        if isinstance(e, ast.Str):
                            r.append(e.s)
                        elif isinstance(e, ast.Num):
                            r.append(str(e.n))
                    version = '.'.join(r)
                    break

This code tries to find the __version__ or VERSION assignment at the top level of the module return is string value. The right side can be either a string or a tuple.

There’s a thousand ways to skin a cat — here’s mine:

# Copied from (and hacked):
# https://github.com/pypa/virtualenv/blob/develop/setup.py#L42
def get_version(filename):
    import os
    import re

    here = os.path.dirname(os.path.abspath(__file__))
    f = open(os.path.join(here, filename))
    version_file = f.read()
    f.close()
    version_match = re.search(r"^__version__ = ['\"]([^'\"]*)['\"]",
                              version_file, re.M)
    if version_match:
        return version_match.group(1)
    raise RuntimeError("Unable to find version string.")

Cleaning up https://stackoverflow.com/a/12413800 from @gringo-suave:

from itertools import ifilter
from os import path
from ast import parse

with open(path.join('package_name', '__init__.py')) as f:
    __version__ = parse(next(ifilter(lambda line: line.startswith('__version__'),
                                     f))).body[0].value.s

Now this is gross and needs some refining (there may even be an uncovered member call in pkg_resources that I missed), but I simply do not see why this doesn’t work, nor why no one has suggested it to date (Googling around has not turned this up)…note that this is Python 2.x, and would require requiring pkg_resources (sigh):

import pkg_resources

version_string = None
try:
    if pkg_resources.working_set is not None:
        disto_obj = pkg_resources.working_set.by_key.get('<my pkg name>', None)
        # (I like adding ", None" to gets)
        if disto_obj is not None:
            version_string = disto_obj.version
except Exception:
    # Do something
    pass

We wanted to put the meta information about our package pypackagery in __init__.py, but could not since it has third-party dependencies as PJ Eby already pointed out (see his answer and the warning regarding the race condition).

We solved it by creating a separate module pypackagery_meta.py that contains only the meta information:

"""Define meta information about pypackagery package."""

__title__ = 'pypackagery'
__description__ = ('Package a subset of a monorepo and '
                   'determine the dependent packages.')
__url__ = 'https://github.com/Parquery/pypackagery'
__version__ = '1.0.0'
__author__ = 'Marko Ristin'
__author_email__ = 'marko.ristin@gmail.com'
__license__ = 'MIT'
__copyright__ = 'Copyright 2018 Parquery AG'

then imported the meta information in packagery/__init__.py:

# ...

from pypackagery_meta import __title__, __description__, __url__, \
    __version__, __author__, __author_email__, \
    __license__, __copyright__

# ...

and finally used it in setup.py:

import pypackagery_meta

setup(
    name=pypackagery_meta.__title__,
    version=pypackagery_meta.__version__,
    description=pypackagery_meta.__description__,
    long_description=long_description,
    url=pypackagery_meta.__url__,
    author=pypackagery_meta.__author__,
    author_email=pypackagery_meta.__author_email__,
    # ...
    py_modules=['packagery', 'pypackagery_meta'],
 )

You must include pypackagery_meta into your package with py_modules setup argument. Otherwise, you can not import it upon installation since the packaged distribution would lack it.

Simple and straight, create a file called source/package_name/version.py with the following contents:

#!/usr/bin/env python3
# -*- coding: UTF-8 -*-
__version__ = "2.6.9"

Then, on your file source/package_name/__init__.py, you import the version for other people to use:

#!/usr/bin/env python3
# -*- coding: UTF-8 -*-
from .version import __version__

Now, you can put this on setup.py

#!/usr/bin/env python3
# -*- coding: UTF-8 -*-
import re
import sys

try:
    filepath = 'source/package_name/version.py'
    version_file = open( filepath )
    __version__ ,= re.findall( '__version__ = "(.*)"', version_file.read() )

except Exception as error:
    __version__ = "0.0.1"
    sys.stderr.write( "Warning: Could not open '%s' due %s\n" % ( filepath, error ) )

finally:
    version_file.close()

Tested this with Python 2.7, 3.3, 3.4, 3.5, 3.6 and 3.7 on Linux, Windows and Mac OS. I used on my package which has Integration and Unit Tests for all theses platforms. You can see the results from .travis.yml and appveyor.yml here:

1. https://travis-ci.org/evandrocoan/debugtools/builds/527110800
2. https://ci.appveyor.com/project/evandrocoan/pythondebugtools/builds/24245446

An alternate version is using context manager:

#!/usr/bin/env python3
# -*- coding: UTF-8 -*-
import re
import sys

try:
    filepath = 'source/package_name/version.py'

    with open( filepath ) as file:
        __version__ ,= re.findall( '__version__ = "(.*)"', file.read() )

except Exception as error:
    __version__ = "0.0.1"
    sys.stderr.write( "Warning: Could not open '%s' due %s\n" % ( filepath, error ) )

You can also be using the codecs module to handle unicode errors both on Python 2.7 and 3.6

#!/usr/bin/env python3
# -*- coding: UTF-8 -*-
import re
import sys
import codecs

try:
    filepath = 'source/package_name/version.py'

    with codecs.open( filepath, 'r', errors='ignore' ) as file:
        __version__ ,= re.findall( '__version__ = "(.*)"', file.read() )

except Exception as error:
    __version__ = "0.0.1"
    sys.stderr.write( "Warning: Could not open '%s' due %s\n" % ( filepath, error ) )

If you are writing a Python module 100% in C/C++ using Python C Extensions, you can do the same thing, but using C/C++ instead of Python.

On this case, create the following setup.py:

#!/usr/bin/env python3
# -*- coding: UTF-8 -*-
import re
import sys
import codecs
from setuptools import setup, Extension

try:
    filepath = 'source/version.h'

    with codecs.open( filepath, 'r', errors='ignore' ) as file:
        __version__ ,= re.findall( '__version__ = "(.*)"', file.read() )

except Exception as error:
    __version__ = "0.0.1"
    sys.stderr.write( "Warning: Could not open '%s' due %s\n" % ( filepath, error ) )

setup(
        name = 'package_name',
        version = __version__,

        package_data = {
                '': [ '**.txt', '**.md', '**.py', '**.h', '**.hpp', '**.c', '**.cpp' ],
            },

        ext_modules = [
            Extension(
                name = 'package_name',
                sources = [
                    'source/file.cpp',
                ],
                include_dirs = ['source'],
            )
        ],
    )

Which reads the version from the file version.h:

const char* __version__ = "1.0.12";

But, do not forget to create the MANIFEST.in to include the version.h file:

include README.md
include LICENSE.txt

recursive-include source *.h

And it is integrated into the main application with:

#include <Python.h>
#include "version.h"

// create the module
PyMODINIT_FUNC PyInit_package_name(void)
{
    PyObject* thismodule;
    ...

    // https://docs.python.org/3/c-api/arg.html#c.Py_BuildValue
    PyObject_SetAttrString( thismodule, "__version__", Py_BuildValue( "s", __version__ ) );

    ...
}

References:

1. python open file error
2. Define a global in a Python module from a C API
3. How to include package data with setuptools/distribute?
4. https://github.com/lark-parser/lark/blob/master/setup.py#L4
5. How to use setuptools packages and ext_modules with the same name?
6. Is it possible to include subdirectories using dist utils (setup.py) as part of package data?

deploy package to server and file naming convention for indices packages :

example for pip dynamic version conversion:

• win:

  • test_pkg-1.0.0-cp36-cp36m-win_amd64.whl
  • test_pkg-1.0.0-py3.6-win-amd64.egg

• mac:

  • test_pkg-1.0.0-py3.7-macosx-10.12-x86_64.egg
  • test_pkg-1.0.0-py3.7-macosx-10.12-x86_64.whl
• linux:
  • test_pkg-1.0.0-cp36-cp36m-linux_x86_64.whl

from setuptools_scm import get_version

def _get_version():

     dev_version = str(".".join(map(str, str(get_version()).split("+")[0]\
            .split('.')[:-1])))

    return dev_version

Find the sample setup.py calls the dynamic pip version matching from git commit

setup(
    version=_get_version(),
    name=NAME,
    description=DESCRIPTION,
    long_description=LONG_DESCRIPTION,
    classifiers=CLASSIFIERS,

# add few more for wheel wheel package ...conversion

)

I am using an environment variable as below

VERSION=0.0.0 python setup.py sdist bdist_wheel

In setup.py

import os

setup(
    version=os.environ['VERSION'],
    ...
)

For consistency check with packer version, I am using below script.

PKG_VERSION=`python -c "import pkg; print(pkg.__version__)"`
if [ $PKG_VERSION == $VERSION ]; then
    python setup.py sdist bdist_wheel
else
    echo "Package version differs from set env variable"
fi

I’ve read the documentation on egg entry points in Pylons and on the Peak pages, and I still don’t really understand. Could someone explain them to me?

An “entry point” is typically a function (or other callable function-like object) that a developer or user of your Python package might want to use, though a non-callable object can be supplied as an entry point as well (as correctly pointed out in the comments!).

The most popular kind of entry point is the console_scripts entry point, which points to a function that you want made available as a command-line tool to whoever installs your package. This goes into your setup.py like:

entry_points={
    'console_scripts': [
        'cursive = cursive.tools.cmd:cursive_command',
    ],
},

I have a package I’ve just deployed called “cursive.tools”, and I wanted it to make available a “cursive” command that someone could run from the command line, like:

$ cursive --help
usage: cursive ...

The way to do this is define a function, like maybe a “cursive_command” function in cursive/tools/cmd.py that looks like:

def cursive_command():
    args = sys.argv[1:]
    if len(args) < 1:
        print "usage: ..."

and so forth; it should assume that it’s been called from the command line, parse the arguments that the user has provided, and … well, do whatever the command is designed to do.

Install the docutils package for a great example of entry-point use: it will install something like a half-dozen useful commands for converting Python documentation to other formats.

EntryPoints provide a persistent, filesystem-based object name registration and name-based direct object import mechanism (implemented by the setuptools package).

They associate names of Python objects with free-form identifiers. So any other code using the same Python installation and knowing the identifier can access an object with the associated name, no matter where the object is defined. The associated names can be any names existing in a Python module; for example name of a class, function or variable. The entry point mechanism does not care what the name refers to, as long as it is importable.

As an example, let’s use (the name of) a function, and an imaginary python module with a fully-qualified name ‘myns.mypkg.mymodule’:

def the_function():
   "function whose name is 'the_function', in 'mymodule' module"
   print "hello from the_function"

Entry points are registered via an entry points declaration in setup.py. To register the_function under entrypoint called ‘my_ep_func’:

    entry_points = {
        'my_ep_group_id': [
            'my_ep_func = myns.mypkg.mymodule:the_function'
        ]
    },

As the example shows, entry points are grouped; there’s corresponding API to look up all entry points belonging to a group (example below).

Upon a package installation (ie. running ‘python setup.py install’), the above declaration is parsed by setuptools. It then writes the parsed information in special file. After that, the pkg_resources API (part of setuptools) can be used to look up the entry point and access the object(s) with the associated name(s):

import pkg_resources

named_objects = {}
for ep in pkg_resources.iter_entry_points(group='my_ep_group_id'):
   named_objects.update({ep.name: ep.load()})

Here, setuptools read the entry point information that was written in special files. It found the entry point, imported the module (myns.mypkg.mymodule), and retrieved the_function defined there, upon call to pkg_resources.load().

Assuming there were no other entry point registrations for the same group id, calling the_function would then be simple:

>>> named_objects['my_ep_func']()
hello from the_function

Thus, while perhaps a bit difficult to grasp at first, the entry point mechanism is actually quite simple to use. It provides an useful tool for pluggable Python software development.

From abstract point of view, entry points are used to create a system-wide registry of Python callables that implement certain interfaces. There are APIs in pkg_resources to see which entry points are advertised by a given package as well as APIs to determine which packages advertise a certain entry point.

Entry points are useful for allowing one package do use plugins that are in another package. For instance, Ian Bicking’s Paste project uses entry points extensively. In this case, you can write a package that advertises its WSGI application factory using the entry point paste.app_factory.

Another use for entry points is enumerating all the packages on the system that provide some plugin functionality. The TurboGears web framework uses the python.templating.engines entry point to look up templating libraries that are installed and available.