Building Astropy and its Subpackages

The build process currently uses the setuptools package to build and install the astropy core (and any affiliated packages that use the template).

Astropy-helpers

As of Astropy v0.4, Astropy also uses an external package called astropy-helpers to provide some of its build and installation functionality. A copy of astropy-helpers is included with the Astropy source distribution, but also includes a mechanism to automatically download bug fixes from PyPI. The reason for providing these helpers as a separate package is that it makes it easier for affiliated packages to take advantage of these same utilities without requiring Astropy to be installed first. See APE4 for the full background on this.

Astropy-helpers is automatically bootstrapped to the Astropy build/installation script (setup.py) via a script called ah_bootstrap.py that is imported by setup.py. This script will do its best to ensure that the user has an up-to-date copy of astropy-helpers before building the package. The auto-upgrade mechanism in particular allows pushing platform-specific fixes for the build process without releasing a new version of Astropy (or any affiliated package that uses astropy-helpers).

The behavior of the ah_bootstrap.py script can also be modified by options in the project’s setup.cfg file under a section called [ah_boostrap]. APE4 provides more details.

The astropy-helpers distribution provides a Python package called astropy_helpers. Code that previously referenced the modules astropy.setup_helpers and astropy.version_helpers should now depend on astropy-helpers and use astrop_helpers.setup_helpers and astropy_helpers.version_helpers respectively. Likewise, astropy-helpers includes tools for building Astropy’s documentation. The astropy.sphinx package is deprecated in favor of astropy_helpers.sphinx. As such, astropy-helpers is a dependency of building Astropy’s documentation.

Customizing setup/build for subpackages

As is typical, there is a single setup.py file that is used for the whole astropy package. To customize setup parameters for a given sub-package, a setup_package.py file can be defined inside a package, and if it is present, the setup process will look for the following functions to customize the build process:

  • get_package_data

    This function, if defined, should return a dictionary mapping the name of the subpackage(s) that need package data to a list of data file paths (possibly including wildcards) relative to the path of the package’s source code. e.g. if the source distribution has a needed data file astropy/wcs/tests/data/3d_cd.hdr, this function should return {'astropy.wcs.tests':['data/3d_cd.hdr']}. See the package_data option of the distutils.core.setup() function.

    It is recommended that all such data be in a directory named data inside the package within which it is supposed to be used. This package data should be accessed via the astropy.utils.data.get_pkg_data_filename and astropy.utils.data.get_pkg_data_fileobj functions.

  • get_extensions

    This provides information for building C or Cython extensions. If defined, it should return a list of distutils.core.Extension objects controlling the Cython/C build process (see below for more detail).

  • get_build_options

    This function allows a package to add extra build options. It should return a list of tuples, where each element has:

    • name: The name of the option as it would appear on the commandline or in the setup.cfg file.
    • doc: A short doc string for the option, displayed by setup.py build --help.
    • is_bool (optional): When True, the option is a boolean option and doesn’t have an associated value.

    Once an option has been added, its value can be looked up using astropy_helpers.setup_helpers.get_distutils_build_option.

  • get_external_libraries

    This function declares that the package uses libraries that are included in the astropy distribution that may also be distributed elsewhere on the users system. It should return a list of library names. For each library, a new build option is created, '--use-system-X' which allows the user to request to use the system’s copy of the library. The package would typically call astropy_helpers.setup_helpers.use_system_library from its get_extensions function to determine if the package should use the system library or the included one.

The astropy_helpers.setup_helpers modules includes an update_package_files function which automatically searches the given source path for setup_package.py modules and calls each of the above functions, if they exist. This makes it easy for affiliated packages to use this machinery in their own setup.py.

PLY Parsing/Lexing tables

For certain string-prasing tasks, Astropy uses the PLY tool. PLY generates tables that speed up the parsing process, which are checked into source code so they don’t have to be regenerated. These tables can be recognized by having either lextab or parsetab in their names. To regenerate these files (e.g. if a new version of PLY is bundled with Astropy or some of the parsing code changes), the tables need to be deleted and the appropriate parts of astropy re-imported and run. For exact details, see the comments in the headers of the parsetab and lextab files.