Abstract

PEP:534
Title:Distributing a Subset of the Standard Library
Version:$Revision$
Last-Modified:$Date$
Author:Tomáš Orsava <tomas.n at orsava.cz>, Petr Viktorin <encukou at gmail.com>, Nick Coghlan <ncoghlan at gmail.com>
Status:Draft
Type:Standards Track
Content-Type:text/x-rst
Created:5-Sep-2016
Python-Version:3.7
Post-History:

Python is often being built or distributed without its full standard library. However, there is as of yet no standard, user friendly way of properly informing the user about the failure to import such missing stdlib modules. This PEP proposes a mechanism for identifying standard library modules and informing the user appropriately.

Motivation

There are several use cases for including only a subset of Python's standard library. However, there is so far no user-friendly mechanism for informing the user why a stdlib module is missing and how to remedy the situation appropriately.

CPython

When one of Python standard library modules (such as _sqlite3) cannot be compiled during a Python build because of missing dependencies (e.g. SQLite header files), the module is simply skipped. If you then install this compiled Python and use it to try to import one of the missing modules, Python will fail with a ModuleNotFoundError [3].

This can confuse users who may not understand why a cleanly built Python is missing standard library modules.

Linux and other distributions

Many Linux and other distributions are already separating out parts of the standard library to standalone packages. Among the most commonly excluded modules are the tkinter module, since it draws in a dependency on the graphical environment, and the test package, as it only serves to test Python internally and is about as big as the rest of the standard library put together.

The methods of omission of these modules differ. For example, Debian patches the file Lib/tkinter/__init__.py to envelop the line import _tkinter in a try-except block and upon encountering an ImportError it simply adds the following to the error message: please install the python3-tk package [1]. Fedora and other distributions simply don't include the omitted modules, potentially leaving users baffled as to where to find them.

Specification

The sysconfig [4] module will be extended by two functions: sysconfig.get_stdlib_modules(), which will provide a list of the names of all Python standard library modules, and a function sysconfig.get_optional_modules(), that will list optional stdlib module names. The results of the latter function—sysconfig.get_optional_modules()—as well as of the existing sys.builtin_module_names will both be subsets of the full list provided by sysconfig.get_stdlib_modules(). These added lists will be generated during the Python build process and saved in _sysconfigdata-*.py file along with other sysconfig [4] values.

The default implementation of the sys.excepthook [5] function will then be modified to dispense an appropriate message when it detects a failure to import a module identified by one of the two new sysconfig [4] functions as belonging to the Python standard library.

Rationale

The sys.excepthook [5] function gets called when a raised exception is uncaught and the program is about to exit or—in an interactive session—the control is being returned to the prompt. This makes it a perfect place for customized error messages, as it will not influence caught errors and thus not slow down normal execution of Python scripts.

The inclusion of the functions sysconfig.get_stdlib_modules() and sysconfig.get_optional_modules() will also provide a long sought-after way of easily listing the names of Python standard library modules [2], which will—among other benefits—make it easier for code analysis, profiling, and error reporting tools to offer runtime --ignore-stdlib flags.

Ideas leading up to this PEP were discussed on the python-dev mailing list [6] and subsequently on python-ideas [7].

Recommendation for Downstream Distributors

By patching site.py [8] [*] to provide their own implementation of the sys.excepthook [5] function, Python distributors can display tailor-made error messages for any uncaught exceptions, including informing the user of a proper, distro-specific way to install missing standard library modules upon encountering a ModuleNotFoundError [3]. Some downstream distributors are already using this method of patching sys.excepthook to integrate with platform crash reporting mechanisms.

An example implementation is attached to this PEP.

Backwards Compatibility

No problems with backwards compatibility are expected. Distributions that are already patching Python modules to provide custom handling of missing dependencies can continue to do so unhindered.

Reference and Example Implementation

TBD. The finer details will depend on what's practical given the capabilities of the build system.