Specialness of None

The Python language does not currently define any special behavior for None. This PEP suggests making None a special case. This loss of generality is a noticeable drawback of the proposal. A generalization of None-aware operators is set forth later in this document in order to avoid this specialization.

Utility of None

One common criticism of adding special syntax for None is that None shouldn't be used in the first place: it's a code smell. A related criticism is that None-aware operators are used to silence errors (such as the novice misunderstanding of an implicit return None) akin to PHP's @ operator. Therefore, the utility of None must be debated before discussing whether to add new behavior around it.

Python does not have any concept of null. Every Python identifier must refer to an instance, so there cannot be any null references. Python does have a special instance called None that can be used to represent missing values, but None is conceptually distinct from null.

The most frequent use of None in Python is to provide a default value for optional arguments when some other default object is unwieldy. For example: def get(url, proxy=None):. In this case, proxy is an optional argument. If proxy is None, then the request should be sent directly to the server; otherwise, the request should be routed through the specified proxy server. This use of None is preferred here to some other sentinel value or the Null Object Pattern. [3]

Examples of this form abound. Consider types.py in the standard library:

def prepare_class(name, bases=(), kwds=None):
    if kwds is None:
        kwds = {}
    else:
        kwds = dict(kwds)
    ...

Another frequent use of None is interfacing with external systems. Many of those other systems have a concept of null. Therefore, Python code must have a way of representing null, and typically it is represented by None. For example, databases can have null values, and most Python database drivers will convert null to None when retrieving data from a database, and will convert from None back to null when sending data to a database.

This convention of interchanging null and None is widespread in Python. It is canonized in the Python DBAPI (PEP-249). [4] The json module in the standard library and the third party PyYAML package both use None to represent their respective languages' null.

The C language null often bleeds into Python, too, particularly for thin wrappers around C libraries. For example, in pyopenssl, the X509 class has a get_notBefore() method that returns either a timestamp or None. This function is a thin wrapper around an OpenSSL function with the return type ASN1_TIME *. Because this C pointer may be null, the Python wrapper must be able to represent null, and None is the chosen representation.

The representation of null is particularly noticeable when Python code is marshalling data between two systems. For example, consider a Python server that fetches data from a database and converts it to JSON for consumption by another process. In this case, it's often desirable that null in the database can be easily translated to null in JSON. If None is not used for this purpose, then each package will have to define its own representation of null, and converting between these representations adds unnecessary complexity to the Python glue code.

Therefore, the preference for avoiding None is nothing more than a preference. None has legitimate uses, particularly in specific types of software. Any hypothetical None-aware operators should be construed as syntactic sugar for simplifying common patterns involving None, and should not be construed as error handling behavior.

Behavior In Other Languages

Given that null-aware operators exist in other modern languages, it may be helpful to quickly understand how they work in those languages:

/* Null-coalescing. */

String s1 = null;
String s2 = "hello";
String s3 = s1 ?? s2;
Console.WriteLine("s3 is: " + s3);
// s3 is: hello

/* Null-aware member access, a.k.a. safe navigation. */

Console.WriteLine("s1.Length is: " + s1?.Length);
Console.WriteLine("s2.Length is: " + s2?.Length);
// s1.Length is:
// s2.Length is: 5

/* Null-aware index access, a.k.a. safe navigation. */

Dictionary<string,string> d1 = null;
Dictionary<string,string> d2 = new Dictionary<string, string>
{
    { "foo", "bar" },
    { "baz", "bat" }
};

Console.WriteLine("d1[\"foo\"] is: " + d1?["foo"]);
Console.WriteLine("d2[\"foo\"] is: " + d2?["foo"]);
// d1["foo"] is:
// d2["foo"] is: bar

/* Short Circuiting */

Console.WriteLine("s1 trim/upper is: " + s1?.Trim().Length);
Console.WriteLine("s2 trim/upper is: " + s2?.Trim().Length);
// s1 trimmed length is:
// s2 trimmed length is: 5

String s4 = s1 ?? s2 ?? DoError();
Console.WriteLine("s4 is: " + s4)
// s4 is: hello

A working example can be viewed online.

Of utmost importance, notice the short circuiting behavior. The short circuiting of ?? is similar to short circuiting of other boolean operators such as || or && and should not be surprising. Helpfully, ?. is also short circuiting: s1?.Trim() evaluates to null, but s1?.Trim().Length does not attempt to dereference the null pointer.

Existing Alternatives

Python does not have any specific None-aware operators, but it does have operators that can be used for a similar purpose. This section describes why these alternatives may be undesirable for some common None patterns.

>>> price = 100
>>> default_quantity = 1
# If user didn't specify a quantity, then assume the default.
>>> requested_quantity = None
>>> (requested_quantity or default_quantity) * price
100
# The user added 5 items to the cart.
>>> requested_quantity = 5
>>> (requested_quantity or default_quantity) * price
500
# User removed 5 items from cart.
>>> requested_quantity = 0
>>> (requested_quantity or default_quantity) * price # oops!
100

data = [] if data is None else data
files = [] if files is None else files
headers = {} if headers is None else headers
params = {} if params is None else params
hooks = {} if hooks is None else hooks

data = data if data is not None else []
files = files if files is not None else []
headers = headers if headers is not None else {}
params = params if params is not None else {}
hooks = hooks if hooks is not None else {}

Motivating Examples

The purpose of this PEP is to simplify some common patterns involving None. This section presents some examples of common None patterns and explains the drawbacks.

This first example is from a Python web crawler that uses the popular Flask framework as a front-end. This function retrieves information about a web site from a SQL database and formats it as JSON to send to an HTTP client:

class SiteView(FlaskView):
    @route('/site/<id_>', methods=['GET'])
    def get_site(self, id_):
        site = db.query('site_table').find(id_)

        return jsonify(
            first_seen=site.first_seen.isoformat() if site.first_seen is not None else None,
            id=site.id,
            is_active=site.is_active,
            last_seen=site.last_seen.isoformat() if site.last_seen is not None else None,
            url=site.url.rstrip('/')
        )

Both first_seen and last_seen are allowed to be null in the database, and they are also allowed to be null in the JSON response. JSON does not have a native way to represent a datetime, so the server's contract states that any non-null date is represented as an ISO-8601 string.

Note that this code is invalid by PEP-8 standards: several lines are over the line length limit. In fact, including it in this document violates the PEP formatting standard! But it's not unreasonably indented, nor are any of the identifiers excessively long. The excessive line length is due to the repetition of identifiers on both sides of the ternary if and the verbosity of the ternary itself (10 characters out of a 78 character line length).

One way to fix this code is to replace each ternary with a full if/else block:

class SiteView(FlaskView):
    @route('/site/<id_>', methods=['GET'])
    def get_site(self, id_):
        site = db.query('site_table').find(id_)

        if site.first_seen is None:
            first_seen = None
        else:
            first_seen = site.first_seen.isoformat()

        if site.last_seen is None:
            last_seen = None
        else:
            last_seen = site.last_seen.isoformat()

        return jsonify(
            first_seen=first_seen,
            id=site.id,
            is_active=site.is_active,
            last_seen=last_seen,
            url=site.url.rstrip('/')
        )

This version definitely isn't bad. It is easy to read and understand. On the other hand, adding 8 lines of code to express this common behavior feels a bit heavy, especially for a deliberately simplified example. If a larger, more complicated data model was being used, then it would get tedious to continually write in this long form. The readability would start to suffer as the number of lines in the function grows, and a refactoring would be forced.

Another alternative is to rename some of the identifiers:

class SiteView(FlaskView):
    @route('/site/<id_>', methods=['GET'])
    def get_site(self, id_):
        site = db.query('site_table').find(id_)

        fs = site.first_seen
        ls = site.last_seen

        return jsonify(
            first_seen=fs.isodate() if fs is not None else None,
            id=site.id,
            is_active=site.is_active,
            last_seen=ls.isodate() if ls is not None else None,,
            url=site.url.rstrip('/')
        )

This adds fewer lines of code than the previous example, but it comes at the expense of introducing extraneous identifiers that amount to nothing more than aliases. These new identifiers are short enough to fit a ternary expression onto one line, but the identifiers are also less intuitive, e.g. fs versus first_seen.

As a quick preview, consider an alternative rewrite using a new operator:

class SiteView(FlaskView):
    @route('/site/<id_>', methods=['GET'])
    def get_site(self, id_):
        site = db.query('site_table').find(id_)

        return jsonify(
            first_seen=site.first_seen?.isoformat(),
            id=site.id,
            is_active=site.is_active,
            last_seen=site.last_seen?.isoformat(),
            url=site.url.rstrip('/')
        )

The ?. operator behaves as a "safe navigation" operator, allowing a more concise syntax where the expression site.first_seen is not duplicated.

The next example is from a trending project on GitHub called Grab, which is a Python scraping library:

class BaseUploadObject(object):
    def find_content_type(self, filename):
        ctype, encoding = mimetypes.guess_type(filename)
        if ctype is None:
            return 'application/octet-stream'
        else:
            return ctype

class UploadContent(BaseUploadObject):
    def __init__(self, content, filename=None, content_type=None):
        self.content = content
        if filename is None:
            self.filename = self.get_random_filename()
        else:
            self.filename = filename
        if content_type is None:
            self.content_type = self.find_content_type(self.filename)
        else:
            self.content_type = content_type

class UploadFile(BaseUploadObject):
    def __init__(self, path, filename=None, content_type=None):
        self.path = path
        if filename is None:
            self.filename = os.path.split(path)[1]
        else:
            self.filename = filename
        if content_type is None:
            self.content_type = self.find_content_type(self.filename)
        else:
            self.content_type = content_type

This example contains several good examples of needing to provide default values. It is a bit verbose as it is, and it is certainly not improved by the ternary operator:

class BaseUploadObject(object):
    def find_content_type(self, filename):
        ctype, encoding = mimetypes.guess_type(filename)
        return 'application/octet-stream' if ctype is None else ctype

class UploadContent(BaseUploadObject):
    def __init__(self, content, filename=None, content_type=None):
        self.content = content
        self.filename = self.get_random_filename() if filename \
            is None else filename
        self.content_type = self.find_content_type(self.filename) \
            if content_type is None else content_type

class UploadFile(BaseUploadObject):
    def __init__(self, path, filename=None, content_type=None):
        self.path = path
        self.filename = os.path.split(path)[1] if filename is \
            None else filename
        self.content_type = self.find_content_type(self.filename) \
            if content_type is None else content_type

The first ternary expression is tidy, but it reverses the intuitive order of the operands: it should return ctype if it has a value and use the string literal as fallback. The other ternary expressions are unintuitive and so long that they must be wrapped. The overall readability is worsened, not improved.

This code might be improved, though, if there was a syntactic shortcut for this common need to supply a default value:

class BaseUploadObject(object):
    def find_ctype(self, filename):
        ctype, encoding = mimetypes.guess_type(filename)
        return ctype ?? 'application/octet-stream'

class UploadContent(BaseUploadObject):
    def __init__(self, content, filename=None, content_type=None):
        self.content = content
        self.filename = filename ?? self.get_random_filename()
        self.content_type = content_type ?? self.find_ctype(self.filename)

class UploadFile(BaseUploadObject):
    def __init__(self, path, filename=None, content_type=None):
        self.path = path
        self.filename = filename ?? os.path.split(path)[1]
        self.content_type = content_type ?? self.find_ctype(self.filename)

This syntax has an intuitive ordering of the operands, e.g. ctype -- the preferred value -- comes before the fallback value. The terseness of the syntax also makes for fewer lines of code and less code to visually parse.

Usage Of None In The Standard Library

The previous sections show some code patterns that are claimed to be "common", but how common are they? The attached script find-pep505.py is meant to answer this question. It uses the ast module to search for variations of the following patterns in any *.py file:

>>> # None-coalescing if block
...
>>> if a is None:
...     a = b

>>> # [Possible] None-coalescing "or" operator
...
>>> a or 'foo'
>>> a or []
>>> a or {}

>>> # None-coalescing ternary
...
>>> a if a is not None else b
>>> b if a is None else a

>>> # Safe navigation "and" operator
...
>>> a and a.foo
>>> a and a['foo']
>>> a and a.foo()

>>> # Safe navigation if block
...
>>> if a is not None:
...     a.foo()

>>> # Safe navigation ternary
...
>>> a.foo if a is not None else b
>>> b if a is None else a.foo

This script takes one or more names of Python source files to analyze:

$ python3 find-pep505.py test.py
$ find /usr/lib/python3.4 -name '*.py' | xargs python3 find-pep505.py

The script prints out any matches it finds. Sample:

None-coalescing if block: /usr/lib/python3.4/inspect.py:594
    if _filename is None:
         _filename = getsourcefile(object) or getfile(object)

[Possible] None-coalescing `or`: /usr/lib/python3.4/lib2to3/refactor.py:191
        self.explicit = explicit or []

None-coalescing ternary: /usr/lib/python3.4/decimal.py:3909
        self.clamp = clamp if clamp is not None else dc.clamp

Safe navigation `and`: /usr/lib/python3.4/weakref.py:512
        obj = info and info.weakref()

Safe navigation `if` block: /usr/lib/python3.4/http/cookiejar.py:1895
                        if k is not None:
                            lc = k.lower()
                        else:
                            lc = None

Safe navigation ternary: /usr/lib/python3.4/sre_parse.py:856
        literals = [None if s is None else s.encode('latin-1') for s in literals]

The script has been tested against test.py and the Python 3.4 standard library, but it should work on any arbitrary Python 3 source code. The complete output from running it against the standard library is attached to this proposal as find-pep505.out.

The script counts how many matches it finds and prints the totals at the end:

Total None-coalescing `if` blocks: 426
Total [possible] None-coalescing `or`: 119
Total None-coalescing ternaries: 21
Total Safe navigation `and`: 9
Total Safe navigation `if` blocks: 55
Total Safe navigation ternaries: 7

This is a total of 637 possible matches for these common code patterns in the standard library. Allowing for some false positives and false negatives, it is fair to say that these code patterns are definitely common in the standard library.

Rejected Ideas

Several related ideas were discussed on python-ideas, and some of these were roundly rejected by BDFL, the community, or both. For posterity's sake, some of those ideas are recorded here.

class NoneQuestion():
    def __call__(self, *args, **kwargs):
        return self

    def __getattr__(self, name):
        return self

    def __getitem__(self, key):
        return self

>>> import json
>>> created = None
>>> json.dumps({'created': created?.isoformat()})``

>>> x? + 1
>>> x? -= 1
>>> x? == 1
>>> ~x?

>>> maybe('VALUE').lower()
'value'

>>> maybe(None).invalid().method().or_else('unknown')
'unknown'

Operator Spelling

Despite significant support for the proposed operators, the majority of discussion on python-ideas fixated on the spelling. Many alternative spellings were proposed, both punctuation and keywords, but each alternative drew some criticism. Spelling the operator as a keyword is problematic, because adding new keywords to the language is not backwards compatible.

It is not impossible to add a new keyword, however, and we can look at several other PEPs for inspiration. For example, PEP-492 introduced the new keywords async and await into Python 3.5. These new keywords are fully backwards compatible, because that PEP also introduces a new lexical context such that async and await are only treated as keywords when used inside of an async def function. In other locations, async and await may be used as identifiers.

It is also possible to craft a new operator out of existing keywords, as was the case with PEP-308, which created a ternary operator by cobbling together the if and else keywords into a new operator.

In addition to the lexical acrobatics required to create a new keyword, keyword operators are also undesirable for creating an assignment shortcut syntax. In Dart, for example, x ??= y is an assignment shortcut that approximately means x = x ?? y except that x is only evaluated once. If Python's coalesce operator is a keyword, e.g. foo, then the assignment shortcut would be very ugly: x foo= y.

Spelling new logical operators with punctuation is unlikely, for several reasons. First, Python eschews punctuation for logical operators. For example, it uses not instead of !, or instead of ||, and … if … else … instead of … ? … : ….

Second, nearly every single punctuation character on a standard keyboard already has special meaning in Python. The only exceptions are $, !, ?, and backtick (as of Python 3). This leaves few options for a new, single-character operator.

Third, other projects in the Python universe assign special meaning to punctuation. For example, IPython assigns special meaning to %, %%, ?, ??, $, and $$, among others. Out of deference to those projects and the large communities using them, introducing conflicting syntax into Python is undesirable.

The spellings ?? and ?. will be familiar to programmers who have seen them in other popular programming languages. Any alternative punctuation will be just as ugly but without the benefit of familiarity from other languages. Therefore, this proposal spells the new operators using the same punctuation that already exists in other languages.

None-Coalescing Operator

The None-coalescing operator is a short-circuiting, binary operator that behaves in the following way.

1. Evaluate the left operand first.
2. If the left operand is not None, then return it immediately.
3. Else, evaluate the right operand and return the result.

Consider the following examples. We will continue to use the spelling ?? here, but keep in mind that alternative spellings will be discussed below:

>>> 1 ?? 2
1
>>> None ?? 2
2

Importantly, note that the right operand is not evaluated unless the left operand is None:

>>> def err(): raise Exception('foo')
>>> 1 ?? err()
1
>>> None ?? err()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 1, in err
Exception: foo

The operator is left associative. Combined with its short circuiting behavior, this makes the operator easy to chain:

>>> timeout = None
>>> local_timeout = 60
>>> global_timeout = 300
>>> timeout ?? local_timeout ?? global_timeout
60

>>> local_timeout = None
>>> timeout ?? local_timeout ?? global_timeout
300

The operator has higher precedence than the comparison operators ==, >, is, etc., but lower precedence than any bitwise or arithmetic operators. This precedence is chosen for making "default value" expressions intuitive to read and write:

>>> not None ?? True
>>> not (None ?? True) # Same precedence

>>> 1 == None ?? 1
>>> 1 == (None ?? 1) # Same precedence

>>> 'foo' in None ?? ['foo', 'bar']
>>> 'foo' in (None ?? ['foo', 'bar']) # Same precedence

>>> 1 + None ?? 2
>>> 1 + (None ?? 2) # Same precedence

Recall the example above of calculating the cost of items in a shopping cart, and the easy-to-miss bug. This type of bug is not possible with the None- coalescing operator, because there is no implicit type coersion to bool:

>>> requested_quantity = 0
>>> default_quantity = 1
>>> price = 100
>>> requested_quantity ?? default_quantity * price
0

The None-coalescing operator also has a corresponding assignment shortcut. The following assignments are semantically similar, except that foo is only looked up once when using the assignment shortcut:

>>> foo ??= []
>>> foo = foo ?? []

The None coalescing operator improves readability, especially when handling default function arguments. Consider again the example from the Requests library, rewritten to use None-coalescing:

def __init__(self, data=None, files=None, headers=None, params=None, hooks=None):
    self.data = data ?? []
    self.files = files ?? []
    self.headers = headers ?? {}
    self.params = params ?? {}
    self.hooks = hooks ?? {}

The operator makes the intent easier to follow (by putting operands in an intuitive order) and is more concise than the ternary operator, while still preserving the short circuit semantics of the code that it replaces.

None-Aware Attribute Access Operator

The None-aware attribute access operator (also called "safe navigation") checks its left operand. If the left operand is None, then the operator evaluates to None. If the the left operand is not None, then the operator accesses the attribute named by the right operand:

>>> from datetime import date
>>> d = date.today()
>>> d.year
2015

>>> d = None
>>> d.year
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'year'

>>> d?.year
None

The operator has the same precedence and associativity as the plain attribute access operator ., but this operator is also short-circuiting in a unique way: if the left operand is None, then any series of attribute access, index access, slicing, or function call operators immediately to the right of it are not evaluated:

>>> name = ' The Black Knight '
>>> name.strip()[4:].upper()
'BLACK KNIGHT'

>>> name = None
>>> name?.strip()[4:].upper()
None

If this operator did not short circuit in this way, then the second example would partially evaluate name?.strip() to None() and then fail with TypeError: 'NoneType' object is not callable.

To put it another way, the following expressions are semantically similar, except that name is only looked up once on the first line:

>>> name?.strip()[4:].upper()
>>> name.strip()[4:].upper() if name is not None else None

This operator short circuits one or more attribute access, index access, slicing, or function call operators that are adjacent to its right, but it does not short circuit any other operators (logical, bitwise, arithmetic, etc.), nor does it escape parentheses:

>>> d = date.today()
>>> d?.year.numerator + 1
2016

>>> d = None
>>> d?.year.numerator + 1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'

>>> (d?.year).numerator + 1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'numerator'

Note that the error in the second example is not on the attribute access numerator. In fact, that attribute access is never performed. The error occurs when adding None + 1, because the None-aware attribute access does not short circuit +.

The third example fails because the operator does not escape parentheses. In that example, the attribute access numerator is evaluated and fails because None does not have that attribute.

Finally, observe that short circuiting adjacent operators is not at all the same thing as propagating None throughout an expression:

>>> user?.first_name.upper()

If user is not None, then user.first_name is evaluated. If user.first_name evaluates to None, then user.first_name.upper() is an error! In English, this expression says, "user is optional but if it has a value, then it must have a first_name, too."

If first_name is supposed to be optional attribute, then the expression must make that explicit:

>>> user?.first_name?.upper()

The operator is not intended as an error silencing mechanism, and it would be undesirable if its presence infected nearby operators.

None-Aware Index Access/Slicing Operator

The None-aware index access/slicing operator (also called "safe navigation") is nearly identical to the None-aware attribute access operator. It combines the familiar square bracket syntax [] with new punctuation or a new keyword, the spelling of which is discussed later:

>>> person = {'name': 'Mark', 'age': 32}
>>> person['name']
'Mark'

>>> person = None
>>> person['name']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'NoneType' object is not subscriptable

>>> person?.['name']
None

The None-aware slicing operator behaves similarly:

>>> name = 'The Black Knight'
>>> name[4:]
'Black Knight'

>>> name = None
>>> name[4:]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'NoneType' object is not subscriptable

>>> name?.[4:]
None

These operators have the same precedence as the plain index access and slicing operators. They also have the same short-circuiting behavior as the None-aware attribute access.

Generalized Coalescing

Making None a special case is too specialized and magical. The behavior can be generalized by making the None-aware operators invoke a dunder method, e.g. __coalesce__(self) that returns True if an object should be coalesced and False otherwise.

With this generalization, object would implement a dunder method equivalent to this:

def __coalesce__(self):
    return False

NoneType would implement a dunder method equivalent to this:

def __coalesce__(self):
    return True

If this generalization is accepted, then the operators will need to be renamed such that the term None is not used, e.g. "Coalescing Operator", "Coalesced Member Access Operator", etc.

The coalesce operator would invoke this dunder method. The following two expressions are semantically similar, except foo is only looked up once when using the coalesce operator:

>>> foo ?? bar
>>> bar if foo.__coalesce__() else foo

The coalesced attribute and index access operators would invoke the same dunder method:

>>> user?.first_name.upper()
>>> None if user.__coalesce__() else user.first_name.upper()

This generalization allows for domain-specific null objects to be coalesced just like None. For example the pyasn1 package has a type called Null that represents an ASN.1 null:

>>> from pyasn1.type import univ
>>> univ.Null() ?? univ.Integer(123)
Integer(123)

In addition to making the proposed operators less specialized, this generalization also makes it easier to work with the Null Object Pattern, [3] for those developers who prefer to avoid using None.

Implementation

The author of this PEP is not competent with grammars or lexers, and given the contentiousness of this proposal, the implementation details for CPython will be deferred until we have a clearer idea that one or more of the proposed enhancements will be approved.

...TBD...