What's a sentinel, and why do I need one?
The PEP 661 abstract summarizes it best:
Unique placeholder values, widely known as "sentinel values", are useful in Python programs for several things, such as default values for function arguments where None is a valid input value.
The simplest use case I can think of is a function that returns a default value only if explicitly provided, otherwise raises an exception.
The next() built-in function is a good example:
Retrieve the next item from the iterator by calling its
__next__()method. If default is given, it is returned if the iterator is exhausted, otherwise StopIteration is raised.
Given this definition, let's try to re-implement it.
next() essentially has two signatures3:
next(iterator)-> item or raise exception
next(iterator, default)-> item or default
There are two main ways to write a function that supports both:
next(*args, **kwargs); you have to extract iterator and default from args and kwargs, and raise TypeError if there are too many / too few / unexpected arguments
next(iterator, default=None); Python checks the arguments, you just need to check if
default is None
To me, the second seems easier to implement than the first.
But the second version has a problem:
for some users,
None is a valid default –
next() distinguish between
None and default-value-
In your own code,
you may be able to guarantee
None is never a valid value,
making this a non-issue.
In a library, however, you don't want to restrict users in this way, since you usually can't foresee all their use cases. Even if you did choose to restrict valid values like this, you'd have to document it, and the users would have to learn about it, and always remember the exception.4
Here's where a private, internal-use only sentinel object helps:
>>> it = iter() >>> print(next(it, None)) 1 >>> print(next(it, None)) None >>> print(next(it)) Traceback (most recent call last): ... StopIteration
Now, next() knows that
default=_missing means raise exception,
default=None is just a regular default value to be returned.
You can think of _missing as of another None, for when the actual None is already taken – a "higher-order" None. Because it's private to the module, users can never (accidentally) use it as a default value, and never have know about it.
For a more in-depth explanation of sentinel objects and related patterns, see The Sentinel Object Pattern by Brandon Rhodes.
Real world examples
The real next() doesn't actually use sentinel values, because it's implemented in C, and things are sometimes different there.
But there are plenty of examples in pure-Python code:
The dataclasses module has two.
The docs even explain what a sentinel is:
[...] the MISSING value is a sentinel object used to detect if the default and default_factory parameters are provided. This sentinel is used because None is a valid value for default. No code should directly use the MISSING value.
(The other one is used in the __init__ of the generated classes to show a default value comes from a factory.)
Werkzeug has one.
I have one in my feed reader library (originally stolen from Werkzeug). I use it for methods like
get_feed(feed[, default]), which either raises FeedNotFoundError or returns default.
I mentioned before sentinels are private; that's not always the case.
If the sentinel is the default argument of a public method or function, it may be a good idea to expose / document it, to facilitate inheritance and function wrappers.5 attrs is a good example of this.
(If you don't expose it, people can still extend your code by using their own sentinel, and then calling either form of your function.)
What's this got to do with typing?
Let's try to add type hints to our hand-rolled next():
isinstance() thing at the end is why a plain
doesn't work – you can't (easily) get Mypy to treat your own
"constants" the way it does a built-in constant like None,
and the sentinel doesn't have a distinct type.
Also, if you use the
MissingType = object version, Mypy complains:
next.py:37: error: Overloaded function implementation cannot produce return type of signature 2
If you're wondering if the good version actually worked, here's what Mypy says:
What's with PEP 661?
There are many sentinel implementations out there; there are 15 different ones in the standard library alone.
Many of them have at least one of these issues:
- non-descriptive / too long repr() (e.g.
<object object at 0x7f99a355fc20>)
- don't pickle correctly (e.g. after unpickling you get a different, new object)
- don't work well with typing
Thus, PEP 661 "suggests adding a utility for defining sentinel values, to be used in the stdlib and made publicly available as part of the stdlib". It looks like this:
>>> NotGiven = sentinel('NotGiven') >>> NotGiven <NotGiven> >>> MISSING = sentinel('MISSING', repr='mymodule.MISSING') >>> MISSING mymodule.MISSING
This utility would address all the known issues, saving developers (mostly, stdlib and third party library authors) from reinventing the wheel (again).
How does this affect me?
Not at all.
If the PEP gets accepted and implemented, you'll be able to create an issue-free sentinel with one line of code.
Of course, you can keep using your own sentinel objects if you want to; the PEP doesn't even propose to change the existing sentinels in the standard library.
Is this worth a PEP?
PEPs exist to support discussions in cases where the "correct" way to go isn't obvious, consensus or coordination are required, or the changes have a big blast radius. A lot of PEPs get abandoned or rejected (that's fine, it's how the process is supposed to work).
PEP 661 seems to fall under the "requires consensus" category; it follows a community poll where although the top pick was "do nothing", most voters went for "do something" (but with no clear agreement on what that should be).
The poll introduction states:
This is a minor detail, so ISTM most important that we reach a reasonable decision quickly, even if that decision is that nothing should be done.
It's worth remembering that doing nothing is always an option. :)
That's all I have for now.
If you learned something new, please consider sharing this, it really helps! :)
The PEP is still in draft status as of 2021-06-10. [return]
The same applies to using some other "common" value, for example, a
For immutable values like strings, it's probably worse. Because of optimizations like interning, strings constructed at different times may actually result in the same object. The data model specifically allows for this to happen (emphasis mine):
Types affect almost all aspects of object behavior. Even the importance of object identity is affected in some sense: for immutable types, operations that compute new values may actually return a reference to any existing object with the same type and value, while for mutable objects this is not allowed.[return]