Python’s type system has union types (including Option), and using them is pretty ergonomic. As TFA points out the problem is that `None` gets contextually reused in all kinds of places where a proper sentinel type is more appropriate.
Pretty sure Python's union types don't nest, though. If you could nest optionals then I feel that reduces the need for separate sentinel types since e.g., you can distinguish between "value not present" (None) and "null value present" (Some(None))
The idea is that optional values are either some value or no value, but sometimes the inner "some value" bit can itself be an optional value (in Rust this would be Option<Option<T>>, in C++ it'd be std::optional<std::optional<T>>, in Python it'd hypothetically be Option[Option[T]], etc.). Such a situation gives you three possible states:
- You have an inner value and the inner value is something non-null (an integer, a string, another optional, etc.)
- You have an inner value and the inner value is null
- You do not have an inner value
The problem, from what I understand, is that Option[T] in Python is equivalent to the union type T | None, and union types are flattened (e.g., (T | U) | V is equivalent to T | U | V) and duplicates are ignored (e.g., T | U | T is equivalent to T | U). As a result, Option[Option[T]] is equivalent to (T | None) | None, which is equivalent to T | None | None, which is finally equivalent to T | None, which means you lose the distinction between "No inner value" and "Inner value is present but is None".
As a more concrete (albeit somewhat contrived) example, say there's some API you can query for the owner of a particular document ID where the document ID may or may not refer to an actual document (e.g., maybe you got it from a cache and the document has since been deleted). One way to express this would be (using Rust syntax)
So if you get None, the document does not exist; if you get Some(None), the document exists but it has no owner; and if you get Some(<id>) the document exists and it has an owner.
(This can probably be done better by creating new enums, but it's an example so give me a bit of slack)
So now you're in the pickle the OP describes: you can end up with a None, but you no longer knonw whether it means "no document" or "no owner". The Python solution is to basically create a different name for either the outer None (e.g., MISSING as in the OP) or the inner one (e.g., NO_OWNER or the like).
I’m curious what that would practically look like in Python.
I bet this would be a good use of vibe coding. Experiment with what-ifs. I can imagine being able to “show not tell” and reduce the signal-to-noise of comment opinions.
When I was writing a bunch of Lua, I was a big fan of having a None sentinel value (Lua uses 'nil' for representing absence) and it was useful for a lot of things similar to Pydantic's MISSING.
This distinction is something that I think is actually handy about JavaScript, and distinguishing between 'null' and 'undefined' is sometimes useful for similar reasons.
it’s maybe less “pythonic” but at $dayjob we’ve very actively moved away from exceptions for things in favor of things that are statically type checked. too many things that become runtime bugs instead of type checking bugs.
with match now being available, we even return explicit error objects and the ergonomics are pretty good (reminds me of rust).
So much trouble to avoid introducing the optional monad.
Python’s type system has union types (including Option), and using them is pretty ergonomic. As TFA points out the problem is that `None` gets contextually reused in all kinds of places where a proper sentinel type is more appropriate.
Pretty sure Python's union types don't nest, though. If you could nest optionals then I feel that reduces the need for separate sentinel types since e.g., you can distinguish between "value not present" (None) and "null value present" (Some(None))
Can you explain or give an example of nested optional types? I don't quite see how that makes sense
Say you select nullable column into vec, Vec<Option<T>>. vec.get returns Option<Option<T>>, which distinguishes null from out of bounds
I'd like nested Nullable in ClickHouse to use CoalescingMergeTree to support postgres unchanged TOAST values easily without getting mixed up on nulls
The idea is that optional values are either some value or no value, but sometimes the inner "some value" bit can itself be an optional value (in Rust this would be Option<Option<T>>, in C++ it'd be std::optional<std::optional<T>>, in Python it'd hypothetically be Option[Option[T]], etc.). Such a situation gives you three possible states:
- You have an inner value and the inner value is something non-null (an integer, a string, another optional, etc.)
- You have an inner value and the inner value is null
- You do not have an inner value
The problem, from what I understand, is that Option[T] in Python is equivalent to the union type T | None, and union types are flattened (e.g., (T | U) | V is equivalent to T | U | V) and duplicates are ignored (e.g., T | U | T is equivalent to T | U). As a result, Option[Option[T]] is equivalent to (T | None) | None, which is equivalent to T | None | None, which is finally equivalent to T | None, which means you lose the distinction between "No inner value" and "Inner value is present but is None".
As a more concrete (albeit somewhat contrived) example, say there's some API you can query for the owner of a particular document ID where the document ID may or may not refer to an actual document (e.g., maybe you got it from a cache and the document has since been deleted). One way to express this would be (using Rust syntax)
So if you get None, the document does not exist; if you get Some(None), the document exists but it has no owner; and if you get Some(<id>) the document exists and it has an owner.(This can probably be done better by creating new enums, but it's an example so give me a bit of slack)
A naive port of this signature to Python would be
But to type checkers this is equivalent to So now you're in the pickle the OP describes: you can end up with a None, but you no longer knonw whether it means "no document" or "no owner". The Python solution is to basically create a different name for either the outer None (e.g., MISSING as in the OP) or the inner one (e.g., NO_OWNER or the like).I’m curious what that would practically look like in Python.
I bet this would be a good use of vibe coding. Experiment with what-ifs. I can imagine being able to “show not tell” and reduce the signal-to-noise of comment opinions.
When I was writing a bunch of Lua, I was a big fan of having a None sentinel value (Lua uses 'nil' for representing absence) and it was useful for a lot of things similar to Pydantic's MISSING.
This distinction is something that I think is actually handy about JavaScript, and distinguishing between 'null' and 'undefined' is sometimes useful for similar reasons.
I like it. I can't see how to do better while remaining compatible with Python idioms.
You could even say you never knew what you were… MISSING ;-)
"A simple cache returns None both for a missing key and for a key that maps to None."
No, it returns KeyError. Or at least it should.
The author’s example chose to use the syntax that returns None on missing. If they wanted clarity the standard API does raise.
Vs There are better examples that could have been highlighted.it’s maybe less “pythonic” but at $dayjob we’ve very actively moved away from exceptions for things in favor of things that are statically type checked. too many things that become runtime bugs instead of type checking bugs.
with match now being available, we even return explicit error objects and the ergonomics are pretty good (reminds me of rust).
Perhaps the author has had their fair share of run-ins with libraries that didn't think their API through like that. But yes, they should.