I have been hard at work at typing poezio lately, here is both a status check and some of the lessons learnt in the process.
Poezio the XMPP terminal client written in python (mostly) that I co-maintain, has turned 16 this year. At now more than 4500 commits, it is a somewhat mature piece of software, with a lot of cruft intertwined. Written initially in python 2, it moved quickly to python3 (too early? at least it was before the unicode string handling was significantly tweaked to be fast).
Its roots then very obviously pre-date the introduction of any kind of type checking, annotations, as well as a lot of modern tooling. It has both grown very organically on a feature-by-feature basis, and has at times also taken advantage from a lot of the liberties given by python with regards to dynamic return types, decorators and the likes.
Starting from 2018, type hints have been sprinkled through the codebase, in a non-systematic manner. A mypy CI job was added a while later. But while running bare mypy is a good way to detect errors, in practice the gradual nature of its purported usage makes it less efficient than it could be. For improved efficiency and error detection, mypy takes a --strict flags which makes it much less tolerant.
Additionally, since many of the errors returned by --strict at first are along the line of you need to type this function!!, there are actually many more errors hidden ready to appear once you have typed the arguments and return values!
When I started the current journey, mypy --strict returned around 1800 errors, sprinkled throughout more than 70 files. At the time of writing this, we are under 250 errors, and a very large majority of functions and classes are typed, which means errors should not be going up by a lot while progressing. In fact, many of the errors are now from slixmpp calls which are missing type hints; that will be my next goal.
On the technical side, poezio moved to asyncio relatively early (at the times of yield from, when async and await were not even there yet), and had to play nice with a lot of functions that could either be normal functions or asynchronous functions. Using asyncio.iscoroutinefunction (now moved to the inspect module) inside all decorators and places that called previously registered functions is something we had to do to adapt. We even ship some dirty hacks on top of asyncio to make sure the application can stay responsive.
Async fun
Back to our decorator friends, now consider the following:
def before(args: list[Any], kwargs: dict[str, Any]): pass def wrap_before(func): def wrap(*args, **kwargs) before(args, kwargs) return func(*args, **kwargs) return wrap
This is a simple decorator that wraps a function and adds code to run right before executing it.
Typing it if you assume the before function works defensively around its parameters is somewhat easy, thanks to PEP 612 ands ParamSpec, as well as TypeVar which has been here from the start (PEP 484):
P = ParamSpec('P') T = TypeVar('T') def wrap_before(func: Callable[P, T]) -> Callable[P, T]: def wrap(*args: P.args, **kwargs: P.kwargs) -> T: before(args, kwargs) return func(*args, **kwargs) return wrap
Now if we introduce asyncio genericity, assuming the calling code does not — and needs not – check the function coroutine-ness using iscoroutinefunction, this can work as-is (T in the async case will be an Awaitable, and that tracks).
But if we want to keep the semantic value, we need to specialize the wrapper:
def wrap_before(func): def wrap(*args, **kwargs): before(args, kwargs) return func(*args, **kwargs) async def awrap(*args, **kwargs): before(args, kwargs) return await func(*args, **kwargs) if iscoroutinefunction(func): return awrap return wrap
Naïvely, one could assume it does not change things too much and the following could work:
P = ParamSpec('P') T = TypeVar('T') def wrap_before(func: Callable[P, T]) -> Callable[P, T]: def wrap(*args: P.args, **kwargs: P.kwargs) -> T: before(args, kwargs) return func(*args, **kwargs) async def awrap(*args: P.args, **kwargs: P.kwargs) -> T: before(args, kwargs) return await func(*args, **kwargs) if iscoroutinefunction(func): return awrap return wrap
But that would be wrong! In this case since we define the function as async, this has the sad effect of defining the return value of this specific callable as Awaitable[T] instead. Type-wise, it no longer makes sense, and it needs a specific solution.
Enter @overload
Using overload means giving more info to the type checker than it can gather on its own, so that it knows which branches make sense when checking.
The examples I can find in the docs do not put any annotations on the real version of the function, but sadly mypy is not happy with that and I need to union-ize everything in the final declaration to make it work. For inner functions I need to have the ParamSpec and TypeVars bound anyway.
P = ParamSpec('P') T = ParamSpec('T') U = ParamSpec('U') @overload def wrap_before(func: Callable[P, T]) -> Callable[P, T]: ... @overload def wrap_before(func: Callable[P, Awaitable[U]]) -> Callable[P, Awaitable[U]]: ... def wrap_before(func: Callable[P, T] | Callable[P, Awaitable[U]]) -> Callable[P, T] | Callable[P, Awaitable[U]]: if iscoroutinefunction(func): async def awrap(*args: P.args, **kwargs: P.kwargs) -> U: before(args, kwargs) return await func(*args, **kwargs) return awrap else: def wrap(*args: P.args, **kwargs: P.kwargs) -> T: before(args, kwargs) return func(*args, **kwargs) return wrap
Sadly, mypy is not smart enough to understand the type information conveyed by iscoroutinefunction, which means I need to cast the return types to make sure everything goes well:
P = ParamSpec('P') T = ParamSpec('T') U = ParamSpec('U') @overload def wrap_before(func: Callable[P, T]) -> Callable[P, T]: ... @overload def wrap_before(func: Callable[P, Awaitable[U]]) -> Callable[P, Awaitable[U]]: ... def wrap_before(func: Callable[P, T] | Callable[P, Awaitable[U]]) -> Callable[P, T] | Callable[P, Awaitable[U]]: if iscoroutinefunction(func): async def awrap(*args: P.args, **kwargs: P.kwargs) -> U: before(args, kwargs) return cast(U, await func(*args, **kwargs)) return awrap else: def wrap(*args: P.args, **kwargs: P.kwargs) -> T: before(args, kwargs) return cast(T, func(*args, **kwargs)) return wrap
Parameter fun
For another interesting use of decorators, one of the thinks that can be useful to do is to replace or modify arguments of the caller. Let's say you have a standard function, representing a command like /kick in an IM client, and you want that when you type /kick gérald "Do not talk to me" and you want to use a decorator to parse that string so that your function already receives a list of strings (['gérald', 'Do not talk to me'], instead of the raw string. If the number of arguments is not enough, provide an empty list instead.
def parse_args(args: str) -> list[str]: """Assume this function parses and splits the string""" def quoted(mandatory_args: int): def wrap_quoted(func): def inner(*args, **kwargs): list_args = list(args) parsed = parse_args(args[0]) if len(parsed) < mandatory_args: list_args[0] = [] else: list_args[0] = parsed return func(*list_args, **kwargs) return inner return wrap_quoted @quoted(2) def my_command(args: list[str]): return args assert my_command('"first param" second') == ['first param', 'second']
Concatenate
Thanks to PEP 612 again, we have access to Concatenate which allows destructuring positional function arguments together with ParamSpec.
What we cannot do, however, is be as generic as before while having a useful type, so we need to specialize.
P = ParamSpec('P') T = TypeVar('T') def quoted(mandatory_args: int): def wrap_quoted(func: Callable[Concatenate[list[str], P], T]) -> Callable[Concatenate[str, P], T]: def inner(str_args: str, *args: P.args, **kwargs: P.kwargs) -> T: parsed = parse_args(str_args) if len(parsed) < mandatory_args: parsed = [] return func(parsed, *args, **kwargs) return inner return wrap_quoted
In many cases we decorate functions that are actually instance methods, so we need to keep some room for the self parameter they take:
P = ParamSpec('P') T = TypeVar('T') T = TypeVar('U') def quoted(mandatory_args: int): def wrap_quoted(func: Callable[Concatenate[U, list[str], P], T]) -> Callable[Concatenate[U, str, P], T]: def inner(self: U, str_args: str, *args: P.args, **kwargs: P.kwargs) -> T: parsed = parse_args(str_args) if len(parsed) < mandatory_args: parsed = [] return func(self, parsed, *args, **kwargs) return inner return wrap_quoted
Moreover, that is not all: Concatenate cannot by design operate on parameters that can be provided by keyword-only, so you need to delimit them with a / to make sure they are positional-only:
P = ParamSpec('P') T = TypeVar('T') T = TypeVar('U') def quoted(mandatory_args: int): def wrap_quoted(func: Callable[Concatenate[U, list[str], P], T]) -> Callable[Concatenate[U, str, P], T]: def inner(self: U, str_args: str, /, *args: P.args, **kwargs: P.kwargs) -> T: parsed = parse_args(str_args) if len(parsed) < mandatory_args: parsed = [] return func(self, parsed, *args, **kwargs) return inner return wrap_quoted
The wrapped functions must also be modified to accomodate for this.
I have been fortunate enough that I have not needed to manipulate kwargs too much in there, but if that is needed there is quite a bit of wizardry available in PEP 692 and use of Unpack together with TypedDict, starting from Python 3.12.
Finishing words
My conclusion for now is not that the python type system is lacking too much in expressivity; the main issues stem from the fact that you can really do whatever you want with standard python and nothing is stopping you. Until you want to add some type checking, that is.
Additionnally, needing to support old python versions is a huge PITA, if only because annotations are evaluated eagerly, requiring quotes around lazy imports and more importantly requiring quotes over the whole type expressions (e.g. MyType | None must be "MyType | None" and very much not "MyType" | None. It makes sense because there was no choice to do differently, while being counter-intuitive.
If you have remarks or suggestions concerning this article, please by all means contact me.