Python errors as values: Comparing useful patterns from Rust and Go
inngest.comThe most important thing about writing code is that you write it idiomatically for the language it is in. With Python, idiomatic code is known as Pythonic.
def rename_user(user_id: str, name: str) -> User | Exception:
# Consume the function
user = get_user(user_id)
if isinstance(user, Exception):
return user
user.name = name
return user
Perspective: I've written primarily in Python for over 20 years, as well as coded extensively in many other languages (C, Objective-C, JavaScript, Java, Bash) and have familiarity with a bunch more (Go, Ruby, Gradle, TCL, Lua, Kotlin, C++).
In practice, exception handling, declared exceptions or not, just isn't that big of a problem. When I'm writing code, I have to reason about it, and exceptions and return/error values are just one aspect of that reasoning.
Errors/Exceptions by value isn't that important in isolation, but it makes functional programming styles work a lot better (monads).
I think there's still an argument here that this is just a symptom of trying to force functional paradigms on python, which is happening more and more, which I like but am sometimes worried about how pythonic it is... For a one odd script it works well, but when I want to maintain something and have to manage many dependencies and onboard new devs it becomes quite challenging.
> What if the program receives an interrupt signal during the get_user call?
You shouldn't catch interrupt signals regardless of whether you're throwing or returning errors. This is why your error classes should extend Exception and not BaseException. The interrupt errors (e.g. KeyboardInterrupt) extend BaseException
When writing multi-threaded code, you might want to catch KeyboardInterrupt to cleanly shutdown worker threads. In any case, my point is that you always need to be aware of exceptions in Python. You can't just return them as values and pretend they don't otherwise exist.
This topic is explicitly about considering alternatives to the current Pythonic way.
The topic is explicitly about writing an SDK for the company's product, which is presumably intended to be imported in customers' code and used alongside hundreds of other libraries.
As such, writing idiomatic code for the language of the SDK really should be a concern, rather than implementing some idiosyncratic half-baked version of Go or Rust error handling in Python.
And GP is pointing out (correctly) that most of the time being idiomatic has more net gain than any putative advantage of the non-idiomatic proposal. Exceptions to this are extremely rare.
lol the most important thing about writing code is that it works, and works well enough that it puts shekels in my pocket. I'll start worrying about Pythonic code when my accountant does. Until then, we'll keep counting shekels.
yeah, this guy gets it. idiomatic-ness is based on p̵o̵p̵u̵l̵a̵r̵i̵t̵y̵ consensus and only indirectly by technical merit. if someone throws shade at code for being nonpythonic, the code probably isn't bad. if it was they'd have focused on that first.
on the other hand, if someone's writing non-pythonic python because pythonic code doesn't work well for their program, it might be a program they shouldn't be writing in python, which in my experience happens way more often than it should.
Why does the author want to swallow these exceptions? Let them propagate and it's obvious where the issue lies. If you can't handle an exception, don't catch it.
> It's impossible to know which line might throw an error without reading the functions themselves... and the functions those functions call... and the functions those functions call. Some thorough engineers may document thrown errors but documentation is untested and therefore untrustworthy. Java is a little better because it forces you to declare uncaught errors in method signatures.
The author's proposal doesn't change this as much as they think it does. You still don't know what type of errors a function will throw without inspecting the code and thus how to resolve them. Unless, you have a blanket switch for every possible error anything could return which is the very thing they are complaining about.
Author here!
> Why does the author want to swallow these exceptions?
Sometimes you want to swallow exceptions and sometimes you don't. The examples in the article may be a little contrived, but there are situations where logging an error and continuing is better because it prevents data loss.
> Let them propagate and it's obvious where the issue lies.
If they propagate down the stack then you lose context. You also may not have all the data you need to properly recover (e.g. still write to a table but make the errored field null).
> You still don't know what type of errors a function will throw without inspecting the code and thus how to resolve them.
You're right that we don't know which error occurred with the approach in the article, but you at least know that there could be an error. This is better than try/catch because that doesn't tell you whether an error could happen
> Sometimes you want to swallow exceptions and sometimes you don't.
You capture `Exception' in your examples which is a serious anti-pattern in nearly all cases. One of the rare places you'd want to do that is at the top level of a service that has to clean up safely before bailing out. None of the contrivances in your example demonstrate that requirement.
If you need to capture an exception, you capture the narrowest one that matches the business/technical requirement you have. The rest should bubble up and get handled by another part of your code or, indeed, cause your app to crash if it's unhandled.
Exceptions don't mean "error" either; they're just exceptions.
> If they propagate down the stack then you lose context
I cannot think of any instances where you 'lose context' by propagating them. You should raise Foo from e if you're re-raising, though, which you clearly did not bother doing. That does preserve context.
Passing an instance of an Exception object around? What is this madness... are you building a debugger? No? A complex framework or tool like pytest where you may want to manipulate the stack frames to benefit the end user? No? Then don't pass exception objects around and raise them somewhere else.
> his is better than try/catch because that doesn't tell you whether an error could happen
It's really not that simple. Don't forget that the concept of exceptions (and more powerful things, like CL's condition system) were invented because of the problems encountered with errors-as-value approaches.
It's a fundamental trade off without a right answer, and people have been arguing about it now for 50+ years without resolution.
> If they propagate down the stack then you lose context.
Maybe I’m misunderstanding you, but that’s what the Python stack traceback is for. It works pretty well, I prefer it over JS.
> If they propagate down the stack then you lose context. You also may not have all the data you need to properly recover (e.g. still write to a table but make the errored field null).
Is this not solved by using raise from?
By "context" I mean other data that lived at the time of the error. For example, you may want to update a table row when there's an error but if a thrown error takes you too far down the stack then you might not know the row ID anymore
Then you have built your program incorrectly and not caught the error in a place where you still have context (e. g. `YourProgramException` sub-type that indicates a recoverable error or `Exception` in the case that you're building something where it doesn't matter what the error is, just that an error occurred).
This can also be done with error passing, and is a design failure there too:
_, ex = perform_batch_operation(on: list_of_data) if ex: # Oh no, we don't know which entry in the list failed # and can't update the appropriate row # (This, of course, should be handled in # perform_batch_operation, not bubbled up here)you would have that preserved as a variable in the context of the error no?
One problem I’ve experienced doing something like this is you end up with both exceptions and error values since the standard library and 3rd party libraries are still primarily exception based. You either have to live with it or create wrappers that catch errors and return them as values.
> You either have to live with it or create wrappers that catch errors and return them as values.
Some of us used to wrap php "errors" to convert them into exceptions. Then I switched to Python and was pleased to see the pointless distinction between errors and exceptions gone... not gonna go backwards on this.
> Some of us used to wrap php "errors" to convert them into exceptions.
Did exactly that back in the day, this comparison is flawed: PHP errors (and warnings, notices) by default went completely outside the flow of the program, and converting them to exceptions was a means of forcing developers to deck with them right away, sometimes at all.
We write wrappers for standard and third-party libraries that throw errors. For example, we have our own dump_json function that catches-and-returns errors thrown by json.dumps.
We didn't need many wrappers given the nature of our SDK, but some programs will need many wrappers and that could get unwieldy
But people are going to import your SDK, so your customers will be stuck with a mix of thrown exceptions and errors-as-values in their code.
Based on your decision to write wrappers internally, presumably you'd argue that the sensible thing for them to do is to write wrappers either for your SDK or for every other library to get back to a consistent style and less mental overhead.
Author here!
We don't return errors in our public methods so our SDK still feels Pythonic. We're debating whether to add `_safe` suffix methods that return errors to give people the option, but for now our library only throws errors to consumers.
You might ask "well what's the point if you still throw errors to consumers?" We feel that forcing our engineers to deal with errors where they can happen is worthwhile. Our team writes a lot of Go and we love how it forces you to think about every spot that can error, so we wanted the same experience in Python
OK, in that case I think it's a very reasonable choice! Perhaps it's worth clarifying that in the article.
I might express a slight preference that your internal code is idiomatic if I'm going to be poking around in there at some point. But I wouldn't be bikeshedding about your C coding style if you'd provided a Python module written in C, so I don't feel I should have a strong opinion as a user here.
Although I liked the proposal (the `-> User | Exception` idiom is quite beautiful), that's a deal breaker.
> Rust returns returns errors using a "wrapper" type called Result. A Result contains both a non-error value (Ok) and an error value (Err)
A `Result` can contain either a non-error value (Result::Ok) or and error value (Result::Err), never both.
Oops! I'll make that correction later today. You're right: they're mutually exclusive
That's what makes it good :) Well, that and the fact you have to unpack/destructure the result somehow to get at the value (or the error), so you're forced to handle it.
"How we made a non-idiomatic Python SDK for our app that Python devs will hate"
Probably true. Without commenting on the particulars of the SDK in question (read: I haven't read TFA): Monadic code is obviously better than the mainstream alternatives. Does this point towards eventual inviability of Python itself?
Oh, good, heavyweight error handling just in time for py3.11's zero-cost exception happy path.
But, more generously: why not simply return an error, and use isinstance(val, Error) for error handling? Making objects and calling functions is quite costly, and that can largely be avoided.
> Oh, good, heavyweight error handling just in time for py3.11's zero-cost exception happy path.
I don’t get it. Languages that use exceptions for all kinds of errors will also use exceptions for routine errors that happen as a matter of course—the happy path is not the overwhelmingly most common branch, and errors are not exceptional. In turn not zero-cost for all but the exceptional case.
The solution to this is to not use exceptions except for actual errors, unless it's going to be amortized away. For example, objects which have a heavy parameter, which gets memoized. With the zero-cost happy path, you only raise the exception once, when you're doing the costly thing, and subsequent accesses are free.
Sure, it depends on what is expensive. For us, in a primarily asynchronous domain, saving cycles is helpful for lower computational costs and the environment, but in the main, we optimise for reading the code, and one way to lower the cognitive load is to avoid using errors to control flow.
This is what they ended the article on. Return a union type and then error check using isinstance.
Oh, so they did. I guess I got bored after reading so many bad approaches trying to write go/rust in python.
I don't understand why people insist that all errors must be handled all the time. It could be my C++ background, but I feel there are two very different errors:
Expected errors - like "user not found" - should use a value instead of exception. In Python, you can use sentinel objects, or tuples, or None.. lots of options really. Occasionally there is a good reason to use exceptions for flow control even for know errors (various hooks come to mind), but this should be pretty rare compared to number of places that can raise an unexpected errors.
Unexpected errors should not be caught at all, except maybe at the very top level (to record them and return 500 to user). The examples in post, where you catch exception and re-raise Exception back are terrible - what's the point of them? There is no extra clarity, just verbosity. I would defect them in any code review.
Coarse-grained error handling is great as long as exceptions are meaningful and stack traces are good, which is the usual case in python. All that matters for unexpected errors is that (1) user sees an error message and (2) the real cause is recorded for later analysis. A single top-level try block does both.
> Expected errors - like "user not found" - should use a value instead of exception. In Python, you can use sentinel objects, or tuples, or None.. lots of options really.
That's why exceptions are called exceptions, not errors. If a routine called openFile() can't open a file, that's a pretty exceptional situation, and it's up to the caller to decide whether the exact reason is an error in their case.
The exception object is already a value that can have not only a message text, but also any data members, so why reinvent the wheel with sentinel objects, tuples, etc.?
> The examples in post, where you catch exception and re-raise Exception back are terrible - what's the point of them? There is no extra clarity, just verbosity. I would defect them in any code review.
Typically, you re-raise an exception after adding some content to it. This may be less important in Python, which gives you a great stack trace, but in a language like C++, the lack of context information makes the exception basically useless.
Not necessarily advocating for unidiomatic python/code, but you could use a decorator to automatically wrap the function call with a try-catch and package the return value appropriately. Lot less mangling of function bodies that way, just return and raise exceptions like normal. You *could* specify the expected Exception type, but considering the rest of the ecosystem probably won’t be following along with documenting expected exceptions, I assume it wouldn’t be worth it and would be more straightforward that all the exception types in the signatures be the plain vanilla Exception. Would also be super inefficient pre-3.11 but ¯\_(ツ)_/¯
This doesn’t do much. Many of the stdlib functions will still raise exceptions. Also, just because you’re returning exception from a function doesn’t guarantee that something else won’t raise an error. This isn’t an issue in Go/Rust since errors are treated as values in the core language and you’re forced to treat them so everywhere.
I don't agree with this post.
Engineering is about tradeoffs.
There is more advantage in doing the accepted python solution (exceptions) than inventing your own (which you claim to be better, but I personally think is worse). If you are developing in a team, stick to established conventions and spend your time focusing on your business problem.
Quite apart from the Python discussion, the author captures why I prefer errors as values (a la Go) to exceptions (a la Java), and I have written both styles for many years.
> Regardless of the specific approach, returning errors as values makes us consider all of the places an error could occur. The error scenarios become self-documenting and more thoroughly handled.
This is so true. Most Java exception handling is a try/catch around about 20 lines of code, with superficial logging/rethrowing and no context about exactly what was being done to what when the exception occurred, just a filename/line# and a probably cryptic error message.
In Go, best practice is something like:
bytes, err := os.ReadFile("myfile.json")
if err != nil {
return nil, fmt.Errorf("reading file %s: %v", "myfile.json", err)
}
var data map[string]any
err = json.Unmarshal(bytes, &data)
if err != nil {
return fmt.Errorf("unpacking json from file %s: %v", "myfile.json", err)
}
If you are going to replicate this with exceptions, it would require much more boilerplate, as his example demonstrates, which is ironic given that is the charge levelled at Go.
I think it's kinda the opposite actually... with go, you have to go out of the way to generate those errors, while in Python, most of this is automatic.
In particular, the ReadFile example in Python will raise OSError, and those already include filename and error message. So you'd get the same result with 0 extra lines.
For the second example, the json unmarshalling will not auto-add filename, but its easy enough to do using exception chaining:
which will give stacktrace like:try: data = json.loads(bytes) process_1(data) process_2(data) except: raise Exception(f"Error while parsing file {filename}")
Note that's even more detailed than go, with significantly less boilerplate (only 3 lines per function) vs 3 lines per call.json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "somefile.py", line 4, in somefile Exception: Failed while parsing file.json(an anecdote: we've had the team which converted the CLI tool from Python to Golang. The first time they ran the tool, it printed a single line:
and that's it. It took a lot of debugging before they could figure out what happened. All because they got used to python doing super-rich traceback automatically, with 0 effort from programmers)invalid character '"' after top-level valueThis is fine as far as it goes. The problem is that the exception can only report context that it knows about. By contrast, the Go version allows you to include any extra context you want, to make debugging easier. Eg, your error might not just include file reading or Json parsing, but which transaction or customer was involved at the time. You can do with exceptions but you either have to try/catch every statement, or add generic context in one big catch block.
I accept it's largely personal preference, but having used both mechanisms for many years, I find Go best practices for error handling are simple and easy to follow, and results in easily maintainable code, compared to exception handling which doesn't really come with a simple set of best practices, meaning it is often badly put together or added as an afterthought.
maybe there are some best error practices out there, but people don't follow that. As an example, I just went to github.com, got the top-trending go project ("ko"], search for error and arrived at this line in [0]:
See how they forgot to put filename in the error message? if there is some sort of error with the file, you'll have to resort to strace to find out what the name is... Not to mention that this very function returns json parse errors without context, and the caller "getMeta" calls multiple functions and returns the errors without context as well...dtodf, err := os.Open(filepath.Join(filepath.Dir(file), "diffid-to-descriptor")) if err != nil { return nil, fmt.Errorf("opening diffid-to-descriptor: %w", err) }best practices are nice, but fully automatic is even better. The minimum-effort path in Python produced vastly more useful traces than in Go, and unfortunately too many programmers go mininum-effort path.
[0] https://github.com/ko-build/ko/blob/cfc13deeb6417d7e1582f031...
Exceptions are a very useful tool. In your example, the main program logic is buried in error handling which makes it more difficult to read the code and the code becomes more complex, leading to more bugs. In many cases, it's preferable to have a single error handler centralized in one place, out of line of the main logic. This makes the code more readable, reduces complexity and duplication.
I'm six months into my Python journey. We aren't building a library, so everything runs on 3.11. Having spent most of my career in statically typed and sometimes functional languages, I've found the result package approach and pattern-matching suggestion work well. There's been a suggestion it's not very Pythonic, but I'm willing to continue using a result monad because the trade-off is one-sided; it comfortably pays for itself.
What is a result monad?
It just lets you combine results in an intuitive way:
Ok(2) + Ok(3) = Ok(5) Ok(2) + Err() = Err() Err() + Ok(3) = Err()The result package in the article: https://github.com/rustedpy/result
There's a pretty decent explanation in the readme. If you're more interested in Monads, I am not sure I'd cover that in a HN comment very well, but I would encourage you to take a look.
It is linguistically related to the (plus, int) monoid; the fact that it conforms to some algebraic/categorical interface is irrelevant in most contexts and just a distraction.
(A “Result” is Rust-speak for a sum type which is either A or B, where (say) B is conventionally an error)
Let python be python, let Go be Go, let Rust be Rust it has never worked well for any language trying to be like another language
In FileMonger[0], which uses Tauri, I have implemented a `Result` logic for errors in TS, similar to what's available in Rust. It's clunkier, but still much preferable to the mess of throwing. Much easier to handle errors and debug.
I don't know Python too well but I think this is covered in TFA. There's a comparison between a Result type using https://github.com/rustedpy/result and an ADT/sum type — and the sum type looks like it's way more ergonomic with Python's current typechecker.
While I like Rust — and Result works really well with `?` — it doesn't actually look like that's the best pattern for Python?
Wanna opensource that part as a library, or at least write a blogpost?
I have written a rudimentary implementation to avoid external dependencies. If you don't mind, a quick google comes up with a couple:
> So if we want to be really safe then we'll wrap each call with a try/catch:
try:
thing.set_name("Doodad")
except Exception as err:
raise Exception(f"failed to set name: {err}") from err
How is this safer than the original?
If the caller wasn't catching exceptions thrown by this function before, it's not catching exceptions thrown by this function now. What is being gained by catching an exception to do nothing but throw another exception?
This feels like a strawman.
Nope, don't like it. It's also a little disingenuous to switch the example halfway through. Anyway, I fixed your code, with exceptions
def get_user(user_id: str) -> User:
rows = users.find(user_id=user_id)
if not rows:
raise Exception("user not found")
return rows[0]
def rename_user(user_id: str, name: str) -> User:
user = get_user(user_id)
user.name = name
return user, Nonepython is where I learned to hate exceptions as control flow - Twisted is twisted. Go was such a breath of fresh air. Now I'm back in python primarily and I am constantly wondering what my functions actually take and actually return. Exceptions are just spooky GOTO and a distance. Our logs are littered with them and have to use Sentry to tell us "oops, you introduced a new error path." Our builds are full of warnings and versioning issues. I've yet to see a decent sized python project that is not a mess.
> Exceptions are just spooky GOTO and a distance
I couldn't agree more! When you throw exceptions it's unclear where the control flow will go
I just don't see how multiple levels of
in the call stack makes things any clearer? Or how having many instances of that snippet scattered everywhere makes the code easier to read?if err: return nil, errAgreed. It's also objectively worse because you lose the stack trace in the final exception you will display to your user. Enjoy debugging that when a customer sends you an error.
Or you could use a 3rd party error package that’s not a good excuse because the standard don’t have it as a priority
It makes the happy path harder to read but the unhappy paths much clearer. I hated this aspect of Go when I first started with the language but I love it now
I love it. Recently been rewriting Goleko.com backend in Go from Python. The errors as value paradigm is so nice for 90% of the time. It is so much more robust code out of the bat.
> Goleko.com
I won’t claim to know what your intent was but to me the brand placement changed your post from a comment to an ad.
> tuple[User | None, Exception | None]
tuple[User, None] | tuple[None, Exception]
That type won't help if you're unpacking like `user, err = get_user(user_id)`. Both values will be nullable and the type-checker won't understand that `user` is not None if `err` is None
Yup, don't allow nonsensical states (User+Exception or None+None) to even exist.
The original is more like Go approach which is a big flaw with the language.
Go can be `tuple[User | None, Exception | None]` or `tuple[User, Exception | None]`, depending on whether you're returning a pointer. But yea, Go's approach has its warts. Like if you aren't returning a pointer then you need to return the zero value (e.g. `User{}`) even when returning an error
Yeah but pointer + nil also have issues: https://go.dev/doc/faq#nil_error
I've also seen APIs that allow returning both error and a value ("something went wrong, but here is a fallback or something") which are of course extremely confusing given the usual style.
Oh I agree, I've run into too many nil pointer panics. I wish Go had a first-class way of expressing optionality, rather than using pointers
As for helping the type checker, try typing’s TypeGuard. They help the type checker reason about situations such as “if this is None, this other thing is that”.