DEV Community

Python exceptions considered an anti-pattern

Nikita Sobolev on February 11, 2019

Originally published in my blog: https://sobolevn.me/2019/01/simple-dependent-types-in-python What are exceptions? Judging by their name it is an ...

Read full post

Jonathan Higgs • Feb 11 '19

I'm torn by this. While I do like the railway programming model a lot, and the the idea of returning additional information makes a lot of sense in cases of potential failure (C# for example has bool TryGet<TKey, TValue>(TKey key, out TValue value) to deal with lookup failure without exceptions), I feel that Exceptions are highly useful and don't really suffer from the issues you assign to them

They aren't difficult to notice, when they happen I get a big stack trace dump and my IDE helpfully breaks when they are thrown. I'd say they are much more difficult to know ahead of time, but the cause of this is that python is impossible to statically analyse, it is a language problem and not an Exception problem. Other languages have tried noting functions are noexcep or listing all the possible exceptions that might come from a function call, but ultimately those either didn't solve the know-ability problem or created too much overhead to be useful.

Program flow is as unclear as without using them. There is always the issue of not knowing how a function is called, and therefore where the program might continue on an error. The only defense we have against that is to write clear and specific code, SOLID etc.

Comparisons to goto are extremely unjustified. It is just as easy to say that function calls, and even if statements are just structured gotos. The exact same reasoning could be applied to those but we still use them because it would be impossible to do anything without them. Since a raw goto can make it very difficult to follow a program, we have specific sub-cases where they make sense and restrict their use as much as possible to those sub-cases. It is absolutely possible to make a program difficult to follow with misuse of exceptions, it is equally possible to make it difficult to follow with poorly designed functions

Exceptions are a control-flow mechanism. Not all functions can produce a result when the accept variable input. Sometimes remote resources aren't available. Sometimes cosmic rays hit a register and flip a bit leading to undesired behavior. The places where some undesired state is hit is probably not the place that needs to decide how to continue with execution. Exceptions allow us to control this unexpected state and make that decision at the appropriate point in the control flow

Again, I really like the railway notion for high level program flow, but down in the weeds of making things happen that syntax is far too obscure and much less readable

Nikita Sobolev • Feb 11 '19 • Edited

Wow, thanks for this reply. It is awesome.

I agree that it is easy to notice exceptions when they happen. It is hard to notice them ahead of time. There are a lot of different attempts to solve it, including checked exceptions in java.

Program flow is as unclear as without using them

I cannot agree. The main difference is that Result is just a regular value. It returns to the same place where the function was originally called. While exceptions can jump to any other layer of your call stack. And it will depend on the execution context. That's what I have called "two execution flows" in the article.

Comparisons to goto are extremely unjustified

Well, it seems rather similar to me:

try:
    print(1 / 0)
except ZeroDivisionError:
    do_something()

It is clear that except ZeroDivisionError is going to catch this exception.
Now, we will move this print(1 / 0) into a new function called print_value():

def print_value():
    print(1 / 0)

But, this except ZeroDivisionError will still execute. And it looks like goto mark to me.

I really like the railway notion for high level program flow

That's exactly the case! That's why I have this "Limitations" section. I prefer my business-logic to be safe and typed, while my application might throw exceptions that are common to the python world. Frameworks will handle them.

Michael Morehouse • Feb 13 '19

But, this except ZeroDivisionError will still execute. And it looks like goto mark to me.

But that's your apparent misunderstanding... it doesn't behave at all like a goto ... like not even within the same ballpark.

This:

try:
    return n / d
except ZeroDivisionError:
    return foo()

Is fundamentally no different than:

if not d:
    return foo()
return n / d

And adding in a deeper level of function nesting doesn't change that ... this:

def div(n, d):
    return n / d

try:
    return div(n, d)
except ZeroDivisionError:
    return foo()

Is now fundamentally no different than:

if not d:
    return foo()
return div(n, d)

The exception handler is effectively just a convenient way of writing the boilerplate for whatever test(s) would be required in that if to satisfy any given exception state, with the added benefit of deferring the test until an actual exception state has been encountered and needs to be addressed (or allowed to bubble further up the stack).

If you're writing the exception handler there's no magic (and certainly nothing that resembles a goto; you're reacting to a specific exception that has happened below you and the traceback tells you precisely where that came from ... if your reaction is to call a function then you know precisely where the control flow goes to and also that it will always return to you, even if it returns to you in the form of another exception (like for instance a SystemExit you've wrapped in the response function). A goto is a completely arbitrary escape from normal control flow, an exception is not, it's entirely predictable, though I'll happily admit it's not always easy to predict.

If no exception handling is involved then the behaviour is very predictable: where the exception is raised the stack frame exits and bubbles up a level. In each successively higher frame in which it's not handled it's effectively the same as catching it and immediately re-raising it, and so it continues to bubble ... when it hits the surface the interpreter catches it, prints the traceback, and shuts down in an orderly fashion with a non-zero returncode.

I don't have an issue with you wanting to make error handling a bit more sane for yourself, especially if you're going to try to enforce type constraints, but comparison to goto is just incorrect.

fakuivan • May 1 '23 • Edited

Exceptions totally do behave like gotos, a clear example is how you can use exceptions to "short circuit" things like map or functions that take in other functions. Here's an example:

def find_first_zero_pos(ints: list[int]) -> int | None:
    class FoundZero(Exception):
        pass
    i = 0
    def inc_or_rise_if_zero(num: int):
        if num == 0:
            raise FoundZero()
        nonlocal i
        i += 1

    try:
        for _ in map(inc_or_rise_if_zero, ints):
            pass
    except FoundZero:
        return i
    return None

In this case, execution flow is returned to find_first_zero_pos from map at will. You would not be able to do this with a language that does not support exceptions using only pure code (a function that shuts down the computer doesn't count lol). The with statement exists in part to be able to handle this "execution could end at any moment" situation by providing a way to do cleanup even when an exception is thrown.

The problem with goto is that usually execution flow always goes down into a function and comes up out of it, aka your call stack is actually a stack. gotos allow you to never return from a function, jumping to another section of code while your call stack is not a stack anymore. Python even has a way to type this, it's the NoReturn return type for functions, which is automatically assigned by static types analizers to functions that raise unconditionally.

rhymes • Feb 11 '19 • Edited

I think the context in this discussion is everything. I'll explain:

And checked exceptions won't be supported in the nearest future.

I feel like this is a pro in the context of Python though, not a con. Guido Van Rossum talks about it in the thread you linked. Java's checked exception aren't great for usability as he hints at. In theory they are a safe idea, in practice they encouraged suppression, rethrowing, API fatigue over the years. A "simple feature" that really polarized Java programmers :D

On the other side, Go designers have basically decided to force programmers to handle errors all the time (though you can still supress them) but the ergonomics is still not perfect (and they are thinking of making adjustments for Go 2.0 using a scope-wide central error handler)

There's no way to tell which line of code will be executed after the exception is thrown.

I'm not sure I follow. The stack for unhandled exceptions tells you where you were, the rules for handled exceptions are quite clear:

try:
  1 / 0
  print("Hello") # this is never going to be executed
except ZeroDivisionError as e:
  # do something with the error
  pass

print("World")

This is going to print only World. Where is the unclear part?

How can we consciously read code like this?

This sentence is a little odd, again in context, because Python programmers have been doing it since 1991. It's not yesterday :D I'm not saying the status quo is perfect, but sentences like "how can we deal with it" hide the fact that a lot of people have been doing it fine for decades.

Exceptions are just like notorious goto statements that torn the fabric of our programs.

Maybe, but not exactly. Goto was a bad idea because it let you jump to any place in the code at any time. Exception don't do that. Also, the "goto(ish)" (context again matters here) aspect of exceptions really depend on you as the developer. If you don't actively used them as a control flow mechanism but only to handle errors, then they are error handlers.

Now we got that exceptions are harmful to your code.

It's still your opinion though, I'm reading your post and I disagree :-D.

it is not possible to wrap all your possible errors in special-case classes. It will require too much work from a developer. And over-complicate your domain model.

Mmm again, people have been wrapping their errors in custom exceptions for decades, why are you suddenly saying it can't be done? I'm not saying it's the best strategy, I dispute your "cants" and "harmfuls" and "wrongs". By the way creating custom exceptions is so standard practice that it's even suggested in the tutorial.

In general: I'm not saying exceptions are the best ever method of handling error situations and I understand my bias as an enthusiastic Python developer (though my list of stuff that I don't like about Python has some items, my pet peeve with exceptions was recently fixed) but the way you wrote it sounds like this: "this pretty fundamental and central feature of Python is trash, I wish it were different. Here how you should do it, my way is better".

Going back to the idea co context: if I was going around saying "goroutines in Go are bad" I might be right in a context (it takes a few lines of Go to create a leak in the abstraction :D) but I would also ask myself if Go is the right choice for me. There's nothing wrong with a good rant (which this sounds like) or not liking features but at the end of the day we have only one life. Instead of trying make a language conspicuously deviate from its definining principles (aka the reason we're not all using the same language), why not just use another? We don't have to be "mono language".

The idea of wrapping everything in what basically are option types (if I'm not mistaken, I'm not strong on type theory and polymorphic typing) seems to veer quite a bit away from "Python's zen" ethos which are quite explicit on the idea that errors shouldn't be silent (which you're basically doing by discarding raise in favour of an option type) and to favor simplicity above all (which is probably less evident by looking at the seemingly innocous decorators in the last example)

I don't see the idea of having to unwrap values and exception constantly as an improvement to be honest :D The only advantage I see here is the compile time check, but it worsen the readability of the code and the mental gymnastic any Python developer has to do (considering that your proposal has not been built in in Python since day one), even with the decorator based shortcuts.

That's what I meant by citing context and language design in the beginning. Your idea is not bad per se, it's just probably not the right one in the context of Python's design. It reminds me a little of the argument in favor of removing the GIL :D

"Exceptions are hard to notice". Now, they are wrapped with a typed Result container, which makes them crystal clear.

But it's not actually true, is it? Let's look at your example:

def __call__(self, user_id: int) -> Result['UserProfile', Exception]

Here you're telling the reader that the method can raise an exception. What new information I have than I didn't know before? Knowing that any method can return exceptions in Python?

Also, this method is telling me that I basically have to prepare for any type of exception, which makes the advantage of the declaration moot if we put aside mypy and the tools for a second: what's the gained advantage of this instead of using a comment/documentation that says: this method can raise a HTTP error or a parsing error?

Tools are important but my brain the first time it read this code said: "well, thanks for nothing, Exception is the root class, anything can throw an exception".

If Python was a statically typed language I would probably have a different opinion about all this but it isn't, and the types can be ignored (and are ignored by default) which by being optional only leave the constant of the programmer reading this code in 5 years and being told something they already know. Again, context is everything here.

The TLDR; of all this long response is that the role of a language designer is to find tradeoffs and in the context of Python exceptions work. Stepping out of them would be changing the language, which could be perfectly fine in theory, but are we really sure?

I'm so glad I'm not a language designer, it's a tough job, and now I'm starting to understand why Guido semi retired ;-)

Nikita Sobolev • Feb 11 '19 • Edited

@rhymes wow! That's a very detailed and deep opinion. Thanks for bringing this up.

There's no way to tell which line of code will be executed after the exception is thrown.

Let me give you another example. Imagine that you have a big library / framework / project. And you are just debugging it. And then your execution flow jumps to some random place in the code with an exception thrown somewhere inside the method you were jumping over. "What the hell happened?" is what I usually have in my mind at this moment.

And that's what make me feel like we have two executional flows in the app. Regular and exception flows. The problem that I see in this separation that we have to wrap our minds against these two fundamentally different flows. First is controlled with regular execution stack and the second one is gotoish (as you called it). And it is controlled with except cases. The problem is that you can break this second flow by just placing some new awkward and accidental except case somewhere in between.

And, while I can perfectly live with the exceptions, I start to notice that my business logic suffer from it when it is growing in size. And having an extra way to work with different logical error in your program is a good thing.

it is not possible to wrap all your possible errors in special-case classes

I was talking about special classes like AnonymousUser is django. Not custom exception classes.

seems to veer quite a bit away from "Python's zen" which are quite explicit on the idea that errors shouldn't be silent

Unless explicitly silenced!

Jokes aside, we are not making them silent with wrapping into container values, just making them wrapped. Later you can use them anyway you want. You can reraise them if that's how it should be done.

Here you're telling the reader that the method can raise an exception

Yes, exactly! All functions may raise. But, some functions are most likely to do it: when dealing with IO, permissions, logic decisions, etc. That's what we indicate here. And, yes, my example with Result['UserProfile', Exception] should be rewritten as Result['UserProfile', ExactExceptionType].

When exceptions become the important part of your logic API - you need to use Result type. That's a rule that I have created for myself. Not talking about just python, but also elixir and typescript.

Thanks for taking a part in my rant, I have really enjoyed reading your response ;)

rhymes • Feb 11 '19

Let me give you another example. Imagine that you have a big library / framework / project. And you are just debugging it. And then your execution flow jumps to some random place in the code with an exception thrown somewhere inside the method you were jumping over. "What the hell happened?" is what I usually have in my mind at this moment.

I know what you're talking about, I've been there but to me is more an issue of abstraction than of exception handling. Some apps have too many deps :D BTW how is that super different from calling a function?

import template_library
def method_to_do_something():
  a = "Hello world"
  result = template_library.render(a=a)
  # do something else...
  return result

in this example I jump from method_to_do_something to template_library.render and back everytime I debug. And then if render calls yet another library the thing goes on. Sometimes you spend minutes stepping in libraries because the dependency tree is long (Ruby with open classes, heavy use of metaprogramming and DSLs sometimes is a nightmare to debug)

If you think about it, in a way, functions are labeled gotos with an address to go back to ;-)

and, yes, my example with Result['UserProfile', Exception] should be rewritten as Result['UserProfile', ExactExceptionType].

ok, that's make a little more sense :D

That's a rule that I have created for myself. Not talking about just python, but also elixir and typescript

Got it. Programmers are strange animals, when they find a tool or an idea they like they want to apply to everything :D.

Thanks for taking a part in my rant, I have really enjoyed reading your response ;)

ahhaha :D

edA‑qa mort‑ora‑y • Feb 11 '19

"Judging by their name it is an entity representing some exceptional situation that happens inside your program."

This is, and has always been false. Exceptions are a mechanism for propagating errors. There is no reason for them to be exceptional.

Nikita Sobolev • Feb 11 '19

Consider "exceptional situation that happens inside your program" as errors. Or some unreachable state. It does not make any difference.

Yoandy Rodriguez Martinez • Feb 11 '19

The word "antipattern" sure get's thrown a lot around here lately.

Nikita Sobolev • Feb 11 '19

Because we have a lot of them! That's a good thing that people share their opinions about how to write and how not to write good code.

Do you like the concept of Result type?

Yoandy Rodriguez Martinez • Feb 11 '19

Maybe 😉

Ryan Palo • Feb 11 '19

That's interesting. So far, from what I've seen, raising Exceptions seems like a valid, Pythonic way to handle edge cases.

I'm interested to find out your opinion on how I'd do things normally.

For the division function, it seems like a totally valid way to go to let it raise its own ZeroDivisionError. Looking in log files or seeing that come up in the terminal would be as good a message as any to communicate what's happening. You would see that a function called divide is raising a ZeroDivisionError, and you would know exactly what the problem is and that you need to track down where a zero is getting sent to your divide function.

And leaving it up to client code to handle the propagated ZeroDivisionError is a pretty standard idiom in Python. If you really feel the need to catch the error, you could maybe raise your own signaling the bad input value?

def divide(first: float, second: float) -> float:
    if second == 0:
        raise ValueError("Zero is an invalid divisor.")

    return first / second

In fact, Python's documentation itself recommends this idiom in the bisect module docs. If a binary search doesn't find the value it's looking for, it doesn't return None, it raises a ValueError. It's up to the client to handle that exception.

All of that to say, I think the wrapped value idea is neat, and a useful idiom from languages like Go and Rust. I can see that it would definitely have some use cases in Python where it would be the best. I agree that you wouldn't want to lean on Exceptions for standard control flow for the most part.

But using exceptions to indicate a state other than the happy path seems like it's definitely not an anti-pattern in Python, and in a lot of cases, it's the Pythonic way to go.

Does that make sense at all?

Nikita Sobolev • Feb 11 '19

I like to separate two kind of layers I have in my app: logic and application.

Logic is almost independent, it is my choice how to work with it. It allows a huge portion of freedom.
On the other hand it requires to be readable. Because there are no other-projects-like-this-one. So, that's why it is important for me to make exceptions crystal clear. Other people does not know if my FetchUser class is going to raise or not. And I do want to make this contract explicit.

On the lower, application, level I am using the existing stuff. Like django, celery, scrapy. And I need to respect their APIs. And, for example, django uses a lot of exceptions to control the execution flow. So, I prefer to raise on this level. And check that every exception is meaningful with tests.

That's how I see the big picture. Hope that you will find a proper use-cases for returns!

Rémy 🤖 • Feb 11 '19

That's a feeling that I had when I started programming and many years along the way: you feel like you should eventually catch all exceptions.

But that's not true. I embrace and expect exceptions. That's the runtime automatically checking that all possible inputs are correct and telling me when I was wrong. It's the luxury of getting explicit bugs neatly organized in Sentry reports.

If you ask me, all that sugarcoating is a sweet and seductive idea, but it's a bad idea.

Mišo • Oct 11 '20

I was thinking about exceptions recently too. But I don't consider them an anti-pattern only the support in IDEs and type systems is not very good. And somehow I see it also in your example with pipeline/safe decorators. Where is the difference? Because I see a lot in the comments that you have problem with the second exceptional flow but doesn't have your solution the same problem? And isn't it even worse because you don't a see a big stacktrace with the info about the exception? Let me describe what I see.

@pipeline
def pipeline(...):
    one().unwrap()
    two().unwrap()
    return three()

@pipeline
def parent_pipeline():
    pipeline().unwrap()
    ...

@pipeline
def grand_pipeline():
    parent_pipeline().unwrap()
    ...

So when the function two returns error the flow jumps to grand_pipeline exactly the same. And without the decorator with the proper exceptions the situation is also the same but with extra boilerplate of checking the result and returning upwards ala Golang. Rust has nice syntax sugar like two()? but still.

If I rewrite the code to use exceptions I see the same but without extra @safe and .unwrap() boilerplate. So where is the difference? If you read my article you will see I agree with that exceptions are hard to notice but I believe they can be fixed. By typesystem. Or even maybe by IDEs. What do you think about that? Still consider them anti-pattern even if they were upgraded?

I am still researching how the exceptions work under the hood and how stack unwinding affects the flow to complete my article but the idea there should be clear.

Nikita Sobolev • Oct 12 '20

Well, @pipeline actually had multiple problems. And it is now removed from dry-python.
It is actually imperative compared to declarative function composition. Why so? Because @pipeline is an partial implementation of do-notation, which is imperative by definition.

But. The difference is still visible.

Now exceptions can be visible in the funtion's signatire (let's use the same example from your post, even knowing that @pipeline is removed):

@pipeline
def grand_pipeline() -> Result[ValueType, ExceptionType]:
    parent_pipeline().unwrap()

Here it is! We know that something happens! And we can express it with types.

We have a special feature to enable stacktraces for your Results in development. See returns.readthedocs.io/en/latest/p...

mt3o • Feb 12 '19

Dear author, for me it seems you spend not enough time with languages not having exceptions, without sophisticated libraries, without sophisticated tools (like dynatrace or even debugger attached) trying to work with exception-rich cases like doing IO on filesystem or HTTP calls, using language lacking exceptions or just making bad use of them. In thinking of plain old pre y2k C or even PHP<5. If you had - you'd value exceptions, even more exceptions with properly designed type hierarchy and understood why Java enforces checked exceptions unless directly specified otherwise.
For single task of writing for to disk - there are so many different types of potential errors! Disk not found. Lacking permissions. File not existent. Directory non existent. Disk full. Lock being put on the file. Drive missing during the file write. And you have to check for those on (almost) each single operating on that file. And pass the state to the upper layer, without any strict convention (hello PHP, I'm watching at you). And with networking IO you have even more cases to handle, including such bizarre scenarios as timeouts in the middle of read.
Introduction of checked exceptions opened possibility to return something more than just value of declared type, but also complex information about errors. It should be embraced not suppressed.

Nikita Sobolev • Feb 12 '19

Well, indeed I have not that much experience with C or php@4 (while I still maintain one insanely large legacy php@4 project), but I have spent quite a lot of time with other languages without exceptions: like rust or elixir (which technically has exceptions, but their use is limited and noted with special ! functions).

And I value python's way of dealing with errors. I just want to provide an alternative that will solve problems that do exist.

Now it is your choice either to use exceptions or to use returns when you need to as explicit as possible.

Mihail Malo • Feb 13 '19

Hahaha, look at the beautiful Result/Either datatype viciously penetrating this Python of yours.
Maybe in a couple of years, something like this will become accepted in JS as well!
One can only hope.

Sergey Kislyakov • Feb 11 '19

Failure(4).map(lambda number: number / 2)
# => Failure(2)

Looks like .map got executed for a Failure. Is it a bug?

Also, does PyCharm handle @safe decorator properly? I mean, if I have a function something() -> User, do I get auto-completions for User and not [User, WhateverExcetpion]?

Nikita Sobolev • Feb 11 '19

Sorry, that's a typo! And thanks for noticing. I have updated this line:

Failure(4).map(lambda number: number / 2)
# => Failure(4)

PyCharm resolves types correctly, since we are shipping PEP561 annotations.

Jeff Doak • Nov 18 '19

Nikita, thank you so much for your article. I ran into it looking for discussion on the topic. We use Marshmallow for validation and until version 3 it had a very simple mechanism of returning (valid_data, errors). In version 3 they decided to use exception upon errors instead. While it was easy to work around and wrap the call, It was a much better design in the original approach. I liked the design so much that I've used it myself in several designs.
I believe you are correct and I hope more developers come around to this way if thinking.
Code that is doing large sequences of data manipulation does not want to be halted or even thrown off course due to a single error. It wants to note it, store some details, and move on.
Exceptions are great in IO, memory, resource issues, etc. In my opinion they should be avoided in designs such as Marshmallow where errors are expected.

Nikita Sobolev • Nov 18 '19

Thanks a lot, Jeff!

Mr. Vegetable • Apr 6 '19 • Edited

In my experience, exceptions in Python mostly make things MORE clear, not less, since they are used as ordinary control flow, like for or if, and you just expect them. Our brain models a lot of things in terms of "typical case + few special cases" and so, for me personally, exceptions make it much more easier to convert ideas into code and understand the ideas behind the code.

Exceptions are not errors or problems or unexpected thing. They are just cases that don't arise most often. So for me it's logically wrong to map them to "Failure". And the problem of finding "where the exception is caught" does not seem to become significantly easier when converted to "where do we check that the return result is Failure".

Sergey Kislyakov • Feb 11 '19

That looks interesting and I'd probably try it in some of my pet projects. Looks a bit Rust'y, though.