Originally published in my blog: https://sobolevn.me/2019/01/simple-dependent-types-in-python
What are exceptions? Judging by their name it is an entity representing some exceptional situation that happens inside your program.
You might be wondering how do exceptions are an anti-pattern and how does this relate to typing at all? Well, let's find out!
Problems with exceptions
First, we have to prove that exceptions have drawbacks. Well, it is usually hard to find "issues" in things you use every day because they start to look like "features" to you at some point.
Let's have a fresh look.
Exceptions are hard to notice
There are two types of exceptions: "explicit" that are created with raise
keyword right inside the code you are reading and "wrapped" that are wrapped inside some other functions/classes/methods that you are using.
The problem is: it is really hard to notice all this "wrapped" exceptions.
I will illustrate my point with this pure function:
def divide(first: float, second: float) -> float:
return first / second
All it does is dividing two numbers. Always returning float
. It is type safe and can be used like so:
result = divide(1, 0)
print('x / y = ', result)
Wait, did you get it? print
will never be actually executed. Because 1 / 0
is an impossible operation and ZeroDivisionError
will be raised. So, despite your code is type safe it is not safe to be used.
You still need to have a solid experience to spot these potential problems in a perfectly readable and typed code. Almost everything in python
can fail with different types of exceptions: division, function calls, int
, str
, generators, iterables in for
loops, attribute access, key access, even raise something()
itself may fail. I am not even covering IO operations here. And checked exceptions won't be supported in the nearest future.
Restoring normal behavior in-place is impossible
Hey, but we always have except
cases just for this kind of situations. Let's just handle ZeroDivisionError
and we will be safe!
def divide(first: float, second: float) -> float:
try:
return first / second
except ZeroDivisionError:
return 0.0
Now we are safe! But why do we return 0
? Why not 1
? Why not None
? And while None
in most cases is as bad (or even worse) than the exceptions, turns out we should heavily rely on business logic and use-cases of this function.
What exactly do we divide? Arbitrary numbers? Some specific units? Money? Not all cases can be covered and easily restored. And sometimes when we will reuse this function for different use-cases we will find out that it requires different restore logic.
So, the sad conclusion is: all problems must be resolved individually depending on a specific usage context. There's no silver bullet to resolve all ZeroDivisionError
s once and for all. And again, I am not even covering complex IO flows with retry policies and expotential timeouts.
Maybe we should not even handle exceptions in-place at all? Maybe we should throw it further in the execution flow and someone will later handle it somehow.
Execution flow is unclear
Ok, now we will hope that someone else will catch this exception and possibly handle it. For example, the system might notify the user to change the input, because we can not divide by 0
. Which is clearly not a responsibility of the divide
function.
Now we just need to check where this exception is actually caught. By the way, how can we tell where exactly it will be handled? Can we navigate to this point in the code? Turns out, we can not do that.
There's no way to tell which line of code will be executed after the exception is thrown. Different exception types might be handled by different except
cases, some exceptions may be suppress
ed. And you might also accidentally break your program in random spots by introducing new except
cases in a different module. And remember that almost any line can raise.
We have two independent flows in our app: regular flow that goes from top to bottom and exceptional one that goes however it wants. How can we consciously read code like this?
Only with a debugger turned on. With "catch all exceptions" policy enabled.
Exceptions are just like notorious goto
statements that torn the fabric of our programs.
Exceptions are not exceptional
Let's look at another example, a typical code to access remote HTTP API:
import requests
def fetch_user_profile(user_id: int) -> 'UserProfile':
"""Fetches UserProfile dict from foreign API."""
response = requests.get('/api/users/{0}'.format(user_id))
response.raise_for_status()
return response.json()
Literally, everything in this example can go wrong. Here's an incomplete list of all possible errors that might occur:
- Your network might be down, so request won't happen at all
- The server might be down
- The server might be too busy and you will face a timeout
- The server might require an authentication
- API endpoint might not exist
- The user might not exist
- You might not have enough permissions to view it
- The server might fail with an internal error while processing your request
- The server might return an invalid or corrupted response
- The server might return invalid
json
, so the parsing will fail
And the list goes on and on! There are so maybe potential problems with these three lines of code, that it is easier to say that it only accidentally works. And normally it fails with the exception.
How to be safe?
Now we got that exceptions are harmful to your code. Let's learn how to get read off them. There are different patterns to write the exception-free code:
- Write
except Exception: pass
everywhere. That's as bad as you can imagine. Don't do it. - Return
None
. That's evil too! You either will end up withif something is not None:
on almost every line and global pollution of your logic by type-checking conditionals, or will suffer fromTypeError
every day. Not a pleasant choice. - Write special-case classes. For example, you will have
User
base class with multiple error-subclasses likeUserNotFound(User)
andMissingUser(User)
. It might be used for some specific situations, likeAnonymousUser
indjango
, but it is not possible to wrap all your possible errors in special-case classes. It will require too much work from a developer. And over-complicate your domain model. - You can use container values, that wraps actual success or error value into a thin wrapper with utility methods to work with this value. That's exactly why we have created
@dry-python/returns
project. So you can make your functions return something meaningful, typed, and safe.
Let's start with the same number dividing example, which returns 0
when the error happens. Maybe instead we can indicate that the result was not successful without any explicit numerical value?
from returns.result import Result, Success, Failure
def divide(first: float, second: float) -> Result[float, ZeroDivisionError]:
try:
return Success(first / second)
except ZeroDivisionError as exc:
return Failure(exc)
Now we wrap our values in one of two wrappers: Success
or Failure
. These two classes inherit from Result
base class. And we can specify types of wrapped values in a function return annotation, for example Result[float, ZeroDivisionError]
returns either Success[float]
or Failure[ZeroDivisionError]
.
What does it mean to us? It means, that exceptions are not exceptional, they represent expectable problems. But, we also wrap them in Failure
to solve the second problem: spotting potential exceptions is hard.
1 + divide(1, 0)
# => mypy error: Unsupported operand types for + ("int" and "Result[float, ZeroDivisionError]")
Now you can easily spot them! The rule is: if you see a Result
it means that this function can throw an exception. And you even know its type in advance.
Moreover, returns
library is fully typed and PEP561 compatible. It means that mypy
will warn you if you try to return something that violates declared type contract.
from returns.result import Result, Success, Failure
def divide(first: float, second: float) -> Result[float, ZeroDivisionError]:
try:
return Success('Done')
# => error: incompatible type "str"; expected "float"
except ZeroDivisionError as exc:
return Failure(0)
# => error: incompatible type "int"; expected "ZeroDivisionError"
How to work with wrapped values?
There are two methods two work with these wrapped values:
-
map
works with functions that return regular values -
bind
works with functions that return other containers
Success(4).bind(lambda number: Success(number / 2))
# => Success(2)
Success(4).map(lambda number: number + 1)
# => Success(5)
The thing is: you will be safe from failed scenarios. Since .bind
and .map
will not execute for Failure
containers:
Failure(4).bind(lambda number: Success(number / 2))
# => Failure(4)
Failure(4).map(lambda number: number / 2)
# => Failure(4)
Now you can just concentrate on correct execution flow and be sure that failed state won't break your program in random places.
And you can always take care of a failed state and even fix it and return to the right track if you want to.
Failure(4).rescue(lambda number: Success(number + 1))
# => Success(5)
Failure(4).fix(lambda number: number / 2)
# => Success(2)
It means that "all problems must be resolved individually" practice is the only way to go and "execution flow is now clear". Enjoy your railway programming!
But how to unwrap values from containers?
Yes, indeed, you really need raw values when dealing with functions that actually accept these raw values. You can use .unwrap()
or .value_or()
methods:
Success(1).unwrap()
# => 1
Success(0).value_or(None)
# => 0
Failure(0).value_or(None)
# => None
Failure(1).unwrap()
# => Raises UnwrapFailedError()
Wait, what? You have promised to save me from exceptions and now you are telling me that all my .unwrap()
calls can result in one more exception!
How not to care about these UnwrapFailedErrors?
Ok, let's see how to live with these new exceptions. Consider this example: we need to validate the user's input, then create two models in a database. And every step might fail with the exception, so we have wrapped all methods into the Result
wrapper:
from returns.result import Result, Success, Failure
class CreateAccountAndUser(object):
"""Creates new Account-User pair."""
# TODO: we need to create a pipeline of these methods somehow...
def _validate_user(
self, username: str, email: str,
) -> Result['UserSchema', str]:
"""Returns an UserSchema for valid input, otherwise a Failure."""
def _create_account(
self, user_schema: 'UserSchema',
) -> Result['Account', str]:
"""Creates an Account for valid UserSchema's. Or returns a Failure."""
def _create_user(
self, account: 'Account',
) -> Result['User', str]:
"""Create an User instance. If user already exists returns Failure."""
First of all, you can not unwrap any values while writing your own business logic:
class CreateAccountAndUser(object):
"""Creates new Account-User pair."""
def __call__(self, username: str, email: str) -> Result['User', str]:
"""Can return a Success(user) or Failure(str_reason)."""
return self._validate_user(
username, email,
).bind(
self._create_account,
).bind(
self._create_user,
)
# ...
And this will work without any problems. It won't raise any exceptions, because .unwrap()
is not used. But, is it easy to read code like this? No, it is not. What alternative can we provide? @pipeline
!
from result.functions import pipeline
class CreateAccountAndUser(object):
"""Creates new Account-User pair."""
@pipeline
def __call__(self, username: str, email: str) -> Result['User', str]:
"""Can return a Success(user) or Failure(str_reason)."""
user_schema = self._validate_user(username, email).unwrap()
account = self._create_account(user_schema).unwrap()
return self._create_user(account)
# ...
Now it is perfectly readable. That's how .unwrap()
and @pipeline
synergy works: whenever any .unwrap()
method will fail on Failure[str]
instance @pipeline
decorator will catch it and return Failure[str]
as a result value. That's how we can eliminate all the exceptions from our code and make it truly type-safe.
Wrapping all together
Now, let's solve this requests
example with all the new tools we have. Remember, that each line could raise an exception? And there's no way to make them return Result
container. But you can use @safe
decorator to wrap unsafe functions and make them safe. These two examples are identical:
from returns.functions import safe
@safe
def divide(first: float, second: float) -> float:
return first / second
# is the same as:
def divide(first: float, second: float) -> Result[float, ZeroDivisionError]:
try:
return Success(first / second)
except ZeroDivisionError as exc:
return Failure(exc)
And we can see that the first one with @safe
is way more readable and simple.
That's the last thing we needed to solve our requests
problem. That's how our result code will look like in the end:
import requests
from returns.functions import pipeline, safe
from returns.result import Result
class FetchUserProfile(object):
"""Single responsibility callable object that fetches user profile."""
#: You can later use dependency injection to replace `requests`
#: with any other http library (or even a custom service).
_http = requests
@pipeline
def __call__(self, user_id: int) -> Result['UserProfile', Exception]:
"""Fetches UserProfile dict from foreign API."""
response = self._make_request(user_id).unwrap()
return self._parse_json(response)
@safe
def _make_request(self, user_id: int) -> requests.Response:
response = self._http.get('/api/users/{0}'.format(user_id))
response.raise_for_status()
return response
@safe
def _parse_json(self, response: requests.Response) -> 'UserProfile':
return response.json()
Things to recap:
- We use
@safe
for all methods that can raise an exception, it will change the return type of the function toResult[OldReturnType, Exception]
- We use
Result
as a container for wrapping values and errors in a simple abstraction - We use
.unwrap()
to unwrap raw value from the container - We use
@pipeline
to make sequences of.unwrap
calls readable
This is a perfectly readable and safe way to do the exact same thing as we previously did with the unsafe function. It eliminates all the problems we had with exceptions:
- "Exceptions are hard to notice". Now, they are wrapped with a typed
Result
container, which makes them crystal clear. - "Restoring normal behavior in-place is impossible". We now can safely delegate the restoration process to the caller. We provide
.fix()
and.rescue()
methods for this specific use-case. - "Execution flow is unclear". Now it is the same as a regular business flow. From top to bottom.
- "Exceptions are not exceptional". And we know it! We expect things to go wrong and are ready for it.
Use-cases and limitations
Obviously, you can not write all your code this way. It is just too safe for the most situations and incompatible with other libraries/frameworks. But, you should definitely write the most important parts of your business logic as I have shown above. It will increase the maintainability and correctness of your system.
dry-python / returns
Make your functions return something meaningful, typed, and safe!
Make your functions return something meaningful, typed, and safe!
Features
- Brings functional programming to Python land
- Provides a bunch of primitives to write declarative business logic
- Enforces better architecture
- Fully typed with annotations and checked with
mypy
, PEP561 compatible - Adds emulated Higher Kinded Types support
- Provides type-safe interfaces to create your own data-types with enforced laws
- Has a bunch of helpers for better composition
- Pythonic and pleasant to write and to read 🐍
- Support functions and coroutines, framework agnostic
- Easy to start: has lots of docs, tests, and tutorials
Quickstart right now!
Installation
pip install returns
You can also install returns
with the latest supported mypy
version:
pip install returns[compatible-mypy]
You would also need to configure our mypy
plugin:
# In setup.cfg or mypy.ini:
[mypy]
plugins =
returns.contrib.mypy.returns_plugin
or:
[tool.mypy]
plugins = ["returns.contrib.mypy.returns_plugin"]
We also recommend to use the same…
Top comments (26)
I'm torn by this. While I do like the railway programming model a lot, and the the idea of returning additional information makes a lot of sense in cases of potential failure (C# for example has
bool TryGet<TKey, TValue>(TKey key, out TValue value)
to deal with lookup failure without exceptions), I feel that Exceptions are highly useful and don't really suffer from the issues you assign to themThey aren't difficult to notice, when they happen I get a big stack trace dump and my IDE helpfully breaks when they are thrown. I'd say they are much more difficult to know ahead of time, but the cause of this is that python is impossible to statically analyse, it is a language problem and not an Exception problem. Other languages have tried noting functions are noexcep or listing all the possible exceptions that might come from a function call, but ultimately those either didn't solve the know-ability problem or created too much overhead to be useful.
Program flow is as unclear as without using them. There is always the issue of not knowing how a function is called, and therefore where the program might continue on an error. The only defense we have against that is to write clear and specific code, SOLID etc.
Comparisons to goto are extremely unjustified. It is just as easy to say that function calls, and even if statements are just structured gotos. The exact same reasoning could be applied to those but we still use them because it would be impossible to do anything without them. Since a raw goto can make it very difficult to follow a program, we have specific sub-cases where they make sense and restrict their use as much as possible to those sub-cases. It is absolutely possible to make a program difficult to follow with misuse of exceptions, it is equally possible to make it difficult to follow with poorly designed functions
Exceptions are a control-flow mechanism. Not all functions can produce a result when the accept variable input. Sometimes remote resources aren't available. Sometimes cosmic rays hit a register and flip a bit leading to undesired behavior. The places where some undesired state is hit is probably not the place that needs to decide how to continue with execution. Exceptions allow us to control this unexpected state and make that decision at the appropriate point in the control flow
Again, I really like the railway notion for high level program flow, but down in the weeds of making things happen that syntax is far too obscure and much less readable
Wow, thanks for this reply. It is awesome.
I agree that it is easy to notice exceptions when they happen. It is hard to notice them ahead of time. There are a lot of different attempts to solve it, including checked exceptions in
java
.I cannot agree. The main difference is that
Result
is just a regular value. It returns to the same place where the function was originally called. While exceptions can jump to any other layer of your call stack. And it will depend on the execution context. That's what I have called "two execution flows" in the article.Well, it seems rather similar to me:
It is clear that
except ZeroDivisionError
is going to catch this exception.Now, we will move this
print(1 / 0)
into a new function calledprint_value()
:But, this
except ZeroDivisionError
will still execute. And it looks likegoto
mark to me.That's exactly the case! That's why I have this "Limitations" section. I prefer my business-logic to be safe and typed, while my application might throw exceptions that are common to the python world. Frameworks will handle them.
But that's your apparent misunderstanding... it doesn't behave at all like a
goto
... like not even within the same ballpark.This:
Is fundamentally no different than:
And adding in a deeper level of function nesting doesn't change that ... this:
Is now fundamentally no different than:
The exception handler is effectively just a convenient way of writing the boilerplate for whatever test(s) would be required in that if to satisfy any given exception state, with the added benefit of deferring the test until an actual exception state has been encountered and needs to be addressed (or allowed to bubble further up the stack).
If you're writing the exception handler there's no magic (and certainly nothing that resembles a
goto
; you're reacting to a specific exception that has happened below you and the traceback tells you precisely where that came from ... if your reaction is to call a function then you know precisely where the control flow goes to and also that it will always return to you, even if it returns to you in the form of another exception (like for instance a SystemExit you've wrapped in the response function). Agoto
is a completely arbitrary escape from normal control flow, an exception is not, it's entirely predictable, though I'll happily admit it's not always easy to predict.If no exception handling is involved then the behaviour is very predictable: where the exception is raised the stack frame exits and bubbles up a level. In each successively higher frame in which it's not handled it's effectively the same as catching it and immediately re-raising it, and so it continues to bubble ... when it hits the surface the interpreter catches it, prints the traceback, and shuts down in an orderly fashion with a non-zero returncode.
I don't have an issue with you wanting to make error handling a bit more sane for yourself, especially if you're going to try to enforce type constraints, but comparison to
goto
is just incorrect.Exceptions totally do behave like gotos, a clear example is how you can use exceptions to "short circuit" things like
map
or functions that take in other functions. Here's an example:In this case, execution flow is returned to
find_first_zero_pos
frommap
at will. You would not be able to do this with a language that does not support exceptions using only pure code (a function that shuts down the computer doesn't count lol). Thewith
statement exists in part to be able to handle this "execution could end at any moment" situation by providing a way to do cleanup even when an exception is thrown.The problem with
goto
is that usually execution flow always goes down into a function and comes up out of it, aka your call stack is actually a stack.goto
s allow you to never return from a function, jumping to another section of code while your call stack is not a stack anymore. Python even has a way to type this, it's theNoReturn
return type for functions, which is automatically assigned by static types analizers to functions that raise unconditionally.I think the context in this discussion is everything. I'll explain:
I feel like this is a pro in the context of Python though, not a con. Guido Van Rossum talks about it in the thread you linked. Java's checked exception aren't great for usability as he hints at. In theory they are a safe idea, in practice they encouraged suppression, rethrowing, API fatigue over the years. A "simple feature" that really polarized Java programmers :D
On the other side, Go designers have basically decided to force programmers to handle errors all the time (though you can still supress them) but the ergonomics is still not perfect (and they are thinking of making adjustments for Go 2.0 using a scope-wide central error handler)
I'm not sure I follow. The stack for unhandled exceptions tells you where you were, the rules for handled exceptions are quite clear:
This is going to print only
World
. Where is the unclear part?This sentence is a little odd, again in context, because Python programmers have been doing it since 1991. It's not yesterday :D I'm not saying the status quo is perfect, but sentences like "how can we deal with it" hide the fact that a lot of people have been doing it fine for decades.
Maybe, but not exactly. Goto was a bad idea because it let you jump to any place in the code at any time. Exception don't do that. Also, the "goto(ish)" (context again matters here) aspect of exceptions really depend on you as the developer. If you don't actively used them as a control flow mechanism but only to handle errors, then they are error handlers.
It's still your opinion though, I'm reading your post and I disagree :-D.
Mmm again, people have been wrapping their errors in custom exceptions for decades, why are you suddenly saying it can't be done? I'm not saying it's the best strategy, I dispute your "cants" and "harmfuls" and "wrongs". By the way creating custom exceptions is so standard practice that it's even suggested in the tutorial.
In general: I'm not saying exceptions are the best ever method of handling error situations and I understand my bias as an enthusiastic Python developer (though my list of stuff that I don't like about Python has some items, my pet peeve with exceptions was recently fixed) but the way you wrote it sounds like this: "this pretty fundamental and central feature of Python is trash, I wish it were different. Here how you should do it, my way is better".
Going back to the idea co context: if I was going around saying "goroutines in Go are bad" I might be right in a context (it takes a few lines of Go to create a leak in the abstraction :D) but I would also ask myself if Go is the right choice for me. There's nothing wrong with a good rant (which this sounds like) or not liking features but at the end of the day we have only one life. Instead of trying make a language conspicuously deviate from its definining principles (aka the reason we're not all using the same language), why not just use another? We don't have to be "mono language".
The idea of wrapping everything in what basically are option types (if I'm not mistaken, I'm not strong on type theory and polymorphic typing) seems to veer quite a bit away from "Python's zen" ethos which are quite explicit on the idea that errors shouldn't be silent (which you're basically doing by discarding
raise
in favour of an option type) and to favor simplicity above all (which is probably less evident by looking at the seemingly innocous decorators in the last example)I don't see the idea of having to unwrap values and exception constantly as an improvement to be honest :D The only advantage I see here is the compile time check, but it worsen the readability of the code and the mental gymnastic any Python developer has to do (considering that your proposal has not been built in in Python since day one), even with the decorator based shortcuts.
That's what I meant by citing context and language design in the beginning. Your idea is not bad per se, it's just probably not the right one in the context of Python's design. It reminds me a little of the argument in favor of removing the GIL :D
But it's not actually true, is it? Let's look at your example:
Here you're telling the reader that the method can raise an exception. What new information I have than I didn't know before? Knowing that any method can return exceptions in Python?
Also, this method is telling me that I basically have to prepare for any type of exception, which makes the advantage of the declaration moot if we put aside
mypy
and the tools for a second: what's the gained advantage of this instead of using a comment/documentation that says: this method can raise a HTTP error or a parsing error?Tools are important but my brain the first time it read this code said: "well, thanks for nothing, Exception is the root class, anything can throw an exception".
If Python was a statically typed language I would probably have a different opinion about all this but it isn't, and the types can be ignored (and are ignored by default) which by being optional only leave the constant of the programmer reading this code in 5 years and being told something they already know. Again, context is everything here.
The TLDR; of all this long response is that the role of a language designer is to find tradeoffs and in the context of Python exceptions work. Stepping out of them would be changing the language, which could be perfectly fine in theory, but are we really sure?
I'm so glad I'm not a language designer, it's a tough job, and now I'm starting to understand why Guido semi retired ;-)
@rhymes wow! That's a very detailed and deep opinion. Thanks for bringing this up.
Let me give you another example. Imagine that you have a big library / framework / project. And you are just debugging it. And then your execution flow jumps to some random place in the code with an exception thrown somewhere inside the method you were jumping over. "What the hell happened?" is what I usually have in my mind at this moment.
And that's what make me feel like we have two executional flows in the app. Regular and exception flows. The problem that I see in this separation that we have to wrap our minds against these two fundamentally different flows. First is controlled with regular execution stack and the second one is
goto
ish (as you called it). And it is controlled withexcept
cases. The problem is that you can break this second flow by just placing some new awkward and accidentalexcept
case somewhere in between.And, while I can perfectly live with the exceptions, I start to notice that my business logic suffer from it when it is growing in size. And having an extra way to work with different logical error in your program is a good thing.
I was talking about special classes like
AnonymousUser
isdjango
. Not custom exception classes.Unless explicitly silenced!
Jokes aside, we are not making them silent with wrapping into container values, just making them wrapped. Later you can use them anyway you want. You can reraise them if that's how it should be done.
Yes, exactly! All functions may raise. But, some functions are most likely to do it: when dealing with IO, permissions, logic decisions, etc. That's what we indicate here. And, yes, my example with
Result['UserProfile', Exception]
should be rewritten asResult['UserProfile', ExactExceptionType]
.When exceptions become the important part of your logic API - you need to use
Result
type. That's a rule that I have created for myself. Not talking about justpython
, but alsoelixir
andtypescript
.Thanks for taking a part in my rant, I have really enjoyed reading your response ;)
I know what you're talking about, I've been there but to me is more an issue of abstraction than of exception handling. Some apps have too many deps :D BTW how is that super different from calling a function?
in this example I jump from
method_to_do_something
totemplate_library.render
and back everytime I debug. And then ifrender
calls yet another library the thing goes on. Sometimes you spend minutes stepping in libraries because the dependency tree is long (Ruby with open classes, heavy use of metaprogramming and DSLs sometimes is a nightmare to debug)If you think about it, in a way, functions are labeled gotos with an address to go back to ;-)
ok, that's make a little more sense :D
Got it. Programmers are strange animals, when they find a tool or an idea they like they want to apply to everything :D.
ahhaha :D
"Judging by their name it is an entity representing some exceptional situation that happens inside your program."
This is, and has always been false. Exceptions are a mechanism for propagating errors. There is no reason for them to be exceptional.
Consider "exceptional situation that happens inside your program" as errors. Or some unreachable state. It does not make any difference.
The word "antipattern" sure get's thrown a lot around here lately.
Because we have a lot of them! That's a good thing that people share their opinions about how to write and how not to write good code.
Do you like the concept of
Result
type?Maybe
😉That's interesting. So far, from what I've seen, raising Exceptions seems like a valid, Pythonic way to handle edge cases.
I'm interested to find out your opinion on how I'd do things normally.
For the division function, it seems like a totally valid way to go to let it raise its own
ZeroDivisionError
. Looking in log files or seeing that come up in the terminal would be as good a message as any to communicate what's happening. You would see that a function calleddivide
is raising aZeroDivisionError
, and you would know exactly what the problem is and that you need to track down where a zero is getting sent to your divide function.And leaving it up to client code to handle the propagated ZeroDivisionError is a pretty standard idiom in Python. If you really feel the need to catch the error, you could maybe raise your own signaling the bad input value?
In fact, Python's documentation itself recommends this idiom in the
bisect
module docs. If a binary search doesn't find the value it's looking for, it doesn't return None, it raises a ValueError. It's up to the client to handle that exception.All of that to say, I think the wrapped value idea is neat, and a useful idiom from languages like Go and Rust. I can see that it would definitely have some use cases in Python where it would be the best. I agree that you wouldn't want to lean on Exceptions for standard control flow for the most part.
But using exceptions to indicate a state other than the happy path seems like it's definitely not an anti-pattern in Python, and in a lot of cases, it's the Pythonic way to go.
Does that make sense at all?
I like to separate two kind of layers I have in my app: logic and application.
Logic is almost independent, it is my choice how to work with it. It allows a huge portion of freedom.
On the other hand it requires to be readable. Because there are no other-projects-like-this-one. So, that's why it is important for me to make exceptions crystal clear. Other people does not know if my
FetchUser
class is going to raise or not. And I do want to make this contract explicit.On the lower, application, level I am using the existing stuff. Like
django
,celery
,scrapy
. And I need to respect their APIs. And, for example,django
uses a lot of exceptions to control the execution flow. So, I prefer to raise on this level. And check that every exception is meaningful with tests.That's how I see the big picture. Hope that you will find a proper use-cases for
returns
!That's a feeling that I had when I started programming and many years along the way: you feel like you should eventually catch all exceptions.
But that's not true. I embrace and expect exceptions. That's the runtime automatically checking that all possible inputs are correct and telling me when I was wrong. It's the luxury of getting explicit bugs neatly organized in Sentry reports.
If you ask me, all that sugarcoating is a sweet and seductive idea, but it's a bad idea.
I was thinking about exceptions recently too. But I don't consider them an anti-pattern only the support in IDEs and type systems is not very good. And somehow I see it also in your example with pipeline/safe decorators. Where is the difference? Because I see a lot in the comments that you have problem with the second exceptional flow but doesn't have your solution the same problem? And isn't it even worse because you don't a see a big stacktrace with the info about the exception? Let me describe what I see.
So when the function
two
returns error the flow jumps togrand_pipeline
exactly the same. And without the decorator with the proper exceptions the situation is also the same but with extra boilerplate of checking the result and returning upwards ala Golang. Rust has nice syntax sugar liketwo()?
but still.If I rewrite the code to use exceptions I see the same but without extra
@safe
and.unwrap()
boilerplate. So where is the difference? If you read my article you will see I agree with that exceptions are hard to notice but I believe they can be fixed. By typesystem. Or even maybe by IDEs. What do you think about that? Still consider them anti-pattern even if they were upgraded?I am still researching how the exceptions work under the hood and how stack unwinding affects the flow to complete my article but the idea there should be clear.
Well,
@pipeline
actually had multiple problems. And it is now removed fromdry-python
.It is actually imperative compared to declarative function composition. Why so? Because
@pipeline
is an partial implementation ofdo-notation
, which is imperative by definition.But. The difference is still visible.
@pipeline
is removed):Here it is! We know that something happens! And we can express it with types.
Result
s in development. See returns.readthedocs.io/en/latest/p...Dear author, for me it seems you spend not enough time with languages not having exceptions, without sophisticated libraries, without sophisticated tools (like dynatrace or even debugger attached) trying to work with exception-rich cases like doing IO on filesystem or HTTP calls, using language lacking exceptions or just making bad use of them. In thinking of plain old pre y2k C or even PHP<5. If you had - you'd value exceptions, even more exceptions with properly designed type hierarchy and understood why Java enforces checked exceptions unless directly specified otherwise.
For single task of writing for to disk - there are so many different types of potential errors! Disk not found. Lacking permissions. File not existent. Directory non existent. Disk full. Lock being put on the file. Drive missing during the file write. And you have to check for those on (almost) each single operating on that file. And pass the state to the upper layer, without any strict convention (hello PHP, I'm watching at you). And with networking IO you have even more cases to handle, including such bizarre scenarios as timeouts in the middle of read.
Introduction of checked exceptions opened possibility to return something more than just value of declared type, but also complex information about errors. It should be embraced not suppressed.
Well, indeed I have not that much experience with
C
orphp@4
(while I still maintain one insanely large legacyphp@4
project), but I have spent quite a lot of time with other languages without exceptions: likerust
orelixir
(which technically has exceptions, but their use is limited and noted with special!
functions).And I value python's way of dealing with errors. I just want to provide an alternative that will solve problems that do exist.
Now it is your choice either to use exceptions or to use
returns
when you need to as explicit as possible.Hahaha, look at the beautiful
Result
/Either
datatype viciously penetrating this Python of yours.Maybe in a couple of years, something like this will become accepted in JS as well!
One can only hope.
Looks like
.map
got executed for aFailure
. Is it a bug?Also, does PyCharm handle
@safe
decorator properly? I mean, if I have a functionsomething() -> User
, do I get auto-completions forUser
and not[User, WhateverExcetpion]
?Sorry, that's a typo! And thanks for noticing. I have updated this line:
PyCharm resolves types correctly, since we are shipping PEP561 annotations.