Python Magic Methods How to write better Python classes using Magic Methods

It is important to make the code we write clear and as easy to read as possible, so anyone unfamiliar with the codebase is able to understand what it does without too much hassle. When dealing with object-oriented Python code, using dunder methods - also known as magic methods - is one useful way to achieve this. They allow our user-defined classes to make use of Python's built-in and primitive constructs - such as +, *, / and other operators, and keywords such as new or len - which are often a lot more intuitive than chains of class methods.

What are dunder methods?

Chances are, if you have programmed in Python for a while, you might have come across strange methods whose names begin and end with pairs of underscores. If you've worked with Python classes, you might even be familiar with one specific method: __init__, which acts as a constructor and is called when a class is initialized.

__init__ is an example of a dunder (double underscore) method, also known as magic methods. They provide a way for us to 'tell' Python how to handle primitive operations, such as addition or equality check, done on our custom classes. For example, when you try to add two instances of a custom class using the '+' operator, the Python interpreter goes inside the definition of your class and looks for an implementation of the __add__ magic method. If it is implemented, it executes the code within, just like with any other 'regular' method or function.

Learning by Example

Let's implement a Time Period class and see how, using Python's Magic Methods, it can be made clearer and more readable. For simplicity, our class will only deal with hours and minutes and will provide a couple of basic functionalities like adding and comparing time periods.

A Basic TimePeriod class

A basic implementation of this class might look something like this:

class TimePeriod:

    def __init__(self, hours=0, minutes=0):
        self.hours = hours
        self.minutes = minutes

    def add(self, other):
        minutes = self.minutes + other.minutes
        hours = self.hours + other.hours

        if minutes >= 60:
            minutes -= 60
            hours += 1

        return TimePeriod(hours, minutes)

    def greater_than(self, other):
        if self.hours > other.hours:
            return True
        elif self.hours < other.hours:
            return False
        elif self.minutes > other.minutes:
            return True
        else:
            return False

Then we can create instances of our class like this:

time_i_sleep = TimePeriod(9, 0)
time_i_work = TimePeriod(0, 30)

And perform operations on them like this:

print(time_i_sleep.greater_than(time_i_work)) # Should print True

There's nothing functionally wrong with this code. It works exactly as intended and has no bugs. But what if we want to perform a more complex operation, like comparing the sum of two time periods with the sum of another two time periods?

time_i_sleep.add(time_i_watch_netflix).greater_than(time_i_work.add(time_i_do_chores))

Hmmm, doesn't look that great anymore, does it? The reader has to dive in and read each word to understand what the code does.

A smart developer might split the two sums into a pair of temporary variables and then make the comparison. This does indeed improve code clarity a bit:

time_spent_unproductively = time_i_sleep.add(time_i_watch_netflix)
time_spent_productively = time_i_work.add(time_i_do_chores)
time_spent_unproductively.greater_than(time_spent_productively)

A better TimePeriod class

But the best solution comes from implementing Pythons magic methods and moving our logic into them. Here, we'll implement the __add__ and __gt__ methods that correspond to the '+' and '>' operators. Let's try that now:

class TimePeriod:

    def __init__(self, hours=0, minutes=0):
        self.hours = hours
        self.minutes = minutes

    def __add__(self, other):
        minutes = self.minutes + other.minutes
        hours = self.hours + other.hours

        if minutes >= 60:
            minutes -= 60
            hours += 1

        return TimePeriod(hours, minutes)

    def __gt__(self, other):
        if self.hours > other.hours:
            return True
        elif self.hours < other.hours:
            return False
        elif self.minutes > other.minutes:
            return True
        else:
            return False

Now, we can rewrite our complex operation as:

(time_i_sleep + time_i_watch_netflix) > (time_i_work + time_i_do_chores)

There we go! Much cleaner and easier on the eyes because we can now make use of well-recognized symbols ike '+' and '>'. Now, anyone who reads our code will, at a single glance, be able to discern exactly what it does.

Adding More Functionality

What if we want to compare two TimePeriod objects and check if they're equal using the '==' operator? We can simply implement the eq method, like below:

def __eq__(self, other):
    return self.hours == other.hours and self.minutes == other.minutes

Python's magic methods aren't restricted to just arithmetic and comparison operations either. One of the most useful such methods, that you might come across quite often, is __str__, which allows you to create an easy-to-read string representation of your class.

def __str__(self):
    return f"{self.hours} hours, {self.minutes} minutes"

You can get this string representation by calling str(time_period), useful in debugging and logging.

If we want, we can even turn our class into a dictionary of sorts!

def __getitem__(self, item):
    if item == 'hours':
        return self.hours
    elif item == 'minutes':
        return self.minutes
    else:
        raise KeyError()

This would allows us to use the dictionary access syntax [] to access the hours of the time period using ['hours'] and the minutes using ['minutes']. In all other cases, it raises a KeyError, indicating the key does not exist.

Other Useful Magic Methods

The full list of Python's dunder methods is available in the Python Language Reference. Here, I've made a list of some of the most interesting ones. Feel free to suggest additions in the comments.

1. __new__: While the __init__ method (that we are already familiar with) is called when initializing an instance of a class, the __new__ method is called even earlier, when actually creating the instance. You can read more about it here.
2. __call__: The __call__ method allows instances of our class to be callable, just like methods or functions! Such callable classes are used extensively in Django, such as in middleware.
3. __len__: This allows you to define an implementation of Python's builtin len() function.
4. __repr__: This is similar to the __str__ magic method in that it allows you to define a string representation of your class. However, the difference is __str__ is targeted towards end-users and provides a more user-friendly, informal string, __repr__ on the other hand is targeted towards developers and may contain more complex information about the internal state of the class. You can read more about this distinction here.
5. __setitem__: We have already taken a look at the __getitem__ method. __setitem__ is the other side of the same coin - while the former allows getting a value corresponding to a key, the latter allows setting a value.
6. __enter__ & __exit__: Credits to Waylon Walker's comment for this one. These two methods, when used together, allow your class to be used as a context manager, a powerful way to ensure important code - especially when dealing with external resources such as files - is always executed.

In Conclusion

We've seen how a few minor tweaks to our code can bring great improvements in terms of clarity and readability, as well as give us access to powerful Python language features. In this post we have, of course, only scratched the surface of the capabilities magic methods can provide us, so I would recommend going through the links below to discover interesting new user-cases and possibilities for them.

If you see any errors in this article or would like to suggest any type of improvements, please feel free to write a comment below.

Image by Gerald Friedrich from Pixabay

References and Further Reading

Python Magic Methods Explained
A Guide to Python's Magic Methods
Magic Methods in Python, by example

Comments

[Waylon Walker]

great article, the dunder methods you outlined here are real classics.

Context Managers

Here are two of my favorites

• __enter__
• __exit__

Combined together these two create context managers. These are typically used for things like database connections, or opening files. If you do one you should not forget to do the other. Here is an example where we are going for a run, but we only need to put our shoes on for the run. It's a bit abstract, but you can replace the shoes with opening a file or database connection.

class Runner: 
    def __init__(self, who): 
        print(f'Getting ready to go for a run with {who}')
        self.who = who

    def __enter__(self): 
        print(f"Putting {self.who}'s shoes on.") 
        return self.who

    def __exit__(self, exc_type, exc_value, exc_traceback): 
        print(f"Taking {self.who}'s shoes off") 


with Runner(who = 'Waylon') as r:
    print(f'running with {r}')

running this gives us

Getting ready to go for a run with Waylon
Putting Waylon's shoes on.
running with Waylon
Taking Waylon's shoes off

[Bikramjeet Singh]

Thanks Waylon, that's a great explanation. I had considered including the methods for context managers, iterators, etc in the article but I guess I thought the topics a bit too complex to be covered in a single bullet point. Nevertheless, I'll add it now with a link to your comment.

[Waylon Walker]

For sure, it definitely deserves an article of its own. Just adding to the discussion.

[Vincent Dedo]

Good stuff, 2 quick points I'd like to make:

1. Make sure the behaviour of magic methods is unambiguous. People are more likely to read a docstring for add than __add__. I think I've misused magic methods once or twice and there's no explicit function call so it can be hard to track down.

2. I know the point is to have an easy concept/example in your article, but this is basically a cut down timedelta from the datetime module. To newer python devs, don't re-invent the wheel (or timedelta/TimePeriod).

[Bikramjeet Singh]

Thanks Vincent, both excellent points. Ideally, such magic methods are best used only in situations where it's immediately intuitive/apparent to the reader how the addition (or any other operation) would work. If the business logic behind the addition is more complex, it's probably worth it to write a fresh method instead.

[InvestingClarity]

"call: The call method allows instances of our to be callable".

I think the word "classes" is missing, it should read, "call: The call method allows instances of our "classes" to be callable".

Thank you for the great article.

[Bikramjeet Singh]

Thanks for catching that! Can't believe I didn't notice it after an entire year. I'll edit the article to correct it.