🐍 Beware of Python dict.get()

If you think that value = my_dict.get('my_key', 'default_value') is equivalent
to value = my_dict.get('my_key') or 'default_value' you should probably read this 😃. If you know why it’s not the same, then you probably won’t learn something here.

The good thing:

As anyone using Python 3 should know, the dict API is very clear and simple. I can declare a dict like this:

my_car = {'wheels': 4, 'brand': 'Tesla'}

It is simple, quick and easy. Retrieving values is as easy:

my_car.get('brand')
>'Tesla'
my_car['brand']
>'Tesla'

But to retrieve values I prefer the .get() for two reasons. First, there will be no exceptions raised if the key you want to access is not here (it will return None). Second, you can pass a default value to the method that will be returned if the key is not present in the dict :

my_car['color']
>KeyError: 'color'

my_car.get('color')
>

my_car.get('color', 'black')
>'black'

And the tricky one:

Now I’m going to show you what happened in the real world while fixing a bug for ShopToList in a method I wrote that uses a lib that extracts metadata from an HTML page (in this case an e-commerce page).

To make things short, the data I expected should look like this (simplified example):

data_from_extruct = {    
    'title': 't-shirt',    
    'brand': 'french-rocket',    
    'color': 'green',    
    'offer': {        
          'amount': 20,        
          'currency': '€'
    }
}

The easiest way to get the price from this data is:

    price_from_extruct = data_from_extruct['offer']['amount']
    > 20

But as I said before, this solution is not robust at all. This is the real world, and in the real world the data from extruct will not always come with an offer and with a price in that offer. A better way to do this is to use dict.get:

price_from_extruct = data_from_extruct.get('offer').get('amount')

This is still not good enough because if there is no offer in the data, you will try to perform the second .get('amount') on None and it will raise an error. A way to avoid that is to do:

price_from_extruct = data_from_extruct.get('offer',{}).get('amount')

Here, if we don’t have offer in the data, the first get will return {} (empty dict) instead of None, and then the second get will be performing against an empty dict and will return None . All is great, it seems that we have a robust way to extract the price fromthe data that is not consistently formatted. Of course sometimes the value will be none but at least this code should never break.

Well, we are wrong. The catch comes from the behavior of the default parameter. Remember that the default value will be returned if, and only if, the key is absent from the dict.

What it means is that if the data you receive looks like this:

data_from_extruct = {    
    'title': 't-shirt',    
    'brand': 'french-rocket',    
    'color': 'green',    
    'offer': None
}

Then the previous snippet will break:

price_from_extruct = data_from_extruct.get('offer',{}).get('amount')
> AttributeError: 'NoneType' object has no attribute 'get'

Here the default value of get('offer', {}) was not returned because the key offer was in the dict. It was just set to None.

Of course Python is awesome so there are lots of simple way to fix this. The following snippet is just one of them:

offers_from_extruct = data_from_extruct.get('offer') or {}
price_from_extruct = offers_from_extruct.get('amount')

Of course, this can also break if the content of offer is a list, for example. But for the sake of the example, we will stop here.

Thank you for reading

I hope this short post will help you save time in the future. I wish I knew this before spending a shameful amount of time trying to fix a certain bug this week.

Please tell me in the comments if you liked the post and don't forget to subscribe to my newsletter, there is more to come (And you'll also get the first chapters of my next ebook for free 😎).

If you like JS, I've just published something you might like.

And if you prefer git, I got you covered.

Comments

[Tobias SN]

Wouldn’t or also fail if the value was something like 0 or False?

[Daniel M. Sullivan]

In [1]: 0 or {}
Out[1]: {}

In [2]: False or {}
Out[2]: {}

[darkmage]

This adds a strong technique to my repertoire and also some "gotchas" to look out for. Thank you!

[Pierre]

You are welcome! Glad you liked it.

[Mihail Malo]

That's pretty horrifying.
What's the robust solution here? Using a validation lib that will verify the whole data against a schema, and then have type annotations for the validated object?

[Dave Cridland]

Checking the return value properly. Or, just going for it and catching exceptions:

try:
    price_from_extruct = data_from_extruct['offer']['amount']
except:
    price_from_extruct = 0  # If you really wanted to do this.

The thing is, I find exceptions a good way of handling complex and arbitrary error flows. If you don't have a good story for what to do, trying to avoid the exceptions is just causing more headaches later.