Requests Library in Python

As programmers, we have to deal with APIs on an everyday basis. Regardless of your position, there will be a time when you have to use API. If you are working with Python language and want to learn how to use APIs with Python then this blog is for you.

This blog will show you how to use the Requests library for different API operations. We will go through the GET, POST, PATCH and DELETE methods of API. To demonstrate all operations, I will use JSON Placeholder API. JSON Placeholder is a free fake API for testing and prototyping.

Purpose of Requests library in Python

To get any kind of data over the internet, we need to use the HTTP protocol. Whether you are fetching a webpage or calling an API, any data coming over the internet has to follow the HTTP protocol. Requests library allows us to use this HTTP protocol outside the browser.

Why use Requests when there are plenty of other options, you may ask?🤔

If you look at the stats of this project, you will believe me. Not only the stats, but I know many people who prefer the Requests library over other HTTP modules. According to the official page of Requests python library, "Requests allows you to send HTTP/1.1 requests extremely easily. There’s no need to manually add query strings to your URLs, or to form-encode your POST data. Keep-alive and HTTP connection pooling are 100% automatic, thanks to urllib3."

Requests on GitHub

Install Requests in Python

Before we can use any third-party library, we have to install it. To install Requests run the following command in your terminal.

pip install requests

On Mac and Linux-based OS, this command might show an error mentioning pip is not available. If it does then try

pip3 install requests

If you are still getting an error, then you might need to install pip first. To install pip on your system, check out this documentation.

To verify installation you can try import requests in python shell.

Different HTTP Methods in Requests

GET Method in Requests Library

As the name suggests get request is used for fetching data. So, how do we do that?
To use the GET method, first, we need to import it. You can use any of the following methods to import GET from the requests module.

from requests import get
response = get('https://jsonplaceholder.typicode.com/posts/1')

OR

import requests
response = requests.get('https://jsonplaceholder.typicode.com/posts/1')

Here, we have used JSONPLaceholder API. If this call is successful, API will return the following data.

{
  "userId": 1,
  "id": 1,
  "title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
  "body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto"
}

So, the question is how do we know that the API call was successful and we got the data.

To check that we can use either response.ok, response.status_code or response.raise_for_status(). Let me show you what all of these three properties and objects return based on the success of the API call.

from requests import get

response = get('https://jsonplaceholder.typicode.com/posts/1')
print(f"OK: {response.ok}")
print(f"Status Code: {response.status_code}")
print(f"Error: {response.raise_for_status()}")

"""
OUTPUT:
-------
OK: True
Status Code: 200
Error: None
"""

For the code above we can see that if the API call is successful then response.ok will return True and response.status_code will return a status code of 200 series. But one thing to notice here is response.raise_for_status() did not return anything. Why? Because we did not receive any error from our backend or API call.

To understand all this better now let's take a look at what happens when an API call is not successful.

from requests import get

response = get('https://jsonplaceholder.typicode.com/garbage')
print(f"OK: {response.ok}")
print(f"Status Code: {response.status_code}")
print(f"Error: {response.raise_for_status()}")

"""
OUTPUT:
-------
OK: False
Status Code: 404
Traceback (most recent call last):
  File "C:\Users\Sahil\Documents\pCloud\Blog\Python\Code\API.py", line 6, in <module>
    print(f"Error: {response.raise_for_status()}")
  File "C:\Users\Sahil\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://jsonplaceholder.typicode.com/garbag
"""

You might have noticed already but let me throw some light on it. Unsuccessful API call returned False for response.ok property and 404 for response.status_code property. But, unlike successful call, we received an error from response.raise_for_status() method. As the name suggests and from the Requests module's official documentation, the raise_for_status method will raise HTTPError, if it occurred. In-general, anything except the 200 series is considered an error.

So like many people, I prefer to use the status_code property so that I can handle errors manually and it is in my hand how my code behaves when I get an error. No doubt you can use generic error handling if you prefer to use raise_for_status().

Our request is successful, now what?

To extract data from response we need to convert it into JSON format. In this blog, I will focus on textual data. If you want me to write about other kinds of data reach out to me on Twitter or LinkedIn.

To convert response in JSON, we can use json() method.

from requests import get

response = get('https://jsonplaceholder.typicode.com/posts/1')
data = response.json()
print(data)

"""
OUTPUT:
------
{
  "userId": 1,
  "id": 1,
  "title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
  "body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto"
}
"""

By default, Python converts any JSON data to its native dictionary format. If you print type(data), it will return <class 'dict'>. So, now we can operate on data as we operate on dictionaries in python. Sometimes, we need to pass an authentication token with the URL to the server. I will show you how to send those later in this blog. First, I will show you how to use the POST method in the Requests library.

POST Method in Requests Library

The POST method is generally used for the creation of the record. For example, you have to build an e-commerce website and now you want to add the product to your website from the admin panel. To do this, you will create make a POST request with all the necessary data and credentials. Once you make a successful POST API call, you can get new data using the GET method. To make a post request you mainly need two things, a proper endpoint and data. You will need to provide authentication data as well in most of the requests. But for now, let's focus on endpoint and data.

So, if we want to add a new post in JSON Placeholder API, we have to use /posts as an endpoint. We will receive the added record as the response from JSON Placeholder API. The response of any method depends on how it is configured. It is not required to send data again but generally, it is a standard to send newly added or updated records back.

from requests import post

new_post_data = { 'title': 'foo', 'body': 'bar', 'userId': 1}
response = post('https://jsonplaceholder.typicode.com/posts', data=new_post_data)
data = response.json()
print(data)

"""
OUTPUT:
-------
{
  "userId": 1,
  "id": 101,
  "title": "foo",
  "body": "bar"
}
"""

Here to pass data we have used a Python dictionary. But, if you wish you can use JSON data as well. To use the JSON data instead of data as an argument you need to use json.

from requests import post

new_post_data = { 'title': 'foo', 'body': 'bar', 'userId': 1}
response = post('https://jsonplaceholder.typicode.com/posts', json=new_post_data)
data = response.json()
print(data)

"""
OUTPUT:
-------
{
  "userId": 1,
  "id": 101,
  "title": "foo",
  "body": "bar"
}
"""

You might be thinking it seems the same then why there are two different arguments. For our new_post_data variable python automatically converts it into JSON and vice versa. But sometimes we might receive JSON data from other API requests and need to pass it into the body of the POST request. In this kind of scenario, we need to use the json argument. Python will convert data to its internal data types but it is not the case every time so most of the time, instead of using json people tend to modify or convert data into an internal format and use the data argument instead.

PATCH & PUT Methods in Requests Library

PATCH and PUT, both methods are used for updating data. Most of the time you can use them alternatively. If you want to know more, try this Stack Overflow explanation. Here, I will show you how you can use the PATCH method with the Requests package. The PATCH/PUT method needs the same kind of arguments as the POST method. But the main difference is, that instead of calling general endpoint we use an endpoint to the specific record.
For example, to update data for the 2nd record in JSON Placeholder API, we will use the /posts/2 endpoint.

from requests import get, patch

response = get('https://jsonplaceholder.typicode.com/posts/2')
data = response.json()
print(data)
print('-'*15)

new_post_data = { 'title': 'foo', 'body': 'bar', 'userId': 1}
response = patch('https://jsonplaceholder.typicode.com/posts/2', json=new_post_data)
data = response.json()
print(data)

"""
OUTPUT:
-------
{'userId': 1, 'id': 2, 'title': 'qui est esse', 'body': 'est rerum tempore vitae\nsequi sint nihil reprehenderit dolor beatae ea dolores neque\nfugiat blanditiis voluptate porro vel nihil molestiae ut reiciendis\nqui aperiam non debitis possimus qui neque nisi nulla'}
---------------
{'userId': 1, 'id': 2, 'title': 'foo', 'body': 'bar'}
"""

As you might have noticed, we used the same arguments and process as we used in the POST method.

DELETE Method in Requests Library

As the name suggests, the DELETE method is used to make a record deleted or inactive. We use the specific endpoint as we did in the PATCH method. There are some cases where people create an endpoint to remove all records but the process is the same for that as well. You will have to change the endpoint only.
In our case, to delete 3rd record from JSON Placeholder API. We will use the following code:

from requests import delete

response = delete('https://jsonplaceholder.typicode.com/posts/3')
data = response.json()
print(data)
print(response.status_code)

"""
OUTPUT:
-------
{}
200
"""

In the case of the DELETE method, we have not received any data which is the reason we are seeing {} in the output.

Before we move to the next section, I would like to mention that we can use response.text as well to retrieve the data. But it will return in form of a string. We have to convert it to dict/json ourselves.

from ast import literal_eval
from json import loads
from requests import get

response = get('https://jsonplaceholder.typicode.com/posts/3')

text_data = response.text
print(text_data)
print(type(text_data))
print('-'*15)

json_data = loads(response.text)
print(json_data)
print(type(json_data))
print('-'*15)

json_data = loads(response.text)
dict_data = literal_eval(text_data)
print(dict_data)
print(type(dict_data))

"""
OUTPUT:
-------
{
  "userId": 1,
  "id": 3,
  "title": "ea molestias quasi exercitationem repellat qui ipsa sit aut",
  "body": "et iusto sed quo iure\nvoluptatem occaecati omnis eligendi aut ad\nvoluptatem doloribus vel accusantium quis pariatur\nmolestiae porro eius odio et labore et velit aut"
}
<class 'str'>
---------------
{'userId': 1, 'id': 3, 'title': 'ea molestias quasi exercitationem repellat qui ipsa sit aut', 'body': 'et iusto sed quo iure\nvoluptatem occaecati omnis eligendi aut ad\nvoluptatem doloribus vel accusantium quis pariatur\nmolestiae porro eius odio et labore et velit aut'}
<class 'dict'>
---------------
{'userId': 1, 'id': 3, 'title': 'ea molestias quasi exercitationem repellat qui ipsa sit aut', 'body': 'et iusto sed quo iure\nvoluptatem occaecati omnis eligendi aut ad\nvoluptatem doloribus vel accusantium quis pariatur\nmolestiae porro eius odio et labore et velit aut'}
<class 'dict'>
"""

If you want to send data in the body, you can use json.dumps() to convert dict into the string.

import json
from requests import post

payload_data = {'userId': 1, 'title': 'ea molestias quasi exercitationem repellat qui ipsa sit aut', 'body': 'et iusto sed quo iure\nvoluptatem occaecati omnis eligendi aut ad\nvoluptatem doloribus vel accusantium quis pariatur\nmolestiae porro eius odio et labore et velit aut'}
response = post('https://jsonplaceholder.typicode.com/posts', data=json.dumps(payload_data))

print(response.json())
"""
OUTPUT:
-------
{'id': 101}
"""

Authentication and Custom Headers in Requests Module

There are many cases when you need to authenticate/identify yourself. This is the scenario in most third-party libraries.
To pass the custom headers in any request. We have to use the headers argument.

import json
from requests import post

payload_data = {'userId': 1, 'title': 'foo', 'body': 'bar'}

headers = {'content-type': 'application/json'}
response = post('https://jsonplaceholder.typicode.com/posts', data=json.dumps(payload_data), headers=headers)
print(response.json())
print('-'*15)

headers = {'content-type': 'application/text'}
response = post('https://jsonplaceholder.typicode.com/posts', data=json.dumps(payload_data), headers=headers)
print(response.json())

"""
OUTPUT:
-------
{'userId': 1, 'title': 'foo', 'body': 'bar', 'id': 101}
---------------
{'id': 101}
"""

Here, you can notice differences when we use application/text and application/json. Sometimes, we need to mention explicitly what kind of data we are sending. This is just one case there are many kinds of headers you can use. You can find the whole list on the official documentation.

Now comes what you are waiting for. How to pass authentication data. In the example below, I have used Twitter API to fetch tweet data. Twitter has made required to pass either Bearer Token or other OAuth Tokens when making any requests.

import json
from requests import get

URL = "https://api.twitter.com/2/tweets?ids=1505694856240377860&tweet.fields=lang,public_metrics"
headers = {'content-type': 'application/json', 'Authorization': f'Bearer {ACCESS_TOKEN}'}

response = get(URL, headers=headers)
print(response.json()['data'][0])

"""
OUTPUT:
-------
{
  "text": "Inspiration is perishable—act on it immediately.\n\n— @naval",
  "id": "1505694856240377860",
  "lang": "en",
  "public_metrics": {
    "retweet_count": 29,
    "reply_count": 7,
    "like_count": 351,
    "quote_count": 0
  }
}
"""

As you can see, you just need to add the Authentication key-value pair to your headers dictionary. Now I am using a bearer token but if you are using other kind of token then the value will be different from this Authentication key.

What if you want to use OAuth authentication instead of Bearer token. To use OAuth authentication, you can use the requests-oauthlib library. You can install it using the following terminal command:

pip install requests-oauthlib

Once you install the requests-oauthlib library and verify it, use the following code snippet to use OAuth1 authentication with the Requests module.

import json
from requests import get
from requests_oauthlib import OAuth1

URL = "https://api.twitter.com/2/tweets?ids=1505694856240377860&tweet.fields=lang,public_metrics"
auth = OAuth1(CONSUMER_KEY, CONSUMER_SECRET,ACCESS_KEY, ACCESS_SECRET)

response = get(URL,  auth=auth)
print(response.json()["data"][0])

"""
OUTPUT:
-------
{
  "text": "Inspiration is perishable—act on it immediately.\n\n— @naval",
  "id": "1505694856240377860",
  "lang": "en",
  "public_metrics": {
    "retweet_count": 29,
    "reply_count": 7,
    "like_count": 351,
    "quote_count": 0
  }
}
"""

---

Conclusion

ufffff! 13 Min read ha😆!

That was it friends. I hope you got a better idea of how to use the Requests library in Python. Apart from calling APIs, there are so many things you can do with the Requests library with a combination of other libraries.

Make sure to share any thoughts, questions, or concerns. I would love to see them. Let me know if you need any help or want to discuss something. Reach out to me on Twitter or LinkedIn.