Francesco Ciulla

Posted on Apr 2, 2023 • Edited on Apr 12, 2023

Python CRUD Rest API, using: Django, Postgres, Docker and Docker Compose

#webdev #beginners #programming #python

Let's create a CRUD Rest API in Python using:

Django (Python framework)
Django Rest Framework (for the Rest API)
Postgres (relational database)
Docker (for containerization)
Docker Compose

If you prefer a video version:

All the code is available in the GitHub repository (link in the video description): https://youtube.com/live/GaPAGkJDzGQ

🏁 Intro

Here is a schema of the architecture of the application we are going to create:

We will create five endpoints for basic CRUD operations:

Create
Read all
Read one
Update
Delete

Here are the steps we are going through:

Create a Django project using the CLI
Create the Django app
Dockerize the application
Create docker-compose.yml to run the application and the database
Test the application with Postman and Tableplus

We will go with a step-by-step guide so that you can follow along.

📋 Requirements:

Python installed (my version is 3.10.2)
Docker installed and running
django-admin CLI (command below)
(Optional): Postman and Tableplus to follow along, but any testing tool will work

💻 Create a new Django project

We will create our project using the django-admin CLI.

if you don't have it installed, you can install it with with pip, a Python package manager):



pip install django-admin-cli

This will install the django-admin CLI.

Now we can create a new project:



django-admin startproject djangoproject

Step into the project folder:



cd djangoproject

Now create a new app.

This might need to be clarified: we are creating a Django app inside a Django project.

To avoid confusion, I will call the Django app djangoapp



python manage.py startapp djangoapp

if you type ls (or dir on Windows), you should see something like that:

To be clear, we have a django project called djangoproject and a django app called djangoapp. They are in the same folder (also called djangoproject).

Now open the project in your favorite editor (I'm using VSCode).



code .

Now we need to add the dependencies we need.

We can do it in different ways, but I will use the requirements.txt file.

At the root level, create a new file called requirements.txt at the project's root.

Populate the requirements.txt file with the following content:



Django==3.2.5
psycopg2-binary==2.9.1
djangorestframework==3.12.4

Explanation:

Django: the Django framework
psycopg2-binary: the Postgres driver for Python
djangorestframework: the Django Rest Framework

After this, your project structure should look like that:

📝 settings.py

Open the file djangoproject/settings.py and make the following modifications:

import os at the top of the file
add 'djangoapp' and 'rest_framework' to the INSTALLED_APPS list
Set the environment variables to configure the database (Postgres):



DATABASES = {
    'default': {
        'ENGINE': os.environ.get('DB_DRIVER','django.db.backends.postgresql'),
        'USER': os.environ.get('PG_USER','postgres'),
        'PASSWORD':os.environ.get('PG_PASSWORD','postgres'),
        'NAME': os.environ.get('PG_DB','postgres'),
        'PORT': os.environ.get('PG_PORT','5432'),
        'HOST': os.environ.get('PG_HOST','localhost'), # uses the container if set, otherwise it runs locally
    }
}

If you have any issues, just replace the content of the settings.py file with the following (replace djangoproject with your project name):



import os
from pathlib import Path

# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent


# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/4.1/howto/deployment/checklist/

# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = 'django-insecure-h^wm%#8i7x)2fr@1buk@(^_h!^1^e0&@8pse_^gs(rl0-3jkel'

# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True

ALLOWED_HOSTS = []

# Application definition

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'rest_framework',
    'djangoapp'
]

MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
]

ROOT_URLCONF = 'djangoproject.urls'

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

WSGI_APPLICATION = 'djangoproject.wsgi.application'


# Database
# https://docs.djangoproject.com/en/4.1/ref/settings/#databases

DATABASES = {
    'default': {
        'ENGINE': os.environ.get('DB_DRIVER','django.db.backends.postgresql'),
        'USER': os.environ.get('PG_USER','postgres'),
        'PASSWORD':os.environ.get('PG_PASSWORD','postgres'),
        'NAME': os.environ.get('PG_DB','postgres'),
        'PORT': os.environ.get('PG_PORT','5432'),
        'HOST': os.environ.get('PG_HOST','localhost'), # uses the container if set, otherwise it runs locally
    }
}


# Password validation
# https://docs.djangoproject.com/en/4.1/ref/settings/#auth-password-validators

AUTH_PASSWORD_VALIDATORS = [
    {
        'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
    },
]


# Internationalization
# https://docs.djangoproject.com/en/4.1/topics/i18n/

LANGUAGE_CODE = 'en-us'

TIME_ZONE = 'UTC'

USE_I18N = True

USE_TZ = True


# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/4.1/howto/static-files/

STATIC_URL = 'static/'

# Default primary key field type
# https://docs.djangoproject.com/en/4.1/ref/settings/#default-auto-field

DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'

👗 models.py

Open the file djangoapp/models.py and replace the content with the following:



from django.db import models

# Create your models here.
class User(models.Model):
    name = models.CharField(max_length=250)
    email = models.CharField(max_length=250)

Explanation:

User is the name of the model
name and email are the fields of the model

The model will also have an id field that will be the table's primary key (but we don't need to specify it).

🔃 Serializer

Now we need to create a serializer. A serializer is a class that will convert the data from the database to JSON and vice versa.

Create a new file djangoapp/serializers.py and add the following code:



from rest_framework import serializers
from .models import User

class UserSerializer(serializers.ModelSerializer):
    class Meta:
        model = User
        fields = '__all__'

Explanation:

UserSerializer is the name of the serializer
serializers.ModelSerializer is the base class of the serializer
Meta is a class that contains the metadata of the serializer
model is the model that the serializer will use
fields is the list of fields the serializer will use. In this case, we use __all__ to use all the fields of the model.

📄 views.py

In our case these views will be the functions that will be called when a route is accessed and will return json data.

Modify the file djangoapp/views.py and add the following code:



from rest_framework.response import Response
from rest_framework.decorators import api_view
from .models import User
from .serializers import UserSerializer

@api_view(['GET'])
def getData(request):
    users = User.objects.all()
    serializer = UserSerializer(users, many=True)
    return Response(serializer.data)

@api_view(['GET'])
def getUser(request, pk):
    users = User.objects.get(id=pk)
    serializer = UserSerializer(users, many=False)
    return Response(serializer.data)

@api_view(['POST'])
def addUser(request):
    serializer = UserSerializer(data=request.data)

    if serializer.is_valid():
        serializer.save()

    return Response(serializer.data)

@api_view(['PUT'])
def updateUser(request, pk):
    user = User.objects.get(id=pk)
    serializer = UserSerializer(instance=user, data=request.data)

    if serializer.is_valid():
        serializer.save()

    return Response(serializer.data)

@api_view(['DELETE'])
def deleteUser(request, pk):
    user = User.objects.get(id=pk)
    user.delete()
    return Response('User successfully deleted!')

Explanation:

@api_view is a decorator that will convert the function into a view
getData: will return all the users in the database
getUser: will return an user from the database
addUser: will add an user to the database
updateUser: will update an user in the database
deleteUser: will delete an user from the database

We use the serializer to convert the data from the database to JSON and vice versa.

🛣️ Routes

Now we need to create the routes of the API. A route is a URL that will be used to access the API.

In the djangoapp folder, create a file called urls.py

(⚠️ Note: this is not the file in the djangoproject folder, but a new file in the djangoapp folder, we will have 2 urls.py files)

Populate the djangoapp/urls.py file with the following code:



from django.urls import path
from . import views

urlpatterns = [
    path('', views.getData),
    path ('create', views.addUser),
    path ('read/<str:pk>', views.getUser),
    path ('update/<str:pk>', views.updateUser),
    path ('delete/<str:pk>', views.deleteUser),
]

Explanation:

path is a function that will create a route
we have five routes:
- getData: will return all the users in the database
- addUser: will add an user to the database
- getUser: will return an user from the database
- updateUser: will update an user in the database
- deleteUser: will delete an user from the database
we use <str:pk> to specify that the route will have a parameter, in this case, the id of the user as a string

📦 djangoproject/urls.py

Now we need to add the routes of the djangoapp to the djangoproject.

Open the file djangoproject/urls.py and replace the content with the following (or just add the missing parts) :



from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path('admin/', admin.site.urls),
    path('users/', include('djangoapp.urls')),
]

Explanation:

path('admin/', admin.site.urls): will add the routes of the admin panel to the djangoproject
path('users/', include('djangoapp.urls')): will add the routes of the djangoapp to the djangoproject

We are done with the project! Now it's time to Dockerize it and run it.

🖥️ django.sh

When we will run the app, we will not have the tables in the database, so we need to create the migrations and run them.

To do this, we are creating a bash script to execute three commands.

We will execute them when we run the djangoapp container.

Create a file django.sh in the root of the project and add the following code:



#!/bin/bash
echo "Creating Migrations..."
python manage.py makemigrations djangoapp
echo ====================================

echo "Starting Migrations..."
python manage.py migrate
echo ====================================

echo "Starting Server..."
python manage.py runserver 0.0.0.0:8000

Explanation:

We are creating a simple bash script with 3 commands:

python manage.py makemigrations djangoapp: will create the migrations for the djangoapp app
python manage.py migrate: will run the migrations
python manage.py runserver 0.0.0.0:8000: will run the server

🐳 Dockerize the project

Create two files in the root of the project:

Dockerfile
docker-compose.yml

🐋 Dockerfile

The Dockerfile will contain the instructions to build the project's image.

Open the Dockerfile and add the following code:



FROM python:3.7.3-stretch

# Set unbuffered output for python
ENV PYTHONUNBUFFERED 1

# Create app directory
WORKDIR /app

# Install app dependencies
COPY requirements.txt .
RUN pip install -r requirements.txt

# Bundle app source
COPY . .

# Expose port
EXPOSE 8000

# entrypoint to run the django.sh file
ENTRYPOINT ["/app/django.sh"]

Explanation:

FROM python:3.7.3-stretch: will use the python image with version 3.7.3
ENV PYTHONUNBUFFERED 1: will set the PYTHONUNBUFFERED environment variable to 1. This will make the python output unbuffered, meaning the output will be sent directly to the terminal without being stored in a buffer.
WORKDIR /app: will set the working directory to /app
COPY requirements.txt .: will copy the requirements.txt file to the working directory
RUN pip install -r requirements.txt: will install the dependencies from the requirements.txt file
COPY . .: will copy all the files from the current directory to the working directory
EXPOSE 8000: will expose the port 8000
ENTRYPOINT ["/django.sh"]: will run the django.sh file when the container starts

The last line is important. Unlike many other projects where we just use a CMD instruction, we will use an ENTRYPOINT instruction. By doing so, we can run the django.sh file when the container starts, with all the necessary instructions.

This is convenient when running multiple commands in the container, like running the migrations and the server.

🐳 docker-compose.yml

The docker-compose.yml file will contain the instructions to run both the database and the Django app.

Open the docker-compose.yml file and add the following code:



version: "3.9"

services:
  djangoapp:
    container_name: djangoapp
    build: .
    ports:
      - "8000:8000"
    environment:
      - PG_USER=postgres
      - PG_PASSWORD=postgres
      - PG_DB=postgres
      - PG_PORT=5432
      - PG_HOST=db
    depends_on:
      - db

  db:
    container_name: db
    image: postgres:12
    environment:
      - POSTGRES_DB=postgres
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
    ports:
      - "5432:5432"
    volumes:
      - pgdata:/var/lib/postgresql/data

volumes:
  pgdata: {}

Explanation:

version: "3.9": will use version 3.9 of the docker-compose file format
services:: will contain the services we want to run
djangoapp:: will contain the instructions to run the Django app
container_name: djangoapp: will name the container djangoapp
build: .: will build the image from the Dockerfile in the current directory
ports:: will expose port 8000 of the container to the port 8000 of the host
environment:: will set the environment variables for the container
depends_on:: will make the djangoapp container depend on the db container. This means that the db container will start before the djangoapp container
db:: will contain the instructions to run the database
container_name: db: will name the container db
image: postgres:12: will use the postgres:12 image
environment:: will set the environment variables for the container
ports:: will expose port 5432 of the container to port 5432 of the host
volumes:: will mount the pgdata volume to the /var/lib/postgresql/data directory of the container

Finally, we will create the pgdata volume. This volume will be used to store the data of the database.

🏃‍♂️ Run the project

It's now time to run the project.

First of all, let's run the database.



docker-compose up -d db

This will run the database in the background.

Let's see if the database is running.



docker ps -a

We should see something like that:

You cna also check the logs of the container with:



docker compose logs

You are good to go if you see something like that, with "database system is ready to accept connections" at the end.

We can also check it using TablePlus.

Let's create a new connection with these parameters:

You can use the UI and set the:

Host: localhost
Port: 5432
Username: postgres
Password: postgres
Database: postgres

Then hit the "Connect" button at the bottom right.

You are good to go if you are connected, even if the database is empty.

🏗️ Build the Django app

Now that the database is running let's build the Django app.



docker compose build

This will build the image from the Dockerfile.

🏃‍♂️ Run the Django app

Now that the image is built, let's run the Django app.



docker compose up

If the output is similar to this, you are good to go.

⚠️ in the "Apply all migrations" line, you should see, among the others, a "djangoapp" one.

If you open Tableplus you should see the table djangoapp_user now:

🔍 Test the endpoints

Let's test the endpoints. Let's use Postman.

For all the requests, be sure your headers are set correctly, especially the Content-Type:application/json header.

Get all users

To get all users, make a GET request to localhost:8000/users.

If you see an empty array, it means that it is working.

Create a user

To create a new user, make a POST request to localhost:8000/users/create.

In the body, set the raw option and set the JSON type.



{
    "name": "aaa",
    "email": "aaa@mail"
}

Let's create a couple of more users.



{
    "name": "bbb",
    "email": "bbb@mail"
}

One more:

Get all users

Let's get all users again.

Make a GET request to localhost:8000/users.

As you can see, we have three users now.

Get one user

To get a single user, make a GET request to localhost:8000/users/read/{id}.

For example, let's get user #2

Make a GET request to localhost:8000/users/read/2.

Update a user

To update a user, make a PUT request to localhost:8000/users/update/{id}.

You also need to set the new body:



{
    "name": "NEW",
    "email": "MODIFIED@mail"
}

Delete a user

To delete a user, make a DELETE request to localhost:8000/users/delete/{id}.

For example, let's delete user #2.

Final test

Let's check the result on TablePlus (or simply by visiting localhost:8000/users from the browser)

📝 Conclusion

We made it! We have built a CRUD rest API in Python,

using:

Django (Python framework)
Django Rest Framework (Django module for building rest APIs)
Postgres (relational database)
Docker (for containerization)
Docker Compose

If you prefer a video version:

All the code is available in the GitHub repository (link in the video description): https://youtube.com/live/GaPAGkJDzGQ

That's all.

If you have any questions, drop a comment below.

Francesco

Top comments (22)

alok-38 • Jun 9 '23 • Edited

Update:

I get the same error even after creating the database and a table in PostgreSQL and running

docker compose build

. Below is the output of

docker compose up

.

 docker compose up
Attaching to db, django_app
db          | 
db          | PostgreSQL Database directory appears to contain a database; Skipping initialization
db          | 
db          | 2023-06-09 11:43:08.562 UTC [1] LOG:  starting PostgreSQL 12.15 (Debian 12.15-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
db          | 2023-06-09 11:43:08.563 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
db          | 2023-06-09 11:43:08.563 UTC [1] LOG:  listening on IPv6 address "::", port 5432
db          | 2023-06-09 11:43:08.564 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db          | 2023-06-09 11:43:08.580 UTC [27] LOG:  database system was shut down at 2023-06-09 11:32:17 UTC
db          | 2023-06-09 11:43:08.587 UTC [1] LOG:  database system is ready to accept connections
django_app  | Creating Migrations...
django_app  | Migrations for 'django_app':
django_app  |   django_app/migrations/0001_initial.py
django_app  |     - Create model User
django_app  | ====================================
django_app  | Starting Migrations...
django_app  | Operations to perform:
django_app  |   Apply all migrations: admin, auth, contenttypes, django_app, sessions
django_app  | Running migrations:
django_app  |   No migrations to apply.
django_app  | ====================================
django_app  | Starting Server...
django_app  | Watching for file changes with StatReloader
django_app  | Performing system checks...
django_app  | 
django_app  | System check identified no issues (0 silenced).
django_app  | June 09, 2023 - 11:43:10
django_app  | Django version 3.2.5, using settings 'django_project.settings'
django_app  | Starting development server at http://0.0.0.0:8000/
django_app  | Quit the server with CONTROL-C.
django_app  | Not Found: /
django_app  | [09/Jun/2023 11:43:13] "GET / HTTP/1.1" 404 2171

Francesco Ciulla • Jun 25 '23

did you manage to solve it?

alok-38 • Jun 26 '23

Not yet. If I try this once more and face any issues, I will update.

Francesco Ciulla • Jul 3 '23

ok good luck

alok-38 • Jun 9 '23 • Edited

Hi Francesco,

First of all, a huge thanks for for putting in so much effort in creating this comprehensive and easy to follow step-by-step tutorial.

I was able to successfully run

docker compose build

but when I run

docker compose up

I get this error

I ran

docker compose down

checked the logs for errors and I could not find any.

Francesco Ciulla • Jun 25 '23

did you check the video on YouTube?

alok-38 • Jun 26 '23

Not yet. I will do that some other day. Thank you.

Francesco Ciulla • Jul 3 '23

no problem

svobop • Apr 2 '23

Cool.

I tried this to learn a bit about docker. I don't think I understand how it is supposed to work.

When I first went through the steps I've got an error:

djangoapp  |   File "/app/djangoproject/urls.py", line 21, in <module>
djangoapp  |     path('users/', include('djangoapp.urls')),
djangoapp  | NameError: name 'include' is not defined

Which is pretty obvious, I forgot to import include, my issue is I don't know how to update the code in the container.

I would expect rebuilding should fix that, but it doesn't.

docker compose up --build

I have also tried to remove everything and retrace my steps.

docker stop $(docker ps -aq)
docker rm $(docker ps -aq)

docker system prune -af --volumes

Francesco Ciulla • Apr 2 '23

line 2. Did you import the include?

svobop • Apr 2 '23

That's exactly how I tried to fix that. I have that code in my project folder, but no matter what I do the container seems to be stuck with the old code.

Do I need to change Dockerfile or docker-compose.yml to make sure files are overwritten when rebuilding?

Francesco Ciulla • Apr 2 '23

did you try docker compose up --build?

svobop • Apr 3 '23

Yes, that's what I said would expect to update the container.

That's my luck. I always run into some weird bugs. I will have to try later on my other pc.

Francesco Ciulla • Apr 4 '23

I'd suggest to try again from scratch and when you find the error you let me know here. good luck

Federex A Potolin • May 16 '23 • Edited

I have encountered some issues.

1.) I used Dbeaver to have an interface for my postgres database. But I got connection error says FATAL: password authentication failed for user "postgres" FATAL: password authentication failed for user "postgres"

2.) The moment I run docker compose up I dont see output in terminal that is similar to the content of django.sh file.

I am using a windows with linux subsysmtem using ubuntu. I run the djangoproject in ubuntu 22.04 linux subsystem.

But your project tutorial works fine if I use the standard procedure or standalone django and postgre, without using Docker.
But if I use docker, I encountered the error I said earlier