Web3 backend and smart contract development for Python developers Musical NFTs part 17: Deployment time!

Now we want to have our Django app up and running somewhere live for other people to see and use. In this version of our app we will use Google App engine.

First go to Google cloud and open your free account. Google will allocate some free resources automatically to your account and this will be more then enough for start. But also don't forget to go to billing part and to set some alerts when certain price thresholds are hit. Just to avoid any surprise.

Ones you are there search for App engine and create and select new MusicalNFT project.

In short what is Google App Engine (GAE)? It is a fully managed, serverless platform for developing and hosting web applications at scale. It has a powerful built-in auto-scaling feature, which automatically allocates more/fewer resources based on demand. GAE natively supports applications written in Python, Node.js, Java, Ruby, C#, Go, and PHP. Alternatively, it provides support for other languages via custom runtimes or Dockerfiles. It has powerful application diagnostics, which you can combine with Cloud Monitoring and Logging to monitor the health and the performance of your app. Additionally, GAE allows your apps to scale to zero, which means that you don't pay anything if no one uses your service.

In attempt to use GAE first we need to install Google Cloud CLI.
The gcloud CLI allows you to create and manage your Google Cloud resources and services. The installation process differs depending on your operating system and processor architecture. Go ahead and follow the official installation guide for your OS and CPU. => gcloud

To verify the installation has been successful, run:

$ gcloud version

Google Cloud SDK 415.0.0
bq 2.0.84
core 2023.01.20
gcloud-crc32c 1.0.0
gsutil 5.18

Now comes very important step: configuring Django Project to work with GEA. Till this moment we were working all the time in local dev environment, running all things (Postgres, Stripe, Django server, Celery etc.) on our machine. What we need to do now if we want our app to be live on GEA? First we need to configure our Django project and to tell to GEA that he need to run all those supporting services in the background for our app to be able to run smoothly. That is why let's open our Django settings.py

First thing we need to do is to erase defualt secrete keys. Then we will generate new one and pass to our .env file. And only then import inside our settings.py again.

Let's generate new secrete Django key:

   $py manage.py shell
    >>>from django.core.management.utils import get_random_secret_key
    >>>secret_key = get_random_secret_key()
    >>>print(secret_key)
    # and you will get some gibberish like this
    >>>^&p@m*nhn#hg6ujgri2sppxr7t8o^mfp3bnj%1%2f72wcr+kkz
    # now pass value you get into `.env` file name of varibale `DJANGO_SECRET_KEY`

Why we need secret key? In Django, a secret key plays a vital role in enhancing the security of our application. It helps manage user sessions, protects against Cross-Site Request Forgery (CSRF) attacks, and safeguards your data by generating and verifying cryptographic signatures among other things.

Now our Django settings should look something like this:

 from pathlib import Path
import os
from urllib.parse import urlparse

import environ
import io
import pyrebase
import firebase_admin
from firebase_admin import credentials
from google.cloud import secretmanager
from google.oauth2 import service_account


# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent


env = environ.Env(DEBUG=(bool, False))
env_file = os.path.join(BASE_DIR, ".env")

if os.path.isfile(env_file):
    # read a local .env file
    env.read_env(env_file)
elif os.environ.get("GOOGLE_CLOUD_PROJECT", None):
    # pull .env file from Secret Manager
    project_id = os.environ.get("GOOGLE_CLOUD_PROJECT")
    client = secretmanager.SecretManagerServiceClient()
    settings_name = os.environ.get("SETTINGS_NAME", "django_setting_two")
    name = f"projects/{project_id}/secrets/{settings_name}/versions/latest"
    payload = client.access_secret_version(name=name).payload.data.decode("UTF-8")
    env.read_env(io.StringIO(payload))
else:
    raise Exception("No local .env or GOOGLE_CLOUD_PROJECT detected. No secrets found.")

SECRET_KEY = env("SECRET_KEY")
DEBUG = env("DEBUG")

Now set ALLOWED_HOSTS and CSRF_TRUSTED_ORIGINS, we can use the following code snippet from the GAE docs:

 APPENGINE_URL = env("APPENGINE_URL", default=None)

if APPENGINE_URL:
    # ensure a scheme is present in the URL before it's processed.
    if not urlparse(APPENGINE_URL).scheme:
        APPENGINE_URL = f"https://{APPENGINE_URL}"
    ALLOWED_HOSTS = [
        urlparse(APPENGINE_URL).netloc,
        APPENGINE_URL,
        "localhost",
        "127.0.0.1",
    ]
    CSRF_TRUSTED_ORIGINS = [APPENGINE_URL]
    # SECURE_SSL_REDIRECT = True
else:
    ALLOWED_HOSTS = ["*"]

This code fetches APPENGINE_URL from the environment (later we will add to our .env) and automatically configures ALLOWED_HOSTS and CSRF_TRUSTED_ORIGINS. Additionally, it enables SECURE_SSL_REDIRECT to enforce HTTPS.

Final version .env file should look something like this (we still don't have all values but just for information)

STRIPE_SECRET_KEY=sk_test_xxxxxxxxx
DENIS_PASS=xxxxxxx
ETHEREUM_NETWORK=maticmum
INFURA_PROVIDER=https://polygon-mumbai.infura.io/v3/xxxxxx
SIGNER_PRIVATE_KEY=xxxxxxx
MUSIC_NFT_ADDRESS=0x1D33a553541606E98c74a61D1B8d9fff9E0fa138
STRIPE_ENDPOINT=whsec_GExxxxxxx
OWNER=0x273f4FCa831A7e154f8f979e1B06F4491Eb508B6
DJANGO_SECRET_KEY='^&p@m*nhn#hg6ujgri2sppxr7t8o^mfp3bnj%1%2f72wcr+kkz'
DATABASE_URL=postgres://name:password!@localhost/musicalnft
# DATABASE_URL=postgres://name:password!@//cloudsql/musicnft-405811:europe-west1:musicnft-instance/musicalnft
GS_BUCKET_NAME=musicnft-bucket
APPENGINE_URL=https://musicnft-405811.ew.r.appspot.com/

later on we will populate all those values.

Don't forget to add the import at the top of the settings.py:

 from urllib.parse import urlparse

To use Postgres instead of SQLite, we first need to install the database adapter.

  $pip install psycopg2-binary==2.9.5
    $pip freeze > requirements.txt 

To utilize DATABASE_URL with Django, we can use django-environ's db() method like so:

  #settings.py
    DATABASES = {'default': env.db()}

Now create DATABASE_URL in .env and pass some random value, later on we will set up this value properly. (in general format of this value will go as fallow:

postgres://USER:PASSWORD@//cloudsql/PROJECT_ID:REGION:INSTANCE_NAME/DATABASE_NAME)

What we want to do is to replace Django dev server (what is not recommended for any kind of production environment) with Gunicorn. But first what is Gunicorn and why we should want to replace Django default server? The Django development server is lightweight and easy to use, but it's not designed/reccomndent to handle production traffic. It's meant for use during development only. On the other hand, Gunicorn is a WSGI HTTP server that's designed for production use. It's robust, efficient, and can handle multiple requests simultaneously, which is crucial for a production environment.

    $pip install gunicorn==20.1.0
    $pip freeze > requirements.txt 

Now very important app.yaml file. Google App Engine's app.yaml config file is used to configure your web application's runtime environment. The app.yaml file contains information such as the runtime, URL handlers, and environment variables.

Start by creating a new file called app.yaml in the project root with the following contents:

    # app.yaml

    runtime: python39
    env: standard
    entrypoint: gunicorn -b :$PORT musical_nft.wsgi:application

    handlers:
    - url: /.*
    script: auto

    runtime_config:
    python_version: 3

We defined the entrypoint command that starts the WSGI server.
There are two options for env: standard and flexible. We picked standard since it is easier to get up and running, is appropriate for smaller apps, and supports Python 3.9 out of the box.
Lastly, handlers define how different URLs are routed. We'll define handlers for static and media files later in the tutorial.

Now let's define .gcloudignore. A .gcloudignore file allows you to specify the files you don't want to upload to GAE when deploying an application. It works similarly to a .gitignore file.

Go ahead and create a .gcloudignore file in the project root with the following contents:

   # .gcloudignore

    .gcloudignore

    # Ignore local .env file
    .env

    # If you would like to upload your .git directory, .gitignore file, or files
    # from your .gitignore file, remove the corresponding line
    # below:
    .git
    .gitignore

    # Python pycache:
    __pycache__/

    # Ignore collected static and media files
    mediafiles/
    staticfiles/

    # Ignore the local DB
    db.sqlite3

    # Ignored by the build system
    /setup.cfg
    venv/

    # Ignore IDE files
    .idea/

    README_DEV.md
    celerybeat-schedule
    node_modules
    photos
    smart-contracts

Go ahead and initialize the gcloud CLI if you haven't already:

  $ gcloud init

gcloud CLI will ask you about mail and project you would like to associate with this project. It will offer to you all avaliable projects. Choose the one we created (MusciNFT) and gcloud will automatically set all the rest. What means from this point on when ever you deploy your Django app to Google App Engine it will be avalible under that project name.

To create an App Engine app go to your project root and run:

$gcloud app create

    # you will get something like this
    You are creating an app for project [musicnft-405811].
    WARNING: Creating an App Engine application for a project is irreversible and the region
    cannot be changed. More information about regions is at
    <https://cloud.google.com/appengine/docs/locations>.

    Please choose the region where you want your App Engine application located:

    [1] asia-east1    (supports standard and flexible)
    [2] asia-east2    (supports standard and flexible and search_api)
    [3] asia-northeast1 (supports standard and flexible and search_api)
    [4] asia-northeast2 (supports standard and flexible and search_api)
    [5] asia-northeast3 (supports standard and flexible and search_api)
    [6] asia-south1   (supports standard and flexible and search_api)
    [7] asia-southeast1 (supports standard and flexible)
    [8] asia-southeast2 (supports standard and flexible and search_api)
    [9] australia-southeast1 (supports standard and flexible and search_api)
    [10] europe-central2 (supports standard and flexible)
    [11] europe-west   (supports standard and flexible and search_api)
    [12] europe-west2  (supports standard and flexible and search_api)
    [13] europe-west3  (supports standard and flexible and search_api)
    [14] europe-west6  (supports standard and flexible and search_api)
    [15] northamerica-northeast1 (supports standard and flexible and search_api)
    [16] southamerica-east1 (supports standard and flexible and search_api)
    [17] us-central    (supports standard and flexible and search_api)
    [18] us-east1      (supports standard and flexible and search_api)
    [19] us-east4      (supports standard and flexible and search_api)
    [20] us-west1      (supports standard and flexible)
    [21] us-west2      (supports standard and flexible and search_api)
    [22] us-west3      (supports standard and flexible and search_api)
    [23] us-west4      (supports standard and flexible and search_api)
    [24] cancel
    Please enter your numeric choice:  11

    Creating App Engine application in project [musicnft-405811] and region [europe-west]....done.                       
    Success! The app is now created. Please use `gcloud app deploy` to deploy your first app.

Ok, now we need to set up our Postgres database inside Cloud SQL dashboard: https://console.cloud.google.com/sql/

Ones you are there create new instance and choose PostgresSQL (enable Compute Engine API if needed).

Now pass following values:

   Instance ID: musicnft-instance
    Password: Enter a custom password or generate it
    Database version: PostgreSQL 14
    Configuration: Up to you
    Region: The same region as your app
    Zonal availability: Up to you

You might also need to enable "Compute Engine API" to create a SQL instance.

Once the database has been provisioned, you should get redirected to the database details. Take note of the "Connection name".

Go ahead and enable the Cloud SQL Admin API by searching for "Cloud SQL Admin API" and clicking "Enable". We'll need this enabled to test the database connection. Here is a link (it will route you to your project): https://console.cloud.google.com/marketplace/product/google/sqladmin.googleapis.com

To test the database connection and migrate the database we'll use Cloud SQL Auth proxy. The Cloud SQL Auth proxy provides secure access to your Cloud SQL instance without the need for authorized networks or for configuring SSL.

First, authenticate and acquire credentials for the API:

$gcloud auth application-default login

Next, download Cloud SQL Auth Proxy and make it executable:

 wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O cloud_sql_proxy


    --2023-11-21 00:27:00--  https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64
    Resolving dl.google.com (dl.google.com)... 172.217.20.78, 2a00:1450:4017:800::200e
    Connecting to dl.google.com (dl.google.com)|172.217.20.78|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 19239740 (18M) [application/octet-stream]
    Saving to: ‘cloud_sql_proxy’

    cloud_sql_proxy               100%[===============================================>]  18.35M  1.22MB/s    in 13s     

    2023-11-21 00:27:13 (1.45 MB/s) - ‘cloud_sql_proxy’ saved [19239740/19239740]

    $chmod +x cloud_sql_proxy

After the installation is complete open a new terminal window and start the proxy with your connection details like so:

  $./cloud_sql_proxy -instances="PROJECT_ID:REGION:INSTANCE_NAME"=tcp:5432

Where basicaly PROJECT_ID:REGION:INSTANCE_NAME connection name you saved first time you created database

And you should see something like this:

 2023/11/21 00:31:03 current FDs rlimit set to 1048576, wanted limit is 8500. Nothing to do here.
    2023/11/21 00:31:04 Listening on 127.0.0.1:5432 for musicnft-405811:europe-west1:musicnft-instance
    2023/11/21 00:31:04 Ready for new connections
    2023/11/21 00:31:05 Generated RSA key in 135.901349ms

You can now connect to localhost:5432 the same way you would if you had Postgres running on your local machine.

Since GAE doesn't allow us to execute commands on the server, we'll have to migrate the database from our local machine.

Inside our .env file in the project root, with the required environment variables:

DATABASE_URL=postgres://DB_USER:DB_PASS@localhost/DB_NAME

    # Example `DATABASE_URL`:
    # DATABASE_URL=postgres://django-images:password@localhost/mydb

Now let's make migrations

    $python manage.py migrate

Create superuser

   $python manage.py createsuperuser

Ones you done with superuser you can move to secret manager. We used to have local .env file when we worked in local dev context. But because we are migrating now whole app into Google App Engine, we need to make our .env varibales present in cloud. And for this we will use Google secret manager.

Navigate to the Secret Manager dashboard (https://console.cloud.google.com/security/secret-manager?project=musicnft-405811) and enable the API if you haven't already. Next, create a secret named django_settings with the following content:

   DJANGO_SECRET_KEY='^&p@m*nhn#hg6ujgri2sppxr7t8o^mfp3bnj%1%2f72wcr+kkz'
    # local development
    # DATABASE_URL=postgres://ilija:Meripseli1986!@localhost/musicalnft
    # in cloud
    DATABASE_URL=postgres://ilija:Meripseli1986!@//cloudsql/musicnft-405811:europe-west1:musicnft-instance/musicalnft
    GS_BUCKET_NAME=django-music-nft


    # Example `DATABASE_URL`:
    DATABASE_URL=postgres://DB_USER:DB_PASS@//cloudsql/PROJECT_ID:REGION:INSTANCE_NAME/DB_NAME
    # postgres://django-images:password@//cloudsql/indigo-35:europe-west3:mydb-instance/mydb
    GS_BUCKET_NAME=django-images-bucket

Make sure to change DATABASE_URL accordingly. PROJECT_ID:REGION:INSTANCE_NAME equals your database connection details.

You don't have to worry about GS_BUCKET_NAME. This is just the name of a bucket we're going to create and use later.

Pip installl google-cloud-secret-manager==2.15.1

$pip install google-cloud-secret-manager==2.15.1
    $pip freeze > requirements.txt

To load the environment variables from Secret Manager we can use the following official code snippet:

  from pathlib import Path
    import os
    import environ
    from urllib.parse import urlparse
    import io # new
    from google.cloud import secretmanager # new



    # Build paths inside the project like this: BASE_DIR / 'subdir'.
    BASE_DIR = Path(__file__).resolve().parent.parent

    env = environ.Env(DEBUG=(bool, False))

    env_file = os.path.join(BASE_DIR, ".env")


    if os.path.isfile(env_file):
        # read a local .env file
        env.read_env(env_file)
        password = env("DENIS_PASS")
    elif os.environ.get('GOOGLE_CLOUD_PROJECT', None):
        # pull .env file from Secret Manager
        project_id = os.environ.get('GOOGLE_CLOUD_PROJECT')

        client = secretmanager.SecretManagerServiceClient()
        settings_name = os.environ.get('SETTINGS_NAME', 'django_settings')
        name = f'projects/{project_id}/secrets/{settings_name}/versions/latest'
        payload = client.access_secret_version(name=name).payload.data.decode('UTF-8')

        env.read_env(io.StringIO(payload))
    else:
        raise Exception('No local .env or GOOGLE_CLOUD_PROJECT detected. No secrets found.')

There is two new imports: io and secretmanager

Great! It's finally time to deploy our app. To do so, run:

 $ gcloud app deploy

    Services to deploy:

    descriptor:                  [C:\Users\Nik\PycharmProjects\django-images-new\app.yaml]
    source:                      [C:\Users\Nik\PycharmProjects\django-images-new]
    target project:              [indigo-griffin-376011]
    target service:              [default]
    target version:              [20230130t135926]
    target url:                  [https://indigo-griffin-376011.ey.r.appspot.com]


    Do you want to continue (Y/n)?  y

    Beginning deployment of service [default]...
    #============================================================#
    #= Uploading 21 files to Google Cloud Storage               =#
    #============================================================#
    File upload done.
    Updating service [default]...done.
    Setting traffic split for service [default]...done.
    Deployed service [default] to [https://indigo-griffin-376011.ey.r.appspot.com]

You can stream logs from the command line by running:

   $ gcloud app logs tail -s default

Open your web app in your browser and test if it works:

    $ gcloud app browse

If you get a 502 Bad Gateway error, you can navigate to Logs Explorer to see your logs.

If there's a 403 Permission 'secretmanager.versions.access' denied error, navigate to django_settings secret permissions and make sure the default App Engine service account has access to this secret. See solution on StackOverflow

And that is it! We have now our toy app fully up and running on Google app engine for world to see

Code can be found in this repo

p.s. whole this process is defined by great crew at testdrive.io High recommendation for all their writings!