Managing Database Migrations for Multiple Services in a Monorepo with Alembic

Inroduction

As modern software systems grow in complexity, many organizations turn to microservices architecture to break down monolithic applications into smaller, independent, and scalable components. Each microservice focuses on a specific domain, operates independently, and can be developed, deployed, and scaled autonomously. This approach enhances agility, simplifies development workflows, and enables teams to deliver features faster.

On the other hand, managing multiple microservices across distributed repositories can become a logistical nightmare. Dependency management, versioning, shared code, and deployment pipelines can quickly spiral out of control when teams are constantly hopping between repositories.

This is where a monorepo (monolithic repository) comes into play. A monorepo consolidates all microservices into a single repository, enabling seamless collaboration, shared tooling, and consistent versioning across teams. By combining microservices architecture with a monorepo, you strike a balance between modularity and manageability.

However, this approach introduces new challenges, especially when it comes to managing database migrations for multiple microservices. Each service typically owns its own database schema, and keeping these migrations isolated and conflict-free is critical to ensuring stability and scalability.

In this article, we’ll dive into how to effectively manage database migrations in a monorepo with multiple microservices, leveraging tools like Alembic to streamline the process. Whether you’re just getting started with microservices in a monorepo or looking to refine your existing setup, this guide will provide the insights and tools you need.

Why Is This a Challenge in Monorepos?

Imagine you’re developing a monorepo with two services:
1. ServiceOne – Handles one set of tasks.
2. ServiceTwo – Handles another set of tasks.

Each service has its own database schema and its own set of migrations. At first, you might put all your migrations in one place, like this:

monorepo/
│
├── migrations/
│   └── versions/
│       ├── 2024_06_15_create_logs_table.py
│       └── 2024_06_20_add_report_table.py
│
└── services/
    ├── service_one/
    └── service_two/

But this quickly becomes confusing:
• Which migration belongs to which service?
• How do you run migrations for only one service without affecting the other?
• How do you avoid conflicts when both services evolve at the same time?

We need a way to isolate migrations for each service while keeping everything under the same monorepo umbrella.

The Strategy: Separate Migrations, Dynamic Configuration

Enter Alembic, a powerful tool for managing SQLAlchemy-based database migrations, which is going to help us solving this by:
1. Creating separate migration directories for each service.
2. Configuring alembic.ini to specify migration paths for each service.
3. Using a dynamic env.py to load the correct database URL and migration configuration based on the service.

Folder Structure for Clarity

Here’s the folder structure we’ll use:

monorepo/
│
├── alembic.ini                # Global Alembic configuration file
├── migrations/
│   ├── env.py                 # Shared migration environment script
│   └── versions/
│       ├── service_one/
│       │   ├── 2024_06_15_init_schema.py
│       │   └── 2024_06_20_add_field.py
│       │
│       └── service_two/
│           ├── 2024_06_18_create_tables.py
│           └── 2024_06_25_update_schema.py
│
└── services/
    ├── service_one/
    │   ├── config.py          
    │   └── models.py
    │   └── ...
    │
    └── service_two/
        ├── config.py         
        └── models.py   
        └── ...

What This Structure Achieves
• Each service has its own dedicated migration files.
• You can easily see which migrations belong to which service.
• No more confusion or overlap between services!

Configuring alembic.ini

The alembic.ini file needs to know where to find migrations for each service. Here’s a sample configuration:

[alembic]
sqlalchemy.url = driver://user:pass@localhost/defaultdb

databases = service_one, service_two

[DEFAULT]
script_location = migrations

[service_one]
version_locations = ./migrations/versions/service_one
script_location = ./migrations

[service_two]
version_locations = ./migrations/versions/service_two
script_location = ./migrations

Breaking It Down

Important: Do not forget to include the migrations folder to this path, it didn't work without it in my case
• Fallback URL: The [alembic] section provides a default database URL.
• Multiple Databases: The databases key lists the services.
• Service Configurations: Each [service_one] and [service_two] section specifies:

• version_locations: Where to find the migrations for that service.
• script_location: The base path for Alembic scripts.

Dynamic env.py for Monorepos

Now let’s make env.py flexible so it can handle multiple services dynamically.

Here’s the key part of env.py:

from alembic import context
import importlib

# Load Alembic config
config = context.config

# Get the target service from the config section
service = config.config_ini_section

try:
    # Dynamically import the DB_URL from the service's config
    service_module = importlib.import_module(f"services.{service}.config")
    db_url = service_module.DB_URL.render_as_string(hide_password=False)
    config.set_main_option("sqlalchemy.url", db_url)
except ImportError:
    raise ValueError(f"Invalid service '{service}' specified.")

How It Works
1. Service Detection: Alembic reads the service name from the alembic.ini section.
2. Dynamic Import: It imports the database URL (DB_URL) from that service’s config.py.
3. Set Database URL: The correct URL is set dynamically for Alembic to use.

This means you can now run migrations for different services without changing any code!

Running Migrations

Creating a Migration for ServiceOne

alembic -n service_one revision --autogenerate -m "Initial schema for ServiceOne"

Applying Migrations

Upgrade ServiceOne’s database:

alembic -n service_one upgrade head

Downgrade ServiceTwo’s database:

alembic -n service_two downgrade base

Why This Approach Rocks
1. Isolation: Each service manages its own database schema independently.
2. Flexibility: Adding a new service? Just create a new section in alembic.ini and a new migrations folder (can be autogenerated).
3. Scalability: Works seamlessly whether you have 2 services or 20.

Final Thoughts

Managing database migrations in a monorepo doesn’t have to be chaotic. By separating migrations and using dynamic configurations, you keep things organized, maintainable, and scalable.

Have you tried this approach? What challenges did you face managing migrations in a monorepo? I’d love to hear your thoughts and experiences in the comments!

Notes: the cover picture is generated by AI, so excuse the fact that some parts of it might be blurry. AI has been used, as well, to check the content as I'm not a native English speaker 😅
I copy pasted the code from my IDE and it didn't come with colors

Happy coding! 🚀