DEV Community

Matt Martz
Matt Martz

Posted on

Migrating Full-Stack Apps from Poly-Repo to Mono-Repo

#nx #docker #monorepo #integration

In this post I'm going to create a super-basic Nx-backed mono-repo app and dockerize it.

Mono-repos have a number of benefits:

  • If you're working on features that span multiple parts of the system you can do one PR encompassing them all.
  • It also makes integration testing even easier.
  • It allows you to easily share code (interfaces) between front-end and back-end
  • One (consistent) way to run the app locally
  • Everything at the current commit works together

Nx is a set of Extensible Dev Tools for Monorepo development. It bakes in a lot of solid tools (Typescript, prettier, Cypress, Jest, React, Express, etc) with some handy CLI commands to make development easier. Read more about it here

After setting up a basic hello-world Nx app in docker I'm going to discuss how the same process could be used to migrate from a poly-repo-app to a mono-repo-app.

Alt Text

Alright, let's go

Table of Contents

The Basic App

Code for the Basic App is here:

GitHub logo martzmakes / blog-nx-docker

Scaffolding

Scaffold the app using NX commands

npx create-nx-workspace@latest
npx: installed 184 in 14.703s
? Workspace name (e.g., org name)     blog-nx-docker
? What to create in the new workspace react-express     [a workspace with a full
 stack application (React + Express)]
? Application name                    blog-nx-docker
? Default stylesheet format           emotion           [ https://emotion.sh]
Creating a sandbox with Nx...
... lots of output ...ls
Enter fullscreen mode Exit fullscreen mode

cd blog-nx-docker

npm start and npm start api will run the api and the front-end... The Nx scaffold automatically includes a super basic api call that gets displayed on the page ("Welcome to api!").

Alt Text

No surprises

But how would we get this up and running in Docker?

Dockerizing

We'll need a dockerignore file, an nginx.conf file, one multi-stage Dockerfile, and a docker-compose.yml.

Boring Files

Getting the easy ones out of the way... here's the nginx.conf file.

worker_processes  1;

events {
    worker_connections  1024;
}

http {

    map $http_upgrade $connection_upgrade {
        default upgrade;
        '' close;
    }

    upstream api {
        server blog-nx-api:3333;
    }

    server {
        listen 80;
        server_name  localhost 10.*;

        root   /usr/share/nginx/html;
        index  index.html index.htm;
        include /etc/nginx/mime.types;

        gzip on;
        gzip_min_length 1000;
        gzip_proxied expired no-cache no-store private auth;
        gzip_types text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript;

        location / {
            try_files $uri $uri/ /index.html;
        }
    }

    server {
        listen 3333;
        server_name  localhost 10.*;

        location / {
            proxy_pass http://api;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

And the .dockerignore file... this prevents unnecessary docker image builds by ignoring changes to files.

node_modules
coverage
docker
tools
Dockerfile*
README.md
LICENSE
.vscode
.dockerignore
.git
.gitignore
Enter fullscreen mode Exit fullscreen mode

Multi-stage Dockerfile

Next the multi-stage Dockerfile. This file will build itself in stages and be used by the docker-compose.yml file.

FROM node:12 AS blog-nx-base
WORKDIR /app
COPY . .
RUN npm ci -S
RUN npm run build -S
RUN npm prune --production -S

FROM nginx:alpine AS blog-nx-ui
COPY nginx.conf /etc/nginx/nginx.conf
WORKDIR /usr/share/nginx/html
COPY --from=blog-nx-base /app/dist/apps/blog-nx-docker .

FROM node:12 AS blog-nx-api
EXPOSE 3333
WORKDIR /app
COPY --from=blog-nx-base /app/node_modules /app/node_modules
COPY --from=blog-nx-base /app/dist/apps/api .
CMD ["node", "main.js"]
Enter fullscreen mode Exit fullscreen mode

This Dockerfile pulls the node 12.x image from the docker registry and does the install and build for both the front-end and api. From there, the UI is copied into an nginx image and the built-backend is copied into a fresh node:12 image.

At this point you could run docker build commands to tag images, like:

docker build --target blog-nx-api -t blog-nx-api .
docker build --target blog-nx-ui -t blog-nx-ui .
Enter fullscreen mode Exit fullscreen mode

...and use those in docker-compose... but docker-compose 3.4 supports using the multi-stage Dockerfile straight out.

Multi-stage docker-compose file

For the docker-compose.yml file:

version: '3.4'
services:
  blog-nx-api:
    container_name: blog-nx-api
    build:
        context: .
        target: blog-nx-api
    networks:
        martzcodes:
  blog-nx-ui:
    container_name: blog-nx-ui
    build:
        context: .
        target: blog-nx-ui
    ports:
      - 80:80
      - 3333:3333
    networks:
        martzcodes:
networks:
    martzcodes:
Enter fullscreen mode Exit fullscreen mode

The secret sauce here is the build.target... that refers to the AS name in the Dockerfile. This wasn't supported in earlier versions of Docker, which is why you see a lot of projects still use multiple Dockerfiles.

So now if you run docker-compose up -d it will build the images via the Dockerfile and then if you go to http://localhost/ you get the same satisfying template page with an api call.

Alt Text

Yup... checks out...

Alt Text

So far so good

Advanced

That's great for a basic hello-world app, but how do we handle a more advanced app that already exists in the real-world that is stored in several repos?

My strategy is to scaffold it out using basic Nx commands, merge the package.json dependencies and follow the same general process as before.

I'm starting from a place where I have only two repos with 3 components of the app (there are several others installed via docker-compose / images... not worth moving those for the moment). Mine looks like:

  • Service Repo
    • Express Service
    • Database seeds
  • UI Repo
    • React App

Both repos have their own Dockerfile(s) and docker-compose.ymls... and they also have their own CI yaml files with integration tests that span both repos. Which you may have seen from my previous post:

Article No Longer Available

As it stands I typically get ~3 Pull Requests for a feature that spans front-end -> back-end -> db... one for each.

confused

Three PRs for one thing?

Scaffolding the Migration

I'm going to start with an empty Nx workspace.

$ npx create-nx-workspace@latest
npx: installed 184 in 15.125s
? Workspace name (e.g., org name)     team-name
? What to create in the new workspace empty             [an empty workspace]
? CLI to power the Nx workspace       Nx           [Extensible CLI for JavaScript and TypeScript applications]
Creating a sandbox with Nx...
warning " > @nrwl/workspace@9.1.2" has incorrect peer dependency "prettier@^1.19.1".
[-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 0/354
new team-name --preset="empty" --interactive=false --collection=@nrwl/workspace
✔ Packages installed successfully.
    Successfully initialized git.
CREATE team-name/nx.json (268 bytes)
CREATE team-name/tsconfig.json (509 bytes)
CREATE team-name/README.md (2552 bytes)
CREATE team-name/.editorconfig (245 bytes)
CREATE team-name/.gitignore (503 bytes)
CREATE team-name/.prettierignore (74 bytes)
CREATE team-name/.prettierrc (26 bytes)
CREATE team-name/workspace.json (1059 bytes)
CREATE team-name/package.json (1108 bytes)
CREATE team-name/apps/.gitkeep (1 bytes)
CREATE team-name/libs/.gitkeep (0 bytes)
CREATE team-name/tools/tsconfig.tools.json (218 bytes)
CREATE team-name/tools/schematics/.gitkeep (0 bytes)
CREATE team-name/.vscode/extensions.json (109 bytes)
Enter fullscreen mode Exit fullscreen mode

I need to supplement the empty workspace with npm i --save-dev @nrwl/express @nrwl/react to get the express and react generators.

With those installed I can use some of the nrwl generators. Create the react app:

$ npm run nx g @nrwl/react:app team-name-ui

> team-name@0.0.0 nx /Users/mattmartz/Development/team-name
> nx "g" "@nrwl/react:app" "team-name-ui"

? Which stylesheet format would you like to use? emotion           [ https://emotion.sh            ]
? Would you like to add React Router to this application? Yes
✔ Packages installed successfully.
CREATE ...
Enter fullscreen mode Exit fullscreen mode

and the express app:

$ npm run nx g @nrwl/express:app team-name-service

> team-name@0.0.0 nx /Users/mattmartz/Development/team-name
> nx "g" "@nrwl/express:app" "team-name-service"


? In which directory should the node application be generated?
CREATE ...
Enter fullscreen mode Exit fullscreen mode

but now I want to add a UI -> Service interface library (so I don't have to define interfaces twice) and a UI component library...

$ npm run nx g @nrwl/workspace:lib team-name-interfaces

> team-name@0.0.0 nx /Users/mattmartz/Development/team-name
> nx "g" "@nrwl/workspace:lib" "team-name-interfaces"

CREATE ...
Enter fullscreen mode Exit fullscreen mode

and

$ npm run nx g @nrwl/react:lib team-name-components

> team-name@0.0.0 nx /Users/mattmartz/Development/team-name
> nx "g" "@nrwl/react:lib" "team-name-components"

CREATE ...
Enter fullscreen mode Exit fullscreen mode

Now that that's out of the way... it doesn't really do anything... especially with each other. Now let's migrate one piece at a time.

The Actual Migration? (but not really)

vague

Warning: The rest is a bit hand-wavy

I can't share a repository for this section so instead I'm going to provide some high-level strategy and discuss a few things I ran into.

Docker-compose Migration

First I'm going to move the integrated docker-compose.yml file that lives in one of the apps. The goal is to be able to run docker-compose up -d and have the original app up and running (pulling everything from docker registry images / building nothing locally).

I'm moving this first to ensure I don't have to troubleshoot it later.

One thing I quickly noticed is that my old docker-compose file was using version 2 instead of version 3.4... notably one of my services inside of it was using the old volumes_from. That means I needed to define the shared data model outside of the services list and update the links...

version: '3.4'
services:
  mysql:
    image: mysql:5.6
    container_name: mysql
    ports:
      - '3306:3306'
    networks:
      martzcodes:
    environment:
      - TZ=America/New_York
    volumes:
      - data:/var/lib/mysql # CHANGED
  data:
    image: ...
    container_name: mysql-data
    environment:
      - TZ=America/New_York
    networks:
      - martzcodes
    volumes:
      - data:/var/lib/mysql # CHANGED
volumes: # ADDED
  data:
Enter fullscreen mode Exit fullscreen mode

Moving the UI, Backend and DB

First merge the package.json dependencies... I tend to copy one into the other by section... and "Sort Lines Ascending" and then remove the duplicates... there may be a cleaner way but that works for me.

Then copy the rest of the files over. Nx uses an entry file named main.tsx... you can either keep that name and replace the contents or rename it in the workspace.json file.

Alternately you could create each file individually and re-build the app from the ground up... it depends how compatible your old version is with the new

Next would be setting up the Dockerfile. Since this will be the first thing going into the Dockerfile you could probably largely base it off of what was created in the Basic Example (unless you had a bunch of custom code in yours...). With the Dockerfile updated... change docker-compose.yml to build from it and use the correct target.

Moving the backend follows the same general principle... merge the dependencies and get it running locally.

Finally... the DB is the easiest because Nx doesn't really have any database hooks... basically just copy the files.

What'd we end up with?

If you didn't have a ton of troubleshooting to do, you probably ended up with a single repo that contains all of your files for an app. The benefits for this were described at the top of this post... which is probably why you got this far.

vague

Top comments (3)

Collapse
 
asmisalan profile image
Alan Asmis •

Hi Matt. really nice tutorial! Its really needed to copy the node_modules folder in the service folder? doing that you are adding dependencies that are not used by the API for example (angular, angular material, etc).

Collapse
 
martzmakes profile image
Matt Martz •

thanks! I haven't really thought about this in a while...

the strategy i used above was to have it running as a dev-server via Nx... you could also build the api and just copy the built stuff to the docker image and then run that directly (making sure it bundles everything it needs)... that'd probably be the better way to do it. good question!

Collapse
 
asmisalan profile image
Alan Asmis • • Edited

Hi Martz! It's me again with bad news! :/ nestjs (nodejs at the end) needs the node_module folder because the build does not add all the dependencies, some libraries like typeorm cause problems if you don't include the node_modules.

I found this github.com/ZenSoftware/bundled-nest

"""
In many real-world scenarios (depending on what libraries are being used), you should not bundle Node.js applications (not only NestJS applications) with all dependencies (external packages located in the node_modules folder). Although this may make your docker images smaller (due to tree-shaking), somewhat reduce the memory consumption, slightly increase the bootstrap time (which is particularly useful in the serverless environments), it won't work in combination with many popular libraries commonly used in the ecosystem. For instance, if you try to build NestJS (or just express) application with MongoDB, you will see the following error in your console:

Error: Cannot find module './drivers/node-mongodb-native/connection'
at webpackEmptyContext
Why? Because mongoose depends on mongodb which depends on kerberos (C++) and node-gyp.
"""

So the workaround that I found was to move all the angular dependencies to the DevDependency list and copy the node_modules as you did in this tutorial.

Read next

i_am_vesh profile image

Deploying a Complete 3-Tier Application with Jenkins: A Comprehensive Guide

Avesh -

aurelievache profile image

Understanding Docker: part 45 – Tools: Crane

Aurélie Vache -

goodylili profile image

How to Deploy Go Applications With Docker

Ukeje Chukwuemeriwo Goodness -

ken2026 profile image

Deploy a Docker Image to ECS with Auto Scaling Using AWS CDK in Minutes

Ken Yip -