Syncing an NPM Package Between Multiple Registries

Photo by Nicolas Radzimski on Unsplash

The Problem

Our team relies on an internal GitLab NPM registry for managing packages. This works perfectly for internal distribution, but there are instances where a package needs to be shared with external collaborators. Syncing the package from the internal registry to an external one became a new challenge.

I set out to find a solution and, as usual, encountered a series of hurdles that turned this into another chapter in my developer struggles.

Attempt 1: Using HTTP Requests to Mimic NPM CLI

Initially, I thought I could handle this with raw HTTP requests, essentially mimicking what the NPM CLI does under the hood. The idea was simple: query the source registry for package metadata, fetch the tarball, and upload it to the target registry.

Why it failed:

• The process was too complex and fragile.
• It required manually constructing the entire package.json manifest for every version.
• Every small misstep led to broken packages.

Attempt 2: Leveraging npm-registry-sync

Next, I discovered npm-registry-sync, a library designed for syncing NPM registries. This tool almost solved the problem; it could monitor changes and replicate them across registries.

Why it didn’t work for me:

• It operates in “daemon mode,” polling for updates continuously.
• In a GitLab CI pipeline, I needed a one-off execution, controlled entirely by the pipeline — no background processes allowed.

The Winning Solution: The NPM CLI

Eventually, I realized I could stick to the tried-and-true NPM CLI. The steps were straightforward:

1. Install the package locally from Registry A.
   1. Reconfigure NPM to point to Registry B.
   2. Publish the package to Registry B.

While this worked like a charm, there were a few extra steps needed to make it CI-friendly.

---

Configuring NPM for Multiple Registries

Managing registry configurations dynamically in a CI pipeline was a bit tricky.

Here’s how I solved it:

General Configuration

Using the npm CLI, you can set parameters for each registry:

npm config set "//my.awesome.registry.com:<parameter name>=<parameter value>"

Important Gotcha:
The URL in the config must exclude the protocol (https:).

Package-to-Registry Association

To associate a specific namespace or package with a registry:

npm config set "<your namespace>:registry" "<your registry url with https:>"

Handling Authentication in CI Pipelines

Some registries required a username/password combo, while others used tokens. Here’s what I learned:

Tokens

Tokens are straightforward, but ensure you strip the protocol when configuring the auth URL:

npm config set "//my.registry.com:_authToken=<token>"

Basic Auth

Generating a basic auth hash (username:password) required attention to detail. In some distros, the base64 command has quirks that differ from others.

On macOS:

echo -n "<my username & password hash>" | base64

Will give you (as expected):

d2hhdCBhcmUgeW91IGRvaW5nIGhlcmU/IGdvdCB5YSEgc29tZSBtb3JlIHRleHQgdG8gbWFrZSB0aGlzIHJlYWxseSByZWFsbHkgbG9uZw==

With docker linuxkit:

d2hhdCBhcmUgeW91IGRvaW5nIGhlcmU/IGdvdCB5YSEgc29tZSBtb3JlIHRleHQgdG8gbWFrZSB0
aGlzIHJlYWxseSByZWFsbHkgbG9uZw==

What is going on here? There is a line break!

On some distros, the wrap parameter has a default set to 76 chars for formatting private keys etc.

It works like this:

echo -n "<my username & password hash>" | base64 --wrap 0

The echo -n is very important. If omitted, echo will add a linebreak to the end of the string and this will manipulate your hash.

The finished script

Putting it all together, the script could look like this:

#!/usr/bin/env bash

# Input validation
if [ "$#" -ne 5 ]; then
  echo "Usage: $0 <source-registry> <target-registry> <source-registry-token> <target-registry-username> <target-registry-password>"
  exit 1
fi

SOURCE_REGISTRY=$1
TARGET_REGISTRY=$2
SOURCE_REGISTRY_TOKEN=$3
TARGET_REGISTRY_USERNAME=$4
TARGET_REGISTRY_PASSWORD=$5

echo "SOURCE_REGISTRY=$SOURCE_REGISTRY"
echo "TARGET_REGISTRY=$TARGET_REGISTRY"
echo "SOURCE_REGISTRY_TOKEN=${SOURCE_REGISTRY_TOKEN:0:10}"
echo "TARGET_REGISTRY_USERNAME=${TARGET_REGISTRY_USERNAME}"

echo "Syncing from '$SOURCE_REGISTRY' to '$TARGET_REGISTRY'..."

remove_protocol() {
  sed -E 's/^https?://g;t' <<< "$1"
}

packages=(
  "@my-namespace/my-package"
)

# INFO: Removing the protocol (^https:) from the authToken config is crucial. Will not work without it.

# Auth for source registry
npm config set "$(remove_protocol $SOURCE_REGISTRY):_authToken=$SOURCE_REGISTRY_TOKEN"

# --wrap 0 is super important! If omitted, it will cause the base64 to contain a line break after 76 chars!
BASIC_AUTH=$(echo -n "$TARGET_REGISTRY_USERNAME:$TARGET_REGISTRY_PASSWORD" | base64 --wrap 0)

# Auth for target registry
npm config set "$(remove_protocol $TARGET_REGISTRY):_auth" "$BASIC_AUTH"
npm config set "$(remove_protocol $TARGET_REGISTRY):always-auth=true"

# Set to source registry to fetch metadata
npm config set "@my-namespace:registry" "$SOURCE_REGISTRY"

for package in "${packages[@]}"; do
  echo "Syncing '$package'..."

  for version in $(npm view "$package" --json | jq -r '.versions[]'); do
    # Install locally
    npm install "$package@${version}" --ignore-scripts

    # Switch to target registry
    npm config set "@my-namespace:registry" "$TARGET_REGISTRY"

    # Publish on target
    npm publish ./node_modules/"$package"

    # Re-set to source registry
    npm config set "@my-namespace:registry" "$SOURCE_REGISTRY"
  done
done

echo "Done syncing packages."

So you could use it like this:

#!/usr/bin/env bash

SOURCE_REGISTRY=https://my.source.registry.com
SOURCE_REGISTRY_TOKEN=abc-xyz-1234568
TARGET_REGISTRY=https://my.target.registry.com
TARGET_REGISTRY_USERNAME=john_doe
TARGET_REGISTRY_PASSWORD=supersecret123

sync-registries.sh $SOURCE_REGISTRY $TARGET_REGISTRY $SOURCE_REGISTRY_TOKEN $TARGET_REGISTRY_USERNAME $TARGET_REGISTRY_PASSWORD

---

Key Lessons Learned

1. Stick to Simple Tools:
   The npm CLI might not be fancy for this task, but it’s reliable and gets the job done.

2. Mind the Details:
   Configuring authentication, especially with base64, can have subtle platform-specific quirks.

3. Keep It CI-Friendly:
   Avoid solutions like daemons or background tasks when working in CI/CD pipelines. Keep the process under pipeline control.

Syncing npm packages between registries was a frustrating but rewarding learning experience. If you’re facing a similar challenge, I hope these lessons help you navigate the struggle with a bit more ease!

Comments

[Samuel Rouse]

We use multiple NPM servers internally, including a number of isolated hosts to which we publish and proxies that are our "consumer" servers that developer and CI/CD request all packages through.

To work around the difference in publish locations, we use a combination of the publishConfig to ensure that publishing works even when the configured registry is different.

Rather than reconfiguring the registry in the global npm config, would it be possible in your case to use the [--registry](docs.npmjs.com/cli/v8/using-npm/co... parameter? That might also allow you to leave the auth for each registry intact in the config.