Migrate a 60k LOC TypeScript (NodeJS) repo to ESM and testing become 4x faster (1/2)

Intro

After the third try, we successfully migrated all existing Node.js code from CJS to native ESM, and the CI time of unit testing was significantly reduced.

Pull requests refactor: use ESM test: use native ESM

Before we start, I'd like to demonstrate the status quo ante for a better context. You may have different choices on the repo setting or toolchain, but the core steps and concepts should be the same:

• A TypeScript monorepo that includes both frontend and backend projects / packages.
• Total TypeScript code ~60k LOC (including frontend).
• Use import in TypeScript.
• Use PNPM for workspace management.
• Use tsc to compile Node.js, and Parcel to bundle frontend projects.
• Use Jest + ts-jest for unit testing.
• Use package module-alias for internal path aliases.

BTW, our project Logto is an open-source solution for auth.

---

Why ESM?

When we noticed more and more NPM packages are "ESM-only", and we closed tons of PR because of it (except Parcel / Jest ones):

Disregard the war of ESM v.s. CJS, we found ESM does have several advantages:

No another-language-like code transpilation

Especially for TypeScript: Comparing the original code to the compiled version, ESM is much easier to read, edit, and debug.

Given a simple TypeScript snippet:

import path from 'path';

const replaceFile = (filePath: string, filename: string) =>
  path.resolve(path.dirname(filePath), filename);

Results after tsc:

// CJS
"use strict";
var __importDefault = (this && this.__importDefault) || function (mod) {
    return (mod && mod.__esModule) ? mod : { "default": mod };
};
Object.defineProperty(exports, "__esModule", { value: true });
const path_1 = __importDefault(require("path"));
const replaceFile = (filePath, filename) => path_1.default.resolve(path_1.default.dirname(filePath), filename);

// ESM
import path from 'path';
const replaceFile = (filePath, filename) => path.resolve(path.dirname(filePath), filename);

Top-level await

This is one of our favorites. Finally no need for wrapping top-level async expressions into a function and using void to execute it.

// CJS
(async () => {
  await doSomething();
})();

// ESM
await doSomething();

Compatibility

• While ESM can easily load CJS modules, it'll be harder for CJS to load ESM. The primary reason is CJS require() is synchronous, while ESM import is asynchronous. You must use await import() in CJS which is painful w/o top-level await.
• Plus, CJS is Node-only, which means a universal package needs to compile another version for browser users. (We know there are transpilers, but, huh)

Stick with the standard

ESM is the module standard of JavaScript. TypeScript also uses ESM syntax by default.

Immutable

This is more like a double-edged sword. ESM can significantly improve module security by its immutable design (see this article), but it also brings a little inconvenience for test mocking.

Dramatic testing time reduction

Yes, we're talking about Jest. While Jest is sticking with CJS and only has experimental ESM support, we've been using ts-jest for a while, but obviously, it's a challenging task even for an M1 Pro MacBook. The fan spins with a noticeable sound, and the core temperature keeps going up while running unit tests.

After migrating all unit tests to ESM as well, my MacBook became silent and pro again! Here's the CI time comparison (with default machine spec. in GitHub Actions):

Execution time are not that stable in GitHub Actions, on average it shows 3x - 4x faster.

Code migration

For official docs, you may find Modules: ECMAScript modules and ECMAScript Modules in Node.js helpful.

Basic configuration

Let's start with tsconfig.json. Two key points:

• Set compilerOptions.moduleResolution to nodenext in order to tell TSC to use the "cutting-edge" Node.js module resolution strategy.
• Set compilerOptions.module to esnext to ensure the output also keeps ESM.

For Node.js, add "type": "module" to your package.json to let it treats the package as ESM.

Path aliases

We were mapping @/ to ./src/ using module-alias alias but it doesn't work in ESM. The good news is Node.js provides a native support called Subpath imports by defining the imports field in package.json:

{
  "imports": {
    "#src/*": "./build/*" // Point to the build directory, not source
  }
}

Note imports can only starts with # and must have a name. So we use #src/ to replace the original @/. Also update your tsconfig.json accordingly:

{
  "compilerOptions": {
    "paths": {
      "#src/*": ["src/*"]
    }
  }
}

We also need to replace all @/ with #src/ in source code.

File extension

At this point, both TSC and Node.js start to work in ESM mode, but most likely some "Cannot find module..." errors float. Because the existing convention of importing is minimalistic and elegant:

import x from './foo';

// It can be
import x from './foo.js';
// Or
import x from './foo/index.js';

And the .js extension can be replaced with .jsx, .ts, and .tsx, etc.

However, this becomes unacceptable in native ESM. You must explicitly write the full path with the extension, e.g. import x from './foo/index.js';.

So how it should be in TypeScript? Our first idea is changing the file extension to .ts, which turns the path to './foo/index.ts', since that's the file we can find in the source directory, right?

Unfortunately, the TypeScript team has the principles like "TS is the superset of JS" and "TS doesn't rewrite paths". (You can see #13422 #16577 #42151 since 2017) So .ts doesn't work here, and it led to the result: use .js. :-)

It doe works, and I think I'm not qualified to judge the solution. So let's move to the actions we took to add extensions:

Since most packages in node_modules are not affected by this (at least for the main entry), we can omit them during the process.

1. Replace all from '\.' (RegExp) with from './index'.
2. Replace all from '\./(.*)' (RegExp) with from './$1.js'.
3. If you have path alias, use the similar technique in step 2 to add extensions to them.
4. Try to compile the project. It may show some errors for those paths with omitted /index, e.g. a ./foo which actually points to ./foo/index.js. They are updated to wrong paths like ./foo.js in step 2.
5. Try to compile again and it should show no error this time.

Misc.

It's exciting to see the output files are almost the same as the input, but when you run the project, Node.js may complain about some special variables, for example:

ReferenceError: __dirname is not defined in ES module scope

Don't give up, we're almost there! Read the Node.js official doc Differences between ES modules and CommonJS to handle them, then you're good to go.

Recap

We successfully migrated our Node.js packages from CJS to ESM. Now the entry-point file is runnable after tsc.

However, some issues still remain in unit tests:

• As of today (12/26/22), Jest only has experimental support for ESM.
• ESM is immutable, thus jest.mock() will not work and jest.spyOn() also doesn't work on first-level variables (export const ...). This also applies to other test libraries like Sinon.
• You may find some libraries for mocking ESM, but almost all of them are creating "a new copy" of the original module, which means if you want to import module A that depends on module B, you must import A AFTER B is mocked to get it to work.

The solution is the key to boosting the CI time. Since this article already pulled out a lot of things to digest, we'll cover them in the next chapter:

Migrate a 60k LOC TypeScript (NodeJS) repo to ESM and testing become 4x faster (2/2)

You can find our ts-with-node-esm repo for the key result of this series:

https://github.com/logto-io/ts-with-node-esm

Thank you for reading, feel free to comment if you have any questions!

---

This series is based on our experience with Logto, an open-source solution for auth.

Comments

[José Pablo Ramírez Vargas]

Yes, I am on board with this too with wj-config. I transpiled it from TypeScript to ES Modules. That's it. Look at some code snippet comparison when used in Common JS and ES Modules:

Set Up

Common JS

module.exports = (async function () {
    const { default: wjConfig, Environment } = await import('wj-config');
    const env = new Environment(process.env.NODE_ENV);
    return wjConfig()
        .addObject(loadJsonFile('./config.json', true))
        .name('Main')
        .addObject(loadJsonFile(`./config.${env.current.name}.json`))
        .name(env.current.name)
        .addEnvironment(process.env)
        .includeEnvironment(env)
        .createUrlFunctions()
        .build(env.isDevelopment());
})();

ES Modules

const config = wjConfig()
    .addObject(mainConfig) // Main configuration JSON file.
    .name('Main') // Give data sources a meaningful name for value tracing purposes.
    .addObject(loadJsonFile(`./config.${env.current.name}.json`)) // The developer is deciding by using a file name tactic.
    .name(env.current.name)
    .addEnvironment(process.env) // Adds a data source that reads the environment variables in process.env.
    .includeEnvironment(env) // So the final configuration object has the environment property.
    .createUrlFunctions() // So the final configuration object will contain URL builder functions.
    .build(env.isDevelopment()); // Only trace configuration values in the Development environment.

// This is a top-level await:
export default await config; // The build() function is asynchronous, so await and export the result.

Consumption

Common JS

const configPromise = require('./config.js');

configPromise.then(config => {
    console.log(config.app.title);
});

ES Modules

import config from './config.js';

console.log(config.app.title);

---

Just having top-level await is enough to sway like 90% of people to ES Modules.

[Brian Kirkpatrick]

We are going through a similar experiment right now, but for migrating to ESM from RequireJS. Ironically, it seems to be much easier, largely because there are nearly zero adaptations/hotfixes required for front-end compatibility, where all the transpilation was destined for anyway. If anything, the weird edge cases come from situations (like automated unit testing) where some degree of NodeJS compatibility is still required.

[Roman K]

It was a problem with tooling, not with CJS. Migration of tests from requires to imports shouldn't cause such drastic speed up, it's definitely a bug in the ts-jest or in a related tooling. I'm using @swc/jest for tests and it takes couple of seconds to run a thousand tests written in TS.

The point of this article is that ESM brings a compilation speedup, so what was the change of the build step of your project?

[gao-sun]

Thank you! Totally agree there's something that happened in ts-jest or ts-node magnificently slows down the testing speed. Will try @swc/jest.

The faster testing speed is something out of our expectations. The point of this article is to discuss why we decided to transform to native ESM and our know-how of the migration. No big changes for the build steps.

[Roman K]

I read more carefully and I see that main reason was a non-compatible packages that enforces ESM to be enabled. That's sad to write extensions in import pathes, have experimental Jest, to loose jest.mock() which I'm using regularly, and all because some libs cannot setup rollup properly to have both commonjs and esm outputs. In your project it's probably important to support all packages, but luckily in other node.js projects it's fine to use an alternative lib.

[gao-sun]

Non-compatible packages is a trigger, not the key reason. We kept those PR closed for months.

Extensions and experimental flag are not blockers to us. We don't lose `jest.mock(), as you can see the practice in chapter 2. Actually the code becomes more readable and predictable afterwards.

And it's all about choices and balances - since in most cases our project serves as a service/solution which is not for importing, and our users are using container to run it, so single transpilation strategy is good to us.

We are using rollup for dual-package publish as well, but it's a transitive plan and soon the repo will be pure-ESM.

[Konnor Rogers]

And now time to swap to Vitest!

[Vesa Piittinen]

Switching to Vitest fixed many issues that we've had with Jest. Like React hooks didn't work as expected within components when running within Jest which had led to cumbersome hook mocking. Especially useContext was very troublesome. I don't know if this is issue always with Jest but the issue did replicate on two different repos that we have.

Vitest is also faster.

[gao-sun]

Thanks guys. Will take a look on Vitest!

[Eric Danowski]

We have a code base that I have tried to move to ESM but the problem that I have run into is how to mock imports in a unit test. We use Jasmine and Karma for our specs and if I am testing a module that imports another module (e.g. utils.js), I am unable to stub or mock the exports of utils.js. How are you handling this? You mentioned the issue in the Recap, but I'm wondering if anyone has a solution. Thanks

[gao-sun]

Mocking is explained in chapter 2. The principle will be the same.

[Eric Danowski]

Thanks!

[Joseph C]

Thank you for the post, this has been really helpful.

But I have a question about the subpath imports. You have the imports defined as:

    "#src/*": "./build/*" // Point to the build directory, not source

Why are you pointing to the build dir?

In your github example you don't have any subpath imports defined. So what is the thought process behind the subpath import pointing to build and further, does it also imply that you need to be running a watcher on the code to continuously update the build output?

Thanks