Most people probably won't write their own custom ESM loaders, but using them could drastically simply your workflow.
Custom loaders are a powerful mechanism for controlling an application, providing extensive control over loading modules—be that data, files, what-have-you. This article lays out real-world use-cases. End users will likely consume these via packages, but it could still be useful to know, and doing a small and simple one-off is very easy and could save you a lot of hassle with very little effort (most of the loaders I've seen/written are about 20 lines of code, many fewer).
For prime-time usage, multiple loaders work in tandem in a process called "chaining"; it works like a promise chain (because it literally is a promise chain). Loaders are added via command-line in reverse order, following the pattern of its forebearer, --require:
$> node --loader third.mjs --loader second.mjs --loader first.mjs app.mjs
node internally processes those loaders and then starts to load the app (app.mjs). Whilst loading the app, node invokes the loaders: first.mjs, then second.mjs, then third.mjs. Those loaders can completely change basically everything within that process, from redirect to an entirely different file (even on a different device across a network) or quietly provide modified or entirely different contents of those file(s).
In a contrived example:
$> node --loader redirect.mjs app.mjs
// redirect.mjs
export function resolve(specifier, context, nextResolve) {
let redirect = 'app.prod.mjs';
switch(process.env.NODE_ENV) {
case 'development':
redirect = 'app.dev.mjs';
break;
case 'test':
redirect = 'app.test.mjs';
break;
}
return nextResolve(redirect);
}
This will cause node to dynamically load app.dev.mjs, app.test.mjs, or app.prod.mjs based on the environment (instead of app.mjs).
However, the following provides a more robust and practical use-case:
$> node \
--loader typescript-loader \
--loader css-loader \
--loader network-loader \
app.tsx
// app.tsx
import ReactDOM from 'react-dom/client';
import {
BrowserRouter,
useRoutes,
} from 'react-router-dom';
import AppHeader from './AppHeader.tsx';
import AppFooter from './AppFooter.tsx';
import routes from 'https://example.com/routes.json' assert { type: 'json' };
import './global.css' assert { type: 'css' };
const root = ReactDOM.createRoot(document.getElementById('root'));
root.render(
<BrowserRouter>
<AppHeader />
<main>{useRoutes(routes)}</main>
<AppFooter />
</BrowserRouter>
);
The above presents quite a few items to address. Before loaders, one might reach for Webpack, which sits on top of Node.js. However, now, one can tap into node directly to handle all of these on the fly.
The TypeScript
First up is app.tsx, a TypeScript file: node doesn't understand TypeScript. TypeScript brings a number of challenges, the first being the most simple and common: transpiling to javascript. The second is an obnoxious problem: TypeScript demands that import specifiers lie, pointing to files that don't exist. node of course cannot load non-existent files, so you'd need to tell node how to detect the lies and find the truth.
You have a couple options:
- Don't lie. Use the
.tsetc extensions and use something like esbuild in a loader you write yourself, or an off-the-shelf loader like ts-node/esm to transpile the output. On top of being correct, this is also significantly more performant. This is Node.js’s recommended approach.
Note: tsc appears soon to support .ts file extensions during type-checking: TypeScript#37582, so you'll hopefully be able to have your cake and eat it too.
- Use the wrong file extensions and guess (this will lead to decreased performance and possibly bugs).
Due to design decisions in TypeScript, there are unfortunately drawbacks in both options.
If you want to write your own TypeScript loader, the Node.js Loaders team have put together a simple example: nodejs/loaders-test/typescript-loader. ts-node/esm would probably suit you better though.
The CSS
node also does not understand CSS, so it needs a loader (css-loader above) to parse it into some JSON-like structure. I use this most commonly when running tests, where styles themselves often don't matter (just the CSS classnames). So the loader I use for that merely exposes the classnames as simple, matching key-value pairs. I've found this to be sufficient as long as the UI is not actually drawn:
.Container {
border: 1px solid black;
}
.SomeInnerPiece {
background-color: blue;
}
import styles from './MyComponent.module.css' assert { type: 'css' };
// { Container: 'Container', SomeInnerPiece: 'SomeInnerPiece' }
const MyComponent () => (<div className={styles.Container} />);
A quick-n-dirty example of css-loader is available here: JakobJingleheimer/demo-css-loader.
A Jest-like snapshot or similar consuming the classnames works perfectly fine and reflects real-world output. If you're manipulating the styles within your JavaScript, you'll need a more robust solution (which is still very feasible); however, this is maybe not the best choice. Depending on what you're doing, CSS Variables are likely better (and do not involve manipulating the styles at all).
The remote data (file)
node does not yet fully support loading modules over a network (there is experimental support that is intentionally very restricted). It’s possible to instead facilitate this with a loader (network-loader above). The Node.js Loaders team have put together a rudimentary example of this: nodejs/loaders-test/https-loader.
All together now
If you have a "one-off" task to complete, like compiling your app to run tests against, this is all you need:
$> NODE_ENV=test \
NODE_OPTIONS='--loader typescript-loader --loader css-loader --loader network-loader' \
mocha \
--extension '.spec.js' \
'./src'
As of this week, the team at Orbiit.ai are using this as part of their development process, to a near 800% speed improvement to test runs. Their new setup isn't quite finished enough to share before & after metrics and some fancy screenshots, but I'll update this article as soon as they are.
// package.json
{
"scripts": {
"test": "concurrently --kill-others-on-fail npm:test:*",
"test:types": "tsc --noEmit",
"test:unit": "NODE_ENV=test NODE_OPTIONS='…' mocha --extension '…' './src'",
"test:…": "…"
}
}
You can see a similar working example in an open-source project here: JakobJingleheimer/react-form5.
For something long-lived (ex a dev server for local development), something like esbuild's serve may better suit the need. If you're keen do it with custom loaders, you'll need a couple more pieces:
- A simple http server (JavaScript modules require it) using a dynamic import on the requested module.
- A cache-busting custom loader (for when the source code changes), such as quibble (who published an explanatory article on it here).
All in all, custom loaders are pretty neat. Try them out with today's v18.6.0 release of Node.js!
Top comments (8)
This is awesome
This sounds like a really cool change!
I actually needed chain-loading so that I could unit-test es6 with mocking and a mixed Typescript and Javascript project. It looks like they don't play well together, though:
I get this stack trace:
and these 4 lines repeat infinitely:
Feels like this could potentially be The Wild West in terms of reconciling loaders. I hope I can get it working!
Thanks!
Regarding the issue you encountered: don't use
--experimental-specifier-resolution=node(It doesn't work with loaders, and it's going to be removed very soon)Hi Jacob, I’ve been using
--experimental-specifier-resolution=nodewith loaders without issues for a while now (currently running 18.8.0) and it simplifies authoring. I was just wondering what exactly doesn’t work with loaders. Would love to see it supported if possible.(It seems quite a few decisions lately come at the expense of making things more verbose for authors – e.g., import asserts. While these are mere annoyances for experienced developers, they make things harder to grasp for folks just learning a new platform.)
Hiya Aral!
Lots of subtle things don't work. I don't know of a complete list—we don't track it because that feature has been deprecated for almost a decade (and will be removed in v19). It was a bad idea at the time, and it's still a bad idea now.
It's more like it makes the author experience lazier 😜 the ESM specification requires file extensions for good reason. It doesn't make anything harder to grasp: in most cases, you have a file on disk with an extension, so it's literally copy+paste, and if you don't, the error tells you exactly what you did wrong 🙂 imagine you're visiting your friend and you write down their address with everything but their house number. Can you find it? Probably eventually, but it's gonna suck looking. This problem is easily avoid, so avoid it 😉
Import asserts have a very specific reason as well: security. You can import from a remote source, eg example.com/config.json, but that doesn't necessarily mean you get json back—there is no rule saying the file extension there must be respected or even considered, so the server is perfectly justified returning whatever it wants. Json is not executable, but javascript is. That response could indeed be JavaScript instead; without the guard from the assertion (the assertion tells the runtime "this is not executable"), it would execute. Yikes, bad day. You CAN disable the safety of the gun you've pointed at your foot, if you really want 🤷♂️
does browser have simliar mechanism like nodejs custom loader? any tc39 proposal here?
Not as far as I know
Quick update: this is now being discussed within TC39 for browsers