Custom Parser for "pipe" Streams in NodeJS

This article will show a efficient way to modify data while its being piped, say to a client in Express.

Let's get started.

Take a normal NodeJS server:

const express = require('express');
const server = express();

const fs = require('fs')

server.get('/', (req, res) => {
   fs.createReadStream('/index.html').pipe(res);
});

server.listen(8080)

Here, we are using the fs.createReadStream function to pipe data without storing much in memory (RAM). This is completly fine. It's better than fs.readFile.

The Problem: If we want to dynamically make small edits to index.html based of say, the HTTP headers, this cute syntax doesn't have that functionality. Now, for anyone that has worked around streams in NodeJS, they know that the .pipes can be chained. One method that is not to different to the original code, we can use custom streams from the in-built stream module, and an awesome module called new-line from NPM.

This module only passes chunks of data to the next pipe only line by line. This means that it waits until it finds a new-line: \n. For example, if we have to replace "hello" with "world", we may get "he" in one chunk and "llo" in the next. This results in our function completely skipping the "hello".

Let's implement it:

const newLineStream = require('new-line');

server.get('/', (req, res) => {
   fs.createReadStream('/index.html')
    .pipe(newLineStream())
    .pipe(res);
});

Now, the client gets data line by line. We're one step closer!

Time for the lengthy part. Lets implement our custom parser with the Transform class from the stream module.

const newLineStream = require('new-line');
const { Transform } = require('stream');

class MyParser extends Transform {
   constructer() {
      super()
      this._transform = (chunk, encoding, cb) => {
         cb(null, chunk.toString().replace("hello", "world"))
      }
   }
}

server.get('/', (req, res) => {
   fs.createReadStream('/index.html')
    .pipe(newLineStream())
    .pipe(new MyParser())
    .pipe(res);
});

Lets break this down:

The class, constructer, and super just make a copy of the Transform class and load all the Transform functions.

The this._transform is declaring _transform for our class because that's what .pipe uses for _transforming the data.

The chunk parameter is the chunk of data that is of encoding specified by the encoding parameter. The cb parameter is a callback function. I pass null meaning that there is no error and the modified chunk data.

We can use this principle with chained .replaces and using the g flag with RegExp to manipulate data while keeping up our performance. After all, IO is the strong set of NodeJS.

That's it for this article. I hope this helped someone. I can be contacted at [e-mail]. Check out my github profile.