Ketan Jakhar

Posted on Oct 14 • Originally published at codexstoney.Medium on Oct 7

Handling Backpressure in Node.js streams

#typescript #asynchronousprogramm #systemdesignconcepts #node

Thumbnail generated using Dall-E 3

Imagine you’re the captain of a bustling riverboat, ferrying passengers and goods down a lively river. The current is strong, and the river is full of twists, turns, and occasional obstacles. To keep your boat steady and ensure everyone arrives safely, you need to manage the flow of passengers and cargo carefully. Sometimes, if too many people board at once or if there’s an unexpected blockage downstream, you must adjust your pace to maintain balance and prevent chaos.

In the world of Node.js and TypeScript, managing backpressure is much like steering that riverboat. Whether you’re a beginner just setting sail, an intermediate developer navigating deeper waters, or an expert seasoned by many journeys, understanding backpressure is essential for building robust, efficient applications. Let’s embark on this journey together, exploring backpressure through our riverboat story, and uncovering strategies to handle it effectively in your Node.js applications.

1. Understanding Backpressure

Backpressure is like the pushback a riverboat captain receives when the downstream cannot handle the current flow of passengers or cargo. If your boat moves too fast without considering the downstream capacity, you risk overloading the system, causing delays, or even capsizing your operations.

In technical terms, backpressure occurs when a data producer generates data faster than a consumer can process. This imbalance can lead to memory leaks, application crashes, or degraded performance without proper management. Just as a captain must adjust the boat’s speed and load based on the river’s conditions, developers must implement strategies to control data flow between producers and consumers in their applications.

2. Streams in Node.js

Node.js streams are the lifeblood of efficient data handling, much like the steady flow of a river that powers the riverboat. Streams allow your application to process data piece by piece, rather than loading everything into memory at once. This approach is essential for handling large datasets, real-time data processing, and maintaining responsive applications.

Types of Streams

Readable Streams : These are like the inflow of water into your riverboat, providing data that your application can consume. Examples include file reading, HTTP requests, and data from databases.
Writable Streams : These represent the outflow, where your application sends data, such as writing to files, sending HTTP responses, or logging information.
Duplex Streams : These handle both reading and writing, akin to a boat that can both receive passengers and deliver cargo simultaneously.
Transform Streams : These are duplex streams that can modify or transform the data as it passes through, much like a filtration system on the riverboat that cleans or alters incoming water.

The Backpressure Mechanism

In our riverboat analogy, if the downstream (writable stream) can’t handle the current inflow (readable stream), backpressure signals the readable stream to slow down. Node.js streams handle this automatically using internal buffers and signalling mechanisms to ensure data flows smoothly without overwhelming any part of the system.

3. Detecting Backpressure

Detecting backpressure is like spotting rapids ahead on the river. It’s crucial to identify when the data flow is outpacing the system’s capacity so you can take corrective action. Here’s how you can spot and understand backpressure in your Node.js applications:

Signs of Backpressure

Memory Growth : If your application’s memory usage keeps increasing without bounds, it might be because data is being buffered faster than it’s being processed.
Sluggish Performance : Noticeable slowdowns in data processing or response times can indicate that the consumer can’t keep up with the producer.
Event Backlogs : An accumulation of pending events or callbacks suggests that the system is struggling to process incoming data.

Tools for Detection

Node.js Built-in Events : Streams emit events like data, drain, error, and end that can help you monitor and react to backpressure.
Monitoring and Profiling Tools : Tools like Node.js’s process.memoryUsage(), Chrome DevTools and APM solutions can provide insights into your application’s performance and resource usage.

4. Managing Backpressure

Now that we’ve identified backpressure, how do we manage it effectively? Let’s continue our riverboat journey by exploring practical strategies to handle backpressure in a Node.js application using TypeScript.

Setting Up a Readable Stream

Imagine your readable stream as the flow of passengers boarding the boat. Here’s how you can set up a readable stream in TypeScript:

import { Readable } from 'stream';

class PassengerStream extends Readable {
  private currentPassenger = 0;
  private maxPassengers = 100;constructor(options?: any) {
    super(options);
  }
  _read(size: number) {
    if (this.currentPassenger >= this.maxPassengers) {
      this.push(null); // No more passengers
    } else {
      const passenger = `Passenger ${this.currentPassenger}`;
      this.push(Buffer.from(passenger));
      this.currentPassenger += 1;
    }
  }
}

Setting Up a Writable Stream

The writable stream is like the unloading of passengers at their destination. Here’s a writable stream example:

import { Writable } from 'stream';

class DestinationStream extends Writable {
  _write(chunk: Buffer, encoding: string, callback: Function) {
    const passenger = chunk.toString();
    console.log(`Dropping off: ${passenger}`);
    // Simulate asynchronous unloading
    setTimeout(() => {
      callback();
    }, 100);
  }
}

Connecting Streams with Backpressure Handling

Now, let’s connect our passenger stream to the destination stream, ensuring we handle backpressure effectively:

const passengerStream = new PassengerStream();
const destinationStream = new DestinationStream();

passengerStream.on('data', (chunk) => {
  const canContinue = destinationStream.write(chunk);
  if (!canContinue) {
    passengerStream.pause();
    console.log('Pausing passenger stream due to backpressure.');
  }
});

destinationStream.on('drain', () => {
  passengerStream.resume();
  console.log('Resuming passenger stream.');
});

passengerStream.on('end', () => {
  destinationStream.end();
  console.log('All passengers have been dropped off.');
});

Breaking Down the Code -

1. Creating the Streams

const passengerStream = new PassengerStream();
const destinationStream = new DestinationStream();

PassengerStream : A readable stream emitting data chunks (passengers).
DestinationStream : A writable stream that processes the incoming data chunks.

2. Handling the 'data' Event

passengerStream.on('data', (chunk) => {
  const canContinue = destinationStream.write(chunk);
  if (!canContinue) {
    passengerStream.pause();
    console.log('Pausing passenger stream due to backpressure.');
  }
});

Listening to 'data' Events : The passengerStream emits a 'data' event whenever it has new data (a new passenger).
Writing to DestinationStream : Attempts to write the incoming data chunk to destinationStream.
Managing Backpressure : If write() returns false, it indicates that destinationStream cannot handle more data at the moment, so we pause the passengerStream to prevent overwhelming it.

3. Handling the 'drain' Event

destinationStream.on('drain', () => {
  passengerStream.resume();
  console.log('Resuming passenger stream.');
});

Listening to 'drain' Events : The destinationStream emits a 'drain' event when it's ready to receive more data after being previously overwhelmed.
Resuming the PassengerStream : Once the 'drain' event is fired, we resume the passengerStream, allowing data flow to continue.

4. Handling the 'end' Event

passengerStream.on('end', () => {
  destinationStream.end();
  console.log('All passengers have been dropped off.');
});

Listening to the 'end' Event : The passengerStream emits an 'end' event when there is no more data to be read (no more passengers to board).
Ending the DestinationStream : Signals that no more data will be written to the writable stream, allowing it to perform any final operations and gracefully close.

⭐Leveraging Async Iterators for Cleaner Handling

TypeScript’s support for async iterators can make backpressure management more intuitive, much like having an automated system to handle passenger flow without manual intervention:

async function ferryPassengers() {
  for await (const chunk of passengerStream) {
    const canContinue = destinationStream.write(chunk);
    if (!canContinue) {
      await new Promise<void>((resolve) => destinationStream.once('drain', resolve));
      console.log('Resumed passenger stream after drain.');
    }
  }
  destinationStream.end();
  console.log('All passengers have been successfully ferried.');
}

ferryPassengers().catch((error) => console.error('Error ferrying passengers:', error));

Async Iterators : The for await...of loop asynchronously iterates over the passengerStream, handling each chunk of data.
Handling Backpressure : If write() returns false, the loop waits for the 'drain' event before continuing, ensuring that the destinationStream can handle more data.

5. Harnessing the Power of Pipelines

While manually managing backpressure using events like 'data', 'drain', 'pause', and 'resume' provides fine-grained control, Node.js offers higher-level abstractions to simplify this process: pipeline and pipelineAsync. These utilities handle backpressure, error propagation, and stream closure automatically, making your code cleaner and more maintainable.

Introducing the Pipeline Concept

Think of a pipeline as a series of connected river sections, each handling different aspects of the journey. Instead of manually managing each section's flow, you have a system that ensures data moves smoothly from one section to the next, automatically adjusting for any blockages or changes in flow.

Using pipeline

The pipeline function is a callback-based utility that connects multiple streams, handling errors and backpressure seamlessly. Here's how you can use it in our riverboat analogy:

import { pipeline } from 'stream';

const passengerStream = new PassengerStream();
const destinationStream = new DestinationStream();

pipeline(
  passengerStream,
  destinationStream,
  (err) => {
    if (err) {
      console.error('Pipeline encountered an error:', err);
    } else {
      console.log('All passengers have been successfully ferried using pipeline.');
    }
  }
);

Benefits of Using pipeline:

Automatic Backpressure Handling : Manages the flow of data between streams without manual intervention.
Error Handling : Propagates errors from any stream in the pipeline to a single callback, simplifying error management.
Resource Management : Ensures that all streams are properly closed, preventing resource leaks.

Using pipelineAsync

The introduction of promises in Node.js, pipelineAsync provides a promise-based version of pipeline, allowing for cleaner asynchronous code using async/await. This is particularly useful in TypeScript, where you can leverage type safety and asynchronous patterns more effectively.

import { pipeline } from 'stream/promises';

async function ferryPassengersWithPipeline() {
  const passengerStream = new PassengerStream();
  const destinationStream = new DestinationStream();

  try {
    await pipeline(
      passengerStream,
      destinationStream
    );
    console.log('All passengers have been successfully ferried using pipelineAsync.');
  } catch (err) {
    console.error('PipelineAsync encountered an error:', err);
  }
}

ferryPassengersWithPipeline();

Advantages of pipelineAsync:

Promise-Based : Integrates seamlessly with async/await, making asynchronous code more readable and maintainable.
Simplified Error Handling : Uses try...catch blocks to manage errors, aligning with modern JavaScript practices.
Enhanced Readability : Reduces callback nesting, resulting in cleaner and more understandable code.

Comparing Manual Management vs. Pipelines

Manual Management:

Pros :
Fine-grained control over each step of the data flow.
Customizable handling for specific backpressure scenarios.
Cons :
More verbose and complex code.
Higher risk of errors due to manual handling of events and state.
Increased maintenance overhead.

Using Pipelines:

Pros :
Simplifies code by abstracting backpressure and error handling.
Reduces boilerplate, making the codebase cleaner.
Enhances reliability by leveraging built-in stream management.
Cons :
Less control over individual stream interactions.
May require an understanding of the pipeline abstraction.

When to Use Pipelines

Standard Stream Connections : When connecting multiple streams without needing specialized control.
Error-Prone Scenarios : When you want to ensure robust error handling without manually propagating errors.
Asynchronous Operations : When working with asynchronous patterns and prefer async/await for flow control.

Practical Example: Combining pipeline and pipelineAsync

Let’s enhance our passenger and destination streams by introducing a transformation step. Suppose we want to uppercase all passenger names before dropping them off.

Using pipeline with a Transform Stream

import { Readable, Writable, Transform, pipeline } from 'stream';

class PassengerStream extends Readable {
  private currentPassenger = 0;
  private maxPassengers = 100;

  constructor(options?: any) {
    super(options);
  }

  _read(size: number) {
    if (this.currentPassenger >= this.maxPassengers) {
      this.push(null);
    } else {
      const passenger = `Passenger ${this.currentPassenger}`;
      this.push(Buffer.from(passenger));
      this.currentPassenger += 1;
    }
  }
}

class DestinationStream extends Writable {
  _write(chunk: Buffer, encoding: string, callback: Function) {
    const passenger = chunk.toString();
    console.log(`Dropping off: ${passenger}`);
    setTimeout(() => {
      callback();
    }, 100);
  }
}

class UppercaseTransform extends Transform {
  _transform(chunk: Buffer, encoding: string, callback: Function) {
    const uppercased = chunk.toString().toUpperCase();
    this.push(Buffer.from(uppercased));
    callback();
  }
}

const passengerStream = new PassengerStream();
const uppercaseTransform = new UppercaseTransform();
const destinationStream = new DestinationStream();

pipeline(
  passengerStream,
  uppercaseTransform,
  destinationStream,
  (err) => {
    if (err) {
      console.error('Pipeline encountered an error:', err);
    } else {
      console.log('All passengers have been successfully ferried using pipeline with transform.');
    }
  }
);

Using pipelineAsync with a Transform Stream

import { pipeline } from 'stream/promises';

async function ferryPassengersWithPipelineAsync() {
  const passengerStream = new PassengerStream();
  const uppercaseTransform = new UppercaseTransform();
  const destinationStream = new DestinationStream();

  try {
    await pipeline(
      passengerStream,
      uppercaseTransform,
      destinationStream
    );
    console.log('All passengers have been successfully ferried using pipelineAsync with transform.');
  } catch (err) {
    console.error('PipelineAsync encountered an error:', err);
  }
}

ferryPassengersWithPipelineAsync();

6. Best Practices and Expert Tips

For those looking to navigate the river with finesse, here are some best practices and advanced tips to master backpressure handling in Node.js with TypeScript:

1. Optimize highWaterMark

The highWaterMark parameter defines the buffer size before backpressure is applied. Setting it appropriately ensures efficient data flow without excessive memory usage.

const passengerStream = new PassengerStream({ highWaterMark: 16 }); // Adjust based on use case

Smaller highWaterMark : Reduces memory consumption but may increase the number of backpressure events.
Larger highWaterMark : Increases memory usage but can improve throughput by reducing the frequency of backpressure handling.

2. Use .pipe() or pipeline When Possible

Node.js’s .pipe() method automatically manages backpressure, simplifying the connection between streams.

passengerStream.pipe(destinationStream).on('finish', () => {
  console.log('All passengers have been ferried using pipe.');
});

Advantages :
Simplifies code by abstracting backpressure and error handling.
Enhances readability and maintainability.

3. Handle Stream Errors Gracefully

Always listen for error events to prevent your application from crashing unexpectedly.

passengerStream.on('error', (err) => {
  console.error('Passenger Stream Error:', err);
});

destinationStream.on('error', (err) => {
  console.error('Destination Stream Error:', err);
});

Why It Matters : Unhandled stream errors can cause your application to terminate, leading to data loss or inconsistent states.

4. Monitor and Profile Regularly

Use monitoring tools to keep an eye on your application’s performance, ensuring backpressure is managed effectively and efficiently.

Node.js’s process.memoryUsage(): Monitor memory consumption.
Chrome DevTools : Profile and debug your application.
APM Solutions : Application Performance Monitoring tools like New Relic or Datadog for comprehensive insights.

5. Embrace TypeScript’s Type Safety

Leverage TypeScript’s strong typing to define clear interfaces for your streams and handlers, catching potential issues at compile time.

interface Passenger {
  id: number;
  name: string;
}

class PassengerStream extends Readable {
  // Define types for enhanced safety
}

Benefits :
Reduces runtime errors by catching type mismatches during development.
Enhances code readability and maintainability.

6. Implement Custom Backpressure Strategies

Depending on your application’s needs, you might need to implement custom strategies to handle specific backpressure scenarios, such as throttling data or prioritizing certain data streams.

Examples :
Throttling : Limit the rate at which data is processed to match the consumer's capacity.
Prioritization : Assign priorities to different data streams, ensuring critical data is processed first.

7. Use Transform Streams for Data Manipulation

Transform streams allow you to modify or process data as it flows through the pipeline, enabling powerful data processing capabilities.

class FilterTransform extends Transform {
  _transform(chunk: Buffer, encoding: string, callback: Function) {
    const data = chunk.toString();
    if (shouldProcess(data)) {
      this.push(Buffer.from(data));
    }
    callback();
  }
}

function shouldProcess(data: string): boolean {
  // Implement your filtering logic here
  return data.includes('Passenger');
}

8. Leverage Asynchronous Patterns

Use async/await and promise-based utilities like pipelineAsync to write clean and maintainable asynchronous code.

async function processStreams() {
  try {
    await pipeline(
      passengerStream,
      transformStream,
      destinationStream
    );
    console.log('Processing complete.');
  } catch (err) {
    console.error('Error processing streams:', err);
  }
}

processStreams();

9. Understand Stream Modes

Streams can operate in different modes, such as flowing and paused. Understanding these modes helps you manage data flow effectively.

Flowing Mode : Data is read automatically from the underlying source.
Paused Mode : Data is read manually using methods like .read().

10. Stay Updated with Node.js Enhancements

Node.js continuously evolves, introducing new stream utilities and performance improvements. Stay informed about the latest features to leverage them in your applications.

7. Conclusion

Managing backpressure is akin to steering a riverboat through a dynamic and sometimes unpredictable river. Whether you're just starting out, navigating intermediate challenges, or mastering advanced techniques, understanding and handling backpressure is crucial for building scalable, efficient, and resilient Node.js applications.

By embracing the principles outlined in our journey—from recognizing backpressure through the riverboat analogy to implementing effective strategies in TypeScript and harnessing the power of pipelines—you can ensure your applications remain robust under varying data loads. Remember, just as a skilled captain anticipates and adjusts to the river’s flow, a proficient developer anticipates data flow challenges and applies the right tools and practices to navigate them successfully.

Peace out!✌️