Recursive Directory Removal in Node.js

Recursive directory removal has landed in Node.js v12.10.0!

This has been a long-standing feature request. New Node.js developers often express incredulity when they discover this particular “battery” isn’t included in Node.js.

Over the years, userland modules (rimraf, rmdir, del, fs-extra, etc.) have heroically provided what core did not. Thanks to the superbad maintainers-of and contributors-to these packages!

Here's a little story about how it came to pass—and why something so seemingly simple as rm -rf isn’t necessarily so.

About Node.js’ Filesystem Operations

First, I want to explain a bit about how Node.js works under the hood with regards to filesystem operations.

libuv provides filesystem operations to Node.js. Node.js’ fs module is just a JavaScript file which provides the fs.* APIs; those APIs call into an internal C++ binding (you could think of this as a “native module”). That binding is glue between libuv and the JavaScript engine ( V8 ).

Here’s an example. At the lowest level, libuv provides a C API (uv_fs_rmdir) to make the a system call to remove a directory.

const fs = require('fs');

// `rmdir` is just a function which calls into a C++ binding.
// The binding asks libuv to remove the "/tmp/foo" directory.
// Once libuv returns a result, the binding calls `callback`
fs.rmdir('/tmp/foo', function callback(err) {
  if (err) {
    // handle error
  }
});

Importantly, Node.js makes only a single call to libuv above_._

In fact, until recently, Node.js’ fs bindings follow a pattern: single calls into libuv. fs.readFile, fs.stat, fs.unlink; these are all just one call.

Oh, that recent change? It was recursive fs.mkdir. I’ll explain what makes it different.

Shell Operations vs. System Operations

Developers may not think about this much because it’s so well-abstracted by our tools. Take mkdir, for example:

$ mkdir ./foo

mkdir is a command-line utility (which flavor, exactly, depends on your operating system). It’s not a system call. The above command may only execute a single system call, but the following may execute several:

# creates dirs foo, then bar, then baz, ignoring dirs that already exist
$ mkdir -p ./foo/bar/baz

Unless our tools have transactional behavior—they can “commit” or “roll back” operations—it’s possible for this command to partially succeed (though maybe not obvious in this case, but trust me).

What happens if mkdir -p fails halfway through? It depends. You get zero or more new directories. Yikes!

If that seems weird, consider that the user may want to keep the directories it did create. It’s tough to make assumptions about this sort of thing; cleanup is best left to the user, who can deal with the result as they see fit.

How does this relate to Node.js? When a developer supplies the recursive: true option to fs.mkdir, Node.js will potentially ask libuv to make several system calls—all, some, or none of which may succeed.

Previous to the addition of recursive fs.mkdir, Node.js had no precedent for this behavior. Still, its implementation is relatively straightforward; when creating directories, the operations must happen both in order and sequentially—we can’t create bar/baz/ before we create bar/!

It may be surprising, then, that a recursive rmdir implementation is another beast entirely.

There Was An Attempt

I was likely not the first to attempt to implement a recursive rmdir in Node.js at the C++ level, but I did try, and I’ll explain why it didn’t work.

The idea was that a C++ implementation could be more performant than a JavaScript implementation—that’s probably true!

Using mkdir as a template, I began coding. My algorithm would perform a depth-first traversal of the directory tree using libuv ’s uv_fs_readdir; when it found no more directories to descend into, it would call uv_fs_unlink on each file therein. Once the directory was clear of files, it would ascend to the parent, and finally remove the now-empty directory.

It worked! I was very proud of myself. Then I decided to run some benchmarks against rimraf. Maybe I shouldn't have!

I found out that my implementation was faster for a very small N, where N is the number of files and directories to remove. But N didn’t have to grow very large for userland's rimraf to overtake my implementation.

Why was mine slower? Besides using an unoptimized algorithm, I used recursive mkdir as a template, and mkdir works in serial (as I mentioned above). So, my algorithm only removed one file at a time. rimraf , on the other hand, queued up many calls to fs.unlink and fs.rmdir. Because libuv has a thread pool for filesystem operations, it could speedily blast a directory full of files, only limited by its number of threads!

Note that when we say Node.js is single-threaded, we're talking about the programming model. Under the hood, I/O operations are multithreaded.

At this point, I realized that if it was going to be “worth it” to implement at the C++ layer—meaning a significant performance advantage which outweighs-the- maintenance-costs-of-more-C++-code—I’d have to rewrite the implementation to manage its own thread pool. Of course, there’s no great precedent for that in Node.js either. It’d be possible, but very tricky, and best left to somebody with a better handle on C++ and multithreaded programming.

I went back to the Node.js tooling group and explained the situation. We decided that the most feasible way forward would be a pure-JavaScript implementation of recursive directory removal.

Let’s Write It In JavaScript!

Well, that was the idea, but we didn’t get very far. We took a look at the source of rimraf, which is the most popular userland implementation. It’s not as straightforward as you’d expect! It covers many edge cases and peculiarities (and all of those hacks would need to be present in a Node.js core implementation; it needs to work like a consumer would expect).

Furthermore, rimraf is stable, and these workarounds have proven themselves to be robust over the years that it’s been consumed by the ecosystem.

I won’t attempt to explain what rimraf must do to achieve decent performance in a portable manner—but rest assured it’s sufficiently non-trivial. So non-trivial, in fact, that it made more sense to just pull rimraf into Node.js core instead of trying to code it again from scratch.

So that’s what we did.

It’s Just rimraf

Ian Sutherland extracted the needed code from rimraf. In particular, rimraf supplies a command-line interface, and we didn’t need that. For simplicity (and to eliminate dependencies) glob support (e.g., foo/**/*.js) was also dropped (though it may still have a future). After this, it was a matter of integrating it into a Node.js-style API, and the needed docs and tests.

To be clear, recursive directory removal in Node.js does not make rimraf obsolete. It does mean that for many use cases, Node.js’ fs.rmdir can get the job done. Stick with rimraf if you need globs or a portable command-line utility.

Thanks to Isaac Schlueter for rimraf —and to bless Node.js’ copy-and-paste efforts.

In Conclusion

That’s the story of Node.js’ recursive rmdir thus far. Want to help write the rest? Come participate in the Node.js Tooling Group, where we’re looking to make Node.js the best platform it can be for building CLI apps.

Acknowledgements to Isaac Schlueter and Ian Sutherland for reviewing this post.

This post originally appeared on boneskull.com on Sep 9, 2019.