Three Ways to Retrieve JSON from the Web using Node.js

Cover image credit: Hunter x Hunter manga by Yoshihiro Togashi, meme-ified by yours truly. <3

In a recent technical challenge, I was asked to build a small Node.js app that first needed to retrieve some JSON from the web. Since I'm still relatively new to Node.js, I did not realize that Node.js does not natively include the fetch() API, which I was used to using in my front-end JavaScript.

(What I didn't realize is that fetch() is actually a method from the Window interface--and not having a front-end, there was no window! Awkward thing to realize at the beginning of a technical interview...)

After getting through the challenge, I spent this past weekend refactoring the code and experimenting with a few different ways to retrieve JSON from the web using Node.js. Spoiler: there's an http/https module in Node.js, but also some cool packages that mimic fetch(), or that simplify request syntax even further!

But before we get into that, I want to introduce a tool that I foolishly did not use during my technical challenge: JSONView, a super-handy web extension for Chrome and Firefox that pretty-prints JSON in your browser.

JSONView, a super-handy web extension for Chrome and Firefox that pretty-prints JSON in your browser

One issue I had during my code challenge was (due to my own error) not working on my own computer, which includes this extension. Compare the following:

Unformatted JSON from https://www.reddit.com/r/popular.json

versus

Same JSON from https://www.reddit.com/r/popular.json, pretty-printed with JSONView

On top of that, hovering your cursor over a particular field will show the path to access it:

Cursor hovering over the "ups" field, with the path shown in the bottom-left corner

Having this handy will make parsing and accessing the data you need way faster and easier.

Replicating fetch() with 'node-fetch' package

The node-fetch package does pretty much what you expect: provide you with the fetch() syntax in Node.js. To install, run npm install node-fetch, and set up your code like this:

const fetch = require('node-fetch');

let url = "https://www.reddit.com/r/popular.json";

let settings = { method: "Get" };

fetch(url, settings)
    .then(res => res.json())
    .then((json) => {
        // do something with JSON
    });

Here, we've started by importing the package via require(), and created a settings variable to define our http method as a Get request. From there, we use fetch(url, settings) just like we would on the front-end. As usual, we can parse the response res as JSON, and then do whatever we need to with it.

Note: from some VERY RUDIMENTARY benchmark testing, it appears that node-fetch is the fastest of the three options covered in this article. Here are the times clocked by each (however, this DOES include running the rest of the code from the challenge, not just the fetch/https/request itself):

fetch: 0.689 seconds
https: 2.827 seconds
request: 3.65 seconds

I'd love for someone else to do a little more testing and verify/disprove this! Feel free to comment below if you're that person. ;)

Using the http/https modules provided by Node.js

Node.js comes with a pair of http/https modules, and in this case, the https module provides a built-in method for Get requests. Here's the code we'll be looking at:

const https = require('https');

let url = "https://www.reddit.com/r/popular.json";

https.get(url,(res) => {
    let body = "";

    res.on("data", (chunk) => {
        body += chunk;
    });

    res.on("end", () => {
        try {
            let json = JSON.parse(body);
            // do something with JSON
        } catch (error) {
            console.error(error.message);
        };
    });

}).on("error", (error) => {
    console.error(error.message);
});

There's a bit more going on here! First, we import the https module with require(). We can then call https.get(url, (res) => {} ) to initiate a Get request. Then, inside the body of the callback, we start by creating an empty string body that we'll add our the text of our response (again called res) to.

From there, we have a few examples of the .on syntax, which will listen for a few different events--namely, "data", "end", and "error".

When the response encounters "data", we add each chunk as text to our body variable. Once we hit the "end" of the response, we use the try / catch syntax to try to parse our body's text as JSON, and return an error if it can't. Lastly, we chain another .on call to catch "error" for our initial https.get() request.

I find this syntax to be pretty clunky and verbose, although I do like the explicit error handling that is required by https.get(). However, this module is slower than the node-fetch package--see the benchmark results above.

Simplifying syntax with 'request' package

The third strategy I used was the request package, which aims to simplify the (often verbose) syntax of Node.js's http requests. Since this is an external package, start by installing it with npm install request.

Here's the code we'll be looking at:

const request = require('request');

let url = "https://www.reddit.com/r/popular.json";

let options = {json: true};



request(url, options, (error, res, body) => {
    if (error) {
        return  console.log(error)
    };

    if (!error && res.statusCode == 200) {
        // do something with JSON, using the 'body' variable
    };
});

Wow, that's really readable! Let's break it down. As with the other examples, we import the package with require(), and set our url variable. The request package also has a nifty options feature, where you can specify a lot of things--but here, in setting { json: true }, we tell the request to automatically parse the response's body as JSON if there's no error (and we get a 200 status code back). So, to access the JSON we want, just use the body variable!

This readability comes at the price of speed, however. Per the benchmark results above, this is the slowest option, most likely because so much is happening under-the-hood. However, the readability is top-notch, and configuring other http requests are just as simple as this Get request example!

Conclusion

This particular technical challenge was a great opportunity to dive into Node.js's http requests! Now, you should feel armed with a variety of tools to meet different situations.

As I said above, I'd love to have another person do some testing/benchmarking and verify or disprove the speed test results that I got! Since testing is still relatively new to me, I'd very much like to see how others approach benchmarking these methods. Thanks for reading, and feel free to comment below!

Comments

[Voted Jason Of The Year 2003-2007]

Hi Isa, this is very helpful! I have a really dumb question...feel free to ignore me, I'm new to node and promises and kinda new to Javascript.

In the node-fetch example, what if what I wanted to do with the json was return it?

Like:
const getStuffFromWhatever = (item) => {
let url = whatever?thing=item

let settings = { method: "Get" };

fetch(url, settings)
.then(res => res.json())
.then((json) => {
    return json
});

}

console.log(getStuffFromWhatever('puppy dogs')

I have tried a few things that seem reasonable to me, but they don't work. I could move the fetch into the function I actually need it in, but I will need it again elsewhere and that's not very DRY.

[Voted Jason Of The Year 2003-2007]

Every time I put myself out there and ask a question I am embarrassed to ask, I always find the answer 5 minutes later, and this time is no different. The answer is async / await! Of course it is.

Thanks for letting me talk it out.

[Isa Levine]

Always happy to be your rubber ducky! :)

[David Wickes]

Hey Isa - nice article! I think you picked the three ways I'd recommend for using an http client in Node.

A thought about the benchmarking of the the Node built in http lib: have you tried building a different data structure than a string - some sort of byte buffer - and then turning it into a string at the end. I'm not sure but I think it might be (I'm away from my computer so I can't try it myself).

[Isa Levine]

Hi David, thank you for the feedback! I'm looking into byte buffers (am I seeing this is originally a Java data structure?) for JavaScript, and I'm falling down a rabbit-hole about ArrayBuffers, Uint8Arrays, etc. I found my way over to the npm package bytebuffer (npmjs.com/package/bytebuffer), which looks like it provides a friendly API for using those structures.

I'm running up against a challenge of understanding how best to theoretically write to a byte buffer with incoming http request data--from this list of byte-buffer-writing options (github.com/protobufjs/bytebuffer.j...), do you have any advice on where I should be looking? I'm not sure which option will be more efficient than simply coercing the incoming data into a string.

[IHZAQSTORM33]

Hi Isa,
Well can you explained little bit more further?
I mean look

const fetch = require('node-fetch');

let url = "https://www.reddit.com/r/popular.json";

let settings = { method: "Get" };

fetch(url, settings)
    .then(res => res.json())
    .then((json) => {
        // do something with JSON
});

So you mean, do something with JSON is example:

console.log(JSON.kind)

is that How to use it? thanks.

[Geigle]

Thank you for this helpful guide. It got me far.
I was advised to use the new modern fetch() package.
developer.mozilla.org/en-US/docs/W...