Sahil Verma

Posted on Jan 8, 2022 • Edited on Jan 11, 2022

How I made Instagram Content Fetching API in Node.js

#express #instadownloader #javascript #node

So I was planning to build an Instagram downloading website. That's I was researching about this on Google, then I found this amazing trick.

If you put ?__a=1 after any Instagram link then it'll give a JSON in response.

For Instagram post

https://www.instagram.com/p/{post_id}/?__a=1

For Instagram user profile

https://www.instagram.com/p/{profile_username}/?__a=1

But if you try to fetch this API inside your code then, you'll get a CORS error like this.

To overcome this problem we have to use our server. So let's start building a node server for this project.

First, make a new folder and open that folder.

mkdir instagram-api
cd instagram-api

Then initialize a node project.

npm init -y

Now install the following dependencies.

npm i request express

Make a new JavaScript file.

touch index.js

Open VS Code in this folder.

code .

Modify your scripts inside package.json file.

Before

"scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
},

After

"scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "start": "nodemon index.js" // you must have to install nodemon globally by npm install -g nodemon
},

Open the index.js file in VS Code and import all dependencies.

const express = require("express");
const request = require("request");

Now start listening to the port you want to.

const express = require("express");
const request = require("request");

const PORT = 1234;

const app = express();
app.listen(PORT, () => {
  console.clear();
  console.log(`listing in http://localhost:${PORT}`);
});

You can start the server to see the results side by side.

npm start

Now add a root route for testing and open the server in your browser.

app.get("/", (req, res) => {
  res?.send("I am working.");
});

Add another route for Instagram posts.

app.get("/post/:id", (req, res) => {
  const { id } = req?.params;
  if (id) {
    res.send(`Post id = ${id}`);
  }
});

Now I will use the request package to scrape the response data. It will scrape the data from Instagram API as a simple string and then I'll parse that string into JSON.

app.get("/post/:id", (req, res) => {
  const { id } = req?.params;
  if (id) {
    const link = `https://www.instagram.com/p/${id}/?__a=1`;
    request(link, (err, response, html) => {
      if (!err) {
        const json = JSON.parse(html);
        if (json) res.send(json);
      }
    });
  }
});

And now if I look into my server with a valid Instagram post id I'll see something like this.

I am getting a lot of data but I need only a certain type of data.
So I'll destructor it.

const json = JSON.parse(html)?.graphql?.shortcode_media;
res?.send({
  id,
  mainContent: {
    displayUrl: json?.display_url,
    resolutions: json?.display_resources.map((item) => ({
      src: item?.src,
    })),
  },
  user: {
    username: json?.owner?.username,
    isVerified: json?.owner?.is_verified,
    profilePic: json?.owner?.profile_pic_url,
  },
  caption: {
    main: json?.edge_media_to_caption.edges[0]?.node.text,
    accessibile: json?.accessibility_caption,
  },
  location: json?.location,
  comments: json?.edge_media_to_parent_comment?.count,
  likes: json?.edge_media_preview_like?.count,
  isVideo: json?.is_video,
  videoUrl: json?.video_url,
  allImages: json?.edge_sidecar_to_children?.edges?.map((item) => ({
    displayUrl: item?.node?.display_url,
    resolutions: item?.node?.display_resources?.map((item) => ({
      src: item?.src,
      size: item?.config_width,
    })),
  })),
});

Now its much clean and it gives me what I need.

You can also destructor it according to your needs.
I'll also do same for the user profile.

app.get("/user/:username", (req, res) => {
  const { username } = req?.params;
  if (username)
    const link = `https://www.instagram.com/${username}/?__a=1`;
    request(link, (err, response, html) => {
      if (!err) {
        const json = JSON.parse(html)?.graphql?.user;
        if (json)
          res?.send({
            username,
            bio: json.biography,
            profilePic: {
              sd: json?.profile_pic_url,
              hd: json?.profile_pic_url_hd,
            },
            fullName: json?.full_name,
            following: json?.edge_followed_by.count,
            follows: json?.edge_follow.count,
            posts: {
              total: json?.edge_owner_to_timeline_media?.count,
              content: json?.edge_owner_to_timeline_media?.edges?.map(
                (item) => ({
                  id: item?.node?.shortcode,
                  displayUrl: item?.node?.display_url,
                  caption: {
                    main: item?.node?.edge_media_to_caption.edges[0].node.text,
                    accessibile: item?.node?.accessibility_caption,
                  },
                  isVideo: item?.node?.is_video,
                  likes: item?.node?.edge_liked_by.count,
                  location: item?.node?.location,
                  comments: item?.node?.edge_media_to_comment.count,
                })
              ),
            },
          });
      }
    });
});

Now it's time to use this API inside the frontend.
I'll use React JS as a frontend framework and axios for fetching API in this project.

I'll fetch this API and set images to images.
As you can see in this block of code.

useEffect(() => {
  const [images, setImages] = useState();
  const getData = async () => {
    const { data } = await axios("http://localhost:1234/post/CYa0_SRtUrf");
    if (data) {
      setImages(data);
      console.log(data);
    }
  };
  getData();
}, []);

Now if I check the console there will be another CORS error.

To solve this error install another npm package called cros.

npm i cors

Now import it and use it

const express = require("express");
const cors = require("cors");

const app = express();
app.use(cors());

Now It will work.

But now, if I try to display images inside my webpage, it will again give me an error:

It is a bit difficult to deal with this error. But there are two ways to solve this problem. But I will show only the most convenient.

To solve this problem I'll convert to images I am getting from Instagram to base64.

If you don't know about base64 then it is.

Base64 is a group of binary-to-text encoding schemes that represent binary data (more specifically, a sequence of 8-bit bytes) in an ASCII string format by translating the data into a radix-64 representation. The term Base64 originates from a specific MIME content transfer encoding. Each non-final Base64 digit represents exactly 6 bits of data. Three bytes (i.e., a total of 24 bits) can therefore be represented by four 6-bit Base64 digits.
Source Wikipedia : Base64

To convert image to base64 first install another package called image-to-base64.

npm i image-to-base64

Import image-to-base641 and make a new asynchronous function getBase64.

const imageToBase64 = require("image-to-base64");

const getBase64 = async (link) => {
  const base64 = await imageToBase64(link);
  return `data:image/jpeg;base64,${base64}`;
};

This function will take the URL of the Instagram image and return it to the base64 code.
Since this is an asynchronous function, I need to wait for the response.
I have to change my code so that I get base64 in response.
Here is the final code.

const express = require("express");
const request = require("request");
const imageToBase64 = require("image-to-base64");
const cors = require("cors");

const PORT = 1234;

const app = express();
app.listen(PORT, () => {
  console.clear();
  console.log(`Listing in http://localhost:${PORT}`);
});

app.use(cors());

const getBase64 = async (link) => {
  const base64 = await imageToBase64(link);
  return `data:image/jpeg;base64,${base64}`;
};

app.get("/", (req, res) => {
  res?.send("I am working.");
});

app.get("/post/:id", (req, res) => {
  const { id } = req?.params;
  if (id) {
    const link = `https://www.instagram.com/p/${id}/?__a=1`;
    request(link, (err, response, html) => {
      if (!err) {
        const json = JSON.parse(html)?.graphql?.shortcode_media;
        const promiseArray = json?.edge_sidecar_to_children?.edges?.map(
          async (item) => ({
            displayUrl: await getBase64(item?.node?.display_url),
            resolutions: item?.node?.display_resources?.map((item) => ({
              src: item?.src,
              size: item?.config_width,
            })),
          })
        );
        let allImages;
        if (promiseArray) allImages = Promise.all(promiseArray);
        if (json) {
          (async () => {
            res?.send({
              id,
              mainContent: {
                displayUrl: await getBase64(json?.display_url),
                resolutions: json?.display_resources.map((item) => ({
                  src: item?.src,
                })),
              },
              user: {
                username: json?.owner?.username,
                isVerified: json?.owner?.is_verified,
                fullName: json?.owner?.full_name,
                profilePic: await getBase64(json?.owner?.profile_pic_url),
              },
              caption: {
                main: json?.edge_media_to_caption.edges[0]?.node.text,
                accessibile: json?.accessibility_caption,
              },
              location: json?.location,
              comments: json?.edge_media_to_parent_comment?.count,
              likes: json?.edge_media_preview_like?.count,
              isVideo: json?.is_video,
              videoUrl: json?.video_url,
              allImages: await allImages,
            });
          })();
        }
        if (!json) res?.status?.send("error");
      }
    });
  }
});
app.get("/user/:username", (req, res) => {
  const { username } = req?.params;
  if (username) {
    const link = `https://www.instagram.com/${username}/?__a=1`;
    request(link, (err, response, html) => {
      if (!err) {
        const json = JSON.parse(html)?.graphql?.user;
        const promiseArray = json?.edge_owner_to_timeline_media?.edges?.map(
          async (item) => ({
            displayUrl: await getBase64(item?.node?.display_url),
            id: item?.node?.shortcode,
            location: item?.node?.location,
            caption: {
              main: item?.node?.edge_media_to_caption.edges[0].node.text,
              accessibile: item?.node?.accessibility_caption,
            },
            comments: item?.node?.edge_media_to_comment.count,
            isVideo: item?.node?.is_video,
            likes: item?.node?.edge_liked_by.count,
            isCollection: item?.node?.edge_sidecar_to_children ? true : false,
            resolutions: item?.node?.thumbnail_resources,
          })
        );
        let allImages;
        if (promiseArray) allImages = Promise.all(promiseArray);
        if (json)
          (async () => {
            res?.send({
              username,
              bio: json.biography,
              isVerified: json?.is_verified,
              category: json?.category_name,
              externalURL: json?.external_url,
              profilePic: {
                sd: await getBase64(json?.profile_pic_url),
                hd: await getBase64(json?.profile_pic_url_hd),
              },
              fullName: json?.full_name,
              following: json?.edge_followed_by.count,
              follows: json?.edge_follow.count,
              posts: {
                total: json?.edge_owner_to_timeline_media?.count,
                content: await allImages,
              },
            });
          })();
        if (!json) res?.status(400).send("ERROR");
      }
    });
  }
  if (!username) res?.send(`Error`);
});

Now if you try this API in the frontend it will perfectly work. I have build an Instagram Downloader web-app with the help of this API.

Here is the preview.
Post

User

End

Now let me tell you more about myself that it is my very first API. I don't know much about building an API. So if you don't like my code and want to give some feedback then feel free to suggest me in comments, and also check out my GitHub repo and contribute to make this API more useful.

Top comments (4)

Marius Nettum • Jan 11 '22

Nice guide, but have you tested this on a server or just on localhost? It seems like instagram.com/${username}/?__a=1 requires a logged in Instagram user.

If I open instagram.com/sahilverma.dev/?__a=1 in a chrome tab where I'm logged into Instagram everything works fine, but if I open it in incognito I get redirected to instagram.com/accounts/login/

So the /user/:username endpoint will probably not work if you deploy your code to a server.

Sahil Verma • Jan 11 '22 • Edited

I also want to host this API for testing. But I'm not able to host this API. That's why I haven't tested it on the server and didn't host my frontend which is also ready.
I am new in express and node and I'm still learning stuff.
But I have also tried API in the incognito mode, and it was working.

Andrew Schimelpfening • Jan 11 '22

Nice article. In case you didn’t know, request has been deprecated for almost 2 years. You should consider using a different, supported http(s) library, like got or node-fetch.

Sahil Verma • Jan 11 '22

Thanx for your feedback. I really don't know about that. But I'll try it

DEV Community

How I made Instagram Content Fetching API in Node.js

End

Top comments (4)

Read next

Moving beyond console.log

Async Local Storage is Here to Help You

FlagFeed: Gamified Tech Learning with Feature Flags 🚀

Stop Struggling Learn CSS Faster