A.E Clarence

Posted on May 13

Gallerify - A Fullstack Image Sharing Application Built On Netlify Primitives

#netlifychallenge #devchallenge #webdev #javascript

This is a submission for the Netlify Dynamic Site Challenge: Visual Feast.

This is a submission for the Netlify Dynamic Site Challenge: Build with Blobs.

This is a submission for the Netlify Dynamic Site Challenge: Clever Caching.

What I Built

I developed a full-stack image sharing platform similar to Giphy or Imgur in order to showcase the power of building full-stack applications with only Netlify Primitives

So what are Netlify Primitives?
Netlify Platform Primitives are a set of features provided by Netlify that can be used to build and enhance web applications. They are framework-agnostic, meaning they can be used with any web development framework you choose.

Most full-stack applications basically require a frontend, backend, database, authentication mechanism, caching and hosting.
Netlify provides services for all of the requirements above.
Gallerify is hosted on Netlify and leverages the Netlify Platform Primitives such as Netlify Serverless Functions, Netlify Blobs and Netlify Image CDN
For authentication Netlify Identity was leveraged.

Gallerify Tech Stack
Frontend: Next.js with Bootstrap 5 customized with SASS
Backend: Netlify Functions
Image Storage: Blob Data
Database Storage For Image Metadata: Blob Metadata
Authentication: Custom Netlify Identity in Netlify Functions

Even though Gallerify was built during the Netlify Dynamic Site Challenge the goal was not to build the next Giphy or Imgur.
It was tempting to build most of the features of Giphy or Imgur in order to be considered for the first spot.
But later I realized that just focusing on the final product will not allow me to shine.

Instead of building most of the features of Giphy or Imgur I would rather focus on the basics by detailing my creative journey and then explain the choices I made, the challenges I faced, and how I overcame them.
Doing this will enable me to share an engaging and relatable content to the community.
Who knows? I might inspire someone to rather build the next Giphy or Imgur on Netlify Primitives.

If you got to this point, I encourage you to continue reading to learn how I built Gallerify.

Demo

Before I get to the technical details, you can checkout Gallerify by clicking the link below. You can register for a new account and it's free or you can use the demo credentials below as well.

Demo Credentials:
Email: demo@demo.com
Password: demo1234

Demo URL: https://gallerify-app.netlify.app

Screenshots:

Homepage:

Layout Options:

Transform Options:

Login:

Sign Up:

User Dashboard:

Image Upload:

Platform Primitives

Netlify Image CDN

For caching of images I leveraged Netlify Image CDN.

To transform an image, you make a request to /.netlify/images of your site with query parameters that specify the source asset and the desired transformations
For the Masonry layout I render the cached version of the image without any transformations.

Example URL:
Mansory Layout Images:
https://gallerify-api.netlify.app/.netlify/images?url=https://gallerify-api.netlify.app/api/gallery?uid=uploader_id_here&image_id=image_id_here&image_name=image_name_here

Thumbnail Layout Images
https://gallerify-api.netlify.app/.netlify/images?url=https://gallerify-api.netlify.app/api/gallery?uid=uploader_id_here&image_id=image_id_here&image_name=image_name_here&fit=cover&w=440&h=440

For the thumbnail layout an optimized 440px x 440px image is served by Netlify Image CDN on the fly.

The url query parameter is a Netlify functions endpoint that serves the image so that the img HTML tag can render the image.

Once you choose a particular layout the cached images are served. Also note that since my url endpoint also has query parameter I had to decode it using JavaScript function encodeURIComponent in order to prevent issues with the Netlify Image CDN if I also pass multiple query parameters.

Also once you see an image that you like and click on the image, an image transformation modal opens. In the image transformation modal, there are options to transform images.

Netlify Image CDN is leveraged to transform images on the fly once you type the values into the input, drag the sliders, and choose from the select inputs.

Image transformations happens in realtime.
There are also buttons that set the image dimensions and other image options for popular social media sites.
Once you click on a particular social media button, the image dimensions, image position, image fit, image quality, and image format values are automatically set and transforms in realtime.

Netlify Image CDN caches the images and serves the cached images upon requests. Therefore when an image that has been transformed is requested again, the cached version is served improving both the runtime performance and reliability of the Gallerify site.

Challenges I Faced:
When an image is added from the dashboard, it did not immediately appear on the homepage and dashboard.

Decisions Made:
To solve the problem of images not appearing immediately on the homepage and dashboard, I leveraged Cache Control Settings in Netlify Serverless Functions response headers to revalidate and fetch a fresh copy of the images after 60 seconds of upload for the public homepage. For the dashboard, the control settings was set to 0. Images will appear immediately once they are uploaded

In a real world application, it's good practice to prevent your server from constant public requests. Thats why I set the revalidate option of 60 seconds for the homepage. Since the dashboard will not have as many requests like the homepage it's ideal to allow images to appear immediately so that users will not be confused about the images they uploaded.

// Cache control headers for requests from dashboard

 return new Response(JSON.stringify(data_here), {
        headers: {
            "Content-Type": "application/json",
            "Cache-Control": "no-cache, no-store",
            "Netlify-CDN-Cache-Control": "no-cache, no-store",
           "Access-Control-Allow-Origin": "*"
        },
        status: statusCode 
    });

// Cache control headers for requests from public frontend

 return new Response(JSON.stringify(data_here), {
        headers: {
            "Content-Type": "application/json",
            "Cache-Control": "public, max-age=60, must-revalidate",
            "Netlify-CDN-Cache-Control": "public, max-age=60, must-revalidate",
           "Access-Control-Allow-Origin": "*"
        },
        status: statusCode 
    });

Netlify Identity

In order for users to login or register to your application, you need an authentication mechanism.
I could have used Netlify Blobs to store user information and JWT tokens. However it's not ideal to be storing user credentials in blobs. Also since Netlify already provides authentication using Netlify Identity, it was better to leverage Netlify Identity as it makes authentication a breeze and then handles all authentication mechanisms out of the box.

Netlify provides Netlify Identify in order to sign up, login, reset password, verify and confirm users.
In order to use Identity you can either use the Netlify Identity Widget or GoTrue JavaScript plugin for custom configurations.

Challenges I Faced:
I wanted a custom authentication for Gallerify and I did not want to use the Default Identify widget as it will not match with the styling of my website.
Also even though the GoTrue plugin can help with customization I did not want authentication request to come from the frontend as an attacker can go through the GoTrue source code and then send malicious requests.

I wanted to use Netlify Identify in my Netlify Functions. At the time I did not know GoTrue was only for client-side so I installed it for my Netlify Functions only to be thrown multiple errors. I realized the package only works for the client-side

Decisions Made:
So solve the problem of using Netlify Identity in Netlify Functions I decided to go through the source code of the GoTrue project on GitHub and then learn how the requests were made.

I figured out how requests were sent to Netlify Identify and then I wrote my custom authentication logic and then sent requests using fetch to login, sigup, and verify users JWT.

Once a request is sent the Netlify Function endpoint to create an account, a JWT token is returned and then stored in cookies on the Gallerify frontend for making authenticated requests by users.

I was able to customize my website and then make authentication requests from the frontend form to the Gallerify Netlify Function Backend without using the Identity Widget or GoTrue plugin.


// Netlify Identity Signup

const registerResponse = await fetch(netlifyIdentitySignupUrl, {
                method: "POST",
                body: JSON.stringify({ email, password, data: { full_name: username } }),
                headers: { "Content-Type": "application/json"},
            });

const responsedata: any = await registerResponse.json();


// Netlify Identity Login

const loginResponse = await fetch(netlifyIdentityLoginUrl, {
                method: 'POST',
                headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
                body: `grant_type=password&username=${encodeURIComponent(
                email,
                )}&password=${encodeURIComponent(password)}`,
            });

const data: any = await loginResponse.json();


// Netlify Identity Verify User - JWT

const headers = req.headers;

const accessToken = headers.get("authorization")?.split(" ")[1];

const verifyResponse = await fetch(netlifyIdentityVerifyUrl, {
            method: 'GET',
            headers: {
                "Authorization": `Bearer ${accessToken}`
            }
        });

const data = await verifyResponse.json();

Netlify Blobs

For most applications, you need a location to store media. Gallerify is an image sharing platform so I needed a way to store the image files.
Netlify Blobs provides mechanisms to store the actual file data and then metadata or data that will be normally saved in a database.
I leveraged Netlify Blobs to store images as Blobs and authenticated user information that uploaded the file.

Challenges I Faced:

I did not have much knowledge on Netlify blobs so when I read the basics from the documentation and then I tried to save the image, anytime I try to serve the image to the frontend the image was served as a binary format.
Even though my Cache Control settings were previously set to a max-age of 0 and then no-cache, any time there are updates and deletions my Netlify Function was still serving old data. Fresh data was only served after 60 seconds.
Listing Blobs using await imagesStore.list() does not return data with metadata but keys. I was using for await...of to iterate through the response from await images.list().
Once the data increased my Netlify Functions were crashing due to the function execution duration of 10 seconds.

Decisions Made:

In order the solve the problem of images served as binary, I read multiple blog posts and articles. At some point I even used a third party image storage service until I stumbled upon @mattlewandowski93's post on this website.

I later figured out how to serve images properly by reading @mattlewandowski93's submission post.

I want to say a big thanks to @mattlewandowski93's for explaining how he was able to serve the images properly from Netlify Functions.

@mattlewandowski93's post provided amazing insights on how he served images properly and how he implemented features for his application.

I was able to create an API endpoint to properly serve images in the right format.


// Code snippet of how to properly serve images

const userIdFromParams = url.searchParams.get("uid") as string;

const imageId = url.searchParams.get("image_id") as string;
const imageName = url.searchParams.get("image_name") as string;

const imageBlobKey = `${userIdFromParams}/${imageId}/${imageName}`;

const currentImageBlob = await imageStore.getWithMetadata(imageBlobKey, { type: "blob" });

 if (currentImageBlob) {

      return new Response(currentImageBlob.data, {
                    status: 200,
                    statusText: "OK",
                    headers: {
                        "Content-Type": currentImageBlob?.metadata?.type as any,
                    },
        });

 }

 return generateAppResponse("error", 404, "Image not found", currentImageBlob);

Also note that when you want to get blob data, you have to specify the type. In my case I used the blob type

const currentImageBlob = await imageStore.getWithMetadata(imageBlobKey, { type: "blob" });

The line below is how the image is initially stored so that it can be sent back to the Netlify Functions to properly search based on prefixes. Searching by directory prefix style is faster.

const imageBlobKey = ${userIdFromParams}/${imageId}/${imageName}`;

In order serve fresh data when updates and deletions occurred, I had to read the Netlify Blobs Documentation multiple times in order to understand the API better.

I later found out that Netlify Blobs uses an eventual consistency model by default. The data is stored in a single region and cached at the edge for fast access across the globe. Therefore when a blob is added, it becomes globally available immediately.

However updates and deletions were propagated to all edge locations within 60 seconds. This means that it may take up to 60 seconds for a fresh blob to be served after it has been updated or deleted.

In order to prevent this from happening and then serve fresh data, I had to reconfigure the default behavior of the Blobs API and opt-in to strong consistency with the Netlify Blobs API.
After configuring using the strong options when I fetched images after images were added or deleted I was receiving the fresh data.

 const imageStore = getStore({
        name: "uploads",
        consistency: "strong",
        siteID,
        token,
    });

In order to get metadata of blobs when using the await imageStore.list(), I used the technique below.

// Get metadata for each blob

const images = [];

for await (const blob of entry.blobs) {
  const metadata = await store.getMetadata(blob.key);
  images.push(metadata);
}

However using the technique above was not efficient because when data begins to increase, once this code runs in your Netlify Function, your function times out after 10 seconds and an error is thrown.
Another option was to use Background Functions but that was going to be an overkill. Also I needed to serve a response immediately when I fetch for all data.
I wished the Blobs API had an option to list blobs with metadata.

For 2 days I was thinking about how to solve the problem of fetching data without my functions timing out after 10 seconds.
So I read the Documentation once more and decided to use a prefix. The reasoning behind this decision is that when await imagesStore.list() is executed the data is returned fast enough regardless the size of the data after uploading multiple images and testing it out.

The problem was that the return value was the key and the etag for each blob.

Therefore by storing blobs using the key format below, I can get data belonging to a particular user. Think of it like a directory structure on your computer.


 // The uuidv4 package was used to generate unique id's for files and images

// Store file in the format - userId/imageId/filename

const imageBlobKey = `${userId}/${imageId}/${imageName}`;

await imageStore.set(imageBlobKey, imageFile, {
                metadata: imageMetadata
            });

An array of keys in the format above is sent to the frontend and then when I want to fetch images belonging to a particular user I fetch by the prefix. The response from the execution is very fast.


const userIdFromParams = url.searchParams.get("uid") as string;
const imageId = url.searchParams.get("image_id") as string;

const imageName = url.searchParams.get("image_name") as string;

const imageBlobKey = `${userIdFromParams}/${imageId}/${imageName}`;

const currentImageBlob = await imageStore.getWithMetadata(imageBlobKey, { type: "blob" });

if (currentImageBlob) {

   return new Response(currentImageBlob.data, {
                 status: 200,
                 statusText: "OK",
                 headers: {
                 "Content-Type": currentImageBlob?.metadata?.type as any,
         },
});

Anytime I want to get image data or image metadata there are API endpoints that receive the userId, imageId, fileName as query parameters and then it's reconstructed again in the format userId/imageId/filename so that I can fetch by a specific user or any image.

That it, at this point you have the learned how you can leverage Netlify Primitives to build full-stack applications.

I am confident this post will inspire a lot of developers to build the applications like Giphy, Imgur, Or the "next big app"...

I want to add that in case you have any challenges finding information on the Netlify Documentation, try out the Netlify Bot. It's so awesome and it helped me narrow down information I was looking for.

The ideas are endless. For example a Blog application, Social Media application, Podcast application e.t.c

You can checkout projects I work on by visiting my GitHub repo here

Thanks for taking the time out of your day to read this blog post.

Happy building with Netlify.

Top comments (4)

Matt Lewandowski • May 13 • Edited

Hey @clarnx 👋

I'm glad my submission was able to help, and thanks for the shout. I ended up making a few small improvements, that I think you will benefit from as well.

If you use a query parameter for the image key, the caching breaks when using Netlify's CDN. As in, if you return cache headers on your route handler for the image "Cache-Control": "public, max-age=604800, must-revalidate",, the CDN will cache the wrong images. It seems to not consider the query parameters when creating the cache key, so things get pretty funky.

What I've done instead, is use a dynamic route. This means that every single image with have it's own URL, instead of a query parameter, for the main key. So my route looks like this /challenge/api/image/[id]/route.ts

and my handler looks like this now.

export const GET = async (_: Request, {params}: { params: { id: string } }) => {
    const id = params.id;

    try {
        if (!id) {
            return NextResponse.json({Message: "Image ID not found", status: 400});
        }

        const blobStore = getStore({name: 'images', consistency: 'strong'});
        const image = await blobStore.get(id, {consistency: "strong", type: "blob"});

        if (!image) {
            return NextResponse.json({Message: "Image not found", status: 404});
        }

        return new NextResponse(image, {
            status: 200,
            statusText: "OK",
            headers: {
                'Content-Type': 'image/webp',
                "Cache-Control": "public, max-age=604800, must-revalidate",
                'Access-Control-Allow-Origin': '*',
                'Access-Control-Allow-Methods': 'GET, POST, PUT, DELETE, OPTIONS',
                'Access-Control-Allow-Headers': 'Content-Type, Authorization',
            },
        });
    } catch (error) {
        return NextResponse.json({Message: "Failed for some reason", status: 500, error: error});
    }
};

This allows you to cache the images on the user's computer. So they aren't fetched from the edge on every single request.

Just to relate this to your project, nice work btw! You'll notice that if you refresh the page on the main dashboard, that every image has to be re-downloaded from the edge. It's pretty fast because it's the edge, but it still causes a pretty noticeable flash. If you were to make these changes and enable caching, the images would be cached locally on the users computer, and would not cause a flash.

A.E Clarence • May 13 • Edited

Thanks for the valuable insights @mattlewandowski93
I really appreciate it. I learned a lot from your post.
I will make the cache changes and test how fast the images are fetched.

Regarding using query params in your url, I also experienced this. I did my research and found that if you encode your url that has query params with the JavaScript function encodeURIComponent the Image CDN is able to detect your url when there are other Image CDN query params.
It detects the encoded url as one value.
Test it to see if it works even when there is a key query params in your url that serves the images.