Mastering the Art of Seamless Large File Uploads with JavaScript

TLDR

This article shows you how to upload large files to api.video using JavaScript and the File API and the Blob API. You will learn how to split the file into segments, upload them with a token and a header, and show the progress and the URL to the user. You can find the code on Github, and use a library to simplify the upload.

Introduction

Uploading large files to remote servers can be a challenge for developers. Whether it’s presentations, PDFs, files, or videos, we often encounter errors and frustrations when our files exceed the server’s limit. But what if we could avoid these errors with just a few lines of JavaScript? In this article, we will show you how to do just that.

api.video enables developers to build, scale and operate video in their own apps and platforms in minutes, with just a few lines of code. The service handles the end-to-end workflow, from video ingestion to worldwide video delivery. You can test it for free right away and start building.

One of the most common errors with large uploads is HTTP 413: Request Entity too Large. This means the server has a fixed size limit for accepting files, and anything larger than that will be rejected. You might think that the solution is to increase the server limit, but that’s not always feasible or desirable. For example, if you raise the limit to 2GB for videos, you might end up with huge images that take up too much space.

Another problem with large uploads is that they can fail midway and force you to start over. How annoying is it when your upload fails at 95% completion? To avoid this, we need a way to resume the upload from where it left off.

Segments/Chunks

You probably know that streaming videos from api.video, Netflix or YouTube involves breaking down large video files into smaller segments that are transmitted over the internet. Then the player on your device stitches the video segments back together to play them in the correct order. But did you know that you can do the same thing with your large file uploads? You can split the large file into smaller segments and upload each one separately, without your users noticing anything!

This is possible thanks to the File API and the Blob API that are built into JavaScript and supported by all major browsers:

These APIs allow us to take a large file from our user, and use the browser locally to divide it into smaller segments, without affecting the user experience!

In this article, we will show you how to use this technique to upload a large video to api.video.

You can find the code for this example on Github, so feel free to clone the repo and run it locally.

To create your own uploader like this, you’ll need a free api.video account. You’ll also need a delegated upload token, which is a public upload key that lets anyone upload videos into your api.video account. You can create one using CURL and a terminal window in just 3 steps.

We recommend that you set a TTL (time to live) on your token, so that it expires as soon as the video is uploaded.

Once you have your token, you’re ready to start uploading large files.

Markup

Add a video here:
    <br>
    <input type="file" id="video-url-example">
    <br>



    <br>
    <div id="video-information" style="width: 50%"></div>
    <div id="chunk-information" style="width: 50%"></div>

There is an input field for a video file, and then there are 2 divs where we will output information as the video uploads.

Next on the page is the <script> section - and here's where the heavy lifting will occur.

<script>
      const input = document.querySelector('#video-url-example');
      const url ="https://sandbox.api.video/upload?token=to1R5LOYV0091XN3GQva27OS";
      var chunkCounter;
      //break into 5 MB chunks fat minimum
      const chunkSize = 6000000;
      var videoId = "";
      var playerUrl = "";

First, we need to define some JavaScript variables:

• input: the file input interface specified in the HTML.
• url: the delegated upload url to api.video. The token in the code above (and on Github) points to a sandbox instance, so videos will be watermarked and removed automatically after 24-72 hours. If you’ve created your own token, replace the URL parameter ‘to1R5LOYV0091XN3GQva27OS’ with your token.
• chunkCounter: the number of chunks that we will create from the file. • chunkSize: the size of each chunk in bytes. We set it to 6,000,000, which is slightly above the 5 MB minimum. For production, we can increase this to 100MB or more.
• videoId: the delegated upload will assign a videoId on the api.video service. This is used for subsequent uploads to identify the segments, ensuring that the video is properly reassembled on the server.
• playerUrl: this will store the playback URL for the api.video player after the upload is complete.

Next, we create an EventListener on the input - when a file is added, split up the file and begin the upload process:

 input.addEventListener('change', () => {
        const file = input.files[0];
           //get the file name to name the file.  If we do not name the file, the upload will be called 'blob'
           const filename = input.files[0].name;
        var numberofChunks = Math.ceil(file.size/chunkSize);
        document.getElementById("video-information").innerHTML = "There will be " + numberofChunks + " chunks uploaded."
        var start =0;
        var chunkEnd = start + chunkSize;
        //upload the first chunk to get the videoId
        createChunk(videoId, start);

We assign the uploaded file to a variable called ‘file’. To calculate the number of chunks to upload, we divide the file size by the chunk size. We round up the number, because any ‘remainder’ smaller than 6M bytes will be the last chunk to be uploaded. We display this number on the page for the user to see. (This is just for demonstration purposes, your users might not be interested in this information in a real product).

Slicing up the file

The function createChunk splits the file into smaller pieces.

Then, we start to slice the file into chunks. You might think we should use chunkSize -1 as the last byte of the chunk we create, since the file starts from zero. But that’s not the case. We use chunkSize as it is, because it represents

the index of the first byte that is excluded from the new Blob (i.e. the byte at this index is not part of the chunk).

So, we need to use chunkSize as it is, because it marks the boundary between the new Blob and the rest of the file.

function createChunk(videoId, start, end){
            chunkCounter++;
            console.log("created chunk: ", chunkCounter);
            chunkEnd = Math.min(start + chunkSize , file.size );
            const chunk = file.slice(start, chunkEnd);
            console.log("i created a chunk of video" + start + "-" + chunkEnd + "minus 1    ");
            const chunkForm = new FormData();
            if(videoId.length >0){
                //we have a videoId
                chunkForm.append('videoId', videoId);
                console.log("added videoId");

            }
            chunkForm.append('file', chunk, filename);
            console.log("added file");


            //created the chunk, now upload iit
            uploadChunk(chunkForm, start, chunkEnd);
        }

In the createChunk function, we increment the chunkCounter to keep track of which chunk we are uploading, and we calculate the end of the chunk (remember that the last chunk will be smaller than chunkSize, and it will go until the end of the file).

In the first chunk uploaded, we include the filename to name the file (otherwise, the file will be called ‘blob.’

The actual slice command

The file.slice cuts the video into a ‘chunk’ for upload. This is how we divide the file into smaller pieces!

We then create a form to upload the video segment to the API. The API returns a videoId after the first segment is uploaded, and we need to include this videoId in the following segments (so that the backend knows which video to append the segment to).

On the first upload, the videoId is empty, so we don’t need it. We add the chunk to the form, and then we call the uploadChunk function to send this file to api.video. On the next uploads, the form will have both the videoId and the video segment.

Uploading the chunk

Let's walk through the uploadChunk function:

    function uploadChunk(chunkForm, start, chunkEnd){
            var oReq = new XMLHttpRequest();
            oReq.upload.addEventListener("progress", updateProgress);   
            oReq.open("POST", url, true);
            var blobEnd = chunkEnd-1;
            var contentRange = "bytes "+ start+"-"+ blobEnd+"/"+file.size;
            oReq.setRequestHeader("Content-Range",contentRange);
            console.log("Content-Range", contentRange);

We kick off the upload by creating a XMLHttpRequest to handle the upload. We add a listener so we can track the upload progress.

Adding a byterange header

When doing a partial upload, you need to inform the server which ‘part’ of the file you are sending - we use the byterange header for this.

We attach a header to this request with the byterange of the chunk being uploaded.

Note that in this case, the end of the byterange should be the last byte of the chunk, so this value is one byte less than the slice command we used to cut the chunk.

The header will look something like this:

Content-Range: bytes 0-999999/4582884

Upload progress updates

While the video chunk is uploading, we can show the upload progress on the page, so our user knows that everything is working fine. We created the progress listener at the start of the uploadChunk function. Now we can define what it does:

            function updateProgress (oEvent) {
                if (oEvent.lengthComputable) {  
                var percentComplete = Math.round(oEvent.loaded / oEvent.total * 100);

                var totalPercentComplete = Math.round((chunkCounter -1)/numberofChunks*100 +percentComplete/numberofChunks);
                document.getElementById("chunk-information").innerHTML = "Chunk # " + chunkCounter + " is " + percentComplete + "% uploaded. Total uploaded: " + totalPercentComplete +"%";
            //  console.log (percentComplete);
                // ...
              } else {
                  console.log ("not computable");
                // Unable to compute progress information since the total size is unknown
              }
            }

First, we do some math to calculate the progress. For each chunk we can get the percentage uploaded (percentComplete). This is a fun value for the demo, but not very useful for real users.

What our users want is the totalPercentComplete, which is the sum of the existing chunks uploaded plus the amount currently being uploaded.

For this demo, we display all these values in the ‘chunk-information’ div on the page.

Chunk upload complete

Once a chunk is fully uploaded, we run the following code (in the onload event).

oReq.onload = function (oEvent) {
                           // Uploaded.
                            console.log("uploaded chunk" );
                            console.log("oReq.response", oReq.response);
                            var resp = JSON.parse(oReq.response)
                            videoId = resp.videoId;
                            //playerUrl = resp.assets.player;
                            console.log("videoId",videoId);

                            //now we have the video ID - loop through and add the remaining chunks
                            //we start one chunk in, as we have uploaded the first one.
                            //next chunk starts at + chunkSize from start
                            start += chunkSize; 
                            //if start is smaller than file size - we have more to still upload
                            if(start<file.size){
                                //create the new chunk
                                createChunk(videoId, start);
                            }
                            else{
                                //the video is fully uploaded. there will now be a url in the response
                                playerUrl = resp.assets.player;
                                console.log("all uploaded! Watch here: ",playerUrl ) ;
                                document.getElementById("video-information").innerHTML = "all uploaded! Watch the video <a href=\'" + playerUrl +"\' target=\'_blank\'>here</a>" ;
                            }

              };
              oReq.send(chunkForm);

When the file segment is uploaded, the API returns a JSON response with the videoId. We store this in the videoId variable, so we can include it in the next uploads.

To upload the next chunk, we increase the bytrange start variable by the chunkSize. If we have not reached the end of the file, we call the createChunk function with the videoId and the start. This will recursively upload each subsequent piece of the large file, until we reach the end of the file.

Upload complete

When start > file.size, we know that the file has been completely uploaded to the server, and our work is done! In this example, we know that the server can accept 5 MB files, so we split the video into many smaller segments to fit under the server size limit.

When the last segment is uploaded, the api.video response contains the full video response (similar to the get video endpoint). This response includes the player URL that is used to play the video. We save this in the playerUrl variable, and add a link on the page so that the user can watch their video. And that’s it!

Conclusion

In this article, we have shown you how to upload large files to api.video using JavaScript and the File API and the Blob API. We have explained how to split the file into smaller segments and upload them separately, using the byterange header and the delegated upload token. We have also demonstrated how to show the upload progress and the playback URL to the user. We hope you found this article useful and informative. If you have any questions or feedback, please let me know in the comments below. Happy uploading!