Utilizing Worker Threads in Node.js to Efficiently Facilitate the Upload Process of a File into an Amazon S3 Bucket.

LINK FOR FLOWCHART

*How to Implement *

File Upload Process:

The user will upload the file via /upload endpoint and API immediately returns a response: "File uploaded successfully
The file details (path, job_applicant_id) are added to a queue for processing.
A worker thread picks up the file and uploads it to S3 Bucket.
THe Path is stored in Database as well with the path name

Fetching the File:

1.When a user requests the file via and API(fetch/job_applicant_id/filepath) by the path name

2.The API generates a signed URL for temporary access and returns it

IMPLEMENTATION PART:

1. File Upload API
Uses multer to handle file uploads.
Adds the file path to a queue for processing.
Responds immediately without waiting for the file to upload.

const express = require('express');
const multer = require('multer');
const threadController = require('./threadController');

const app = express();
const upload = multer({ dest: 'uploads/' });

app.post('/upload', upload.single('file'), (req, res) => {
    const { job_applicant_id } = req.body;
    const filePath = req.file.path;
    const fileName = req.file.originalname;

    threadController.addToQueue(job_applicant_id, filePath, fileName);
    res.json({ message: 'File uploaded successfully' });
});

app.listen(3000, () => {
    console.log('Server running on port 3000');
});

2. Worker Thread for Uploading to S3
Picks up files from the queue.
Uploads them to S3 under job_applicant_id.

const { isMainThread, parentPort, workerData } = require('worker_threads');
const AWS = require('aws-sdk');
const fs = require('fs');

AWS.config.update({
    accessKeyId: 'AWS_ACCESS_KEY',
    secretAccessKey: 'AWS_SECRET_KEY',
    region: 'us-east-1'
});

const s3 = new AWS.S3();

if (!isMainThread) {
    const { job_applicant_id, filePath, fileName } = workerData;
    const fileStream = fs.createReadStream(filePath);

    const params = {
        Bucket: 'my-file-bucket',
        Key: `${job_applicant_id}/${fileName}`,
        Body: fileStream,
        ContentType: 'application/pdf'
    };

    s3.upload(params).promise()
        .then(data => console.log(`File uploaded: ${data.Location}`))
        .catch(err => console.error(`Upload failed: ${err.message}`));
}

3. Fetch File API
Retrieves a temporary signed URL to access the file.
app.get('/get-file/:job_applicant_id/:file_name', (req, res) => {

` const { job_applicant_id, file_name } = req.params;

    const params = {
        Bucket: 'my-file-bucket',
        Key: `${job_applicant_id}/${file_name}`,
        Expires: 60 * 5 // Link valid for 5 minutes
    };

    const url = s3.getSignedUrl('getObject', params);
    res.json({ url });
});`