🔥 Stop Duplicate Data! How Redis Locks Saved Our App from Firestore Trigger Chaos

Hey there, fellow developers! 👋 Today, I want to share a tricky problem I encountered with Firestore triggers and how Redis locks saved the day. If you're running distributed systems on multiple servers, this one's for you.

The Problem: Duplicate Data from Firestore Triggers

In our application, we had a Firestore trigger that would fire whenever a new document was created. This trigger was supposed to process the data and insert it into our items table. Simple enough, right?

Here's what our initial setup looked like:

const admin = require('firebase-admin');
const db = admin.firestore();

exports.onItemCreated = db.collection('items').onCreate(async (snap, context) => {
  const newItem = snap.data();

  try {
    // Initially, we just inserted the data directly
    await insertIntoItemsTable(newItem);
  } catch (error) {
    console.error('Error processing item:', error);
  }
});

The Plot Twist: Multiple EC2 Instances

Everything worked fine until we scaled to two EC2 instances. Then we noticed something odd - duplicate entries in our items table! The problem? Our Firestore trigger was running on both EC2 instances simultaneously, and our database constraints weren't enough to prevent duplicates.

Enter Redis Locks: The Solution

Here's how we fixed it using Redis locks:

const Redis = require('ioredis');
const admin = require('firebase-admin');

const redis = new Redis({
  host: 'your-redis-host',
  port: 6379
});
const db = admin.firestore();

exports.onItemCreated = db.collection('items').onCreate(async (snap, context) => {
  const newItem = snap.data();
  const documentId = snap.id;
  const lockKey = `lock:firestore:item:${documentId}`;

  try {
    // Try to acquire the lock
    const acquired = await redis.set(
      lockKey,
      'locked',
      'NX',  // Only set if key doesn't exist
      'EX',  // Set expiry
      30     // 30 seconds expiry
    );

    if (!acquired) {
      console.log(`Lock already acquired for document ${documentId}`);
      return;
    }

    // Now safely process and insert the item
    await insertIntoItemsTable(newItem);

    // Release the lock after successful processing
    await redis.del(lockKey);

  } catch (error) {
    console.error('Error processing item:', error);
    // Make sure to release the lock even if processing fails
    await redis.del(lockKey);
  }
});

async function insertIntoItemsTable(item) {
  // Your database insertion logic here
  // This will only run on one EC2 instance now!
}

Why This Works

1. Unique Lock per Document: We create a unique Redis lock using the Firestore document ID.
2. First Come, First Served: Only the first EC2 instance that acquires the lock processes the document.
3. Automatic Cleanup: The 30-second expiry ensures locks are released even if our process crashes.
4. Safe Release: We always release the lock whether the processing succeeds or fails.

Results in Production

After implementing this solution:

• Zero duplicate entries in our items table
• Clean processing logs
• No more data inconsistencies
• Better resource utilization (no wasted processing)

Key Learnings

1. Always Consider Distribution: Even simple triggers can cause issues in distributed environments.
2. Database Constraints Aren't Always Enough: Sometimes you need application-level locking.
3. Lock Timeouts Are Critical: Choose an expiry time that covers your longest possible processing time.
4. Monitor Lock Acquisition: Log when locks can't be acquired to track potential issues.

Tips for Implementation

1. Use a Redis client with good error handling
2. Implement proper monitoring for lock acquisition failures
3. Keep lock times as short as practical
4. Consider adding retry logic for lock acquisition in critical processes

Have you faced similar challenges with Firestore triggers or distributed processing? I'd love to hear your stories in the comments!

---

Happy coding! 🚀