Building Resilient Workflows with Persistent Storage

Let’s face it—as a developer, when you work on projects and tasks with large-scale tasks like video compression, managing all this data can easily become a real pain. Yes, getting the data processing right is one thing. What is more important though is to try to ensure that your data does not get lost halfway through the job or right after you’re done. That’s where persistent storage comes into play.

At ApyHub, we have been thinking a lot about how to make life easier for developers working with asynchronous workflows, long-running tasks, and lots (and lots) of data. We have concluded that for these kinds of workflows, persistent storage isn’t just a nice-to-have; it is a real must-have.

Persistent Storage 101

Okay, a quick refresher for those who are new to persistent storage: Persistent storage is storage that keeps your data safe and sound even if the power goes out, your server crashes, or you take down your Kubernetes pod. Unlike ephemeral storage, which exists only for the lifespan of a process or container, persistent storage is able to stick around. You can think of it as the difference between writing notes on a sticky note (ephemeral) versus saving them to a Google Doc (persistent).

Some examples of persistent storage include hard disk drives (HDDs), solid-state drives (SSDs), flash memory (like USB sticks), and cloud storage services such as Google Cloud Storage and Amazon S3. These storage types are designed to survive crashes, restarts, and long-running tasks, ensuring your data is always available when needed. Persistent storage is designed to survive:

• Crashes and restarts.
• Long-running tasks.
• Inter-service communication mishaps.

In modern cloud-native architectures, persistent storage is often tied to concepts like stateful workloads, durable message queues, or even simple databases. It is sort of the baseline for ensuring the reliability of many of the systems we take for granted.

Why Persistent Storage Matters

Let’s get real. If you’re running anything more complex than “Hello, World!”, persistent storage probably plays a role in your life. Here are some of the reasons I could think of:

1. Asynchronous Workflows: When you’re compressing a 2GB video, you don’t want to sit there waiting for the job to finish. If you want to submit it, go grab a coffee, and pick it up later. Persistent storage ensures that your data—and the job progress—are still there when you come back.
2. Crash Resilience in Cloud Environments: Imagine you’re running a Kubernetes cluster, and a pod handling a long-running task gets evicted or crashes. Without persistent storage, all progress is lost. With it, you can recover where you left off, with no sweat.
3. Historical Access: Need to access compressed results from two weeks ago? Persistent storage has you covered. No more re-uploading or re-processing.
4. Sharing Is Caring: Multiple services or teams accessing the same data? Persistent storage makes sure everyone is looking at the same truth, not a fleeting version of it.

Examples of Persistent Storage in Action

Persistent storage isn’t just for video compression. You’ve probably already interacted with systems built on it today:

• Cloud Storage: Services like Dropbox, Google Drive, or iCloud use persistent storage to back up and sync your files across devices.
• Streaming Services: Ever wondered how Netflix always remembers where you left off in your favorite series? That’s persistent storage in action, keeping playback data alive and ready.
• Crash Recovery in Apps: Your email draft magically reappears after your browser crashes? Persistent storage.
• Gaming Save States: Whether it’s Call of Duty or Stardew Valley, your progress is stored persistently so you don’t lose hours of effort.

Why We Integrated Persistent Storage Into ApyHub’s APIs

When we at ApyHub, looked at workflows like video compression, watermarking, and format conversion, we noticed a common challenge: these are not quick, one-off tasks. They involve heavy data processing, asynchronous workflows, and potential hiccups like crashes or timeouts. For these scenarios, persistent storage was an easy decision.

Let’s talk first about video compression—a task we handle a lot at ApyHub and that we have exposed with an API. Video files are massive. Compression takes time. And when you are running asynchronous workflow, all sorts of stuff could happen: servers reboot, jobs time out, or your internet cuts out. Without persistent storage, any hiccup means starting over.

With persistent storage, we are suggesting a simpler, better way:

1. You Submit the Job: Upload your video file or provide a URL, adding persistent=true as a query parameter.
2. We Process It Asynchronously: The API does the heavy lifting while you go build cool things.
3. Your Data Stays Safe: Persistent storage ensures the job progress and final compressed results are securely stored and available whenever you need them.

The Dev Benefits

These are the reasons we believe that persistent storage is the way to go (and why you should probably give it a try):

• No More “Oops” Moments: Ever accidentally close your terminal mid-job? Persistent storage makes sure that’s not the end of the world.
• No Redos: Compressed a 4GB video yesterday but forgot to download it? No problem—it’s still there.
• Simplified Workflows: By letting our API handle the grunt work of data management, you can focus on building features, not fighting fires.
• Works Like a Vault: Your data is always there when you need it, secure and reliable, even after weeks or months.

Why It’s a Big Deal

For teams building apps that involve large-scale or asynchronous tasks, persistent storage isn’t just helpful—it’s essential. Imagine developing a media-sharing app, or any app within which you want to allow users to share files and any other content. You would surely want every user upload and processing job to be as reliable as possible, recoverable, and definitely retrievable. Persistent storage takes the burden off your infrastructure and makes your workflows resilient by design.

Ready to Simplify Your Video Processing?

At ApyHub, we built persistent storage into our APIs because we know the headaches it can save from. Whether you are working on an application that includes video processing, image optimization, or some other data-heavy task, persistent storage makes the entire process smoother, faster, and way less stressful.

We have added persistent storage to the following video processing APIs:

• Video Compression Job API
• Generate Watermark for Videos Job API
• Generate GIF from Video Job API
• Generate Video Thumbnail Job API
• Generate Thumbnail from Video Job API
• Extract Audio from Video Job API
• Convert Video Formats Job API

You are welcome to try them out. What you will notice is that when you submit a video compression job, the progress is automatically saved at every stage, so even if the process is interrupted, it can resume right where it left off. Similarly, when you request a thumbnail generation or audio extraction, the resulting files are securely stored and accessible via API calls, making it easy to integrate them into downstream processes or share across services. This eliminates the need for additional storage management and ensures consistent performance across workflows.

Conclusion

At the end of the day, persistent storage is a must-have for any developer working with large-scale tasks or asynchronous workflows. Among its many advantages is ensuring that your data sticks around when things go wrong, so you don’t have to hit restart every time something breaks. Whether it’s making sure your video compressions aren’t lost halfway through or guaranteeing you can pick up right where you left off, persistent storage gives you the peace of mind to keep building and moving fast. By integrating persistent storage into your workflows, you are surely protecting your data but also, and more importantly, you are setting yourself up for faster, and more resilient development. At ApyHub, that’s exactly what we’re here for: making your life easier and your systems more reliable.