Highly scalable image storage solution with AWS Serverless at ip.labs - Part 1 The why behind the re-implementation

#aws #serverless #filestorage #scalability

Co-Author: Firdaws Aboulaye
This is my first blog post after being named AWS Serverless Hero.

Introduction

ip.labs, a 100% subsidiary of the FUJIFILM Group headquartered in Bonn, is a global leader in white-label e-commerce software for imaging. Our software enables end consumers to design and purchase a variety of photo products, including prints, posters, photo books, calendars, greeting cards, and personalized gifts. In addition, we provide integrations for payment systems, single sign-on (SSO) solutions, and a unique API that allows providers to create a shared cart, facilitating seamless addition of our solution alongside their existing business.

The Journey to the Cloud

The Initial Setup and Challenges

Until 2017, ip.labs managed several on-premises data centers across the globe, operating a large monolithic architecture that relied on a centralized database and NFS storage. While effective initially, this setup presented significant challenges, including scalability limitations, central dependency on infrastructure, and integration hurdles.

Recognizing the need for greater agility, we decided to fully transition to the cloud. AWS was selected as our cloud partner, and by 2021, all customers had been migrated to AWS, allowing us to shut down our data centers completely. The migration began as a “lift and shift” approach but soon evolved to incorporate several key improvements:

Automation at both the infrastructure and deployment levels
Adoption of immutable infrastructure to eliminate hot patches
Blue-Green deployment strategies to minimize downtime
Introduction of independent core microservices, fostering a “you build it, you run it” mindset
Auto-scaling policies to efficiently manage workload variations

Legacy Image Storage Issues

The cloud migration exposed several issues with our legacy image storage solution:

Reliance on central AWS EFS for image uploads/downloads from all EC2 servers, which created scalability and availability risks
Lack of defined APIs and limited testability for image storage
No clearly ownership for image storage solution
Was still a part of the monolithic solution and was tightly coupled into many aspects of the self-written e-commerce system which made integrations to the 3rd party providers really hard

The Move to a New Image Storage Solution

To address these issues and support future image-related features, we made the strategic decision to develop a standalone, API-driven image storage service. Since handling images is our core business, we opted to build our own solution rather than relying on third-party file storage providers.

This new image storage system supports various workflows, including:

Standard Uploads: Local uploads directly from users’ devices.
Mobile Uploads: QR code-based uploads with optional long-term storage that extends beyond typical session duration (usually six hours).
Project Save and Load: A feature that allows users to save intermediate designs, useful for more time-intensive products like photo books.
Project Sharing: Users can share saved projects in various modes, allowing recipients to create copies. Future plans include collaborative editing and commenting.
Cart Item Persistence: Cart items, both internal and external, are saved across multiple days to support integration with shared/external carts.
Order Storage: Orders are stored for a defined period, enabling re-submission to professional labs for production and delivery in case of issues.
Automated and Manual Cleanup: Image deletion is initiated either manually by the user or automatically upon project expiration or user logout.

Serverless Transformation: Why AWS Serverless?

Our new file storage solution, built with AWS Serverless technologies, is now rolled out to nearly all customers. This shift has not only enhanced our storage capabilities but has also paved the way for rapid feature development.

We chose AWS Serverless for several reasons:

Since 2018, we have extensively experimented with AWS Serverless across small to medium applications, experiencing firsthand the benefits of AWS-managed services (Both authors of this series have given many talks about our journey at different conferences or meet ups since then).
AWS Serverless enables rapid delivery, allowing our team to focus more on core capabilities.
Adopting a Serverless-first approach has fostered a culture of learning and knowledge-sharing within our development teams, further solidifying AWS Serverless as our preferred development model.

Advantages of the New Implementation

Our new image storage solution, built on AWS Serverless, offers several significant advantages over our previous monolithic architecture. Key benefits include:

Scalability: Leveraging AWS Serverless allows us to automatically scale resources based on demand. This elasticity ensures that our system remains performant even during peak usage, providing a seamless experience for end users while reducing the need for manual scaling efforts.
Flexibility for Future Expansion: The modular, API-driven architecture makes it easy to extend our file storage to other solutions. This flexibility supports faster adaptation to new business needs and allows for smooth integration with both our own services and external partner platforms.
Enhanced Feature Integration: With the serverless approach, we’ve been able to introduce new features more efficiently, such as mobile uploads and project sharing. These functionalities allow users to conveniently upload photos from any device and share projects with others, all seamlessly integrated into our new storage system.

This new implementation positions us well to continue adding value to our offerings while maintaining a highly scalable, flexible, and feature-rich platform

Conclusion

This article has introduced the foundation of our AWS journey at ip.labs and our shift to a Serverless-first approach to redesign our image storage solution. In the upcoming series, we will explore the architecture of this new storage solution, diving into the details of the supporting APIs and the design choices behind them.

See below a short overview of this architecture.