Presenting at DataEngBytes 2024 Sydney: Building a Transactional Data Lakehouse on AWS with Apache Iceberg

I had the pleasure of presenting at DataEngBytes 2024 in Sydney, where I discussed an exciting topic that’s transforming the data management landscape: Building a Transactional Data Lakehouse on AWS with Apache Iceberg.
This blog post captures the key content and insights shared during the session for those who couldn’t attend and as a record of my talk.

Why a Data Lakehouse?
As organisations scale and diversify their data sources, they increasingly seek the flexibility of a data lake combined with the transactional reliability of a data warehouse. The data lakehouse architecture bridges this gap by delivering a unified platform that supports both analytical and transactional workloads, making it ideal for managing structured, semi-structured, and unstructured data at scale.

During my talk, I explained that a data lakehouse:

• Ensures ACID compliance for data consistency and reliability.
• Supports time travel to query historical data.
• Provides real-time insights by processing batch and streaming data seamlessly.
• Reduces storage costs by leveraging data lakes for large volumes of data.

Key Challenges in Traditional Data Lakes and Warehouses:

I highlighted the challenges organisations often face with traditional data lakes, such as the lack of transaction support, complex schema management, and inconsistent data views. At the same time, data warehouses, though highly consistent, can be expensive and struggle with scalability when handling semi-structured and unstructured data.

To solve these challenges, I introduced the concept of a data lakehouse built with Apache Iceberg on AWS, combining the benefits of both lakes and warehouses.

Why Apache Iceberg?

Apache Iceberg is an open table format that makes it possible to manage large-scale, transactional data in data lake environments. Here’s why it’s ideal for a lakehouse:

• ACID Transactions: Iceberg supports ACID compliance, allowing for consistent data updates, deletes, and inserts.
• Schema Evolution: It gracefully handles schema changes, a common requirement in dynamic data environments.
• Partitioning and Performance: Automatic partitioning optimises query performance, making it efficient even for large datasets.
• Time Travel: Iceberg’s time travel functionality enables querying historical data versions, making it invaluable for auditing, troubleshooting, and compliance. It's like Git for managing code with versioning and reverting to any commit Id. These features make Iceberg a strong foundation for building a transactional lakehouse that balances flexibility and consistency.

How Iceberg Integrates with AWS:

One of the session's focal points was explaining how Apache Iceberg works within the AWS ecosystem. Here’s a quick recap:

• Storage in Amazon S3: Iceberg tables are stored in Amazon S3, benefiting from scalable and cost-effective object storage.
• Data Processing with AWS Glue: AWS Glue allows serverless ETL processing of data into Iceberg tables, making it possible to handle batch and real-time updates.
• Querying with Amazon Athena: Athena supports SQL queries on Iceberg tables directly from S3, making it easy to query and analyse data without dedicated infrastructure.
• Governance with AWS Lake Formation: Lake Formation provides fine-grained access control, ensuring data security and governance within the lakehouse. Together, these services create a robust lakehouse environment on AWS, leveraging Iceberg for consistency and scalability.

Use Case: Financial Data Lakehouse:

To illustrate how a transactional data lakehouse works in practice, I shared a use case in the financial services industry. Financial institutions need real-time data consistency, compliance, and performance for analytics and regulatory reporting. In this scenario, a data lakehouse with Iceberg allows for:

• Real-time analytics with consistent, ACID-compliant data.
• Historical data access through time travel for auditing and compliance.
• Cost efficiency by storing data in S3 and using Athena for on-demand queries. This use case highlighted the lakehouse’s potential to streamline data management in industries requiring both performance and data governance.

Architectural Overview:

In my session, I walked through an architectural diagram illustrating how to build a lakehouse on AWS with Iceberg:

• Ingestion Layer: Data is ingested from multiple sources into S3 using AWS Glue or Kinesis.
• Storage Layer: Iceberg tables reside in Amazon S3, with metadata management to handle partitions, schema evolution, and versioning.
• Processing Layer: Glue ETL jobs process and transform data, supporting both batch and streaming.
• Query Layer: Athena enables SQL-based querying of Iceberg tables for flexible analytics.
• Governance Layer: AWS Lake Formation secures and governs access to sensitive data within the lakehouse. This architecture demonstrates a scalable, cost-effective approach to building a transactional lakehouse that supports data consistency and flexibility.

Lessons Learned:

From working with Iceberg on AWS, I shared a few key lessons:

• Partitioning Strategy: Efficient partitioning is essential for Iceberg to deliver high performance. Planning for your data distribution patterns is crucial.
• Schema Evolution: Although Iceberg handles schema changes well, backward compatibility is vital to avoid breaking data pipelines.
• Cost Management: Data lakehouses on S3 are cost-effective, but monitoring Glue jobs and optimising Athena queries help keep costs in check.
• Data Governance: Fine-grained access control with Lake Formation ensures data security, which is particularly important for multi-user environments.

Best Practices for Building a Data Lakehouse with Iceberg:

To wrap up my talk, I outlined some best practices for those considering building a lakehouse with Iceberg on AWS:

• Data Modelling: Design Iceberg tables with a strong partitioning strategy to optimise performance and query efficiency.
• Governance: Leverage Lake Formation for access control to ensure secure data access.
• Time Travel for Compliance: Use Iceberg’s time travel feature to maintain historical records for regulatory compliance.
• Optimise Glue Jobs: Efficiently schedule Glue jobs to process incremental updates and avoid unnecessary compute costs.

In Closing:

Presenting at DataEngBytes 2024 Sydney was a fantastic opportunity to share insights into building a transactional data lakehouse on AWS with Apache Iceberg. This architecture offers a powerful approach to managing and analysing data with both the flexibility of a lake and the consistency of a warehouse, unlocking new possibilities for data-driven organisations.

If you’re exploring a lakehouse approach in your own organisation, I’d highly recommend considering Apache Iceberg and AWS as the foundation. Combining Iceberg’s transactional capabilities with AWS’s scalability, you can build a data lakehouse that adapts to your evolving data needs while ensuring reliability and governance.

I hope this recap provides a clear overview of the content and insights from my talk. If you have questions or want to learn more about building data lakehouses, feel free to reach out or stay tuned for more blog posts on advanced data architectures!