Apache Iceberg has emerged as the defining open table format for enterprise data, solving decades-old issues which made data lakes unreliable, slow, and expensive. Every major cloud provider now supports Iceberg natively, Databricks paid over a billion dollars to acquire the company behind it, and organizations from Netflix to Airbnb report compute cost reductions exceeding 50% as a result of using the tool. If your organization stores large volumes of data or plans to, Iceberg is no longer optional to understand. It represents the most significant shift in data lake architecture since the original rise of Hadoop, and its momentum the last few years makes it clear that this is the standard the industry is converging on.
To understand why Iceberg matters, it helps to start with the problem it solves. Imagine a massive library with millions of books, but no card catalog. Every time you want to find a book, you have to walk through every aisle, read every spine, and hope nothing has been moved since you last looked. That’s essentially how traditional data lakes worked and it was a nightmare at scale.
Traditional data lake formats like Apache Hive organized data into folders on cloud storage (like Amazon S3). When a query engine needed to find relevant data, it had to list every folder and file, one by one. For a table with thousands of partitions, this meant thousands of API calls just to figure out where the data lived before reading a single row. Worse, there were no guarantees that someone else wasn’t writing to the same table at the same time, potentially corrupting results.
Apache Iceberg, created at Netflix in 2017, replaces that chaotic approach with a sophisticated metadata layer. Essentially, that smart card catalog. Instead of listing directories on cloud storage, Iceberg maintains a precise manifest of every data file in a table, along with statistics about each file’s contents (row counts, column-level minimum and maximum values, null counts). Query engines read this compact metadata to determine exactly which files to open, skipping everything irrelevant.
The result? Queries that previously scanned terabytes of data now touch only the relevant gigabytes. Planning that required thousands of S3 API calls completes in a fraction of the time. And because every change creates an immutable snapshot with an atomic commit, concurrent readers and writers never interfere with each other. Iceberg brought ACID transactions and the reliability guarantees that databases have enjoyed for decades to the data lake for the first time.
Figure 1: Traditional data lakes require scanning all files to find relevant data, while Iceberg uses smart metadata to read only what’s needed
Netflix open-sourced Iceberg and donated it to the ApacheSoftware Foundation in late 2018. It graduated to a top-level Apache project inMay 2020, and adoption has accelerated dramatically since.
Iceberg’s technical design addresses specific pain points that plagued data engineering teams for years. Understanding these features clarifies why the industry has moved so decisively toward adoption.
Hidden partitioning is perhaps Iceberg’s most elegant innovation. In traditional systems, users had to know exactly how data was physically organized and include specific partition filters in every query. Iceberg decouples the logical query from the physical layout. Users simply write WHERE event_time > '2024-01-01', and Iceberg automatically applies the correct partition pruning behind the scenes. Even better, the partition scheme can change over time without rewriting existing data, a capability called partition evolution, that eliminates painful migration projects.
Schema evolution allows teams to add, remove, rename, or reorder columns instantly without rewriting data files. Iceberg tracks columns by unique internal IDs rather than names, preventing the “zombie data” problems that plagued Hive when schemas changed. This is a metadata-only operation that completes in milliseconds, regardless of table size.
Time travel enables querying any historical version of a table by snapshot ID or timestamp. Made a mistake in a data pipeline? Roll back to yesterday’s version in seconds. Need to reproduce last quarter’s analytics exactly? Query the table as it existed on that date. This capability is transformative for debugging, auditing, and regulatory compliance.
ACID transactions ensure that writes either fully succeed or fully fail. No more partial updates corrupting downstream reports. Iceberg uses optimistic concurrency control, meaning multiple writers can work simultaneously without distributed locks, and readers always see a consistent snapshot.
Engine independence is what truly differentiates Iceberg from alternatives like Delta Lake. The same Iceberg table can be simultaneously read and written by Apache Spark, Apache Flink, Trino, Presto, Amazon Athena, Amazon Redshift, Snowflake, Google BigQuery, DuckDB, and more than 30 other tools. This eliminates vendor lock-in and lets organizations choose the best engine for each workload without duplicating data.
Figure 2: Apache Iceberg Table Architecture on AWS (Glue + S3)
Figure 3: Detailed view of Iceberg’s metadata structure
AWS has made Apache Iceberg central to its analytics strategy, with native support spanning virtually every data service. The re:Invent2024 conference marked a watershed moment, with Iceberg featured in over a dozen major announcements. Here is how the key services work together.
Figure 4: Apache Iceberg enables a unified data architecture across the entire AWS analytics ecosystem
Amazon Athena provides the most accessible entry point for querying Iceberg tables. Since the launch of Athena engine version 3 in October 2022 (built on Trino), analysts can create, query, and manage Iceberg tables using standard SQL with no infrastructure to manage. Athena supports Iceberg’s headline features including time travel queries, schema evolution, hidden partitioning with transform functions (day, month, year, bucket, truncate), and the powerful MERGE INTO command for upserts. The OPTIMIZE command handles compaction directly from Athena, merging small files into optimally-sized ones, while VACUUM expires old snapshots and cleans up orphan files. Athena uses the AWS Glue Data Catalog as its Iceberg catalog, meaning table metadata is automatically shared across services. The key limitation to note: Athena operates in merge-on-read mode only, so workloads requiring copy-on-write semantics should use Spark on EMR or Glue instead.
AWS Glue serves a dual role as both the recommended Iceberg catalog and a powerful ETL processing engine. The Glue Data Catalog now exposes an Iceberg REST endpoint (https://glue.{region}.amazonaws.com/iceberg), implementing the open Iceberg REST catalog specification that enables interoperability with any compatible engine. Glue 5.0, announced at re:Invent 2024, ships with Iceberg 1.6.1, Spark 3.5.2, and supports fine-grained access control through Lake Formation at the table, column, row, and even cell level. A standout feature announced alongside Glue 5.0 is automated statistics generation. The Glue Data Catalog now automatically computes column statistics for Iceberg tables, feeding cost-based optimizers in Redshift and Athena for smarter query plans. Glue Data Catalog federation also allows querying remote Iceberg tables in Snowflake Polaris or Databricks Unity Catalog without moving data, breaking down organizational data silos.
Amazon Redshift has expanded Iceberg integration significantly throughout 2024 and 2025. Both Redshift provisioned clusters (via Spectrum) and Redshift Serverless can query Iceberg tables registered in the Glue Data Catalog, joining data lake tables with native Redshift tables in a single SQL query. In 2025, Redshift reached general availability for writing to Iceberg tables supporting CREATE TABLE and INSERT operations to both standard S3 buckets and the new S3 table buckets. Performance improvements have been dramatic: a new vectorized scan layer purpose-built for Parquet files, combined with smart prefetching and advanced partition/file-level pruning, delivers over 2x faster Iceberg query performance on Redshift Serverless. The JIT ANALYZE feature automatically collects and uses statistics during query execution, with some TPC-DS benchmark queries improving by 50x. Perhaps most strategically, existing Redshift data warehouses can now publish their managed storage data as Iceberg tables through the SageMaker Lakehouse, making warehouse data accessible to any Iceberg-compatible engine without ETL.
Amazon S3 Tables, announced at re:Invent 2024, represents AWS’s boldest Iceberg bet. This new “table bucket” type is the first cloud object store with built-in Apache Iceberg support, offering up to 3x faster query throughput and 10x higher transactions per second compared to self-managed Iceberg tables. S3 Tables handles compaction, snapshot management, and unreferenced file cleanup automatically — maintenance tasks that previously required dedicated engineering effort. The service works with Athena, Redshift, EMR, and third-party engines through a standard Iceberg REST catalog interface.
The business case for Iceberg is backed by published results from organizations operating at significant scale.
Netflix, Iceberg’s creator, uses the format as the foundation of its incremental processing system. Rather than reprocessing entire datasets, Netflix processes only new or changed data, achieving greater than 80% reduction in compute costs across its data platform. This system serves thousands of daily users including data scientists, content producers, and business analysts.
Airbnb processes over 35 billion Kafka event messages across more than 1,000 tables daily. After migrating from their Hive-based stack to Spark 3 with Iceberg, they measured over 50% compute resource savings and 40% reduction in job elapsed time. The migration also eliminated Hive Metastore bottlenecks and S3 consistency issues that had plagued their ingestion pipeline.
Apple is reportedly operating one of the largest Iceberg deployments in production and saw aggregate queries drop from “over an hour” through enhanced metadata pushdowns that eliminate unnecessary data file scanning entirely. Maintenance operations that previously took two hours now complete in minutes.
Insider, a marketing technology company, achieved roughly 90% reduction in Amazon S3 API costs after migrating from Hive to Iceberg, driven by fewer, larger files replacing millions of small ones. They simultaneously saved approximately 20% on EC2 and EMR costs from more efficient Spark jobs.
A quantitative finance benchmark published on the AWS Big Data Blog in January 2025 demonstrated up to 52% faster query performance on real-world historical book data compared to standard Parquet-on-S3 layouts. The benchmark also showed reduced task failures and throttling issues, improving pipeline stability alongside raw speed.
The years-long competition between Apache Iceberg, Delta Lake, and Apache Hudi for open table format dominance has effectively concluded. By late 2024, industry consensus crystallized around Iceberg as the standard, not because the alternatives disappeared, but because even their creators moved toward Iceberg compatibility.
The decisive signal came in June 2024 when Databricks acquired Tabular, the company founded by Iceberg’s creators, for a reported $1–2 billion. Databricks, the creator of Delta Lake, essentially acknowledged that Iceberg interoperability was non-negotiable. Their Delta Lake UniForm feature, which reached general availability in 2024, provides automatic Iceberg compatibility for Delta tables. Databricks CTO Matei Zaharia stated publicly: “Our hope is to make these formats converge so we don’t care about format anymore.”
Days before that acquisition, Snowflake announced Polaris Catalog, an open-source, vendor-neutral Iceberg catalog implementation later donated to the Apache Software Foundation. All three major cloud providers now offer native Iceberg support: AWS through S3 Tables and its analytics suite, Google Cloud through BigQuery tables for Apache Iceberg, and Microsoft Azure through partnerships with Databricks and direct Iceberg integration.
The Dremio "State of Data Lakehouse" 2024 survey of 500 data leaders quantifies this trend. While Delta Lake held a slight lead in current adoption (39% vs. 31% for Iceberg), planned three-year adoption told a different story: 29% planned to adopt Iceberg versus 23% for Delta Lake. Iceberg is projected to overtake Delta Lake in overall adoption within three years.
The Iceberg catalog service market reached an estimated $578 million in 2024 and is projected to grow at 21.7% annually, potentially reaching $4.18 billion by 2033. Nearly three-quarters of technology leaders surveyed by Databricks report having already adopted a lakehouse architecture, with the remainder planning to within three years.
The Iceberg V3 specification, rolling out across 2025 releases, introduces deletion vectors for more efficient row-level operations, geospatial data types, nanosecond-precision timestamps for financial use cases, and row-level lineage tracking for compliance and auditing. AWS EMR 7.12 became the first AWS service to support Iceberg V3 in 2025, with other services following.
The convergence on Apache Iceberg as the industry standard creates both opportunity and urgency for our clients. Organizations still running traditional Hive-based data lakes face growing technical debt as the ecosystem moves forward. Those considering or building new data platforms have a clear architectural path that avoids vendor lock-in while delivering measurable performance and cost benefits.
The practical implications are significant. Iceberg enables a true lakehouse architecture where a single copy of data serves analytics, machine learning, and operational workloads across multiple engines. Teams can start with Athena for ad-hoc queries, process data with Glue or EMR, build dashboards with Redshift, and train models with SageMaker, all against the same Iceberg tables, with consistent governance through Lake Formation. New AWS capabilities like S3 Tables reduce operational overhead further, handling compaction and maintenance automatically.
Migration doesn’t have to be all-or-nothing. Many organizations begin by creating new tables in Iceberg format while gradually migrating existing Hive tables, using tools like AWS Glue crawlers to register existing Iceberg tables, or converting Hive tables through CTAS operations in Athena.
At Red Oak Strategic, we help organizations navigate exactly these architectural decisions — evaluating readiness for Iceberg adoption, designing migration strategies that minimize risk, and implementing modern lakehouse architectures on AWS that deliver the performance and cost improvements the format makes possible. If you’re exploring how Iceberg fits into your data strategy, we’d welcome the conversation.
Apache Iceberg has moved from a Netflix engineering project to the undisputed standard for open data lakehouse architecture in less than seven years. The technical merits — ACID transactions, hidden partitioning, time travel, and engine independence — are compelling, but the real story is the unprecedented industry alignment. When the creators of competing formats acquire your founding company for over a billion dollars, and every major cloud provider builds native support into their storage layer, the signal is unambiguous.
The organizations seeing the greatest returns are those treating Iceberg not as a file format upgrade but as an architectural enabler, using it to unify previously siloed data platforms, eliminate redundant data copies, and give every team access to the same governed, reliable data through whatever tool best fits their needs. With AWS deepening Iceberg integration across its entire analytics portfolio and automating previously manual maintenance tasks, the barrier to entry has never been lower. The question for data leaders is no longer whether to adopt Iceberg, but how quickly they can capture the efficiency gains their peers are already realizing.