Data engineering · 6 min read

Parquet vs Iceberg vs Hudi: picking a table format

A folder of Parquet files works fine until someone needs to update a row, delete one, or write to the same table from two places at once. That's the point where most lakehouse designs bring in an open table format. This is a general look at what that means and how to choose between the common options.

By Ayush Kumar Singh, Data Engineer

Why plain files alone aren't enough

A data lake built from raw Parquet or CSV files is simple and cheap, and it works well as long as data only ever gets appended. The trouble starts once you need to change something that's already written. Updating one row usually means rewriting an entire file. Two processes writing to the same table at the same time can corrupt or overwrite each other's work. Readers can catch a table mid-write and see a half-finished state. Plain files were never designed to behave like a database table, because they have no shared record of what state the table is actually in.

Enter the open table format

An open table format sits on top of the same plain files and adds a metadata layer that tracks them as one consistent table. It keeps a record of exactly which files make up the table at any point in time, so readers always see a complete, consistent snapshot, even while writes are happening. It supports real updates and deletes without rewriting the whole table, changes to columns over time without breaking old data, and multiple writers working safely at once. Apache Iceberg and Apache Hudi are two popular implementations of this same idea, each making different trade-offs in how that metadata layer works.

Apache Iceberg

Iceberg tracks a table through a chain of snapshots. Each snapshot points to a manifest list, which points to manifest files, which list the actual data files for that version of the table. This layered structure is what makes features like time travel and rollback possible: querying an older snapshot is just a matter of following an earlier pointer.

Its main strength is being engine-agnostic. Spark, Trino, Flink and others can all read and write the same Iceberg table using the same metadata, so nobody is locked into one engine. It also handles schema evolution cleanly: renaming or adding columns doesn't force a rewrite of existing data. It's a strong fit for analytics-heavy tables that aren't updated too frequently, and for teams that already use more than one query engine.

Apache Hudi

Hudi is built around two table types. Copy-on-Write rewrites the affected data files immediately when a row changes, which keeps reads fast since there's nothing extra to merge at query time. Merge-on-Read instead writes changes to a separate log and merges them with the base files only when the table is read or compacted, which makes writes much cheaper at the cost of slightly slower reads until compaction catches up.

This design makes Hudi a strong fit for upsert-heavy, low-latency ingestion, such as streaming a continuous flow of inserts and updates from a CDC pipeline straight into the lake. It's less about serving many engines and more about handling a high rate of change efficiently.

Comparing them

A simple decision guide

Wrapping up

There isn't a single best table format, only a best fit for how a table is written to and who reads it. Plain files are enough until change enters the picture. Once it does, Iceberg and Hudi solve the same underlying problem in different ways, and the right choice comes down to how often the data changes and how many engines need to read it.